HDDS-15071. [SCM] Add configuration and global EC reconstruction limit by jojochuang · Pull Request #10122 · apache/ozone

jojochuang · 2026-04-24T17:06:04Z

What changes were proposed in this pull request?

HDDS-15071. [SCM] Add configuration and global reconstruction limit

Please describe your PR in detail:

Introduce the foundational configuration properties in ReplicationManagerConfiguration:

hdds.scm.replication.decommission.ec.reconstruction.enabled
hdds.scm.replication.decommission.ec.reconstruction.load.factor (default 0.9)
hdds.scm.replication.reconstruction.global.limit (default 0, disabled)

Implement an atomic counter in ReplicationManager to track active ReconstructECContainersCommand tasks cluster-wide. Update UnhealthyReplicationProcessor
to check this global limit before dequeuing containers for reconstruction, ensuring aggregate bisectional bandwidth is protected.

Default behavior is unchanged. hdds.scm.replication.reconstruction.global.limit defaults to 0, which disables cluster-wide reconstruction limit checking (same pattern as hdds.scm.replication.inflight.limit.factor). Set a positive value to opt in to throttling; a value of 50 is recommended when enabling HDDS-15072 EC decommission reconstruction.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15071

How was this patch tested?

Existing tests. TestReplicationManager.testReconstructionGlobalLimitDisabledByDefault(), TestReplicationManager.testInflightReconstructionLimit(), TestUnderReplicatedProcess.testMessageNotProcessedIfReconstructionLimitReached()

smengcl

lgtm

jojochuang · 2026-07-06T22:32:05Z

Pushed a small change to disable the global reconstruction threshold, to keep the behavior the same.

smengcl

Thanks @jojochuang . I found two more issues

smengcl · 2026-07-07T06:54:08Z

+      if (reconstructionLimitReached(replicationManager)) {
+        LOG.info("The maximum number of pending reconstruction commands ({}) " +
+                "are scheduled. Ending the iteration.",
+            replicationManager.getReconstructionInFlightLimit());
+        break;
+      }


Should this skip/requeue only reconstruction work rather than halt all under-replication recovery? break here ends the whole loop

So hitting the reconstruction limit will stop processing the entire under-replicated queue, including containers that just need a normal replica copied, which have nothing to do with reconstruction bandwidth.

Copilot

Pull request overview

Adds a cluster-wide throttling mechanism for EC reconstruction in SCM’s ReplicationManager, backed by new configuration knobs. This helps prevent excessive aggregate EC reconstruction work when the feature is enabled, while keeping default behavior unchanged (limit disabled by default).

Changes:

Introduces new ReplicationManagerConfiguration properties for EC decommission reconstruction enablement, load factor, and a global EC reconstruction limit.
Tracks inflight EC reconstruction commands via an atomic counter and per-command fragment accounting, and enforces the global limit when sending reconstruction commands.
Adds/extends unit tests covering default-disabled behavior, limit accounting, and limit-triggered rejection paths.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java	Adds global EC reconstruction limit config + inflight tracking and enforcement in reconstruction command sending / completion handling.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationManager.java	Adds tests for default behavior, inflight reconstruction tracking, rejection at limit, and counter reset on leader transitions.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestUnderReplicatedProcessor.java	Adds a processor test related to reconstruction-limit behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java:601

The PR description says UnhealthyReplicationProcessor will check the global reconstruction limit before dequeuing work; however, the processors only check the replication inflight limit and there is no reconstruction-limit check in UnhealthyReplicationProcessor/UnderReplicatedProcessor. As implemented, the limit is enforced only when sending the command (by throwing CommandTargetOverloadedException), which can still dequeue/process/requeue items and create churn. Please either implement the dequeue-time check or update the PR description/testing notes accordingly.

    if (isReconstructionLimitReached()) {
      metrics.incrECReconstructionCmdsDeferredTotal();
      throw new CommandTargetOverloadedException(
          "Global reconstruction limit (" + getReconstructionInFlightLimit()
              + ") reached for container " + containerInfo.getContainerID());
    }
    List<DatanodeDetails> targets = command.getTargetDatanodes();
    List<Pair<Integer, DatanodeDetails>> targetWithCmds =
        getAvailableDatanodesForReplication(targets);
    if (targetWithCmds.isEmpty()) {
      metrics.incrECReconstructionCmdsDeferredTotal();
      throw new CommandTargetOverloadedException("No target with capacity " +
          "available for reconstruction of " + containerInfo.getContainerID());
    }
    DatanodeDetails target = selectAndOptionallyExcludeDatanode(
        rmConf.getReconstructionCommandWeight(), targetWithCmds);
    sendDatanodeCommand(command, containerInfo, target);
  }

  public void sendThrottledReconstructionCommand(ContainerInfo containerInfo,
      ReconstructECContainersCommand command)
      throws CommandTargetOverloadedException, NotLeaderException {
+    if (isReconstructionLimitReached()) {
+      metrics.incrECReconstructionCmdsDeferredTotal();
+      throw new CommandTargetOverloadedException(
+          "Global reconstruction limit (" + getReconstructionInFlightLimit()
+              + ") reached for container " + containerInfo.getContainerID());
+    }


+    ContainerInfo container = ReplicationTestUtil.createContainerInfo(
+        repConfig, 1, HddsProtos.LifeCycleState.CLOSED, 10, 20);
+
+    // Send one reconstruction command with 2 fragments


Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 9 comments.

+  private boolean tryReserveReconstructionSlot() {
+    int limit = getReconstructionInFlightLimit();
+    if (limit <= 0) {
+      return true;
+    }
+    while (true) {
+      int current = inflightReconstructionCount.get();
+      if (current >= limit) {
+        return false;
+      }
+      if (inflightReconstructionCount.compareAndSet(current, current + 1)) {
+        return true;
+      }
+    }
+  }


+    if (!tryReserveReconstructionSlot()) {
      metrics.incrECReconstructionCmdsDeferredTotal();
-      throw new CommandTargetOverloadedException("No target with capacity " +
-          "available for reconstruction of " + containerInfo.getContainerID());
+      throw new CommandTargetOverloadedException(
+          "Global reconstruction limit (" + getReconstructionInFlightLimit()
+              + ") reached for container " + containerInfo.getContainerID());


+    ReconstructECContainersCommand cmd1 = new ReconstructECContainersCommand(
+        1L, Collections.emptyList(),
+        ImmutableList.of(MockDatanodeDetails.randomDatanodeDetails()),
+        ECUnderReplicationHandler.integers2ByteString(ImmutableList.of(1)), (ECReplicationConfig) repConfig);


+    ReconstructECContainersCommand cmd2 = new ReconstructECContainersCommand(
+        2L, Collections.emptyList(),
+        ImmutableList.of(MockDatanodeDetails.randomDatanodeDetails()),
+        ECUnderReplicationHandler.integers2ByteString(ImmutableList.of(2)), (ECReplicationConfig) repConfig);


+    ReconstructECContainersCommand cmd3 = new ReconstructECContainersCommand(
+        3L, Collections.emptyList(),
+        ImmutableList.of(MockDatanodeDetails.randomDatanodeDetails()),
+        ECUnderReplicationHandler.integers2ByteString(ImmutableList.of(3)), (ECReplicationConfig) repConfig);


+    ReconstructECContainersCommand cmd = new ReconstructECContainersCommand(
+        1L, Collections.emptyList(),
+        ImmutableList.of(MockDatanodeDetails.randomDatanodeDetails()),
+        ECUnderReplicationHandler.integers2ByteString(ImmutableList.of(1)), (ECReplicationConfig) repConfig);


+        1L, Collections.emptyList(),
+        ImmutableList.of(MockDatanodeDetails.randomDatanodeDetails(),
+            MockDatanodeDetails.randomDatanodeDetails()),
+        ECUnderReplicationHandler.integers2ByteString(ImmutableList.of(1, 2)), (ECReplicationConfig) repConfig);


+    startLatch.countDown();
+    assertTrue(doneLatch.await(30, TimeUnit.SECONDS));
+    executor.shutdown();
+


+      final long containerId = t + 100;
+      executor.submit(() -> {
+        try {
+          startLatch.await();
+          for (int i = 0; i < attemptsPerThread; i++) {
+            ReconstructECContainersCommand cmd = new ReconstructECContainersCommand(
+                containerId + i, Collections.emptyList(),
+                ImmutableList.of(MockDatanodeDetails.randomDatanodeDetails()),
+                ECUnderReplicationHandler.integers2ByteString(ImmutableList.of(1)),
+                (ECReplicationConfig) repConfig);


Set hdds.scm.replication.reconstruction.global.limit default to 0 so cluster-wide EC reconstruction throttling is opt-in and default behavior matches pre-PR Ozone. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I6a0915cf8b181a1a98b08a4f3ede071e4d460e88

… reset. Move global reconstruction throttling from the under-replicated processor loop to sendThrottledReconstructionCommand so 1-1 replication continues when the cap is reached. Clear reconstruction counters on leader transition when pending ops are cleared. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I881fee0c3ea3c2e350b8fbb95321985569bef636

Skip reconstruction counter decrements when the command is no longer tracked after notifyStatusChanged clear, and floor decrements at zero. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I1530348aa2b09c232ad0410468d90cf098d8d0a3

…and tests. Validate reconstruction config bounds, remove a misleading processor test stub, and reuse ECUnderReplicationHandler.integers2ByteString in unit tests. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I55216c0c4c0af92e003486e04dd9bc0c42a8dc5c

…espace. Reserve the global reconstruction limit with a CAS before sending commands, release on send failure, and add a concurrent unit test. Clean up trailing whitespace in reconstruction limit tests. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: If8c0e4613209b8fb381271c7dab1733eac0e5bde

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

+  private boolean tryReserveReconstructionSlot() {
+    int limit = getReconstructionInFlightLimit();
+    if (limit <= 0) {
+      return true;
+    }


+  public boolean isReconstructionLimitReached() {
+    int limit = getReconstructionInFlightLimit();
+    return limit > 0 && getInflightReconstructionCount() >= limit;
+  }


adoroszlai reviewed Apr 24, 2026

View reviewed changes

Comment thread hadoop-ozone/common/pom.xml Outdated

jojochuang changed the title ~~Hdds 15071~~ HDDS-15014. [SCM] Add configuration and global reconstruction limit Apr 24, 2026

jojochuang changed the title ~~HDDS-15014. [SCM] Add configuration and global reconstruction limit~~ HDDS-15071. [SCM] Add configuration and global reconstruction limit Apr 24, 2026

jojochuang force-pushed the HDDS-15071 branch from daf857e to d4fdcbe Compare April 24, 2026 17:59

adoroszlai mentioned this pull request May 11, 2026

WIP. HDDS-15014. Speed up EC container decommission #10082

Closed

jojochuang requested a review from smengcl May 27, 2026 14:06

smengcl approved these changes May 27, 2026

View reviewed changes

jojochuang marked this pull request as ready for review June 18, 2026 19:26

smengcl reviewed Jul 7, 2026

View reviewed changes

jojochuang requested a review from Copilot July 9, 2026 00:05

Copilot started reviewing on behalf of jojochuang July 9, 2026 00:05 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

jojochuang requested a review from Copilot July 9, 2026 01:14

Copilot started reviewing on behalf of jojochuang July 9, 2026 01:15 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

jojochuang requested a review from Copilot July 9, 2026 01:28

Copilot started reviewing on behalf of jojochuang July 9, 2026 01:29 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

jojochuang changed the title ~~HDDS-15071. [SCM] Add configuration and global reconstruction limit~~ HDDS-15071. [SCM] Add configuration and global EC reconstruction limit Jul 9, 2026

jojochuang and others added 6 commits July 10, 2026 10:14

HDDS-15071. [SCM] Add configuration and global reconstruction limit

ed4e47c

Copilot AI review requested due to automatic review settings July 10, 2026 17:14

jojochuang force-pushed the HDDS-15071 branch from b5ca933 to c196dd4 Compare July 10, 2026 17:14

Copilot started reviewing on behalf of jojochuang July 10, 2026 17:14 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

jojochuang commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

smengcl left a comment

Choose a reason for hiding this comment

Uh oh!

jojochuang commented Jul 6, 2026

Uh oh!

smengcl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smengcl Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jojochuang commented Apr 24, 2026 •

edited

Loading