HDDS-15412. Add per-volume push replication thread pools on DataNode. by jojochuang · Pull Request #10693 · apache/ozone

jojochuang · 2026-07-08T23:11:51Z

What changes were proposed in this pull request?

HDDS-15412. Add per-volume push replication thread pools on DataNode.

Please describe your PR in detail:

Adds optional per-volume replication handler thread pools on the DataNode for push-based container replication. When enabled, each data volume gets its own thread pool, so a slow or backed-off disk no longer blocks push replication for containers on other disks.
This targets the EC decommission scenario described in HDDS-15412: with a single global pool, replication starts fast on all disks, then slows until only one disk stays busy while others idle.
Default is off (hdds.datanode.replication.per.volume.enabled=false), so existing clusters keep current behavior unless the feature is explicitly enabled.

When per.volume.enabled=true:

Push replication is routed to a per-volume thread pool (sized by hdds.datanode.replication.per.volume.streams.limit, default 1)
EC reconstruction and reconcile tasks continue using the global pool (streams.limit)
Volume is resolved from the source container via ContainerSet → container.getContainerData().getVolume()
On volume failure, that volume’s pool is shut down and new push tasks for containers on that volume are rejected
When per.volume.enabled=false (default): all tasks use the global pool, same as today.

Known limitations / follow-ups

per.volume.enabled requires DN restart (not reconfigurable)
Reconfig of per.volume.streams.limit does not re-apply outofservice.limit.factor on decommissioning nodes (node state transitions do)
Rejected tasks are logged but not counted in replication metrics
No per-volume metrics yet (queue depth, active threads)
No integration test in this PR (unit tests only)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15412

How was this patch tested?

Made with Cursor Composer 2.5.

Unit tests added in TestReplicationSupervisor:

Feature disabled → no per-volume pools; legacy behavior
Init logging with volume paths
Per-volume pool size respected
Runtime resize via setPerVolumePoolSize()
Decommission scaling via outofservice.limit.factor
Non-push tasks (reconcile) use global pool when feature enabled
Cross-volume push isolation (blocking on volume A does not block volume B)
Volume failure shuts down pool and rejects subsequent push tasks
TestReplicationConfig: per-volume config validation.

All TestReplicationConfig + TestReplicationSupervisor tests pass.

Introduce optional per-disk replication handler thread pools so slow push replication on one volume does not block replication on other disks. Pull replication was removed upstream; EC/reconcile tasks continue using the global pool. Includes runtime resize support and volume-failure cleanup. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I7cacd6fbeb435f5172fc67141a869cd4d1073811

Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I87c4c544a24f99675ef68940b9412367d60c9b8e

Copilot

Pull request overview

This PR adds an optional per-data-volume execution model for push-based container replication on the datanode. When enabled, replication tasks are dispatched to a per-volume replication handler thread pool, aiming to prevent slow/backed-off disks from stalling push replication work on other disks.

Changes:

Introduces VolumeReplicationThreadPools and updates ReplicationSupervisor to route push replication to per-volume executors when configured.
Adds DN reconfiguration support for hdds.datanode.replication.per.volume.streams.limit and documents it as reconfigurable.
Adds unit tests validating per-volume pool initialization, sizing/resizing, non-push behavior staying on the global pool, isolation across volumes, and volume-failure behavior.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
hadoop-hdds/docs/content/feature/Reconfigurability.md	Documents the new per-volume streams limit as a reconfigurable DN setting.
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationSupervisor.java	Adds unit tests covering per-volume pool behavior, isolation, resize, and failure handling.
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationConfig.java	Adds unit coverage for default and validated per-volume config behavior.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java	Registers and applies runtime reconfig for per-volume streams limit (when feature enabled).
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/VolumeReplicationThreadPools.java	New helper managing per-volume `ThreadPoolExecutor`s for push replication.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationTask.java	Adds `isPushReplication()` helper for routing decisions.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisor.java	Routes eligible push replication tasks to per-volume executors; adds pool shutdown/resize hooks and queue-size aggregation.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationServer.java	Adds new config keys and validation for per-volume enablement and per-volume stream limit.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java	Wires container/volume context into the supervisor and shuts down per-volume pools on volume failure events.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    queuedCounter.get(task.getMetricName()).incrementAndGet();
+    ExecutorService taskExecutor = selectExecutor(task);
+    if (taskExecutor == null) {
+      LOG.warn("Rejected {} in ReplicationSupervisor: no replication handler "
+          + "thread pool available", task);
+      queuedCounter.get(task.getMetricName()).decrementAndGet();
+      inFlight.remove(task);
+      decrementTaskCounter(task);
+      return;
+    }
+    taskExecutor.execute(new TaskRunner(task));
+  }


+    try {
+      pool.shutdownNow();
+      if (!pool.awaitTermination(3, TimeUnit.SECONDS)) {
+        LOG.warn("Per-volume replication thread pool for volume {} did not "


+    } finally {
+      supervisor.stop();
+    }


… is unavailable. When a push replication task cannot use its per-volume thread pool, route it to the global replication handler pool instead of rejecting it at queue time. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Ie3524488eecd667f9c16ee8f19bbb1774f4baa6a

jojochuang

Commit summary: `47251206749`

Title: HDDS-15412. Fall back to global replication pool when per-volume pool is unavailable.

Problem

On HDDS-15412-by-cursor, when per-volume push replication thread pools were enabled, ReplicationSupervisor.addToQueue() would reject push-replication tasks if no per-volume executor was available (e.g. volume failed, pool shut down, or pool missing). That meant:

The task never ran
Logs said "rejecting task" even though rejection happened later in addToQueue()
Behavior was inconsistent with the container-not-found path, which already fell back to the global pool

Change

ReplicationSupervisor.resolveVolumeExecutor()

When the volume is null/failed or the per-volume pool is missing, return the global executor instead of null
Log a unified fallback message: falling back to global replication handler thread pool

ReplicationSupervisor.addToQueue()

Removed the taskExecutor == null rejection path (counter rollback + early return)
Tasks are always submitted via selectExecutor(task).execute(...)

TestReplicationSupervisor.volumeFailureShutsDownPool

After volume failure and per-volume pool shutdown, expects the task to run on the global pool and succeed (getReplicationSuccessCount() == 1 instead of 0)

Behavior after this commit

Scenario	Before	After
Per-volume pool shut down (volume failure)	Task rejected at queue time	Task runs on global pool
`volume.isFailed()`	Task rejected	Task runs on global pool
Container not found	Fallback to global	Unchanged

SCM impact

No change to SCM notification — replicate commands still do not use CommandStatus. SCM still learns about replication via container reports and pending-op timeouts.

Files changed

ReplicationSupervisor.java — 25 lines changed (net −7)
TestReplicationSupervisor.java — 1 assertion updated

smengcl · 2026-07-10T21:07:00Z

+    LOG.info("Shutting down per-volume replication thread pool for failed "
+        + "volume {}", volumeRoot);
+    try {
+      pool.shutdownNow();


shutdownNow() discards the queued Runnables it returns. Those queued TaskRunners never execute, so the finally in ReplicationSupervisor.TaskRunner.run() that calls inFlight.remove(task) and decrements the counters never runs for them. Their (containerId, target) entries stay in inFlight permanently, and since addToQueue early returns when inFlight.add(task) is false, every later push replication command for those containers is dropped until the datanode restarts. This fires exactly in the failed volume path this feature adds.

Consider draining the returned list and running the same cleanup, for example have shutdownVolume return pool.shutdownNow() so shutdownFailedVolumePools can call back into the supervisor to remove each drained task from inFlight and fix the counters. Otherwise a single volume failure can strand re-replication for its containers.

smengcl · 2026-07-10T21:07:15Z

+
+  public void setPerVolumePoolSize(int newSize) {
+    if (volumePools != null) {
+      volumePools.setPoolSize(newSize);


This writes the raw newSize straight to the pools and skips the out-of-service scaling that resize() applies (see the per-volume block in resize). setReplicationMaxStreams sets the config then calls resize(state.get()), so the global pool gets newSize * outOfServiceFactor while decommissioning or in maintenance. Reconfiguring per.volume.streams.limit on a node that is already DECOMMISSIONING leaves the per-volume pools at the unscaled value, which is inconsistent with the global pool during the exact scenario this feature targets.

Suggest mirroring setReplicationMaxStreams: set replicationConfig.setPerVolumeStreamsLimit(newSize) first, then call resize(state.get()) to reuse the existing scaling path instead of calling setPoolSize directly.

smengcl · 2026-07-10T21:07:24Z

+   * @return true if this is a push replication task (source datanode sends
+   *     container to target).
+   */
+  public boolean isPushReplication() {


ReplicateContainerCommand calls Objects.requireNonNull(targetDatanode) in every constructor and in getFromProtobuf, so getTargetDatanode() is never null and this always returns true. The guard if (!replicationTask.isPushReplication()) { return executor; } in ReplicationSupervisor.selectExecutor is therefore unreachable today. If this is meant as a guard for a future pull style task, a short comment saying so would help; otherwise it can be dropped to avoid implying a branch that never runs.

jojochuang requested a review from smengcl July 8, 2026 23:11

jojochuang added the AI-gen label Jul 8, 2026

jojochuang mentioned this pull request Jul 8, 2026

[WIP] HDDS-15412. [DataNode] Disk volume-specific container replication thread pool #10595

Closed

HDDS-15412. Fix checkstyle violations.

320eacd

Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I87c4c544a24f99675ef68940b9412367d60c9b8e

jojochuang requested a review from Copilot July 9, 2026 00:05

Copilot started reviewing on behalf of jojochuang July 9, 2026 00:06 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

jojochuang commented Jul 9, 2026

View reviewed changes

smengcl reviewed Jul 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-15412. Add per-volume push replication thread pools on DataNode.#10693

HDDS-15412. Add per-volume push replication thread pools on DataNode.#10693
jojochuang wants to merge 3 commits into
apache:masterfrom
jojochuang:HDDS-15412-by-cursor

jojochuang commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jojochuang left a comment

Uh oh!

smengcl Jul 10, 2026

Uh oh!

smengcl Jul 10, 2026

Uh oh!

smengcl Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jojochuang commented Jul 8, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Commit summary: 47251206749

Problem

Change

Behavior after this commit

SCM impact

Files changed

Uh oh!

smengcl Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

smengcl Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

smengcl Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Commit summary: `47251206749`