Skip to content

HDDS-15667. containerbalancer status: show stop reason and iteration details after stop#10681

Draft
sreejasahithi wants to merge 3 commits into
apache:masterfrom
sreejasahithi:HDDS-15667
Draft

HDDS-15667. containerbalancer status: show stop reason and iteration details after stop#10681
sreejasahithi wants to merge 3 commits into
apache:masterfrom
sreejasahithi:HDDS-15667

Conversation

@sreejasahithi

@sreejasahithi sreejasahithi commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Currently, when Container Balancer is stopped, ozone admin containerbalancer status only reports: "ContainerBalancer is Not Running."

It would be helpful if after stop, status shows the last known run state:

  • Stop reason (also in non-verbose mode)
  • Start/stop timestamps, balancing duration, and configuration used ( with --verbose flag)
  • Last iteration details ( with --verbose flag)
  • Iteration history details (with --verbose -H flag)

What is the link to the Apache JIRA

HDDS-15667

How was this patch tested?

Tested manually in docker cluster and added test cases.
Sample runs:

bash-5.1$ ozone admin containerbalancer start
Container Balancer started successfully.
bash-5.1$ ozone admin containerbalancer status --verbose -H
ContainerBalancer is Not Running.
Stop reason: INITIALIZATION_FAILED
Message: Failed to initialize a container balancer iteration. Details:  iteration number 1, Did not find any unbalanced Datanodes.
Started at: 2026-07-08 12:26:26
Stopped at: 2026-07-08 12:26:27
Balancing duration: 1s

Container Balancer Configuration values:
Key                                                Value
Threshold                                          10
Max Datanodes to Involve per Iteration(percent)    20
Max Size to Move per Iteration                     500GB
Max Size Entering Target per Iteration             26GB
Max Size Leaving Source per Iteration              26GB
Number of Iterations                               10
Time Limit for Single Container's Movement         65min
Time Limit for Single Container's Replication      50min
Interval between each Iteration                    0min
Whether to Enable Network Topology                 false
Whether to Trigger Refresh Datanode Usage Info     false
Container IDs to Include in Balancing              None
Container IDs to Exclude from Balancing            None
Datanodes Specified to be Balanced                 None
Datanodes Excluded from Balancing                  None

Last iteration info:
Key                                                Value
Iteration number                                   1
Iteration duration                                 1s
Iteration result                                   ITERATION_INTERRUPTED
Size scheduled to move                             0 B
Moved data size                                    0 B
Scheduled to move containers                       0
Already moved containers                           0
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes
 -
Exited data from nodes
 -

Completed iteration history:
-
bash-5.1$ ozone admin containerbalancer start -t 0.1 -d 100 -i 3
Container Balancer started successfully.
bash-5.1$ ozone admin containerbalancer status --verbose -H
ContainerBalancer is Not Running.
Stop reason: CAN_NOT_BALANCE_ANY_MORE
Message: No more eligible container moves were found.
Started at: 2026-07-08 12:45:25
Stopped at: 2026-07-08 12:46:20
Balancing duration: 55s

Container Balancer Configuration values:
Key                                                Value
Threshold                                          0.1
Max Datanodes to Involve per Iteration(percent)    100
Max Size to Move per Iteration                     500GB
Max Size Entering Target per Iteration             26GB
Max Size Leaving Source per Iteration              26GB
Number of Iterations                               3
Time Limit for Single Container's Movement         65min
Time Limit for Single Container's Replication      50min
Interval between each Iteration                    0min
Whether to Enable Network Topology                 false
Whether to Trigger Refresh Datanode Usage Info     false
Container IDs to Include in Balancing              None
Container IDs to Exclude from Balancing            None
Datanodes Specified to be Balanced                 None
Datanodes Excluded from Balancing                  None

Last iteration info:
Key                                                Value
Iteration number                                   2
Iteration duration                                 0s
Iteration result                                   CAN_NOT_BALANCE_ANY_MORE
Size scheduled to move                             0 B
Moved data size                                    0 B
Scheduled to move containers                       0
Already moved containers                           0
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes
 -
Exited data from nodes
 -

Completed iteration history:
Key                                                Value
Iteration number                                   1
Iteration duration                                 30s
Iteration result                                   ITERATION_COMPLETED
Size scheduled to move                             300 MB
Moved data size                                    300 MB
Scheduled to move containers                       3
Already moved containers                           3
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes
0c916231-0bc0-4303-910e-a5e3eedbf0c0 <- 100 MB
39b69c76-67aa-4a04-8675-5ad143a63ad7 <- 100 MB
30b0de7b-a6e5-40c3-ab6b-40b35adf85f7 <- 100 MB
Exited data from nodes
1eccc612-654a-4aa2-919f-2b916aa7f43b -> 100 MB
850f87cb-19ab-4188-bfe3-ff83672f8341 -> 100 MB
a6f2b660-ddfb-4738-817e-9a477428365e -> 100 MB
bash-5.1$ ozone admin containerbalancer status
ContainerBalancer is Not Running.
Stop reason: CAN_NOT_BALANCE_ANY_MORE
Message: No more eligible container moves were found.
bash-5.1$ ozone admin containerbalancer start -t 0.1 -d 100 -i 2
Container Balancer started successfully.
bash-5.1$ ozone admin containerbalancer status --verbose -H
ContainerBalancer is Not Running.
Stop reason: COMPLETED_ALL_ITERATIONS
Message: Completed all configured number of iterations.
Started at: 2026-07-08 12:55:17
Stopped at: 2026-07-08 12:56:59
Balancing duration: 1m 42s

Container Balancer Configuration values:
Key                                                Value
Threshold                                          0.1
Max Datanodes to Involve per Iteration(percent)    100
Max Size to Move per Iteration                     500GB
Max Size Entering Target per Iteration             26GB
Max Size Leaving Source per Iteration              26GB
Number of Iterations                               2
Time Limit for Single Container's Movement         65min
Time Limit for Single Container's Replication      50min
Interval between each Iteration                    0min
Whether to Enable Network Topology                 false
Whether to Trigger Refresh Datanode Usage Info     false
Container IDs to Include in Balancing              None
Container IDs to Exclude from Balancing            None
Datanodes Specified to be Balanced                 None
Datanodes Excluded from Balancing                  None

Last iteration info:
Key                                                Value
Iteration number                                   2
Iteration duration                                 36s
Iteration result                                   ITERATION_COMPLETED
Size scheduled to move                             100 MB
Moved data size                                    0 B
Scheduled to move containers                       3
Already moved containers                           3
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes
1eccc612-654a-4aa2-919f-2b916aa7f43b <- 40 MB
850f87cb-19ab-4188-bfe3-ff83672f8341 <- 40 MB
a6f2b660-ddfb-4738-817e-9a477428365e <- 20 MB
Exited data from nodes
0c916231-0bc0-4303-910e-a5e3eedbf0c0 -> 40 MB
39b69c76-67aa-4a04-8675-5ad143a63ad7 -> 40 MB
30b0de7b-a6e5-40c3-ab6b-40b35adf85f7 -> 20 MB

Completed iteration history:
Key                                                Value
Iteration number                                   1
Iteration duration                                 41s
Iteration result                                   ITERATION_COMPLETED
Size scheduled to move                             160 MB
Moved data size                                    160 MB
Scheduled to move containers                       3
Already moved containers                           3
Failed to move containers                          0
Failed to move containers by timeout               0
Entered data to nodes
0c916231-0bc0-4303-910e-a5e3eedbf0c0 <- 80 MB
39b69c76-67aa-4a04-8675-5ad143a63ad7 <- 20 MB
30b0de7b-a6e5-40c3-ab6b-40b35adf85f7 <- 60 MB
Exited data from nodes
1eccc612-654a-4aa2-919f-2b916aa7f43b -> 20 MB
850f87cb-19ab-4188-bfe3-ff83672f8341 -> 80 MB
a6f2b660-ddfb-4738-817e-9a477428365e -> 60 MB

Green CI : https://github.com/sreejasahithi/ozone/actions/runs/28775666320

…and iteration details (of last and history) after stop
@adoroszlai adoroszlai changed the title HDDS-15667. Containerbalancer status command should show stop reason and iteration details after stop HDDS-15667. containerbalancer status: show stop reason and iteration details after stop Jul 7, 2026

@ArafatKhan2198 ArafatKhan2198 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @sreejasahithi
Some comments for you

Comment thread hadoop-hdds/interface-admin/src/main/resources/proto.lock Outdated

@rakeshadr rakeshadr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sreejasahithi for working on this. Added a few comments, please go thru it.

Comment thread hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto
LocalDateTime dateTime =
LocalDateTime.ofInstant(startedAtInstant, ZoneId.systemDefault());
System.out.println("ContainerBalancer is Running.");
} else if (response.hasContainerBalancerStatusInfo()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we think of adding explicit state(enum ContainerBalancerRunState) instead of overloading boolean. If this requires more code change, testing etc then please raise another sub-task(sub-jira) and improve the code.

enum ContainerBalancerRunState {
  NOT_STARTED = 0;
  RUNNING = 1;
  STOPPING = 2;
  STOPPED = 3;
}
optional ContainerBalancerRunState runState = 6;

Then CLI:

switch (balancerStatusInfo.getRunState()) {
case RUNNING:
  System.out.println("ContainerBalancer is Running.");
  break;
case STOPPING:
  System.out.println("ContainerBalancer is Stopping.");
  break;
case STOPPED:
  System.out.println("ContainerBalancer is Not Running.");
  break;
default:
  System.out.println("ContainerBalancer is Not Running.");
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, will keep this for a follow up jira. and keep the current PR focused on post stop status.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants