pkg/ddl, pkg/dxf: add local sort disk check and logging for backfill workers (#68320)

[expxiaoli]

This is a cherry-pick of #68320.

What problem does this PR solve?

Issue Number: ref #68293

Problem Summary:

When ADD INDEX runs with local sort on TiDB workers, the worker does not have an admission check for local disk headroom. A new backfill task may start even when the remaining disk is not enough for the ongoing local-sort workload plus the extra 10% headroom reserved for partial import.

Besides, there is no summary view of how much local disk space the backfill actually used, which makes disk pressure diagnosis harder.

What changed and how does it work?

This PR cherry-picks #68320 to improve both prevention and observability for local-sort ADD INDEX backfill.

The change:

• Adds local-sort disk usage sampling to the read-index backfill pipeline and logs a structured summary after the pipeline finishes.
• Adds a TiDB worker-side disk precheck when the worker receives a local-sort backfill task.
• Snapshots current executor tasks from the DXF slot manager and derives the required slots for each local-sort job.
• Calculates required disk bytes from running local-sort jobs plus the new job.
• Returns an error early when available disk space is insufficient, so local-sort ADD INDEX can fail fast before entering a higher-risk execution path.
• Adds regression coverage for the new disk check and DXF slot snapshot.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

Unit tests and checks:

• GOPATH=/Users/xiaoli/go GOSUMDB=sum.golang.org ./tools/check/failpoint-go-test.sh pkg/ddl/ingest -run TestRiskOfDiskFull -count=1
• GOPATH=/Users/xiaoli/go GOSUMDB=sum.golang.org ./tools/check/failpoint-go-test.sh pkg/dxf/framework/taskexecutor -run TestSlotManager -count=1
• GOPATH=/Users/xiaoli/go GOSUMDB=sum.golang.org ./tools/check/failpoint-go-test.sh pkg/ddl -run '^$' -count=1
• GOPATH=/Users/xiaoli/go GOSUMDB=sum.golang.org make lint
• git diff --check HEAD~2..HEAD
• git range-diff origin/pr/68320~2..origin/pr/68320 HEAD~2..HEAD

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

`ADD INDEX` with local sort now checks TiDB worker local disk space before backfill starts and fails fast when the remaining disk is insufficient for the local-sort workload plus the extra 10% headroom reserved for partial import.

[pantheon-ai]

@expxiaoli I've received your pull request and will start the review. I'll conduct a thorough review covering code quality, potential issues, and implementation details.

⏳ This process typically takes 10-30 minutes depending on the complexity of the changes.

_{ℹ️ Learn more details on Pantheon AI.}

[tiprow]

Hi @expxiaoli. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

Important

Review skipped

Too many files!

This PR contains 261 files, which is 111 over the limit of 150.

To get a review, narrow the scope:
• coderabbit review --type committed # exclude uncommitted changes
• coderabbit review --dir # limit to a subdirectory
• coderabbit review --base # compare against a closer base

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 3fc7758e-3213-4271-b41e-47417b4e5c57

📥 Commits

Reviewing files that changed from the base of the PR and between e370684 and ac150c8.

⛔ Files ignored due to path filters (4)

• go.sum is excluded by !**/*.sum
• lightning/pkg/checkpoints/checkpointspb/file_checkpoints.pb.go is excluded by !**/*.pb.go
• tests/realtikvtest/importintotest/spark-legacy-date.gz.parquet is excluded by !**/*.parquet
• tests/realtikvtest/importintotest/spark-legacy-datetime.gz.parquet is excluded by !**/*.parquet

📒 Files selected for processing (261)

• .github/licenserc.yml
• DEPS.bzl
• Makefile
• README.md
• br/pkg/restore/BUILD.bazel
• br/pkg/restore/internal/import_client/BUILD.bazel
• br/pkg/restore/internal/import_client/import_client_test.go
• br/pkg/restore/misc_test.go
• br/pkg/restore/snap_client/BUILD.bazel
• br/pkg/restore/snap_client/import_test.go
• br/pkg/restore/snap_client/pipeline_items.go
• br/pkg/restore/split/BUILD.bazel
• br/pkg/restore/split/client.go
• br/pkg/restore/split/client_test.go
• br/pkg/restore/split/mock_pd_client.go
• br/pkg/restore/split/split.go
• br/pkg/restore/split/split_test.go
• cmd/tidb-server/BUILD.bazel
• cmd/tidb-server/main.go
• cmd/tidb-server/main_test.go
• go.mod
• lightning/pkg/checkpoints/BUILD.bazel
• lightning/pkg/checkpoints/checkpoints.go
• lightning/pkg/checkpoints/checkpoints_file_test.go
• lightning/pkg/checkpoints/checkpoints_sql_test.go
• lightning/pkg/checkpoints/checkpoints_test.go
• lightning/pkg/checkpoints/checkpointspb/BUILD.bazel
• lightning/pkg/checkpoints/checkpointspb/file_checkpoints.proto
• lightning/pkg/checkpoints/main_test.go
• lightning/pkg/errormanager/BUILD.bazel
• lightning/pkg/errormanager/errormanager.go
• lightning/pkg/errormanager/errormanager_test.go
• lightning/pkg/errormanager/resolveconflict_test.go
• lightning/pkg/importer/BUILD.bazel
• lightning/pkg/importer/check_info_test.go
• lightning/pkg/importer/checksum_helper.go
• lightning/pkg/importer/chunk_process.go
• lightning/pkg/importer/chunk_process_test.go
• lightning/pkg/importer/dup_detect.go
• lightning/pkg/importer/get_pre_info.go
• lightning/pkg/importer/import.go
• lightning/pkg/importer/import_test.go
• lightning/pkg/importer/meta_manager_test.go
• lightning/pkg/importer/precheck.go
• lightning/pkg/importer/precheck_impl.go
• lightning/pkg/importer/precheck_impl_test.go
• lightning/pkg/importer/table_import.go
• lightning/pkg/importer/table_import_test.go
• lightning/pkg/importer/tidb.go
• lightning/pkg/importer/tidb_test.go
• lightning/pkg/server/BUILD.bazel
• lightning/pkg/server/checkpoint_control.go
• lightning/pkg/server/checkpoint_control_test.go
• lightning/pkg/server/lightning.go
• lightning/pkg/server/lightning_serial_test.go
• lightning/pkg/web/BUILD.bazel
• lightning/pkg/web/progress.go
• lightning/tests/lightning_write_timeout/run.sh
• pkg/config/BUILD.bazel
• pkg/config/config.go
• pkg/config/config.toml.example
• pkg/config/config.toml.nextgen.example
• pkg/config/config_test.go
• pkg/config/deploymode/BUILD.bazel
• pkg/config/deploymode/doc.go
• pkg/config/deploymode/mode.go
• pkg/config/deploymode/mode_test.go
• pkg/ddl/BUILD.bazel
• pkg/ddl/backfilling_dist_executor.go
• pkg/ddl/backfilling_dist_scheduler.go
• pkg/ddl/backfilling_operators.go
• pkg/ddl/backfilling_read_index.go
• pkg/ddl/create_table.go
• pkg/ddl/ddl_tiflash_api.go
• pkg/ddl/executor.go
• pkg/ddl/index.go
• pkg/ddl/index_modify_test.go
• pkg/ddl/ingest/BUILD.bazel
• pkg/ddl/ingest/backend.go
• pkg/ddl/ingest/backend_mgr.go
• pkg/ddl/ingest/config.go
• pkg/ddl/ingest/disk_root.go
• pkg/ddl/ingest/mem_root_test.go
• pkg/ddl/partition.go
• pkg/ddl/reorg_util.go
• pkg/ddl/reorg_util_test.go
• pkg/ddl/tiflash_replica_test.go
• pkg/distsql/context/BUILD.bazel
• pkg/distsql/context/context.go
• pkg/distsql/context/context_test.go
• pkg/distsql/distsql.go
• pkg/domain/BUILD.bazel
• pkg/domain/domain.go
• pkg/domain/infosync/BUILD.bazel
• pkg/domain/infosync/info.go
• pkg/domain/infosync/info_test.go
• pkg/domain/infosync/tiflash_manager.go
• pkg/dxf/framework/handle/BUILD.bazel
• pkg/dxf/framework/handle/handle.go
• pkg/dxf/framework/handle/status.go
• pkg/dxf/framework/handle/status_testkit_test.go
• pkg/dxf/framework/integrationtests/framework_test.go
• pkg/dxf/framework/proto/node.go
• pkg/dxf/framework/proto/task_test.go
• pkg/dxf/framework/scheduler/nodes.go
• pkg/dxf/framework/scheduler/scheduler.go
• pkg/dxf/framework/storage/BUILD.bazel
• pkg/dxf/framework/storage/history.go
• pkg/dxf/framework/storage/nodes.go
• pkg/dxf/framework/storage/subtask_state.go
• pkg/dxf/framework/storage/table_test.go
• pkg/dxf/framework/taskexecutor/manager.go
• pkg/dxf/framework/taskexecutor/slot.go
• pkg/dxf/framework/taskexecutor/slot_test.go
• pkg/dxf/framework/taskexecutor/task_executor.go
• pkg/dxf/framework/testutil/context.go
• pkg/dxf/framework/testutil/table_util.go
• pkg/dxf/importinto/BUILD.bazel
• pkg/dxf/importinto/job.go
• pkg/dxf/importinto/planner.go
• pkg/dxf/importinto/subtask_executor.go
• pkg/dxf/importinto/wrapper.go
• pkg/dxf/importinto/wrapper_test.go
• pkg/executor/BUILD.bazel
• pkg/executor/adapter.go
• pkg/executor/adapter_internal_test.go
• pkg/executor/adapter_test.go
• pkg/executor/distsql_test.go
• pkg/executor/executor_failpoint_test.go
• pkg/executor/explain.go
• pkg/executor/explain_unit_test.go
• pkg/executor/importer/BUILD.bazel
• pkg/executor/importer/chunk_process.go
• pkg/executor/importer/chunk_process_testkit_test.go
• pkg/executor/importer/engine_process.go
• pkg/executor/importer/import.go
• pkg/executor/importer/import_test.go
• pkg/executor/importer/importer_testkit_test.go
• pkg/executor/importer/sampler.go
• pkg/executor/importer/sampler_test.go
• pkg/executor/importer/table_import.go
• pkg/executor/infoschema_reader.go
• pkg/executor/internal/exec/executor.go
• pkg/executor/staticrecordset/BUILD.bazel
• pkg/executor/staticrecordset/recordset.go
• pkg/executor/test/distsqltest/BUILD.bazel
• pkg/executor/test/distsqltest/distsql_test.go
• pkg/executor/test/memtest/BUILD.bazel
• pkg/expression/aggregation/BUILD.bazel
• pkg/expression/aggregation/base_func.go
• pkg/expression/aggregation/base_func_test.go
• pkg/ingestor/ingestcli/BUILD.bazel
• pkg/ingestor/ingestcli/client_test.go
• pkg/lightning/backend/BUILD.bazel
• pkg/lightning/backend/backend.go
• pkg/lightning/backend/external/BUILD.bazel
• pkg/lightning/backend/external/byte_reader.go
• pkg/lightning/backend/external/byte_reader_test.go
• pkg/lightning/backend/external/engine.go
• pkg/lightning/backend/external/engine_test.go
• pkg/lightning/backend/external/iter_test.go
• pkg/lightning/backend/external/reader.go
• pkg/lightning/backend/local/BUILD.bazel
• pkg/lightning/backend/local/checksum.go
• pkg/lightning/backend/local/checksum_test.go
• pkg/lightning/backend/local/duplicate.go
• pkg/lightning/backend/local/duplicate_test.go
• pkg/lightning/backend/local/engine.go
• pkg/lightning/backend/local/job_worker.go
• pkg/lightning/backend/local/job_worker_test.go
• pkg/lightning/backend/local/local.go
• pkg/lightning/backend/local/local_test.go
• pkg/lightning/backend/local/localhelper.go
• pkg/lightning/backend/local/localhelper_test.go
• pkg/lightning/backend/local/region_job.go
• pkg/lightning/backend/tidb/BUILD.bazel
• pkg/lightning/backend/tidb/tidb.go
• pkg/lightning/backend/tidb/tidb_test.go
• pkg/lightning/importdef/BUILD.bazel
• pkg/lightning/importdef/tidb.go
• pkg/lightning/membuf/buffer.go
• pkg/lightning/membuf/buffer_test.go
• pkg/lightning/membuf/limiter.go
• pkg/lightning/mydump/BUILD.bazel
• pkg/lightning/mydump/csv_parser_test.go
• pkg/lightning/mydump/parquet_parser.go
• pkg/lightning/mydump/parquet_parser_test.go
• pkg/lightning/mydump/parquet_type_converter.go
• pkg/lightning/mydump/parquet_writer.go
• pkg/lightning/mydump/parser.go
• pkg/lightning/mydump/spark_rebase.go
• pkg/lightning/mydump/spark_rebase_micros_generated.go
• pkg/metrics/ru_v2.go
• pkg/parser/ast/functions.go
• pkg/parser/ast/misc.go
• pkg/parser/ast/misc_test.go
• pkg/parser/parser.go
• pkg/parser/parser.y
• pkg/parser/parser_test.go
• pkg/server/BUILD.bazel
• pkg/server/conn_stmt_test.go
• pkg/server/handler/tests/dxf_test.go
• pkg/server/handler/tikvhandler/dxf.go
• pkg/server/http_status.go
• pkg/server/internal/resultset/resultset.go
• pkg/session/BUILD.bazel
• pkg/session/session.go
• pkg/session/session_test.go
• pkg/session/test/temporarytabletest/BUILD.bazel
• pkg/session/test/variable/BUILD.bazel
• pkg/session/test/variable/variable_test.go
• pkg/session/tidb_test.go
• pkg/sessionctx/variable/nextgen_test.go
• pkg/sessionctx/variable/slow_log.go
• pkg/sessionctx/variable/sysvar.go
• pkg/sessionctx/variable/sysvar_test.go
• pkg/sessionctx/variable/tests/slowlog/slow_log_test.go
• pkg/sessionctx/variable/varsutil_test.go
• pkg/sessiontxn/isolation/BUILD.bazel
• pkg/sessiontxn/isolation/base.go
• pkg/sessiontxn/isolation/base_test.go
• pkg/sessiontxn/isolation/main_test.go
• pkg/sessiontxn/staleread/BUILD.bazel
• pkg/sessiontxn/staleread/provider.go
• pkg/sessiontxn/staleread/provider_test.go
• pkg/statistics/handle/handletest/statstest/BUILD.bazel
• pkg/store/copr/BUILD.bazel
• pkg/store/copr/coprocessor.go
• pkg/store/copr/mpp.go
• pkg/store/copr/ruv2_metrics.go
• pkg/store/driver/txn/BUILD.bazel
• pkg/store/driver/txn/ruv2_metrics.go
• pkg/store/driver/txn/ruv2_metrics_test.go
• pkg/store/helper/helper.go
• pkg/util/execdetails/BUILD.bazel
• pkg/util/execdetails/execdetails.go
• pkg/util/execdetails/execdetails_test.go
• pkg/util/execdetails/ruv2_metrics.go
• pkg/util/execdetails/tiflash_stats.go
• pkg/util/execdetails/util.go
• pkg/util/memory/BUILD.bazel
• pkg/util/printer/BUILD.bazel
• pkg/util/printer/printer.go
• pkg/util/printer/printer_test.go
• pkg/util/stmtsummary/statement_summary.go
• pkg/util/stmtsummary/v2/record.go
• pkg/util/timeutil/time_zone.go
• pkg/util/timeutil/time_zone_test.go
• pkg/util/topsql/stmtstats/BUILD.bazel
• pkg/util/topsql/stmtstats/stmtstats_test.go
• tests/integrationtest/r/expression/issues.result
• tests/integrationtest/t/expression/issues.test
• tests/realtikvtest/addindextest2/global_sort_test.go
• tests/realtikvtest/addindextest4/BUILD.bazel
• tests/realtikvtest/addindextest4/integration_test.go
• tests/realtikvtest/importintotest/BUILD.bazel
• tests/realtikvtest/importintotest/parquet_test.go
• tests/realtikvtest/sessiontest/session_fail_test.go
• tests/realtikvtest/txntest/replica_read_test.go
• tests/realtikvtest/txntest/shared_lock_test.go
• tests/realtikvtest/txntest/stale_read_test.go

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign bb7133, d3hunter, leavrth, nolouch, terry1purcell, windtalker for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• br/OWNERS
• lightning/OWNERS
• pkg/config/OWNERS
• pkg/ddl/OWNERS
• pkg/distsql/OWNERS
• pkg/domain/OWNERS
• pkg/dxf/OWNERS
• pkg/executor/importer/OWNERS
• pkg/expression/OWNERS
• pkg/ingestor/OWNERS
• pkg/lightning/OWNERS
• pkg/metrics/OWNERS
• pkg/parser/OWNERS
• pkg/sessionctx/variable/OWNERS
• pkg/statistics/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@expxiaoli: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-error-log-review	ac150c8	link	false	/test pull-error-log-review
pull-lightning-integration-test	ac150c8	link	true	/test pull-lightning-integration-test
pull-integration-e2e-test	ac150c8	link	true	/test pull-integration-e2e-test
idc-jenkins-ci-tidb/check_dev	ac150c8	link	true	/test check-dev
pull-build-next-gen	ac150c8	link	true	/test pull-build-next-gen
idc-jenkins-ci-tidb/check_dev_2	ac150c8	link	true	/test check-dev2
pull-mysql-client-test-next-gen	ac150c8	link	true	/test pull-mysql-client-test-next-gen
pull-integration-realcluster-test-next-gen	ac150c8	link	true	/test pull-integration-realcluster-test-next-gen
pull-br-integration-test	ac150c8	link	true	/test pull-br-integration-test
pull-mysql-client-test	ac150c8	link	true	/test pull-mysql-client-test
idc-jenkins-ci-tidb/build	ac150c8	link	true	/test build
idc-jenkins-ci-tidb/mysql-test	ac150c8	link	true	/test mysql-test
pull-unit-test-ddlv1	ac150c8	link	true	/test pull-unit-test-ddlv1
pull-unit-test-next-gen	ac150c8	link	true	/test pull-unit-test-next-gen
idc-jenkins-ci-tidb/unit-test	ac150c8	link	true	/test unit-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.