Problem
Polaris writes asynchronous task metadata (such as file cleanup tasks after a DROP TABLE operation) to the database as TASK entities. These tasks are scheduled to run asynchronously. However, if a node crashes, the JVM restarts, or a transient error occurs during execution, these tasks become orphaned and remain stuck in the database indefinitely.
Although the metastore layer implements CAS-based task leasing (PolarisMetaStoreManager#loadTasks), there is currently no background thread or daemon in Polaris that polls this endpoint to recover and execute orphaned tasks.
Proposal
Introduce a distributed background task sweeper daemon (PolarisTaskSweeper) to periodically lease, run, and clean up stuck tasks across all active realms.
Design Highlights
- CAS Leasing: Periodically calls
loadTasks which uses internal CAS updates (LAST_ATTEMPT_START_TIME, ATTEMPT_COUNT) to lease tasks. Multiple Polaris nodes can safely sweep concurrently without double-executing tasks.
- Async Processing: Claims are processed asynchronously on the shared
task-executor thread pool to avoid blocking the scheduler thread during long cloud storage cleanups.
- Observability: Fires standard
BEFORE_ATTEMPT_TASK and AFTER_ATTEMPT_TASK events so sweeps are visible to audit loggers and OpenTelemetry listeners.
- Thundering Herd Mitigation: Applies a random jitter (up to 20% of the interval) before runs so multi-node deployments do not hit the database simultaneously.
- Configuration: Exposes
polaris.tasks.sweeper.enabled (default: true) and polaris.tasks.sweeper.interval (default: 1m) configuration properties.
I have a fully tested local implementation of this daemon (including unit tests and documentation updates) ready on a branch and can submit a PR.
Problem
Polaris writes asynchronous task metadata (such as file cleanup tasks after a
DROP TABLEoperation) to the database asTASKentities. These tasks are scheduled to run asynchronously. However, if a node crashes, the JVM restarts, or a transient error occurs during execution, these tasks become orphaned and remain stuck in the database indefinitely.Although the metastore layer implements CAS-based task leasing (
PolarisMetaStoreManager#loadTasks), there is currently no background thread or daemon in Polaris that polls this endpoint to recover and execute orphaned tasks.Proposal
Introduce a distributed background task sweeper daemon (
PolarisTaskSweeper) to periodically lease, run, and clean up stuck tasks across all active realms.Design Highlights
loadTaskswhich uses internal CAS updates (LAST_ATTEMPT_START_TIME,ATTEMPT_COUNT) to lease tasks. Multiple Polaris nodes can safely sweep concurrently without double-executing tasks.task-executorthread pool to avoid blocking the scheduler thread during long cloud storage cleanups.BEFORE_ATTEMPT_TASKandAFTER_ATTEMPT_TASKevents so sweeps are visible to audit loggers and OpenTelemetry listeners.polaris.tasks.sweeper.enabled(default:true) andpolaris.tasks.sweeper.interval(default:1m) configuration properties.I have a fully tested local implementation of this daemon (including unit tests and documentation updates) ready on a branch and can submit a PR.