fix(taskworker): Measure child busy and wait time in shared memory

[enochtangg]

Context

Taskworker autoscaling is blocked by the occupancy metric. It reads below reality and the gap widens with load, so process-segments-push in s4s2 sits around 0.45 against a KEDA threshold of 0.8 and never scales out, even with its child queue pinned full.

Root Issue

In the first iterations of tracking child process utilization, each child pushed busy/idle events through a multiprocessing.Queue to a parent thread that competes for CPU with the children it measures. Under load that thread falls behind as backlog grows, and each event is billed to whichever flush interval eventually drains it rather than the one it happened in.

Changes

Children now write their own cumulative busy and wait totals into a shared-memory slot and the parent diffs and flushes on each interval, so each round of accounting costs O(children) per second instead of O(tasks) per second and there is no queue left to fall behind.

To accomplish this:

• Each child gets its own RawArray('double') at spawn, five doubles wide: a version counter, cumulative busy, cumulative wait, the open segment's start, and its kind. The parent holds the only reference, on the TrackedChild, so the slot lives exactly as long as the child does. There is no pool to size and no free list to hand a slot back to, so no exit path can lose one.
• Each child owns one slot and is its sole writer. Therefore, we can just track totals as plain floats and republish the whole slot on each transition.
• The version counter is used which guards against the parent trying to flush the slot at the same time a child is trying to update it. The writer bumps the version to odd before touching a field and to even after; the reader retries up to three times if it sees an odd or changed version.
  • A failed read defers rather than drops. The reader leaves its baselines untouched, so the following sample covers both intervals.
• The parent folds each child's open segment forward at read time by adding now - segment_start to whichever counter is open. A child inside a long task therefore contributes to every interval it spans instead of dumping its whole duration into the interval it happens to finish in.
• Wait is tracked as the mirror of busy rather than inferred as elapsed - busy, and both counters sum over running children only, to match occupancy's divisor.
• A slot is freed when its child is reaped, not on the exiting transition. The child publishes once more on its way out, and multiprocessing's heap hands a freed block straight back to the next allocation, so an early release would let a departing child write into its replacement's slot.
• Occupancy is divided by the summed per-child eligible window instead of elapsed * running_count. A child started part-way through a flush could never have filled it, and billing it whole reads as idle time the pool never had.
  • Eligibility is measured from each child's last successful sample, so a deferred read carries its window forward with it and busy + wait <= eligible survives a retry.

Testing

Six sandbox tests on a single worker pod and broker pod were run to validate accuracy of metric.

Test 1 is the control: a 40ms task at ~10% CPU, which pushes ~1600 messages/s but leaves the parent CPU to spare, and is the one case the old code got right. Tests 2-5 run a 100ms all-CPU task, which is the adversarial case, since the accounting thread competes directly with the children it measures. Test 6 adds aggressive recycling at 150 tasks per child which turns a child over every ~35s, for 471 spawns in 11 minutes and as many slots allocated and freed.

arm	C	task	recycling	occupancy	true	error	error before
1	64	40ms, ~10% CPU	off	0.6301	0.6288	+0.0012	+0.014
2	16	100ms, all CPU	off	0.9931	0.9938	-0.0007	-0.566
3	24	100ms, all CPU	off	0.9950	0.9954	-0.0004	-0.470
4	12	100ms, all CPU	off	0.9959	0.9965	-0.0005	-0.289
5	32	100ms, all CPU	off	0.9951	0.9954	-0.0002	-0.361
6	32	100ms, all CPU	150 tasks	0.9812	0.9954	-0.014	not run

[evanh]

Something to think about: it's OK if occupancy goes above 1.0. You can have greater than 100% utilization. Not sure if that simplifies any logic here.

[evanh]

Can this all be simplified? This is a lot of code to measure "executing a task" vs. "not executing a task".

[enochtangg]

Discussed offline: there has been a significant change in the implementation since last reviewed. We have replaced the whole accounting path with shared memory, and occupancy now divides by the summed per-child eligible window rather than elapsed * running_count, so it's probably worth a fresh pass.

On going above 1.0, a child is either executing or blocked in get_task at any instant, so its window has a physical bound busy <= eligible, which means above 1.0 can only be the accounting double-billing. That's why the clamp is paired with accounting_overflow: the clamp keeps the value bounded and the counter makes the double-billing visible instead of letting it read as a healthy 1.0. Most in-flight reads never get that far, since the seqlock catches them and the parent leaves its baselines alone so the next flush covers both intervals rather than dropping one.

[cursor]

Stale Bugbot comment from a previous run.

[evanh]

Is there some way to avoid slot acquisition? I am worried that we now have a failure mode where there is an accounting bug and processes stop getting assigned slots.

[enochtangg]

Slot exhaustion should not stop children from spawning or running tasks. If no slot can be acquired, it should continue; however, the metric becomes broken and it won't be able to scale out by utilization. In theory, this shouldn't happen as we assign 2x the slots what we need, it's possible it's a failure mode that could come up somehow unexpectedly. I added a metric to monitor that: taskworker.worker.child.timing_slot_exhausted.

To mitigate this though, we could assign each child process its own RawArray('d', 5) at spawn. This should also reduce the surface area that might cause potential memory leaks as well.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7afcd9e. Configure here.}