Skip to content

fix(builder): abandon the load operation when a resource load times out - #1553

Draft
joaodinissf wants to merge 1 commit into
masterfrom
fix/parallel-loader-timeout-count
Draft

joaodinissf wants to merge 1 commit into
masterfrom
fix/parallel-loader-timeout-count

Conversation

@joaodinissf

@joaodinissf joaodinissf commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Why the change

After a resource load times out, the builder silently leaves some other resource out of the index; this makes a timeout cancel the build visibly instead.

Special things to note

  • Behaviour change: a load timeout while writing resource descriptions now cancels the build (rolled back, next build is a full build). On master that phase carried on with a stale index; the linking phase already cancelled this way on master, so both phases now behave the same.
  • A load that always stalls longer than the timeout (300 s with no result from any worker) now fails every build instead of quietly degrading the index. An interrupted wait is still reported as a timeout; that pre-existing issue is left for a follow-up.
  • Raising the timeout alone does not fix the miscount, it only makes it rarer: on master every timeout leaves the counter one short for the rest of the operation.

Change outline

The owed-result counter in ParallelLoadOperation.next():

 result = resourceQueue.poll(waitTime)
-toProcess--                       # also on timeout: counter now one short
+if result != null: toProcess--
+else: timedOut = true
 if result == null
+  if timedOut: cancel()           # toProcess = 0, shutdownNow; "remaining loads are abandoned"
   throw LoadOperationException(null, TimeoutException)

What the builder does after a timeout:

 MonitoredClusteringBuilderState
   linking loop     timeout -> log, then !hasNext() -> cancel build      (unchanged)
   writeResources   timeout -> log
-                    loop ends when hasNext() is false: rest silently unindexed
+                    isAbandonedByTimeout(ex, loadOperation) -> OperationCanceledException

Tests (ParallelResourceLoaderTest, each run against an unbounded queue and production's synchronous hand-off):

  • timeoutAbandonsEveryOutstandingLoad: one timeout ends the operation and queued loads never start. Fails on master.
  • failedLoadDoesNotAbandonTheOperation: an ordinary load failure keeps the other loads.
  • resultWithinTimeoutIsDelivered: sanity check.
  • BuilderLoadTimeoutTest: only a timeout with nothing left to load counts as abandonment; a timeout with loads left, an ordinary load failure and other wrapped exceptions do not.

🤖 Generated with Claude Code

joaodinissf added a commit that referenced this pull request Sep 26, 2026
formal/README.md describes the method, target status, models and how to
reproduce. formal/BUGS.md catalogues 51 findings (49 confirmed, 1
plausible, 1 refuted) plus 9 observations, all verified by three
independent skeptics, with traces, test status, fix plans, a proposed
fix-PR sequence and links to the fix PRs opened so far (#1550, #1551,
#1552, #1553). REPORT.md is the chronological log of rounds 1-2. The
patches are reference fixes used to show each disabled test turns green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
if (result != null) {
toProcess--;
consecutiveTimeouts = 0;
} else if (++consecutiveTimeouts >= MAX_CONSECUTIVE_TIMEOUTS) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not get the point of this PR, could we not just increase the timeout if we want to try longer?

ParallelLoadOperation.next() decremented the outstanding-result counter
even when poll() timed out. From then on the counter was one short of the
results still coming, so the builder either aborted the cluster as if it
had been cancelled or, in writeResources, left some other resource
silently unindexed. A longer timeout makes this rarer but cannot prevent
it.

Decrement only when a result is delivered. A timeout cannot be attributed
to a URI, and a load that never finishes would otherwise keep the builder
waiting forever, so a timeout now cancels the whole operation and says so.
writeResources treats that like the linking phase already did on master:
it cancels the build instead of dropping the remaining resources, so the
next build is a full build rather than one with a stale index. The check
is a public predicate, isAbandonedByTimeout, so that it can be tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joaodinissf
joaodinissf force-pushed the fix/parallel-loader-timeout-count branch from 4282d4c to 95ccf71 Compare September 28, 2026 19:15
@joaodinissf joaodinissf changed the title fix(builder): keep waiting for a timed-out resource load instead of dropping it fix(builder): abandon the load operation when a resource load times out Sep 28, 2026
joaodinissf added a commit that referenced this pull request Sep 29, 2026
formal/README.md describes the method, target status, models and how to
reproduce. formal/BUGS.md catalogues 51 findings (49 confirmed, 1
plausible, 1 refuted) plus 9 observations, all verified by three
independent skeptics, with traces, test status, fix plans, a proposed
fix-PR sequence and links to the fix PRs opened so far (#1550, #1551,
#1552, #1553). REPORT.md is the chronological log of rounds 1-2. The
patches are reference fixes used to show each disabled test turns green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants