A batch job can stay RUNNING for ever with no log line, because the future's throwable is discarded

[aborroy]

Type: bug. Effort: S. Affects all three batch ingesters.

Summary

A batch sync job can be left RUNNING for ever, with no log line of any kind, if the task throws something that is not an Exception. Two things combine to make it silent:

• runJob catches Exception, which does not catch Error or any other Throwable.
• The task is launched with CompletableFuture.runAsync(...) and the returned future is discarded, so a throwable that escapes the catch is captured into that future and never observed, never logged, never surfaced.

The job record therefore keeps status: RUNNING and completedAt: null indefinitely. A caller polling /api/sync/status/{id} waits for ever; a test harness times out and blames the source system.

Observed

A real SharePoint run against Microsoft Graph, 2026-09-21:

GET /api/sync/status/652f0d4b-...
{ "status": "RUNNING", "completedAt": null, "discoveredCount": 52, "syncedCount": 51, "failedCount": 0 }

Nine minutes after the last log line, and it never changed afterwards. Diagnosis:

• Container CPU 0.15%, so nothing was working.
• 50 Chunking produced lines and 50 Completed sync for node lines, so the per-document work had finished.
• docker kill -s QUIT thread dump: the single connector-batch-1 thread is parked in LinkedBlockingQueue.take inside ThreadPoolExecutor.getTask, i.e. back in the pool waiting for new work. No thread anywhere in ConnectorBatchIngestionService, ConnectorDiscoveryService or NodeSyncService.
• No WARN or ERROR from our code, no OutOfMemoryError, no java.lang.Error, no stack trace.

So the task ended and neither job.complete() nor job.fail() ran. With the catch (Exception) in place, a Throwable that is not an Exception is the only way to reach that state, and the discarded future is why nothing was printed.

Note the absence of an Error in the log is consistent with this rather than evidence against it: a throwable captured by a CompletableFuture nobody inspects is never printed at all.

Where

Same shape in all three:

• plugin-runtime/plugin-batch-ingester ConnectorBatchIngestionService:85
• alfresco/alfresco-batch-ingester BatchIngestionService:84 and :97
• nuxeo/nuxeo-batch-ingester NuxeoBatchIngestionService:67 and :74

Fix

Two independent changes, and both are worth making because either alone leaves a gap:

1. Observe the future. CompletableFuture.runAsync(...).whenComplete((ignored, thrown) -> { if (thrown != null) { job.fail(); log.error("...", thrown); } }). This catches everything, including what the inner catch misses, and it is the only place that can.
2. Catch Throwable in runJob, not Exception, so the job is marked failed at the point of failure with the source context still in scope. Rethrow after marking if that is preferred for an Error.

A job left RUNNING is worse than a job marked FAILED, because the second is actionable and the first is indistinguishable from slow.

Acceptance criteria

• A task that throws an Error leaves the job FAILED, not RUNNING, and logs the throwable with the job id.
• The same holds for a Throwable that is neither Exception nor Error.
• A test per ingester, injecting a throwing collaborator, asserting the terminal status and that something was logged.
• Consider whether a job with no progress for N minutes should be reported distinctly, since "stalled" and "working" are currently the same state to a caller. Out of scope for the fix, worth a decision.

Not in scope

What the original throwable was on that run is unknown and unknowable from here, which is the point of the issue. Once the future is observed, the next occurrence names itself.

[aborroy]

Now reproducible, and on a real file. Resumed the same crawl and it died at exactly the same point, which upgrades this from "an unknown throwable once" to something diagnosable.

The evidence

Second run: discovered 52, skipped 51, synced 0, failed 0, status RUNNING for ever. The 51 already-ingested documents took the staleness skip, so the only document with work to do was the 52nd -- and the job died on it, immediately.

Both runs, the last log line names the same document:

Created hxpr document a9cc1ff0-... for node b!...:01KESNXQBKVML2FFEJ7BB3FGZIEPIQ7Y3Y
                      at /connector-sync/my-onedrive/samples/Multilanguage.docx_signed.csig
Updated hxpr document a9cc1ff0-... for node b!...:01KESNXQBKVML2FFEJ7BB3FGZIEPIQ7Y3Y
   <-- nothing after this, ever

Second thread dump, identical to the first: the single connector-batch-1 worker parked in LinkedBlockingQueue.take inside ThreadPoolExecutor.getTask, no frame anywhere in our code. Task ended, job.complete() and job.fail() both unreached, nothing logged, failedCount still 0.

Ruled out: the Graph token was valid throughout both stalls (probed 200 between them, and it only expired later), there were no GraphExceptions, no 429s, no budget waits, and CPU was 0.15%.

The likely throwable

Multilanguage.docx_signed.csig is a digital-signature sidecar, and the stall is immediately after the document is written, which is where extraction runs. Tika on an unusual or malformed binary is a well-known source of StackOverflowError and OutOfMemoryError. Both are Error, so:

• NodeSyncService's per-node handling catches Exception, which is why failedCount stayed 0 rather than incrementing;
• runJob catches Exception, so the job was not marked failed;
• CompletableFuture.runAsync's future is discarded, so the throwable was never printed.

Three layers, each of which would individually have surfaced it.

Why this matters more than the status code

One unusual file in a source silently kills the whole crawl, and the operator sees a job that says RUNNING. The remaining documents are never ingested and nothing anywhere says why. On this run that was 1 of 52; on a customer corpus it could be document 900 of 100,000.

So the fix in the issue body is necessary but not sufficient. Extraction of a single document must not be able to end the pass:

• Observe the future, as described above.
• Catch Throwable per node in the sync loop, count it as a failure, and continue to the next document. A corpus of 100,000 documents must not be hostage to one of them. This is the part that turns a dead crawl into failedCount: 1.
• Catch Throwable in runJob so the terminal status is FAILED rather than RUNNING if something does escape.
• A test that feeds an extractor which throws an Error, asserting the pass continues and the job terminates.

Note the second item has a judgement call in it: an OutOfMemoryError is not safely recoverable in general, since the heap may be exhausted for every subsequent document too. Catching it per node and continuing is right for a StackOverflowError and arguable for an OOM. Marking the job FAILED with the node named is the minimum either way, and is strictly better than the current silence.

[aborroy]

Fixed in fbe3f44, both halves.

The future is now observed (whenComplete), which is the only place a throwable that escaped the task can be
seen at all, and the inner catch is widened to Throwable so the job is marked at the point of failure with the
source context still in scope. An Error is rethrown after marking, so a JVM-level problem still reaches the
executor rather than being swallowed on the way.

All five call sites: ConnectorBatchIngestionService, and both in each of BatchIngestionService and
NuxeoBatchIngestionService.

One thing this fix had to touch that the issue did not mention, and it is the part worth reading. All three
ingesters call job.complete() before the reconciliation sweep, deliberately, so that a sweep failure cannot
turn a successful ingestion into a failed job. That invariant was held by nothing except the sweep's own
catch (Exception). Widening the caller's catch to Throwable on its own would therefore have let a sweep
throwable reach it and downgrade an already-COMPLETED job -- a regression in the opposite direction from this
bug, in a method whose diff would have looked untouched. The sweep now catches Throwable too and deliberately
does not rethrow, and aSweepThrowableLeavesACompletedJobCompleted pins it. Do not narrow either catch back.

Tests: one per ingester, injecting a collaborator that throws an Error, plus one for a Throwable that is
neither Exception nor Error -- the case neither clause used to reach.

On the fourth acceptance criterion

Consider whether a job with no progress for N minutes should be reported distinctly, since "stalled" and
"working" are currently the same state to a caller.

Recorded as considered and not built, which is what that criterion asked for.

It is a different kind of change from this one. This bug was a throwable going unobserved, and the fix makes
every terminating job reach a terminal state. A stall detector is new state to model: it needs a definition of
progress, a threshold per ingester, a clock, and a new value a caller has to learn to interpret -- and it would
have caught nothing here, because the job in this issue was not slow, it had already ended.

What made that run undiagnosable was the silence, and that is gone: the next occurrence names itself in the log
with the job id. If a genuine stall shows up later -- a job whose task is alive and making no progress -- that is
worth filing on its own evidence rather than pre-emptively.

Closing.