fix(selector): close sherdlock lock-contention gaps (Phases 3-8, #2395)

[adecaro]

Summary

Fixes the sherdlock token selector's lock-contention hot-spot that caused spurious "insufficient funds" errors under concurrent load (#2395).

Six mechanisms made a small number of "hot" tokens absorb the vast majority of lock collisions. This PR closes all of them:

1. Anti-join — the DB query that feeds the selector now excludes already-locked tokens, so selectors stop queuing up to fight over the same row.
2. Amount-ordered candidates + shuffle — tokens are fetched smallest-first so a small payment doesn't wastefully grab a large token; equal-amount candidates are shuffled so contention doesn't just shift to whichever token happens to sort first.
3. Sufficiency-window randomization — when the smallest sufficient token is found, a bounded lookahead picks uniformly among similarly-sized candidates, preventing all concurrent selectors from deterministically targeting the exact same token.
4. Blacklisting — a token that lost a lock race is skipped for the rest of that Select() call, instead of being retried in a tight loop for minutes.
5. Immediate lock release on settlement — when a transaction reaches a terminal status (confirmed, deleted, or retry-exhausted), its locks are released immediately via the finality listener and recovery handler, rather than sitting until the 3-minute lease-expiry sweep during which those tokens remain invisible to all other selectors.
6. Postgres lock strategies — two new acquisition modes (onConflict, skipLocked) that avoid server-side unique-constraint violations on lost races, plus batch locking (LockBatch) that claims a covering window of candidates in a single round-trip.

Consolidates the #2398-#2403 stack (Phases 3-8 of #2395) into a single PR on top of #2397's diagnostics/baseline, so the remaining lock-contention work reviews as one unit instead of five stacked PRs. Supersedes #2398, #2399, #2400, #2402, #2403.

Follow-up: two review-flagged gaps closed

• Selection was still deterministic under realistic, distinct token amounts. The Phase 4 bucketedIterator/NewPermutation shuffle only randomizes within contiguous runs of byte-equal Quantity; with mostly-distinct amounts (the CERT-incident shape) every bucket is size 1, so every concurrent selector always targeted the single smallest sufficient token — reproducing the exact hot-token pattern the issue warned against. Fixed in the selector layer (sherdlock/selector.go): when the ascending candidate scan reaches a token that alone covers the remaining requested amount, it now looks ahead over a small bounded window (count-capped by sufficiencyWindow, magnitude-capped by maxSufficiencyRatio so it can't grab a wildly oversized token) and picks uniformly at random among the sufficient candidates in that window. New test TestSizeOrderedSelection_SufficiencyWindowShuffle proves real spread among several distinct, individually-sufficient amounts while the existing deterministic smallest-fit test still holds.
• Listener.OnError never released locks, unlike OnStatus/recovery — a transaction whose finality notification is permanently undeliverable (retry budget exhausted) kept its locks held until the next lease-expiry sweep, reproducing Phase 5's original gap via a different trigger. OnError now calls the same releaseLocks helper as runOnStatus; new tests TestOnError_ReleasesLocks and TestOnError_LockReleaseErrorDoesNotPropagate cover it.

Follow-up: independent review, 6 fixes (TDD, each with its own commit)

• Listener.runOnStatus's default: branch released locks on non-terminal network.Busy/Unknown statuses — these reach OnStatus in normal operation, not just error paths; releasing locks mid-commit let a concurrent Select hand the same tokens to a second transaction. Fixed by adding explicit case network.Busy, network.Unknown: that returns nil without touching locks or stores. New tests: TestOnStatus_DoesNotReleaseLocksOnNonTerminalStatus, TestOnStatus_ReleasesLocksOnceOnRetryExhaustion, TestOnStatus_ReleasesLocksExactlyOnceOnUnrecognizedStatus.
• HasEnoughSpendableTokens fast-fail double-counted tokens the current call had already locked — the comparison was against remaining (quantity minus what this call already selected), but the lock-ignoring total already includes the already-selected tokens, so the check could never fire once anything had been won. Fixed by comparing against the full quantity. New test: TestSelectorFastFail_PartiallyFilledRequest.
• The sufficiency-window shuffle was inert whenever the smallest sufficient token was already >5x the requested amount — the magnitude cap was computed from remaining, not from the anchor token itself, so the window collapsed to size 1 exactly in the regime it was built to fix. Fixed by anchoring the cap on the anchor token's own quantity. New test: TestSizeOrderedSelection_SufficiencyWindowWhenEveryTokenDwarfsTheRequest.
• A batch-lock store error silently dropped the whole candidate window — contradicted its own comment by never requeuing the window tokens, costing a full refetch cycle and risking a transient store outage being misreported as lock contention. Fixed by refetching (bounded by the existing maxImmediateRetries budget). New tests: TestBatchLockStoreError_WindowIsRefetchedNotDropped, TestBatchLockStoreError_TerminatesWithinRetryBudget.
• HasAnySpendableTokens was dead code — added to the public driver.TokenStore interface in Phase 4a, superseded by HasEnoughSpendableTokens in Phase 6, with zero remaining production callers. Removed from the interface, all implementations, and mocks.
• Pre-existing lint breakage in benchmark_test.go — 4 ireturn + 1 thelper issue that would have failed CI. Fixed mechanically.

Test plan

All items verified locally against the rebased branch (Go 1.27.1, Docker 29.8.1, Fabric v3.1.1 binaries).

• go build ./... — plus the nested x/token/services/network/evm module, which ./... from the root does not reach

• go test -race ./token/services/selector/... ./token/services/ttx/... ./token/services/storage/... — 0 failures, no data races, including the Docker-backed TestHotTokenContention*/TestStaticHotTokenContentionPareto suites, the ordering/OnError tests, and the review-fix tests above

• make checks — both halves green: checks-fast (license, gofmt, goimports, misspell, ineffassign, protos-lint, buf-format, ✓ All Go modules are tidy) and checks-heavy (go vet, go fix, staticcheck, govulncheck — the 2 reachable vulnerabilities are the ones already in govulncheck-allowlist.txt, treated as a pass as on main). This needed the staticcheck v0.7.0 → v0.8.1 bump in this PR: v0.7.0 panics in its own IR builder under a Go 1.27.1 module.

• make unit-tests-race (full suite) — every package green, with no data races and no panics. It takes two runs to show this, because two pre-existing tests have contradictory path requirements and neither can be satisfied at once outside CI: TestTranslatePath (token/services/identity/config) asserts the translated path contains "panurus", while TestTMSScopedProviderWiringIsIntact (token/services/metricsdoc) greps the repo with filepath.WalkDir, which does not follow symlinks. Run from the worktree path, the only failure is the former; run through a …/panurus symlink, the only failure is the latter; each passes in the other path, so the union covers every package. CI hits neither, since its checkout is a real directory named panurus.

• Integration tests (fabtoken/dlog, TEST_FILTER="T1") — both suites pass against this branch:

  • make integration-tests-fabtoken-fabric-t1 → Ran 3 of 15 Specs — 3 Passed, 0 Failed, 12 skipped (33m55s)
  • make integration-tests-dlog-fabric-t1 → Ran 3 of 44 Specs — 3 Passed, 0 Failed, 41 skipped (35m18s)

  The 3 specs per suite are the T1 label across all three infra types (websocket, libp2p, replicas). The transaction … is not valid [Deleted] errors in the dlog log are asserted by the suite itself (integration/token/fungible/tests.go:550 passes "is not valid" as an expected message), not failures.

Fixes #2395

🤖 Generated with Claude Code

[AkramBitar]

Summary

The PR fixes a bug where a small number of tokens were getting "stuck" under concurrent load — many selectors would pile up racing to lock the same few tokens, causing spurious "insufficient funds" errors.

Root causes fixed:

• Selectors could see and fight over tokens already locked by someone else
• All selectors always picked the same smallest token (deterministic ordering)
• Lost lock races were retried in a tight loop on the same token for minutes
• Settled transactions never released their locks, keeping tokens invisible to others

How:

• DB query now hides locked tokens from the selector entirely
• Candidates are shuffled within similarly-sized groups so concurrent selectors spread out
• Tokens that lost a race are blacklisted for the rest of that call
• Locks are released immediately on settlement, not after a 3-minute sweep
• New Postgres lock modes that avoid server-side constraint violation errors
• On Postgres, a whole window of candidate tokens is locked in one round trip instead of one at a time

One thing to know when monitoring: on Postgres the batch lock only reports which tokens it won, so a token that was already spent looks the same as one that was simply locked by someone else. It is counted as a lock conflict, which means stale_candidates_total stays at zero on that backend.