Design: startup I/O schedule - sequential background lane + RED-shape freshness tiers (stop paying per-file verification on every start)

[msarson]

Problem (measured on the real 40-project IBS solution, VM disk, 2026-07-14 retests)

Startup runs four I/O sweeps to verify disk caches that are almost always clean: SDI 4,104 stats, FRG 3,016 stats, RefIndex 3,016, plus the open-tab validator pass reads. Two structural findings from the [P4] -> [352-353] -> [352-355] log sequence:

1. Per-op cost is machine-dependent and does not warm away. The same SDI stat loop measured 97-224ms on a warm dev box and 2,008-6,972ms on an idle VM; run-2-vs-run-1 comparisons showed a see-saw (some phases SLOWER on the second run), i.e. a fresh per-I/O tax per process (AV filter is the prime suspect), not a cold cache that warms. Cold/taxed disk IS the product experience: after boot, after a VSIX update, on customer VMs.

2. Concurrency multiplies the pain. Perf: SDI startup revalidation (4,104 stats) gates the async validator pass - 7s diagnostics delay on cold/taxed disk #355 moved the SDI sweep off the gating path - and the freed validators + FRG + RefIndex + background SDI sweep then ran SIMULTANEOUSLY: classConstants code actions starved 9-26s (were 150-250ms), FRG went 303ms -> 25-27s, the SDI background sweep took 19-21s vs 7s foreground. On an op-rate-bound disk, parallel sweeps do not overlap waiting - they queue behind each other plus seek/filter overhead. The old serialized startup was an ACCIDENTAL rate limiter; the fix is a deliberate one.

The lever we own is per-operation COUNT and SCHEDULING, not cache warmth.

Design

Part 1 - one sequential background I/O lane

All background sweeps run strictly one at a time, in this order, starting only after the open tab`s async validation pass has completed:

1. (foreground, unchanged) sync diagnostics at open; solution load; SDI cache-trusted load (Perf: SDI startup revalidation (4,104 stats) gates the async validator pass - 7s diagnostics delay on cold/taxed disk #355)
2. open-tab async validator pass (the diagnostics the user is waiting for)
3. FRG build
4. RefIndex build
5. SDI background drift sweep (Perf: SDI startup revalidation (4,104 stats) gates the async validator pass - 7s diagnostics delay on cold/taxed disk #355) - pure hygiene, zero urgency
6. (later additions ride the same lane)

Interactive requests (code actions, F12, hover, lens resolves) are never queued behind the lane - the lane yields to them, not vice versa. This is also the home for the banked "interactive-request priority" item (F12 measured 14.9s queued behind validator storms on 07-13).

Part 2 - freshness tiers keyed off RED path shape

The redirection entries already classify every directory, no heuristics needed:

• Volatile tier - project-relative paths (.\genfiles\src, .\classes, .\SharedCode\*, .): the only dirs that change between sessions (appgen regen writes ONLY genfiles\src - confirmed by Mark; git pull; hand edits). Keep per-file mtime checks at startup. Scale: a few hundred files instead of thousands.
• Vendor tier - install-rooted paths (macro-derived absolute: %ROOT%\libsrc\win, %ROOT%\Accessory\libsrc\win, images, template) plus the ClarionProperties.xml <libsrc> node dirs: change only when an installer runs. Startup does ONE stat per directory against a stored fingerprint (dir mtime + file count); tripwire fires -> deep-sweep that directory only. Backstops for the in-place-overwrite edge (dir mtime does not change when an existing file is rewritten): a sentinel file check (e.g. equates.clw mtime) and an explicit "Clarion: Refresh Index" command for patch day.

Applies to the SDI sweep first (4,104 -> ~500 stats + ~12 dir stats). FRG/RefIndex files are solution sources = all volatile tier by definition; their win comes from Part 1, not Part 2.

Part 3 - cheap hardening riding along

• Cache identity becomes mtime + size (both come from the same stat; strictly stronger) - Mark`s suggestion.
• Watch the RED files themselves and invalidate resolution caches on change - the decompiled IDE does exactly this (FileSystemWatcher per RED); we currently reparse only on solution load.

Phasing

• A. Sequential background lane (biggest UX win: kills the 9-26s code-action starvation and restores FRG to warm-cache speed)
• B. SDI freshness tiers (kills ~3,600 startup stats; needs the volatile/vendor classifier on RedirectionEntry paths + per-dir fingerprints in the SDI disk cache + the Refresh Index command)
• C. mtime+size identity + RED-file watcher

Each phase lands via the usual per-issue branch/TDD cycle; A is independently shippable and should go first.

Acceptance

• Startup on a cold/taxed disk: no interactive request (code action, F12, hover) starved behind background sweeps; diagnostics-settled time no worse than [352-355]; background sweeps complete without overlapping each other.
• SDI startup I/O: per-file stats only for volatile-tier dirs; vendor tier = one stat per dir when fingerprints match.
• External changes still detected: volatile per-file at startup; vendor via tripwire/sentinel/Refresh Index; mid-session unchanged (Stale TokenCache for files modified outside the IDE: uri+version(1) hit test returns old tokens against fresh disk content #340 watcher).
• Environment guidance documented (README/wiki): Defender exclusions for the dev tree + %TEMP%\clarion-extension-* caches.

Context

Design agreed with Mark 2026-07-14 (post-#345/#352/#353/#355 retests; pause called after the [352-355] contention log). Ground truth inputs: his sln + Clarion REDs (F:\TestApps\Direct10Meta\red), the "Redirection System" doc, and the decompiled IDE`s RedirectionFile.cs. Related: #295 (off-thread architecture), #354 (hot-path logging sweep), banked queue items (lens-count reuse, missingIncludes phase-5 memoize, chain-index persistence).

[msarson]

Phase A shipped — 5c80f55 on version-1.0.1.

The one uncontrolled concurrent actor was the #355 SDI drift sweep, launched from inside buildIndex the instant the cache-trusted index loaded. It is now deferrable (deferBackgroundValidation): a cache-trusted build records the sweep as pending; the startup orchestrator drains it via runDeferredValidations() as the LAST lane step, after FRG, RefIndex and the revalidation pass — so at most one background sweep touches the disk at a time. The existing chain was already sequential (settle → FRG → RefIndex → reval, #297 order); this closes the last gap.

Expected in the next retest: no SDI background validation complete line overlapping the FRG progress lines; instead a Phase: SDI deferred drift sweep complete (lane tail) line AFTER sdiReady revalidation pass complete; FRG build time drops back toward its warm ~300ms + solo disk tax (was 25-27s while contended); classConstants code actions no longer starved 9-26s.

• A. Sequential background lane
• B. SDI freshness tiers
• C. mtime+size identity + RED-file watcher

Proceeding to Phase B.

[msarson]

Phases B and C staged behind the Phase-A retest — with a concrete technical reason, not just discipline.

Phase C forces a cold rebuild that would corrupt the Phase-A measurement. C adds file SIZE to the SDI disk-cache identity, which is a cache-schema change → DISK_CACHE_VERSION bump → the existing on-disk cache is invalidated → the next startup runs a full COLD scan, not the cache-trusted path. Shipping C now would mask the exact behaviour Phase A optimizes. C must land AFTER a cache-warm Phase-A retest.

Phase B''s value dropped sharply after Phase A and its sizing depends on the retest. A moved the 4,104-stat sweep off every critical path — it now runs alone at the lane tail, contending with nothing. B''s win is therefore "shorten the idle tail + cut total disk churn", not "stop starving the user" (which A already does). B is also a real new freshness model (per-dir fingerprints keyed off RED path shape, with an in-place-overwrite gap that needs the Refresh Index command as backstop) — exactly the kind of substantial I/O-behaviour change that the [352-355] regression taught us not to stack on an unmeasured baseline. The retest tells us whether the tail-sweep duration is even worth B''s complexity.

Both are scoped and ready to execute the moment the Phase-A retest is in:

• B1 (server): tier classifier (volatile = under project root; vendor = install-rooted, defaults volatile for safety) + per-dir fingerprint (mtime + file count) in the disk cache; vendor dirs skip per-file stats when the fingerprint matches.
• B2 (client+server): "Clarion: Refresh Index" command — the honest backstop for the in-place-overwrite edge.
• C: mtime+size cache identity + RED-file watcher (the decompiled IDE watches its REDs; we reparse only on solution load).

VSIX rebuilt, banner [357A BUILD].

[msarson]

Phase A VERIFIED ([357A BUILD] retest, 2026-07-14) — and the data KILLS Phase B

Retest opened IBSCommon.clw directly (the 873KB / 68,486-token program file). The structural wins are unmistakable:

Metric	[352-355] (contended)	[357A] (deferred to lane tail)
SDI drift sweep	19,663-21,247ms	101ms (stat_loop=100, files=4104, drift=false)
Sweep placement	overlapping FRG progress	AFTER sdiReady revalidation pass complete
FRG build	24,850-26,828ms	737ms
RefIndex	contended	264ms
classConstants code actions	9,000-26,000ms (starved)	118-453ms

Phase: SDI deferred drift sweep complete (lane tail) | ms=101 fired last, alone, exactly as designed. No stampede.

Phase B is obviated — recommend NOT building it

The whole justification for B was "4,104 startup stats cost 2-7s." The retest shows those 4,104 stats cost 101ms once they run alone at the tail — the 19-21s was pure CONTENTION, which Phase A removed. B would shave ~500 stats off that, saving <90ms at the idle tail, in exchange for a new freshness model (per-dir fingerprints + the in-place-overwrite gap + a Refresh Index command). Not worth it. Gating B on the retest was the right call — building it blind would have shipped complexity to save 88ms.

Phase C — low value now, still forces a cold rebuild

With the sweep at 101ms, mtime+size identity is pure correctness hardening (not perf) and still costs a one-time cold rebuild. The RED-file watcher remains independently worthwhile. Proposing: keep C as a small optional hardening follow-up, decoupled from #357.

Recommendation: close #357 as "A shipped, B obviated by measurement, C spun out"

The real remaining startup cost this retest exposed is unrelated to the I/O schedule — filed separately (chain-index cold build + unfiltered template equates on the big program file).