prediction-alignment

A read-only audit that checks every published Vesuvius Challenge surface-prediction Zarr against the source volume it says it predicts.

A surface prediction is only useful if a reader can put it back on the volume it came from. Neither store checks that. This tool does, for the whole public population, in about two minutes, over anonymous HTTPS, with no credentials, no zarr, no s3fs, no boto3, and no writes.

What the run found

Full population, 2026-09-06, bucket https://vesuvius-challenge-open-data.s3.amazonaws.com. Machine-readable output in reports/scan.json, console log in reports/scan-full.log.

surface-prediction stores audited	43 (36 samples, 3 models)
stores with no contract finding	1 / 43
chunk-presence probes issued	237,825
prediction chunks decoded and inspected	1,869
wall time, 5 workers	117.5 s

check	contract findings	stores affected
shape_agreement — prediction grid equals source grid at the declared level	0	0 / 43
axes_agreement — axis names, types, order match the source	0	0 / 43
values — a store with a declared threshold is binarised	0	0 / 43
naming — path, catalog parameters and model table agree	0	0 / 43
encoding — the store is written the way the catalog says it is	84	42 / 43
coverage — the prediction only fires where the source has data	38	38 / 43
world_scale — the pair can be co-registered from metadata alone	0 (56 advisory)	14 / 43

Two defects and one advisory pattern, all reproducible from the JSON report. Full write-up with exact locations in FINDINGS.md.

1. 1,635 prediction chunks holding 1,702,945,391 non-zero voxels sit over regions where the source volume has no data at all. Present in 38 of 43 stores. Every one was decoded, not inferred: 1,635 confirmed non-zero, 0 explicitly-written empty chunks. In a further 3 stores the same pattern appears but is reported as advisory, because the registration control below could not establish the axis convention there.
2. Every store's recorded codec is wrong in the catalog. All 42 entries with a creation_info block declare zarr_compressor: "lz4" and zarr_compression_level: 7; all 42 stores are actually blosc/zstd at clevel 1.
3. Advisory: the 14 L2 predictions restart their coordinate scale at 1.0, although their level 0 holds the same voxels as the source's level 2 (scale 4.0). Nothing in OME-NGFF is violated — the axes carry no unit and each store's own chain is self-consistent — but the pair cannot be put in one frame without knowing L out of band.

Clean results are results too: shapes, axes, thresholds and naming are exact across the entire population. Prediction level k matches source level k+L on every axis of every level of all 43 stores, and every sampled chunk in all 43 stores holds only {0, 255}.

What is checked

The catalog lists a surface-prediction-zarr next to the ome-zarr of the volume it hangs off, plus a parameters block naming the model, the source pyramid level L the model ran on, and the probability threshold. Everything below compares the two stores as a pair.

1. Shape agreement. Prediction level k must cover exactly the same voxel grid as source level k+L, on every axis, for every level. One voxel of disagreement puts every downstream read somewhere else in the scroll, and nothing in either store would say so. Levels finer than the source pyramid reaches are reported as advisory, not as a violation.

2. Axes and world scale. Axis names, types and ordering must match the source. Separately, prediction level k is compared against the coordinate scale the source declares at level k+L. A mismatch is advisory: OME-NGFF does not require two stores to share a frame, so this reports an interop hazard, not a violation.

3. Encoding. dtype, chunk grid, order, fill_value, codec and filters must be uniform across the prediction's own levels; fill_value must match the source's, because that is what makes an absent chunk mean the same thing in both stores; and the codec, compression level and level count recorded in the catalog's creation_info must be what the store actually is.

4. Value semantics. A store the catalog gives a threshold_value for must hold a binary mask. Sampled chunks are decoded and their value set reported. A store with no declared threshold is reported, never failed.

5. Coverage. Both stores are sparse — an absent chunk is entirely fill_value. Compared on a common voxel grid, a present prediction chunk whose entire footprint lands on absent source chunks means the model marked surface inside a region the masked CT says is empty. Every such candidate is then decoded: only chunks that really carry non-zero voxels are reported as a violation, and candidates that decode to all-fill_value are reported separately as an advisory. The reverse view — occupied source volume with no prediction over it — is always advisory.

5b. Registration control. The coverage check rests on one assumption that nothing in either store states: prediction index (z,y,x) addresses source index (z,y,x), same origin, same handedness. A shape check cannot rule out a flipped or transposed convention, because these volumes are square in y and x. So the orphan count is recomputed under each convention a shape check cannot exclude — z-flip, y-flip, x-flip, and yx-swap — and all counts are published in the report. If any alternative fits a store strictly better than the identity mapping, that store's coverage result is demoted to advisory: the prediction may simply be registered differently from what was assumed. Identity is strictly the best fit on 39 of 43 pairs, usually by a wide margin, tied on 1, and beaten on 3.

5c. Deep confirmation (--deep N). The comparison level is chosen to fit a probe budget, so an absent source chunk is only known to be empty at that level. Because both pyramids are built by downsampling, faint level-0 material could in principle average to zero and never be written at a coarse level. --deep N re-probes the N largest orphans in each store against source level 0 instead, over the same voxel box. A single present chunk falsifies the orphan.

6. Naming and provenance. The store path is the only provenance a consumer sees without the catalog. The volume id, model id, level and threshold encoded in the path must agree with the catalog parameters, the model must exist in the catalog model table, and it must be registered for the surface-prediction task.

Findings are split the way the source data deserves:

• contract — machine-checkable, safe to gate automation on. Fails the run.
• advisory — unusual, but not a violation of any stated contract. Never fails a run.

Running it

Python 3.11+. numpy is required. zstandard is required only for the checks that decode chunk contents (values, and orphan confirmation in coverage); the geometric and metadata checks run without it, and a store whose codec cannot be decoded is reported as an advisory skip rather than silently passed.

pip install numpy zstandard

python3 -m prediction_alignment list                 # enumerate the population
python3 -m prediction_alignment check PHerc0800      # one store, full JSON
python3 -m prediction_alignment scan --workers 5 --out reports/scan.json
python3 -m tests.test_alignment                      # self-check, no network

Exit status: 0 clean, 2 contract violation, 1 tool error.

The catalog is read from ../_data/s3_metadata.json (the cached s3_metadata.json published with the dataset, plain or gzipped); override with --catalog. Useful flags: --budget caps chunk-presence probes per store and so picks the pyramid level coverage is compared at; --max-decode caps how many orphan candidates are decoded; --deep N re-probes the N largest orphans per store against source level 0; --samples sets how many chunks the value check reads; --no-coverage skips the network-heavy part.

The exact commands behind the numbers above:

python3 -m prediction_alignment scan --workers 5 --max-decode 400 \
        --out reports/scan.json                                      # 117.5 s
python3 -m prediction_alignment scan --workers 5 --max-decode 400 \
        --deep 1 --out reports/scan-deep.json                        # 216.6 s

What this does not prove

• It does not show that any of this changes a result. No published reconstruction, segmentation or ink prediction was re-run against a corrected store. Whether the out-of-mask predictions or the scale mismatch affect downstream output has not been measured.
• It does not show that the orphan chunks are a model error. The tool proves a geometric fact: prediction data over a region the published masked volume leaves entirely empty. Whether that is the model firing outside the specimen or the volume mask cutting material the model correctly found is not something a read-only audit can settle. See the last section of FINDINGS.md for the experiment that would.
• Coverage is checked at chunk granularity, not per voxel. A prediction chunk is only reported when every source chunk under its whole footprint is absent, which is a conservative test: finer misalignment inside an occupied region would not be seen. The pyramid level the comparison runs at is chosen per store to fit the probe budget, so it is level 1 for some stores and level 4 for others; the level used is recorded in each result.
• Deep confirmation is a sample, not a sweep. The --deep 1 run descends one orphan per store to source level 0 — 39 orphans, 151,659 level-0 chunk probes, none falsified, two skipped for exceeding the probe cap. The remaining 1,596 orphans are established at the compared level only.
• The registration control is a comparison, not a proof. It rules out the axis conventions a shape check cannot exclude. It cannot rule out a translation, a rotation, or a resampling difference.
• It assumes an absent chunk means fill_value. That is what Zarr v2 specifies and what these stores are written to, and fill_value is 0 in every source volume and every prediction, but the audit reads absence from HTTP 403/404 and cannot distinguish "not written" from "not readable".
• It cannot verify the numeric threshold. The stores are binarised and the underlying probability maps are not published, so th0.2 versus th0.45 is taken on trust; only the fact of binarisation is checked.
• Value sampling is a sample. Chunks are read at the coarsest pyramid level by default. A separate manual spot check of nine populated level-0 chunks in three stores also found only {0, 255}, but no store was read exhaustively.
• It does not re-check either pyramid's internal fidelity. That is a different tool's job.

Licence

MIT. See LICENSE.