Compare per channel, with a magnitude gate and two aggregate terms - #19
Conversation
…terms The per-pixel predicate was a bisection at luminance 128, so Share was the fraction of pixels whose ink CLASSIFICATION differed and not the fraction that differed. A decoder rendering every pixel at 120 where poppler renders 20 scored 0.000 on an error of 100 levels everywhere — blind in exactly the direction a lossy decoder fails. difference now subtracts channel by channel and carries four terms out of one walk: Peak, the largest single-channel difference, which is the criterion; a count of pixels above Gate, reported as Share and never spent as a budget; and MSE and a SIGNED mean, in levels squared and levels, which are FFmpeg's omse and ome and pdfium's mse units. The signed mean is the term that catches bias, which is what the bisection could not see at all. Neither aggregate is a pass criterion: no bound has been measured for pictures that were extracted rather than rendered, and borrowing pdfium's would repeat the mistake the withdrawn 1% tolerance was. Gate is 2, derived rather than borrowed. pdfium's 3 buys rasteriser slack we have no use for since pdfimages extracts; ISO/IEC 10918-2 allows a conformant IDCT one level per sample either side of the reference (libavcodec/tests/dct.c:259), so two conformant decoders may differ by two. The three things the rule did not settle: - Colour conversion. Each picture now carries the colour space pdfimages reports (ImageOutputDev.cc:152-190), and the pictures poppler had to convert to reach RGB are tallied in their own bucket with their own agreement figure and their own magnitudes. Only gray, rgb and "-" count as direct; index is counted converted because the base space is not reported, and a picture that cannot be classified must not be credited as agreement. An unread listing leaves everything converted, which is deliberately loud. - Polarity. Inverted is computed in the same pass against the judge's complement and stays its own signal. The direct comparison is tried first, so a uniform mid-grey — within the gate of its own complement — is reported as agreeing rather than as a convention. - Alpha. An ordinary picture is compared in R, G and B; a stencil in one derived coverage channel, ours the alpha render puts the shape in and theirs 255 minus the luminance, because poppler's black is where a stencil paints. Also: pictures are ordered by the number pdfimages wrote into the file name rather than lexically, which mattered for the pages that hold more than a thousand of them, and the baseline record now carries the gate it was taken at, because a record taken by another instrument is not comparable with one taken by this.
render labels Stencil on three different things — a /ImageMask true stencil,
a /SMask and a /Mask — and does NOT put them in the same channels. Read out
of the buffers rather than assumed:
us-opm/SF2801PR.pdf Im0 /ImageMask true, 325x240, every RGB nought,
two alpha values: the shape is in ALPHA.
us-opm/sf2822.pdf Im0/SMask eight-bit grey, 116x73, alpha 255 at all
8468 pixels: the levels are in RGB.
Taking the alpha of the second gives 255 everywhere, which is why us-opm's
one agreeing (samples) mask read as a 48% disagreement with a peak of 255 and
a mean of +88.6 — an artefact of the reduction and not a decoder.
So a mask is reduced by ONE formula applied to both sides: alpha times 255
minus the luminance, over 255. On a stencil the luminance is nought and it is
the alpha; on a soft mask the alpha is 255 and it is the inverted luminance;
on poppler's side, always opaque grey, it is the inverted luminance. All three
layouts land on the same quantity, and the judge parameter goes away because
the two sides are now read the same way.
The rule section becomes the derivation of D and N; the description of what is compared becomes a description of what the code does, including the three things #16 said the rule did not settle — colour conversion counted apart, polarity kept as its own signal, and one ink-coverage formula for a mask. The tables of figures are not touched here. They are the ink-classification measure and are replaced by the re-measurement, not by an edit.
One gate applies to every filter, which is a loosening for the lossless ones: JPEG 2000 conformance is required exact on most of ISO/IEC 15444-4's files and CCITT and JBIG2 have no rounding at all, so a 100% on those used to be bit equality and under a gate of 2 it is not necessarily. So a bucket now counts Identical beside Exact — how many agreeing pictures had no channel differ AT ALL — and the record carries it. Without it a reader cannot tell an agreement rate that is bit equality from one the gate carried, which is the question a uniform gate raises and the baseline has to answer.
The section claimed D reached 2 as a per-case exception for the two lossy filters. What landed is one constant applied to every filter, so the document now says that, names the consequence — it is a loosening for the lossless ones — and points at the count that measures the loosening rather than leaving a reader to assume bit equality.
|
[2026-08-31 13:18:04 CEST] Answering the five questions. The re-measurement is not finished — the 1. The band was an artefact of the bisection, and what replaced it argues for
|
| population | filter | differing | share med | share worst | peak med | peak worst | mse med | mean med |
|---|---|---|---|---|---|---|---|---|
ia-uscourts |
DCTDecode |
11 | 0.007024 | 0.227292 | 23 | 81 | 0.2898 | −0.0001 |
ia-medical |
DCTDecode |
1 | 0.070221 | 0.070221 | 62 | 62 | 1.3258 | +0.0215 |
ia-uscourts |
(samples) mask |
2 | 0.958333 | 0.958333 | 255 | 255 | 62315.6 | −240.39 |
ia-biodiversity |
JPXDecode |
10 | 0.994521 | 0.999976 | 255 | 255 | 17813.9 | +34.26 |
ia-uscourts |
(samples) |
2 | 1.000000 | 1.000000 | 207 | 207 | 16181.4 | +96.87 |
There is still a gap in share — 0.070 to 0.958 — but it is a gap between
different objects and it licenses nothing. The two rows below it have peaks
of 23 and 62, eleven and thirty-one times the gate. A 1% share budget would
forgive ia-uscourts's DCTDecode row outright: 0.7% of its pixels differ,
and they differ by up to 81 levels. That is Cairo's objection stated in our own
data — "otherwise some problems could be masked" — and it is the argument for
keeping N at 0 and letting the peak decide.
So: the empty band did not survive, the reasoning that read a threshold off it
was measuring the bisection, and under a magnitude measure a share threshold is
not merely unjustified but has nothing near it to arbitrate. Reported as
found; the convenient answer would have been that the band survived and 1% was
right all along.
The forms corpus is where most of the old low group lived (fr-cerfa DCT at
0.000135, gh-pdfcpu at 0.000909, uk-govuk at 0.000013). Whether it behaves
the same way is a measurement, not a prediction, and it is running.
2. What the re-measurement covers, and what it does not
Re-measured with this instrument — per channel, gate D = 2, N = 0,
render v0.20.0, pdfimages 26.04.0 — four populations, the whole
/Users/Shared/pdfscans corpus except ia-americana:
| population | documents | unopenable | refused | pictures | direct compared | exact | identical | agreement | converted |
|---|---|---|---|---|---|---|---|---|---|
ia-texts |
12 | 7 | 0 | 14 | 11 | 11 | 2 | 100.0% | 0 |
ia-uscourts |
250 | 0 | 0 | 134 | 78 | 63 | 54 | 80.8% | 16 |
ia-medical |
250 | 0 | 0 | 745 | 529 | 528 | 33 | 99.8% | 1 |
ia-biodiversity |
250 | 0 | 4 | 781 | 644 | 634 | 157 | 98.4% | 0 |
identical is new and it is the column to read beside the rate. One gate
applies to every filter, which is a loosening for the lossless ones, so the
record now says how many agreeing pictures differed by nothing at all.
ia-medical's JPXDecode is the striking one: 496 pictures, 496 exact, 1
identical. JPEG 2000 agrees with poppler within two levels everywhere and is
bit-equal almost nowhere. Under the bisection that column read 15.4% across
the fleet. Both numbers are true of different questions, which is why they are
not in the same table.
Its mirror holds too: JBIG2Decode mask is 187 exact of 187 compared and 187
identical, and JBIG2Decode 9 of 9, all identical. The gate bought the
lossless filters nothing, which is the answer to the question a uniform gate
raises.
Still carrying old figures: every population of
/Users/Shared/pdfforms — ca-cra, fr-cerfa, fr-impots, gh-openpdf,
gh-pdfbox, gh-pdfcpu, gh-pypdf, gh-qpdf, gh-safedocs, gh-verapdf,
int-wipo, uk-govuk, us-dol, us-irs, us-opm, us-ssa, us-uscis,
us-uscourts — plus pdfscans-ia-americana. Their records under baseline/
are the ink bisection at render v0.19.0 and must not be subtracted from
anything above. They are running now and this PR is not finished until they
land or are named as failed.
A new finding that is not about the measure. ia-biodiversity reports
4 refused — documents poppler opens and ours will not. That population had
never completed, so the old baseline's "refused is 0 across all 2724
documents of the 20 populations that ran" did not cover it. Which four
documents, and why, is not yet established; identifying them is a second pass
over that corpus and I have not run it rather than guess.
3. The three populations that never completed
| population | old outcome | now |
|---|---|---|
pdfscans-ia-biodiversity |
killed at 24.9 GB, rc=137 |
completed, peak 3.9 GB |
pdfscans-ia-americana |
hit the 45-minute cap, rc=124, memory flat |
in progress — a first attempt reached 2.8 GB before my own session ended and killed the driver (rc=143, not the cap); restarted under a 90-minute cap |
pdfforms-gh-openpdf |
killed at 27.5 GB, rc=137 |
not reached yet — it is in the forms queue now |
render v0.20.0's fix is confirmed for the first of the two allocation
failures: 24.9 GB to 3.9 GB, and the population produced a full record. The
second has not been tried yet and I am not claiming it in advance.
4. The three problems #16 left open
Colour conversion — recorded and counted apart, not absorbed. Each picture
now carries the colour space pdfimages reports for it
(utils/ImageOutputDev.cc:152-190: gray, rgb, cmyk, lab, icc,
index, sep, devn, and - for a mask). Only gray, rgb and - count
as direct; everything else is converted and goes in its own bucket,
with its own agreement figure and its own magnitudes. The agreement rate is
computed over the direct bucket and never over the two together. Widening D
to swallow ICC arithmetic would have destroyed the gate, which is what #16 said
must not happen.
Two honesties written into the doc comment. index counts as converted even
though its base is often DeviceRGB, because pdfimages does not report the
base and a picture that cannot be classified must not be credited. And
poppler folds CalGray onto gray and CalRGB onto rgb, so a few pictures
counted direct did pass a CIE conversion — that is poppler's resolution, not a
claim of ours. A picture whose listing row could not be read at all also lands
in the converted bucket, so a failure of pdfimages -list reads as every
filter wholly converted rather than as quiet generosity.
Inverted — its own signal, computed in the same pass. The complement is
tested against the same gate, and inverted means the direct comparison failed
the gate and the complemented one passed it. The direct comparison is tried
first, which matters: a uniform mid-grey is within the gate of its own
complement, and without that ordering a picture that agrees would be filed away
as a polarity convention and dropped out of the rate. It also generalises — a
stencil convention carried through a codec that rounds is still recognised,
where Share == 1 needed exactness.
Alpha — one formula, after reading the buffers rather than the flags. This
is where the first draft was wrong and the correction is the second commit.
render sets Stencil on three different things and does not put them in
the same channels:
us-opm/SF2801PR.pdfIm0,/ImageMask true, 325×240 — every RGB nought,
exactly two alpha values: the shape is in alpha;us-opm/sf2822.pdfIm0/SMask, eight-bit grey, 116×73 — alpha 255 at all
8468 pixels: the levels are in RGB.
Taking the alpha of the second gives 255 everywhere, and it made us-opm's one
agreeing (samples) mask read as a 48% disagreement with a peak of 255 and a
mean of +88.6 — an artefact of the reduction, not a decoder. So a mask is
reduced by one formula applied to both sides, alpha × (255 − luminance) ÷ 255: on a stencil the luminance is nought and it is the alpha, on a soft mask
the alpha is 255 and it is the inverted luminance, and on poppler's side —
always opaque grey — it is the inverted luminance. All three layouts land on
the same quantity, and the judge parameter went away because the two sides
are now read identically. An ordinary picture is compared in R, G and B, and
alpha is left out of those because pdfimages writes an opaque picture for
anything that is not a mask.
5. CI
Green on the PR head by REST:
$ gh api repos/go-pdfkit/conformance/commits/89476149f4c23975d9f8e34cf61f258befb95dd6/check-runs
1
Build + test + exact 100% coverage gate completed success 2026-08-31T11:15:42Z
Locally the same head passes go vet, -race, the nine cross-compile targets,
CGO_ENABLED=0, GOWORK=off, exact 100% statement coverage on every package,
and gofmt -l empty under the Go 1.27.0 toolchain rather than the local
1.26.4. Not merged.
What is not done
The forms corpus and ia-americana are running; baseline/ still holds the
old records and baseline/README.md still describes the old instrument. Both
are replaced when the run lands, and the populations that fail will be named in
the document rather than omitted. The four ia-biodiversity refusals are
counted and not diagnosed.
No timings anywhere. A fleet sweep is running on this machine throughout,
so every duration measured here would be a measurement of that sweep as much as
of this. Counts and pixel comparisons are unaffected by load; wall-clock is not.
Every record under baseline/ was taken by the ink bisection at render v0.19.0 and none of it can be subtracted from a per-channel figure, so all of it is replaced. Both seams are now behind the records rather than in front of them: one instrument, one library version, 23 populations, 3280 documents, zero failures. The three populations that never completed all completed. ia-biodiversity went from 24.9 GB and rc=137 to a 3.9 GB peak and a full record, gh-openpdf from 27.5 GB likewise, so render v0.20.0's tree-for-a-graph fix is confirmed on both. ia-americana ran to 53 minutes at a 5.2 GB peak: it was never a defect, only a slow population of large scans behind a 45-minute cap. Two findings the old instrument could not have made. The empty band that produced the 1% is gone. Fourteen of eighteen rows used to sit below 1% with a factor of 24 of nothing above them; two of twenty-one do now, and the largest gap anywhere in the ordered medians is 6.8. Worse for the withdrawn rule, those two rows have peak medians of 18 and 23 levels — nine and eleven times the gate — so what sits under 1% is no longer rounding but sparse gross error, and a share budget there would forgive pictures wrong by up to 81 levels. N stays at 0. And the two lossy filters separate. JPXDecode goes from 15.4% to 99.2% with 7 of 1225 bit-identical, which is exactly what a conformant lossy decoder looks like. DCTDecode does not move — 33.2% to 34.0% — so it differs from poppler by more than the ISO/IEC 10918-2 IDCT allowance on two thirds of what it was compared on, by 16 to 62 levels in most populations. That is a specific decode finding and it is recorded, not diagnosed. Also recorded and not diagnosed: ia-biodiversity reports 4 refused, the only nonzero refusal in 3280 documents. That population had never completed, so the previous "refused is 0" was true of the 20 that ran and untested here.
… gap Named and diagnosed rather than left as a gap. All four ia-biodiversity documents open and their first page resolves; what fails is render.Images with ErrTooMuchToDecode, because v0.20.0 gives one call a 256-megapixel budget and refuses the whole page when a picture will not fit in what is left. The clearest case is an ordinary 794x1372 picture refused with 223089 of 268435456 pixels remaining. That is the other half of the fix that let the population run at all: the same change that took it from a 24.9 GB kill to a 3.9 GB peak did it by bounding the decode, and a bound that fires reads here as a refusal. So Missing.Ours folds two different things together — a document we cannot read and a page we decline to decode — and only the first is a defect. The counts are right and the label is too coarse; splitting it is not part of #16 and is recorded rather than done.
|
[2026-08-31 14:40:12 CEST] The re-measurement finished. All 23 populations ran to completion, 3280 1. The band — the full-fleet answer, and it is the unconvenient oneThe withdrawn 1% was read off eighteen rows: fourteen medians from 0.000011 to The band is gone as a band. Largest gap anywhere: a factor of 6.8 But I have to report something sharper than "the band moved", because the
The old low group has not moved below the new one — it has left the differing 2. The finding this was built to make: JPX and DCT are not the same problemThey read 15.4% and 33.2% under the bisection and looked alike.
JPEG 2000 agrees within two levels on 99.2% and is bit-equal on 7 of 1225 —
3. All three populations that never finished, finished
4. Two corrections to my previous commentThe four That is the other half of the fix in finding 3: the same change that took And 5. The gate bought the lossless filters nothingThe question a uniform
Every agreeing picture of every lossless filter is bit-equal. Not one 6. CI, by REST, on the final headLocally the same head passes What is still open, named rather than reconstructed
No timings are reported anywhere. A fleet sweep ran on this machine |
1 participant
Implements the measure designed in #16, replacing the one #17 was honest about.
The old per-pixel predicate was
dark(), a binarisation at luma 128, so the recorded share was not the fraction of pixels that differ but the fraction whose black/white classification differs. A decoder rendering every pixel at luma 120 where poppler renders luma 20 scored 0.000 — perfect agreement — on a 100/255 error at every pixel. The measure was blind in exactly the direction a lossy decoder fails.This compares per channel, counts a pixel as differing only when its magnitude exceeds the gate, and carries two aggregate terms beside the count: a mean squared error for accumulated noise, and a signed mean for bias — the failure the old measure could not see at all.
Opened by the coordinator on the author's behalf: the branch was complete and committed but the push was refused in that session. The measurement findings and the CI confirmation follow in a comment.