Our JPEG differs from poppler by more than the ISO IDCT allowance on two thirds of compared pictures

[tannevaled]

Found by the per-channel measure landed in go-pdfkit/conformance#19, over 23 populations and 3280 documents. Recorded, not diagnosed — this issue is the observation, not a theory about it.

The observation

Replacing the old binarised comparison with a per-channel one, with a magnitude gate of 2 levels, moved the two lossy filters in opposite directions:

• JPXDecode: 15.4% → 99.2% agreement, with 7 of 1225 pictures bit-identical. That is what a conformant lossy decoder looks like — the old figure was an artefact of the bisection.
• DCTDecode: 33.2% → 34.0%. It did not move.

So on roughly two thirds of the JPEG pictures compared, we differ from poppler by 16 to 62 levels, where ISO/IEC 10918-2 allows a conformant IDCT at most one level per sample either side of the reference (the bound is enforced in FFmpeg's own conformance test, libavcodec/tests/dct.c:259).

The gate is 2, derived from that standard rather than borrowed. A difference of 16 levels is eight times it; 62 is thirty-one times.

Why this deserves investigation rather than a tolerance

The obvious response — widen the tolerance until DCT agrees — is the one to refuse. JPX now sits at 99.2% under the same gate, so the gate is not too tight in general; it is a JPEG-specific disagreement. And the previous measure hid this for as long as it existed: a binarisation at luma 128 cannot see a systematic level shift, which is exactly the shape of an IDCT or colour-conversion error.

What would identify it

Candidates worth separating before assuming any of them:

• Colour conversion. Is the disagreement concentrated in YCbCr→RGB, or does it appear in greyscale JPEGs too? The measure now buckets by colour space, so this is answerable from the landed records rather than by guessing.
• The IDCT itself — a rounding or scaling difference would show as a small, uniform, signed offset. The measure carries a signed mean precisely for this; read it.
• Chroma upsampling, which would concentrate the error at edges rather than spreading it.
• Which side is wrong. poppler is a judge, not an oracle. pdftoppm renders through a different path than pdfimages extracts; if the two disagree with each other, the finding is about poppler. Settle this before concluding it is ours — the stencil polarity investigation (conformance#13) found exactly that, twice.

The per-population records under baseline/ carry peak, share, MSE and signed mean per filter and per colour space, so the first three questions can be narrowed from data already on disk before a single new measurement is taken.

[tannevaled]

Diagnosed: chroma upsampling, and nothing else

The cause is ours, it is one thing, and poppler is not the odd side. Go's image/jpeg REPLICATES a subsampled chroma sample over the pixels it covers; libjpeg INTERPOLATES, and so poppler does. Every other part of our decode already agrees with libjpeg to within the ISO/IEC 10918-2 allowance, and on a greyscale JPEG it agrees to within one level over 112 real corpus pictures.

1. Which side is wrong — settled first, and it is not poppler

Three ways, in increasing strength:

• poppler has one JPEG path, not two. A /DCTDecode stream becomes a DCTStream at poppler/Stream.cc:348, in the filter chain, not in a tool — so pdfimages and pdftoppm read the same decoder. DCTStream::init (poppler/DCTStream.cc:98) calls jpeg_read_header, sets the colour space, and calls jpeg_start_decompress at :190. It never touches do_fancy_upsampling, which libjpeg defaults to TRUE (libjpeg-turbo/src/jdapimin.c:229). So poppler is libjpeg with libjpeg's defaults.
• Measured, the two tools agree exactly. I built a 1 346-byte PDF drawing one 682-byte 8×8 4:2:0 JPEG at 8 pt on an 8 pt page. pdftoppm -r 576 (8× native), sampled at each 8×8 block's centre, is bit-identical to pdfimages -png of the same picture. Not close — identical, all 192 bytes.
• poppler is bit-identical to libjpeg's own reference decoder. pdfimages -png of that picture equals djpeg (jpeg-turbo 3.1.4.1, the same libjpeg.8.dylib poppler links) byte for byte, and differs from djpeg -nosmooth by 74 levels on 25% of its pixels.

So the judge is consistent with itself and with the library it is. This one is ours.

2. The three questions in the issue, answered from the corpus

I re-decoded every DCTDecode picture pdfimages reports as gray or rgb across both corpora — 559 pictures, 3 280 documents — decoding each raw stream twice, once as we do it and once with libjpeg's filter, and comparing both against pdfimages -png.

Is it colour or does greyscale differ too? Colour, entirely. Split by what the picture actually has to reconstruct:

pictures	what they are	worst channel difference from poppler
112	greyscale — no chroma at all	1 on 104 of them, 0 on 8. Never more.
167	4:4:4 colour — chroma already full size	1 on 112, 2 on 21, 3 on 23, 4 on 11. Never more.
280	subsampled colour — chroma to put back	median 27, p90 50, max 127.

A greyscale JPEG exercises the Huffman decode, the dequantiser and the IDCT and nothing else, and there we are within one level — which is exactly the ISO/IEC 10918-2 allowance the gate was derived from. Our IDCT is conformant. The 279 pictures with nothing to reconstruct carry the whole of the agreement; the 280 with something to reconstruct carry the whole of the disagreement.

Is the signed mean a level shift? No — it is symmetric noise, and the measure said so. Over those 280 pictures the signed mean's median is +0.0013 levels and the median of its magnitude is 0.052, against a median peak of 27: a ratio of 523. 153 run positive and 124 negative. So it is not an IDCT scaling or rounding bias, and the baseline's near-zero mean column was already telling us that.

Is the error spread or concentrated? Concentrated, on a median of 5.5% of pixels (p90 37%) — and where it lands is the tell: at chroma edges, which is what a reconstruction filter disagreement looks like and not what a transform disagreement looks like.

3. The minimal case

An 8×8 picture, half saturated red and half saturated blue, encoded 4:2:0 at quality 100 — 682 bytes:

	peak	pixels over the gate
ours vs djpeg -nosmooth	0	0
ours vs djpeg (default)	74	25%
pdfimages -png vs djpeg (default)	0	0

Our decoder is bit-identical to libjpeg with fancy upsampling switched off, and poppler is bit-identical to libjpeg with it on. There is nothing else in it.

4. What the conventional choice is, and what ours is

Go's image/jpeg hands back an *image.YCbCr, and reading that reads chroma through image.YCbCr.COffset (image/ycbcr.go:101), which divides x and y by the sampling factor — four pixels share one sample exactly.

libjpeg weights the nearer chroma sample 3 and the further one 1 in each direction: h2v1_fancy_upsample (jdsample.c:296), h1v2_fancy_upsample (:336), h2v2_fancy_upsample (:382), selected at :506 and :534 when the chroma plane is more than two samples wide, and int_upsample — which replicates — otherwise (:553). An image edge reads a duplicate of the last real row rather than the MCU padding beside it (jdmainct.c:217).

The field splits by which decoder it uses, not by opinion:

reader	decoder	chroma
poppler	libjpeg, defaults untouched (DCTStream.cc:98)	interpolated
pdfium	libjpeg (core/fxcodec/jpeg/jpeg_common.c:29), never sets the flag	interpolated
mupdf	libjpeg (source/fitz/filter-dct.c:284), never sets the flag	interpolated
pdf.js	its own, src/core/jpg.js:1244 truncates with 0 | (x * componentScaleX)	replicated
us	Go's image/jpeg	replicated

Neither behaviour violates ISO/IEC 10918. The standard specifies the inverse DCT and bounds it; it does not specify reconstruction. So this is not a defect in the sense the gate was built to catch — but three of the four other readers interpolate, and on a 4:2:0 photograph replication is visible 2×2 chroma blocking. We are the outlier.

5. What adopting libjpeg's filter is worth, measured

Same 559 pictures, same judge, both decodes compared against the same pdfimages -png:

	gate 2	gate 3	gate 4
replicate (today)	49.9%	57.2%	61.2%
interpolate	57.1%	86.4%	99.3%

and on the 280 subsampled pictures alone the worst channel difference falls from a median of 27 levels to a median of 3, and the maximum from 127 to 4 once four pictures are set aside (see 7 below).

6. Why gate 2 does not move much, and why that is not an argument for moving the gate

Interpolating takes the median peak on the affected pictures from 27 to 3, which is above a gate of 2 — so the headline barely moves at gate 2 and jumps to 99.3% at 4. That number is not a reason to widen anything, and I am not proposing it. It is a fact about where the gate was derived versus where it is applied:

• ISO/IEC 10918-2 bounds the IDCT at one level per sample, in the sample domain — which for a colour JPEG is Y, Cb and Cr.
• The measure compares in RGB, after the colour transform. One allowed level in each of Y, Cb and Cr becomes up to 1 + 1.402 = 2.4 levels of red and 1 + 1.772 = 2.8 of blue, before rounding.

The corpus shows exactly that ceiling and no more: over the 279 pictures with no chroma to reconstruct, the peak is at most 1 for greyscale (one component) and at most 4 for 4:4:4 colour (three components through the matrix). Two conformant decoders can be 4 levels apart in RGB while each is within 1 of the reference in YCbCr.

That is a finding about the instrument, offered as an observation rather than a fix: the gate is derived in one colour space and applied in another. It does not touch this issue's finding, which stands at any defensible gate — a median of 27 levels is nine times the RGB-transported allowance and twenty-seven times the sample-domain one.

A second-order contributor to the 3-and-4-level residual, for the record: Go's color.YCbCrToRGB and libjpeg's jdcolor.c tables round differently (libjpeg's FIX(1.40200) is 91 882, Go uses 91 881, and Go folds its half-LSB into y * 0x10101 where libjpeg adds ONE_HALF inside the table). Substituting libjpeg's tables on one real 4:4:4 picture took the peak from 4 to 3 and the share over the gate from 0.067% to 0.008%. It is a quarter of a level of systematic bias and it is not worth diverging from the standard library over.

7. A separate defect in the measure, found on the way

Four of the 559 pictures still differ by more than 4 levels after the reconstruction is fixed. Every one of them is a /CalRGB picture, and none is a codec matter:

picture	peak	colour space
gh-openpdf/…PDF_2.0_image_with_BPC.pdf	110	/CalRGB, white point [0.9643 1 0.8251] (D50)
fr-cerfa/cerfa_11616.pdf, cerfa_11612.pdf	10	/CalRGB, gamma 2.22221
fr-cerfa/cerfa_12625.pdf	11	/CalRGB, gamma 2.22221

On cerfa_11616 poppler differs from djpeg's own output by peak 9 over 12.5% of the picture — poppler applies the CalRGB→RGB conversion through colorMap->getRGB (utils/ImageOutputDev.cc:451) and we do not.

pdfimages -list names a CalRGB picture rgb — utils/ImageOutputDev.cc:162-164 maps both csDeviceRGB and csCalRGB to the string "rgb", and :648 lumps 3-component csICCBased in with them for the write path. So the direct bucket, defined in images/images.go as {"gray", "rgb", "-"}, silently contains pictures poppler converts — which is precisely what the converted bucket exists to keep out. Worth its own issue on conformance; it is charging a colour-management difference to a codec.

8. Proposed, not landed

#41 reproduces libjpeg's filter, and only where libjpeg applies it — 4:2:0, 4:2:2 and 4:4:0 above two chroma samples wide; 4:1:1, 4:1:0 and anything with no chroma are asserted to come out byte for byte as they do today. On us-dol/CA-10.pdf's 75×75 logo it takes us from 56 levels away from poppler on 48.3% of pixels to 3 levels away on 0.018%.

It changes the output of every subsampled JPEG in the fleet and neither behaviour is non-conformant, so it is a proposal and I have not merged it.

---

Corpus and instruments under /Users/Shared/dct40, outside every work tree. No timings are reported: another job was on the machine throughout. One aside for whoever runs the measure next — pdfimages -list HANGS on gh-qpdf/qpdf_qtest_qpdf_shared-unnamed-field.pdf and has to be killed; both of my sweeps stalled on it.