Learned layout classifier - #6

Open

ivanvanderbyl wants to merge 53 commits into

mainfrom

ivan/learned-layout-classifier

Commits on Aug 5, 2026

feat(parser): resolve base-14 metrics, figure geometry, and heading guards

ivanvanderbyl
and
claude
committed
513c968
feat(parser): resolve Unicode for TeX-encoded fonts via width fingerprints

ivanvanderbyl
and
claude
committed
635acf4
fix(pdf): stop tall math glyphs stealing prose letters and bridging lines

ivanvanderbyl
and
claude
committed
5b29254
fix(table): suppress equation fake tables and spanned-cell row duplication

ivanvanderbyl
and
claude
committed
c759c5f
docs: record entropy and DPBench results for equation fake-table fix

ivanvanderbyl
and
claude
committed
06a7e77
fix(font): tighten TeX width-fingerprint thresholds for small subsets

ivanvanderbyl
and
claude
committed
fd91c54
fix(pdf): review fixes — table-safe stack folding, total-order cell sort

ivanvanderbyl
and
claude
committed
7e23cd6
fix(table): address review of gutter-persistence gate

ivanvanderbyl
and
claude
committed
1fa423d
Merge branch 'ivan/display-equation-reading-order' into staging/verify2
ivanvanderbyl
committed
e9ad474
Merge branch 'ivan/equation-fake-tables' into staging/verify2
ivanvanderbyl
committed
faf0f07
chore: ignore entropy scratch conversions

ivanvanderbyl
and
claude
committed
0b29ab6
fix(pdf): measure a line by its dominant glyph size, not its maximum

ivanvanderbyl
and
claude
committed
83cb3b7

Commits on Aug 6, 2026

docs: plan a learned layout classifier to replace hand-tuned heuristics

ivanvanderbyl
and
claude
committed
3ad8fae
docs: prove the layout classifier with leaves + go:embed before codegen

ivanvanderbyl
and
claude
committed
a6598c9
docs: rework layout-classifier plan into a two-stage learned cascade

ivanvanderbyl
and
claude
committed
1206f2b
spike(layout): train a Formula classifier on docmill's own geometry

ivanvanderbyl
and
claude
committed
8afd0d7
docs(research): record the Task 0 spike result — it works, teacher is the limit

ivanvanderbyl
and
claude
committed
7bfa607
docs(research): record which branch the spike numbers were measured on

ivanvanderbyl
and
claude
committed
eff369b
Merge branch 'main' of github.com:ivanvanderbyl/docmill into ivan/learned-layout-classifier
ivanvanderbyl
committed
2330bfa
spike(layout): generate the trees as Go and measure it against leaves

ivanvanderbyl
and
claude
committed
056b317
docs(research): measure how the generated model scales, and where it breaks

ivanvanderbyl
and
claude
committed
e60e9c3
feat(layout): train the 12-class LINE model on DocLayNet

ivanvanderbyl
and
claude
committed
5fbbe42
feat(pdf): move layout features into pkg/pdf with the current-class baseline

ivanvanderbyl
and
claude
committed
fab3736
docs(research): Task 1 baseline — the model beats the heuristics on every class

ivanvanderbyl
and
claude
committed
7411772
feat(pdf): add the classify-then-route pipeline behind a flag

ivanvanderbyl
and
claude
committed
702c033
feat(pdf): embed the LINE model and migrate the Formula class

ivanvanderbyl
and
claude
committed
943e1fa
test(pdf): prove the Formula veto runs on DPBench rather than assuming it

ivanvanderbyl
and
claude
committed
4343558
Merge branch 'ivan/learned-layout-classifier' of https://github.com/ivanvanderbyl/docmill into ivan/learned-layout-classifier
ivanvanderbyl
committed
dd028fb
feat(pdf): hand headings, lists and figures to the model as well

ivanvanderbyl
and
claude
committed
d72ee2d
Merge remote-tracking branch 'origin/ivan/learned-layout-classifier' into ivan/learned-layout-classifier
ivanvanderbyl
committed
e575787
feat(cli): run the learned layout classifier from convert

ivanvanderbyl
and
claude
committed
ac3752f

Commits on Aug 7, 2026

fix(pdf): retrain the model after the assembler merge changed its input

ivanvanderbyl
and
claude
committed
d83410f
fix(pdf): ask the FIRST line whether a block is a list item

ivanvanderbyl
and
claude
committed
ac8c4dc
refactor(layout): delete the Task 0 spike now that it is superseded

ivanvanderbyl
and
claude
committed
cae9bc2
feat(layout): verify FinTabNet's cell boxes line up with docmill's geometry

ivanvanderbyl
and
claude
committed
3455f63
feat(table): learn column boundaries from FinTabNet

ivanvanderbyl
and
claude
committed
e17f73f
feat(table): wire up the learned columns, measure them, leave them off

ivanvanderbyl
and
claude
committed
e792650
feat(pdf): add the REGION model, the second stage of the cascade

ivanvanderbyl
and
claude
committed
f783233
measure(layout): the region gate is starved, not weak

ivanvanderbyl
and
claude
committed
59957fc
feat(parser): report every kind of ink, clipped to what is visible

ivanvanderbyl
and
claude
committed
2833d27
measure(layout): the Picture ceiling goes from 22.1% to 69.6% with ink

ivanvanderbyl
and
claude
committed
8f84265
feat(pdf): replace the region proposer with ink, geometry and split lines

ivanvanderbyl
and
claude
committed
36cd172
feat(pdf): classify region proposals and suppress the overlaps

ivanvanderbyl
and
claude
committed
de17ff9
test(pdf): measure what the region stage costs per page, not per call

ivanvanderbyl
and
claude
committed
5702083
fix(pdf): rank suppression by extent quality, not class confidence

ivanvanderbyl
and
claude
committed
0993518
fix(gbm): shift softmax by the original max, not the overwritten slot

ivanvanderbyl
and
claude
committed
dea98eb
feat(pdf): let real classes outvote Background, and measure fairly

ivanvanderbyl
and
claude
committed
65ece42
feat(pdf): page-scope gutters, regularised training, table-weighted IoU head

ivanvanderbyl
and
claude
committed
f2d1c1c
feat(pdf): the retrained region stage gets a Markdown path and a viewer

ivanvanderbyl
and
claude
committed
2d91804
fix(pdf): region Markdown learns from the Shannon sample

ivanvanderbyl
and
claude
committed
41c2a17

Commits on Aug 8, 2026

feat(pdf): region Markdown reaches parity on headings, closes on tables

ivanvanderbyl
and
claude
committed
9164b8a
cleanup(pdf,table): delete every learned component the evidence rejected

ivanvanderbyl
and
claude
committed
b7f7840
docs: the cleanup made -learned-layout the new bar — no losses anywhere

ivanvanderbyl
and
claude
committed
530bb59