Line breaking: TeX's two passes, and the parameters a document sets - #194
Merged
Conversation
… sets The paragraph builder hyphenated every word up front and ran one pass over the result. TeX runs the paragraph twice (tex.web §16986-16999): the FIRST pass knows no hyphenation points at all and works at \pretolerance, so an unhyphenated setting wins whenever one exists — which is why real TeX hyphenates far less than an optimiser holding the hyphens from the start. Only when that pass finds nothing does the second one run, over the hyphenated list, at \tolerance; the emergency pass stays behind it as before. The tolerances and demerits are now read from the count registers they live in rather than hard-coded, so \tolerance, \linepenalty, \adjdemerits, \doublehyphendemerits and \finalhyphendemerits do what a document says. They reach the optimiser through linebreak.Params (go-typeset/linebreak v1.1.0), which also brings TeX's fitness classes and a tolerance compared against the BADNESS rather than the adjustment ratio. Measured against tectonic: arXiv page error 633 → 624 over 157 papers, beamer unchanged at 40 over 79 decks, layout divergence flat within its own noise (+0.068 on a 50-paper sample, −0.013 on another). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The paragraph builder hyphenated every word up front and ran a single pass over
the result, with the tolerances hard-coded in the call. TeX does neither.
Two passes (tex.web §16986-16999). The first pass has no hyphenation points
at all and works at
\pretolerance; only when it finds nothing does the secondrun over the hyphenated list at
\tolerance. That ordering is why real TeXhyphenates far less than an optimiser that holds the hyphens from the start —
an unhyphenated setting wins whenever one is within reach. The emergency pass
stays behind both, unchanged.
The document's own parameters.
\tolerance,\linepenalty,\adjdemerits,\doublehyphendemeritsand\finalhyphendemeritsare read from the countregisters they live in, so a class that sets them gets the paragraphs it asked
for. They reach the optimiser as
linebreak.Params.The library side is go-typeset/linebreak#2, released as v1.1.0: fitness classes
with the best break kept per class, TeX's demerits in full, and — the change
that moves the corpus — feasibility tested on the BADNESS rather than the
adjustment ratio.
\tolerance=200had meant "stretch a line up to 200 times itsstretchability" instead of "up to about 1.26 of it", so no line was ever
rejected for being too loose.
Measured, against tectonic
The two divergence samples disagree in sign, so that barometer is flat within
its own noise; the page count, deterministic over 157 documents, is 9 pages
closer. Two intermediate states were measured and are not what shipped: the
fitness classes alone move nothing (633, 4.140), and the two passes without the
badness fix are worse on both (636, 4.350) — the passes only pay once a
tolerance means what it says.