engine: a macro substituted into a formula brings TeX with it - #179
Merged
Conversation
The maths retry path substitutes a macro's body into the source as text. A class's body is TeX, not maths — \the@inst is \number\c@inst, an author's mark is an \@for loop over \expandafter and \edef — and the maths layer can only refuse those primitives, which costs the whole formula. The census over 200 arXiv papers had \expandafter first by documents (7) and \number second (6), both of them the same affiliation machinery landing in a superscript. The substituted material now goes through the gullet, as \edef would. It is bounded to what was spliced, and an isolated expansion can no longer read past its own sentinel (#174) — which is what makes this safe where expanding the WHOLE source was not: that experiment cost one paper 78% of its glyphs and is recorded in #146. Retokenising the material needs @ to be a letter inside a control word, as TeX reads it with \makeatletter and as go-tex/math now does: \c@inst split at the @ names \c, a macro nobody has. Every control sequence this engine writes carries a trailing space, so a name never bleeds into what follows it. Measured over 200 arXiv papers and 200 beamer talks: arXiv gains 309411 glyphs — 3% of everything the corpus typesets — over 129 documents, 27 of them repaginating; the page error against the 157 tectonic renders falls 651 -> 624 with three more exact matches. On the worst paper the formulas the maths layer refuses go 94 -> 18. beamer does not move a page and loses 771 glyphs over 8 talks, one of them a pandoc template whose "content" is $if(...)$ placeholders.
tannevaled
added a commit
that referenced
this pull request
Sep 1, 2026
#179 says it runs "the substituted material" through the gullet; it ran the WHOLE maths source. The rest of that source is the author's maths, and the engine's text-mode stand-ins won there: \def\cdot{\char183\relax} (latex.go) \def\begin#1{\gotex@checkenv{#1}\csname #1\endcsname} so every \cdot became \char — 935 dropped formulas over 45 papers — and every \begin{bmatrix} became \bmatrix, 676 over 42. Worse than the drops: with \begin expanded, \gotex@checkenv opened the environment and the ENGINE typeset the matrix as text. One paper came out with 411251 glyphs on seven pages, 58000 a page, where it has 69284. Now only the body just substituted is flattened, which is what the seam was for: a class's \the@inst is \number\c@inst and that still resolves. \begin and \end are marked \noexpand inside the body too, for a macro that wraps an environment. Measured over 200 arXiv papers and 200 beamer talks, against main: formulas the maths layer refuses 4312 -> 2715 arXiv page error (157 renders) 624 -> 634, exacts 17 -> 19 the flooded paper 411251 -> 69284 glyphs The page error moves the wrong way by ten because the flood was flattering it. Against the state before #179 the pair now stands at 651 -> 634 with 14 -> 19 exact, for 212 more dropped formulas — a trade the exact matches and the junk removal earn.
tannevaled
added a commit
that referenced
this pull request
Sep 1, 2026
#179 says it runs "the substituted material" through the gullet; it ran the WHOLE maths source. The rest of that source is the author's maths, and the engine's text-mode stand-ins won there: \def\cdot{\char183\relax} (latex.go) \def\begin#1{\gotex@checkenv{#1}\csname #1\endcsname} so every \cdot became \char — 935 dropped formulas over 45 papers — and every \begin{bmatrix} became \bmatrix, 676 over 42. Worse than the drops: with \begin expanded, \gotex@checkenv opened the environment and the ENGINE typeset the matrix as text. One paper came out with 411251 glyphs on seven pages, 58000 a page, where it has 69284. Now only the body just substituted is flattened, which is what the seam was for: a class's \the@inst is \number\c@inst and that still resolves. \begin and \end are marked \noexpand inside the body too, for a macro that wraps an environment. Measured over 200 arXiv papers and 200 beamer talks, against main: formulas the maths layer refuses 4312 -> 2715 arXiv page error (157 renders) 624 -> 634, exacts 17 -> 19 the flooded paper 411251 -> 69284 glyphs The page error moves the wrong way by ten because the flood was flattering it. Against the state before #179 the pair now stands at 651 -> 634 with 14 -> 19 exact, for 212 more dropped formulas — a trade the exact matches and the junk removal earn.
tannevaled
added a commit
that referenced
this pull request
Sep 1, 2026
The engine spells some characters it has no glyph command for as \char:
\def\cdot{\char183\relax} \def\bullet{\char8226\relax} (latex.go)
\protected\def\_{\char95\relax} (classkernel.go)
In a formula those names belong to the maths layer, which renders ⋅ and ∙ itself. A
substituted body is flattened through the gullet (#179, #182), and flattening turned
them into \char — a primitive the layer refuses, costing the whole formula. A paper's
own \expect{}{\cdot} lost every occurrence.
A body that begins with \char is what makes a stand-in recognisable, without a list
to keep in step with the kernel; those names are marked \noexpand for the flattening,
beside \begin and \end.
Measured over 200 arXiv papers and 200 beamer talks: the formulas the maths layer
refuses fall 2715 -> 2635, arXiv gains 860 glyphs over 8 documents and one more exact
page match (20 of 157), the page error 634 -> 633. beamer does not move.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The maths retry path substitutes a macro's body into the source as text. A class's body is TeX, not maths —
\the@instis\number\c@inst, an author's mark is an\@forloop over\expandafterand\edef— and the maths layer can only refuse those primitives, which costs the whole formula. The census over 200 arXiv papers had\expandafterfirst by documents (7) and\numbersecond (6), both of them the same affiliation machinery landing in a superscript:The substituted material now goes through the gullet, as
\edefwould. It is bounded to what was spliced, and an isolated expansion can no longer read past its own sentinel (#174) — which is what makes this safe where expanding the whole source was not: that experiment cost one paper 78% of its glyphs and is recorded in #146.Retokenising the material needs
@to be a letter inside a control word, as TeX reads it under\makeatletterand asgo-tex/mathnow does (go-tex/math#8):\c@instsplit at the@names\c, a macro nobody has. Every control sequence this engine writes carries a trailing space, so a name never bleeds into what follows it.Measured — 200 arXiv papers, 200 beamer talks
+3% of everything the corpus typesets, over 129 documents with 27 repaginating. On the worst paper the formulas the maths layer refuses go 94 → 18. beamer does not move a page; of its 8 changed talks one is a pandoc template whose "content" is
$if(...)$placeholders.Test
mathexpandprimitives_test.go: a macro whose body holds\number,\the,\expandafteror\romannumeralmust leave the maths layer with nothing to drop, and must reach the page. All four fail onmain.