parse: add corpus-census math symbols (\mathbbm, \intercal, \Coloneqq) - #1
Merged
Conversation
An arXiv corpus census of equations the math layer dropped whole (an unknown
command inside a formula takes the entire equation with it) surfaced several
standard symbols and double-struck alphabets we lacked:
- \mathbbm (1045 papers) / \mathbbmss / \mathbbb — the bbm/bbold double-struck
fonts; approximated with the Unicode blackboard block like \mathbb/\mathds,
so the ubiquitous \mathbbm{1} indicator renders 𝟙 instead of dropping.
- \intercal (225 papers) → ⊺, \dotplus → ∔ (binary operators).
- \Coloneqq / \Coloneq (mathtools) → ⩴ (relation).
Red-without/green-with tests: TestCorpusCensusSymbols renders each, and
TestAlphabets asserts the \mathbbm{1}=𝟙 rune mapping. 100% coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Sep 1, 2026
…la (#10) A \def written inside a formula — a paper's own inline shorthand reaching the math layer, \def\R{\mathbb{R}}\R or \def\abs#1{\lvert#1\rvert}\abs{x} — was an unknown command that dropped the whole equation. Macros defined in the preamble are already substituted upstream by go-tex/engine; the ones that live inside the formula were not handled, and \def is a top math-drop cause across the arXiv corpus. Add a TeX "mouth" pass (macro.go) that runs before the parser: it records each \def and expands every use — arguments spliced into #1…#n of the body, the result re-processed so nested and chained macros resolve — with a budget that stops a self- or mutually-recursive macro from spinning. It handles undelimited parameters (the inline form that occurs); a \def it cannot read (delimited parameters, no body) or a use with too few arguments is left verbatim for the parser to report, never guessed at. The pass is transparent to a def-free stream. Verified: parameterless and parameterised macros, chained and nested macros, expansion inside \frac arguments, single-token vs {group} arguments, and every rejecting path (bad name, delimited params, missing/unbalanced body, too few args, self-recursion) all covered; full suite, race, go vet, gofmt clean at 100%. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add standard math symbols and double-struck alphabets that an arXiv corpus census found the math layer was dropping — an unknown command inside a formula drops the whole equation, so these were silent losses on real papers:
\mathbbm\mathbbm{1}→ 𝟙)\SI/\num\intercal\dotplus\Coloneqq/\Coloneq\mathbbmss/\mathbbb\mathbbm/\mathbbmss(bbm) and\mathbbb(bbold) are all double-struck fonts; we approximate each with the Unicode blackboard block already used for\mathbb/\mathds, so\mathbbm{1}renders the indicator 𝟙 rather than refusing the equation.Tests
TestCorpusCensusSymbolsrenders each new symbol/alphabet (red-without:unknown command \intercal; green-with).TestAlphabetsextended to assert the rune mapping (\mathbbm{1}=𝟙 = U+1D7D9,\mathbbm{R}=U+211D).-race; 100% coverage;gofmt/go vetclean.🤖 Generated with Claude Code