fix(translate): stop shipping units the model returns untranslated

[martyanovandrey]

yfm translate ships pages in the source language. A fragment the model returns unchanged is detected, logged and then written to the output file as is, so a heading, a one-line include or a whole page can end up in the translated docs still in russian. This PR removes the cause and adds a retry for what is left.

How it was measured

The eval harness (npm run translate:eval --real) against Eliza deepseek-v4-flash, ru->en corpus, 16 pages, 500 units, default thresholds, 8 runs of the same configuration:

	before	after
runs that failed	7 of 8	2 of 8
markup violations	1 run	none
glossary violations	6 runs	none
untranslated text	syntax/links.md whole page in 3 runs, a one-line include in 4, the about.md heading in 6	the about.md heading in 1 run
worst page similarity	0.114	0.937

Before the fix the failures were structural, and every glossary violation in the series was a side effect of a page that was never translated. After it, one run still echoes the about.md heading (# О Diplodoc), which the model occasionally returns as is even in a request of its own. The second failure of the series has nothing to do with translation: the judge left one pair of 409 unscored.

What changes

The glossary moves to the system prompt

The rendered glossary (Use these required term translations:\n- заметка → note ...) used to sit in the user message directly before the fragments. A short fragment that follows a list of source → target pairs is read as another entry of that list and echoed back:

request	fragment	returned verbatim
no glossary	# О Diplodoc	2 of 20
glossary of 6 pairs in the user message	# О Diplodoc	20 of 20
glossary of 1 pair in the user message	# О Diplodoc	15 of 15
glossary of 6 pairs in the system prompt	# О Diplodoc	0 of 15
glossary of 6 pairs in the user message	a full sentence	1 of 15

Presence is what matters, not size or relevance: one unrelated pair is as bad as six. Long prose is nearly immune, which is why headings, table cells and one-line includes were the pages that failed.

buildMessages() already solves this for context files: they land in the system prompt because they are identical for every batch and play well with provider-side prompt caching, and a {{contextFiles}} placeholder in either template overrides the placement. The glossary now works the same way, so {{glossary}} in a custom system or user prompt still takes full control.

Units returned untranslated are re-requested

retryUntranslated() sits next to the existing repairDamaged() and reuses retryFragments(): fragments that came back byte-identical to their source while still carrying source-script text are re-requested once, in a request of their own, with the same prompt. Isolation alone helps - the same heading failed 6 runs out of 8 inside the full corpus and 1 run in 10 as a request of its own.

The retry deliberately carries no instruction about the previous attempt. Telling the model that its answer came back unchanged makes it echo far more often: 15 of 20 against 2 of 20 with the plain prompt.

Acceptance mirrors the rule that triggered the retry. A stricter rule - rejecting any answer that still carries source-script text - would discard legitimate translations of pages that quote the source language on purpose and ship their source text instead.

Reporting

The run report gains fixes.untranslatedRetried and fixes.untranslatedKept next to the markup counters, and the run summary mentions them when they are non-zero. units.untranslated keeps its meaning and is now the residual after the retry.

Cost and compatibility

• Translation caches reset once. The default prompts are part of the cache fingerprint, so the first run after the upgrade re-translates everything. This is the established cost of a prompt change in this CLI, and it is why both fixes ship together.
• One extra request per batch that contains an echoed fragment. The retry runs inside the same concurrency slot and the same rate gate as every other request, and re-sends only the failing fragments. Content that legitimately stays in the source script (a proper noun, a code sample) is re-requested once per batch on every run, the same way such content already stays out of the translation cache.
• Consumers with a custom user prompt that does not mention {{glossary}} get their glossary in the system message from now on.

Eval harness

The harness captures translation units through a local echo endpoint, and an echoed fragment is indistinguishable from a refused translation, so the retry re-requested every unit of the russian side and the capture recorded it twice. The capture server now skips a fragment that the previous request for the same file already recorded. The window is deliberately one request wide: the capture run is sequential, so a retry always arrives right after its batch, while translate() never asks for the same unit twice on its own (it caches deferred units by text for the whole run), so a wider window could only drop units that are genuinely new.

The reference stops expecting translated inline code

The system prompt tells the model not to translate text inside inline code, while the corpus reference translated it in ten spans (`текст_ссылки`, `{wide-content title="название таблицы"}`), so a run failed whenever the model followed its own instructions. The reference now keeps the source text there, and tests/eval/README.md records the rule together with the prompt line it mirrors. The untranslated check tolerates a line the reference keeps in the source language, so a model that does translate such a span still passes.

Design notes and the full measurement log: docs/specs/2026-09-16-translate-untranslated-units-design.md.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
96.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud