feat(translate): eval series as the gate before a model change

[martyanovandrey]

Re-opens the work of #2279, which was merged into the branch of #2277 instead of master: that branch was deleted when #2277 landed as a squash, so these commits never reached master. Same content, rebased onto master, nothing added.

The eval harness can tell whether a translation configuration is broken, but it could not be used as the gate it was written for. One run is not a measurement - the same configuration produced a different result in each of 8 runs, because the batches a run composes depend on timing - and the harness failed runs for reasons that have nothing to do with the translation.

What changes

--repeats <n> runs the same configuration n times, each run in its own <workdir>/run-<i>, and applies the thresholds to the totals over the series. Totals, not per-run verdicts: a rule like "every run passes" is stricter than a single run, so a series would flake more than the run it was meant to stabilize. The default stays 1 and a single run is byte-identical to today, including the JSON report shape.

Unscored judge pairs stop failing a run. A pair the judge fails to score is its own hiccup: in the measured series one run failed on 1 unscored pair of 409 while the translation was clean. They now fail a run only above 5% of the pairs sent, which is a malfunction rather than a hiccup.

The ritual is documented. docs/translate-eval.md gains the command, the pass rule, and - more importantly - the three metrics that are deliberately not gates.

The gate

npm run translate:eval -- --real --repeats 3 --max-untranslated 1 \
  --provider openai --api-base <gateway> --model <candidate> --auth <token file>

Over the three runs: markup violations 0, glossary violations 0, untranslated lines at most 1, judge average at least 70.

A live series of the current configuration (Eliza deepseek-v4-flash, ru->en corpus):

run  markup  glossary  untranslated  similarity  judge
1    0       0         0             0.976       100.0
2    0       0         0             0.978       100.0
3    0       0         1             0.975       99.8

Series of 3 runs: markup 0, glossary 0, untranslated 1 (allowed 1)
Judge: 1229 units scored by deepseek-v4-flash, average 99.9/100, 1 below threshold 70
Pages with defects: about.md (run 3)

Verdict: PASS

That is the criterion doing its job: one heading the model occasionally echoes fits the budget, while any regression this corpus has produced returned whole pages in the source language (33 lines on one page).

What is deliberately not a gate

• Similarity measures the distance from the reference phrasing, not quality: glm-5-2 scored 0.936 average with zero defects where deepseek-v4-flash scored 0.98 with zero defects. It compares a model against its own history; it must never accept or reject a candidate.
• The judge average never sees the defect that matters most: a unit returned untranslated is not scored at all, by design (makeJudgeCollector). In the earlier series it reported 100/100 for runs that shipped a whole page in russian.
• Unscored judge pairs, as above.

Scope

No CI automation, no cron, no baseline file, and nothing that gates a release: a real-model run cannot live in this repository's CI, which has no access to an internal gateway. The mock-mode e2e stays the guard of the harness itself, and npm run translate:eval -- --repeats 2 runs offline without credentials.

Design notes, including the measurements behind every number above: docs/specs/2026-09-16-translate-eval-series-design.md.

[martyanovandrey]

already shiped here #2279

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
99.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud