Skip to content

feat(translate): code mode option and unit-local cache keys, translation 1.9.0 DOCSTOOLS-6809 - #2295

Closed
martyanovandrey wants to merge 3 commits into
masterfrom
fix/translate-unit-local-ids
Closed

martyanovandrey wants to merge 3 commits into
masterfrom
fix/translate-unit-local-ids

Conversation

@martyanovandrey

@martyanovandrey martyanovandrey commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Bumps @diplodoc/translation to 1.9.0 and wires up both things it brings. Supersedes #2291 (the code mode part is that PR's commit, rebased here).

Code mode (DOCSTOOLS-6809, diplodoc-platform/translation#284)

--code <no|all|precise|adaptive> for translate (and seed), also code in the translate section of .yfm. The four values mirror the code option of @diplodoc/translation:

  • no: nothing, code blocks are copied as they are;
  • all: the whole block, keys and values included;
  • precise: only <placeholders> and comments of bash/shell fences, as before;
  • adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams. Commented-out code stays as is.

Defaults: adaptive for LLM providers (yandexgpt, openai, openrouter, anthropic), precise for yandex, so machine translation runs are not affected unless the project opts in. seed takes the value of the translate section of .yfm and defaults to adaptive, since seeds feed the LLM cache and the mode is part of the unit texts. extract and compose are untouched. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Unknown values fail early both from the CLI and from .yfm, with the CONFIG error code in the run report.

The run report gets an optional code field with the mode of the run (additive, schemaVersion unchanged), documented in docs/translate-run-report.md; docs/translate-seed.md lists --code among the options that must match between seed and translate.

E2E: three cases in tests/e2e/translation.spec.ts run yfm translate against a local OpenAI-compatible mock model (tests/fixtures/mock-model.ts): code comments in the default (adaptive) mode, the same page with --code precise, and mermaid diagrams. Every case compares the translated page with the source line by line: only comment and label lines may differ, everything else is byte-identical. Unit tests cover config resolution per provider, the seed inheriting and defaulting the mode, the unit loader in both modes and the report field.

Unit-local cache keys (diplodoc-platform/translation#285)

Inline placeholder ids (g-N / x-N) are numbered through the document, so a unit's text - and with it the translation cache and seed key - depends on how much markup sits above it. Adding one release section at the top of yt-server.md (1344 units) rewrote the keys of every unit below it:

source seed hit rate requests
unchanged 90% 6
section added at the top, before 43% 37
section added at the top, after 83% 11

loadTranslationUnits now extracts with unitLocalIds: true, so ids restart for every unit and unchanged text keeps its key. Seeding and translation go through the same helper, so keys stay in parity. translate extract and the yandex provider are untouched: the XLIFF handed to external tools keeps document-wide ids - checked byte-identical (XLIFF and skeleton) against 5.60.0 with translation 1.8.0.

The eval corpus reference page en/syntax/vars.md now repeats "Suppose the variable presets file defines:" where the source page repeats it too: with unit-local ids the pipeline serves the second copy from the run cache and never asks the model for it, so the positional capture of both sides has to collapse the same way. The harness's own guidance for this case is to align the reference with the source.

Compatibility

  • Machine translation (yandex) runs: no change.
  • LLM runs get adaptive code handling by default: comments and mermaid labels start being translated. --code precise restores the previous behaviour.
  • Persistent --cache-dir caches get new keys for every unit whose ids did not already start at 1 (and for code blocks under the new default mode): one cold run after the update, then the cache is warm again. Seeds (yfm translate seed) are rebuilt from the target files on every run and are not affected.

Verified: full unit suite (2232), translation and eval e2e (25), typecheck, lint.

…STOOLS-6809

`--code <precise|adaptive>` (also `code` in the `translate` section of
.yfm) selects how much of fenced code blocks goes to translation:

- precise: placeholders and shell comments, as before;
- adaptive: also line comments of yaml, python, go, sql and other
  languages and labels of mermaid diagrams (@diplodoc/translation
  adaptive code mode, diplodoc-platform/translation#284).

LLM providers default to adaptive, yandex keeps precise, so machine
translation runs are not affected unless the project opts in. The seed
command follows the LLM default, since seeds feed the LLM cache and the
mode is part of the unit texts. A single fence overrides the mode with
translate=precise / translate=adaptive in its info string.

Two e2e cases run `yfm translate` against a local OpenAI-compatible mock
model that translates by a dictionary and records unknown fragments:
comments in yaml, python, ts, sql and bash fences and labels of mermaid
diagrams are translated while commented-out code, identifiers and the
code itself stay byte for byte. A third case checks that the precise
mode leaves comments of other languages untouched.

The e2e cases need @diplodoc/translation with the adaptive handlers and
fail on 1.8.0.
Inline placeholder ids (`g-N`/`x-N`) are numbered through the document,
so a unit's text - and with it the translation cache and seed key -
depends on how much markup sits above it. Adding one section at the top
of a file rewrote the keys of every unit below: on a YTsaurus release
notes file the seed hit rate fell from 90% to 43% and 767 units went to
the model instead of 229.

`loadTranslationUnits` now extracts with `unitLocalIds`, so the ids
restart for every unit and unchanged text keeps its key. Seeding and
translation share the helper, so keys stay in parity. `translate extract`
is untouched: the XLIFF handed to external tools keeps document-wide ids
(verified byte-identical against the previous release).

Persistent `--cache-dir` caches get new keys for units whose ids did not
already start at 1: one cold run, then warm again. Seeds are rebuilt on
every run and are not affected.
Brings the adaptive code mode and the `unitLocalIds` extract option,
so the cases skipped on 1.8.0 now run.

The eval corpus reference page `en/syntax/vars.md` repeats the sentence
"Suppose the variable presets file defines:" where the source page
repeats it too: with unit-local ids the pipeline serves the second
copy from the run cache and never asks the model for it, so the
positional capture of both sides must collapse the same way.
@martyanovandrey
martyanovandrey force-pushed the fix/translate-unit-local-ids branch from e23d132 to 17b30de Compare September 22, 2026 15:07
@martyanovandrey
martyanovandrey marked this pull request as ready for review September 22, 2026 15:07
@martyanovandrey
martyanovandrey requested review from separatrixxx and stenin-nikita and removed request for a team September 22, 2026 15:07
@martyanovandrey martyanovandrey changed the title fix(translate): key the cache on unit-local placeholder ids feat(translate): code mode option and unit-local cache keys, translation 1.9.0 DOCSTOOLS-6809 Sep 22, 2026
@sonarqubecloud

Copy link
Copy Markdown

@martyanovandrey

Copy link
Copy Markdown
Contributor Author

Folded into #2291, which already has an approval: the same two commits are on top of its branch.

@martyanovandrey
martyanovandrey deleted the fix/translate-unit-local-ids branch September 22, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant