Skip to content

feat(translate): code mode option for comments and mermaid labels DOCSTOOLS-6809 - #2291

Merged
martyanovandrey merged 3 commits into
masterfrom
DOCSTOOLS-6809-translate-e2e
Sep 22, 2026
Merged

martyanovandrey merged 3 commits into
masterfrom
DOCSTOOLS-6809-translate-e2e

Conversation

@martyanovandrey

@martyanovandrey martyanovandrey commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

What

Makes the new code handling of @diplodoc/translation (diplodoc-platform/translation#284, DOCSTOOLS-6809) configurable and covers it with e2e.

Option. --code <no|all|precise|adaptive> for translate (and seed), also code in the translate section of .yfm. The four values mirror the code option of @diplodoc/translation:

  • no: nothing, code blocks are copied as they are;
  • all: the whole block, keys and values included;
  • precise: only <placeholders> and comments of bash/shell fences, as before;
  • adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams. Commented-out code stays as is.

Defaults: adaptive for LLM providers (yandexgpt, openai, openrouter, anthropic), precise for yandex, so machine translation runs are not affected unless the project opts in. seed takes the value of the translate section of .yfm (its own scope is translate.seed, which does not see the parent otherwise) and defaults to adaptive, since seeds feed the LLM cache and the mode is part of the unit texts. extract and compose are untouched. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Unknown values fail early both from the CLI (commander choices) and from .yfm, with the CONFIG error code in the run report.

Report. The run report gets an optional code field with the mode of the run (additive, schemaVersion unchanged), documented in docs/translate-run-report.md; docs/translate-seed.md lists --code among the options that must match between seed and translate.

E2E. Three cases in tests/e2e/translation.spec.ts run yfm translate against a local OpenAI-compatible mock model (tests/fixtures/mock-model.ts, reusable by other specs). The mock translates fragments by a dictionary and records unknown fragments as misses, so a unit that the engine fails to extract (or extracts wrongly, e.g. br/ out of <br/>) fails the test.

  • mocks/translation/code-comments, default (adaptive): yaml, python, ts, sql and bash fences, a fence inside a list item and a text fence without a known comment syntax. Comments are translated; commented-out code and the code itself stay as is.
  • the same mock with --code precise: only the bash comments and placeholders change, the other comments stay untouched.
  • mocks/translation/mermaid: a sequence diagram with an %%{init}%% directive, participant RPC as RPC, notes, messages and a loop; a flowchart with a frontmatter title, all node shapes, edge labels and a bracketed subgraph; a pie with a title.

Every case compares the translated page with the source line by line: only comment and label lines may differ, everything else is byte-identical.

Unit tests: config resolution of code per provider, arg, config and invalid values (src/commands/translate/index.spec.ts); the seed inheriting, overriding and defaulting the mode (commands/seed.command.spec.ts); the unit loader in both modes (utils/units.spec.ts); the report field. The eval harness e2e (translate-eval.spec.ts) passes with the adaptive default.

Unit-local cache keys (diplodoc-platform/translation#285)

Inline placeholder ids (g-N / x-N) are numbered through the document, so a unit's text - and with it the translation cache and seed key - depends on how much markup sits above it. Adding one release section at the top of yt-server.md (1344 units) rewrote the keys of every unit below it:

source seed hit rate requests
unchanged 90% 6
section added at the top, before 43% 37
section added at the top, after 83% 11

loadTranslationUnits now extracts with unitLocalIds: true, so ids restart for every unit and unchanged text keeps its key. Seeding and translation go through the same helper, so keys stay in parity. translate extract and the yandex provider are untouched: the XLIFF handed to external tools keeps document-wide ids - checked byte-identical (XLIFF and skeleton) against 5.60.0 with translation 1.8.0.

The eval corpus reference page en/syntax/vars.md now repeats "Suppose the variable presets file defines:" where the source page repeats it too: with unit-local ids the pipeline serves the second copy from the run cache and never asks the model for it, so the positional capture of both sides has to collapse the same way. The harness's own guidance for this case is to align the reference with the source.

Dependency

@diplodoc/translation bumped to 1.9.0 (adaptive code mode and the unitLocalIds option), so the cases skipped on 1.8.0 now run.

Compatibility

  • Machine translation (yandex) runs: no change.
  • LLM runs get adaptive code handling by default: comments and mermaid labels start being translated. --code precise restores the previous behaviour.
  • Persistent --cache-dir caches get new keys for every unit whose ids did not already start at 1 (and for code blocks under the new default mode): one cold run after the update, then the cache is warm again. Seeds (yfm translate seed) are rebuilt from the target files on every run and are not affected.

@martyanovandrey martyanovandrey changed the title test(translate): cover code comments and mermaid labels DOCSTOOLS-6809 feat(translate): code mode option for comments and mermaid labels DOCSTOOLS-6809 Sep 22, 2026
@martyanovandrey
martyanovandrey force-pushed the DOCSTOOLS-6809-translate-e2e branch 3 times, most recently from b8c33a2 to 986a852 Compare September 22, 2026 09:51
…STOOLS-6809

`--code <precise|adaptive>` (also `code` in the `translate` section of
.yfm) selects how much of fenced code blocks goes to translation:

- precise: placeholders and shell comments, as before;
- adaptive: also line comments of yaml, python, go, sql and other
  languages and labels of mermaid diagrams (@diplodoc/translation
  adaptive code mode, diplodoc-platform/translation#284).

LLM providers default to adaptive, yandex keeps precise, so machine
translation runs are not affected unless the project opts in. The seed
command follows the LLM default, since seeds feed the LLM cache and the
mode is part of the unit texts. A single fence overrides the mode with
translate=precise / translate=adaptive in its info string.

Two e2e cases run `yfm translate` against a local OpenAI-compatible mock
model that translates by a dictionary and records unknown fragments:
comments in yaml, python, ts, sql and bash fences and labels of mermaid
diagrams are translated while commented-out code, identifiers and the
code itself stay byte for byte. A third case checks that the precise
mode leaves comments of other languages untouched.

The e2e cases need @diplodoc/translation with the adaptive handlers and
fail on 1.8.0.
@martyanovandrey
martyanovandrey force-pushed the DOCSTOOLS-6809-translate-e2e branch from 986a852 to c459521 Compare September 22, 2026 11:01
Inline placeholder ids (`g-N`/`x-N`) are numbered through the document,
so a unit's text - and with it the translation cache and seed key -
depends on how much markup sits above it. Adding one section at the top
of a file rewrote the keys of every unit below: on a YTsaurus release
notes file the seed hit rate fell from 90% to 43% and 767 units went to
the model instead of 229.

`loadTranslationUnits` now extracts with `unitLocalIds`, so the ids
restart for every unit and unchanged text keeps its key. Seeding and
translation share the helper, so keys stay in parity. `translate extract`
is untouched: the XLIFF handed to external tools keeps document-wide ids
(verified byte-identical against the previous release).

Persistent `--cache-dir` caches get new keys for units whose ids did not
already start at 1: one cold run, then warm again. Seeds are rebuilt on
every run and are not affected.
Brings the adaptive code mode and the `unitLocalIds` extract option,
so the cases skipped on 1.8.0 now run.

The eval corpus reference page `en/syntax/vars.md` repeats the sentence
"Suppose the variable presets file defines:" where the source page
repeats it too: with unit-local ids the pipeline serves the second
copy from the run cache and never asks the model for it, so the
positional capture of both sides must collapse the same way.
@martyanovandrey
martyanovandrey marked this pull request as ready for review September 22, 2026 15:14
@martyanovandrey
martyanovandrey requested review from Yurok868 and kadymov and removed request for a team September 22, 2026 15:14
@martyanovandrey
martyanovandrey enabled auto-merge (squash) September 22, 2026 15:14
@sonarqubecloud

Copy link
Copy Markdown

@martyanovandrey
martyanovandrey merged commit a339f33 into master Sep 22, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants