feat(translate): code mode option for comments and mermaid labels DOCSTOOLS-6809 - #2291
Merged
Merged
Conversation
martyanovandrey
force-pushed
the
DOCSTOOLS-6809-translate-e2e
branch
3 times, most recently
from
September 22, 2026 09:51
b8c33a2 to
986a852
Compare
…STOOLS-6809 `--code <precise|adaptive>` (also `code` in the `translate` section of .yfm) selects how much of fenced code blocks goes to translation: - precise: placeholders and shell comments, as before; - adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams (@diplodoc/translation adaptive code mode, diplodoc-platform/translation#284). LLM providers default to adaptive, yandex keeps precise, so machine translation runs are not affected unless the project opts in. The seed command follows the LLM default, since seeds feed the LLM cache and the mode is part of the unit texts. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Two e2e cases run `yfm translate` against a local OpenAI-compatible mock model that translates by a dictionary and records unknown fragments: comments in yaml, python, ts, sql and bash fences and labels of mermaid diagrams are translated while commented-out code, identifiers and the code itself stay byte for byte. A third case checks that the precise mode leaves comments of other languages untouched. The e2e cases need @diplodoc/translation with the adaptive handlers and fail on 1.8.0.
martyanovandrey
force-pushed
the
DOCSTOOLS-6809-translate-e2e
branch
from
September 22, 2026 11:01
986a852 to
c459521
Compare
goldserg
approved these changes
Sep 22, 2026
Inline placeholder ids (`g-N`/`x-N`) are numbered through the document, so a unit's text - and with it the translation cache and seed key - depends on how much markup sits above it. Adding one section at the top of a file rewrote the keys of every unit below: on a YTsaurus release notes file the seed hit rate fell from 90% to 43% and 767 units went to the model instead of 229. `loadTranslationUnits` now extracts with `unitLocalIds`, so the ids restart for every unit and unchanged text keeps its key. Seeding and translation share the helper, so keys stay in parity. `translate extract` is untouched: the XLIFF handed to external tools keeps document-wide ids (verified byte-identical against the previous release). Persistent `--cache-dir` caches get new keys for units whose ids did not already start at 1: one cold run, then warm again. Seeds are rebuilt on every run and are not affected.
Brings the adaptive code mode and the `unitLocalIds` extract option, so the cases skipped on 1.8.0 now run. The eval corpus reference page `en/syntax/vars.md` repeats the sentence "Suppose the variable presets file defines:" where the source page repeats it too: with unit-local ids the pipeline serves the second copy from the run cache and never asks the model for it, so the positional capture of both sides must collapse the same way.
martyanovandrey
marked this pull request as ready for review
September 22, 2026 15:14
martyanovandrey
requested review from
a team and
diplodoc-bot
as code owners
September 22, 2026 15:14
martyanovandrey
requested review from
Yurok868 and
kadymov
and removed request for
a team
September 22, 2026 15:14
martyanovandrey
enabled auto-merge (squash)
September 22, 2026 15:14
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What
Makes the new code handling of
@diplodoc/translation(diplodoc-platform/translation#284, DOCSTOOLS-6809) configurable and covers it with e2e.Option.
--code <no|all|precise|adaptive>fortranslate(andseed), alsocodein thetranslatesection of.yfm. The four values mirror thecodeoption of@diplodoc/translation:no: nothing, code blocks are copied as they are;all: the whole block, keys and values included;precise: only<placeholders>and comments of bash/shell fences, as before;adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams. Commented-out code stays as is.Defaults:
adaptivefor LLM providers (yandexgpt, openai, openrouter, anthropic),preciseforyandex, so machine translation runs are not affected unless the project opts in.seedtakes the value of thetranslatesection of.yfm(its own scope istranslate.seed, which does not see the parent otherwise) and defaults toadaptive, since seeds feed the LLM cache and the mode is part of the unit texts.extractandcomposeare untouched. A single fence overrides the mode withtranslate=precise/translate=adaptivein its info string. Unknown values fail early both from the CLI (commander choices) and from.yfm, with theCONFIGerror code in the run report.Report. The run report gets an optional
codefield with the mode of the run (additive,schemaVersionunchanged), documented indocs/translate-run-report.md;docs/translate-seed.mdlists--codeamong the options that must match betweenseedandtranslate.E2E. Three cases in
tests/e2e/translation.spec.tsrunyfm translateagainst a local OpenAI-compatible mock model (tests/fixtures/mock-model.ts, reusable by other specs). The mock translates fragments by a dictionary and records unknown fragments as misses, so a unit that the engine fails to extract (or extracts wrongly, e.g.br/out of<br/>) fails the test.mocks/translation/code-comments, default (adaptive): yaml, python, ts, sql and bash fences, a fence inside a list item and atextfence without a known comment syntax. Comments are translated; commented-out code and the code itself stay as is.--code precise: only the bash comments and placeholders change, the other comments stay untouched.mocks/translation/mermaid: a sequence diagram with an%%{init}%%directive,participant RPC as RPC, notes, messages and aloop; a flowchart with a frontmatter title, all node shapes, edge labels and a bracketed subgraph; apiewith a title.Every case compares the translated page with the source line by line: only comment and label lines may differ, everything else is byte-identical.
Unit tests: config resolution of
codeper provider, arg, config and invalid values (src/commands/translate/index.spec.ts); the seed inheriting, overriding and defaulting the mode (commands/seed.command.spec.ts); the unit loader in both modes (utils/units.spec.ts); the report field. The eval harness e2e (translate-eval.spec.ts) passes with the adaptive default.Unit-local cache keys (diplodoc-platform/translation#285)
Inline placeholder ids (
g-N/x-N) are numbered through the document, so a unit's text - and with it the translation cache and seed key - depends on how much markup sits above it. Adding one release section at the top ofyt-server.md(1344 units) rewrote the keys of every unit below it:loadTranslationUnitsnow extracts withunitLocalIds: true, so ids restart for every unit and unchanged text keeps its key. Seeding and translation go through the same helper, so keys stay in parity.translate extractand theyandexprovider are untouched: the XLIFF handed to external tools keeps document-wide ids - checked byte-identical (XLIFF and skeleton) against 5.60.0 with translation 1.8.0.The eval corpus reference page
en/syntax/vars.mdnow repeats "Suppose the variable presets file defines:" where the source page repeats it too: with unit-local ids the pipeline serves the second copy from the run cache and never asks the model for it, so the positional capture of both sides has to collapse the same way. The harness's own guidance for this case is to align the reference with the source.Dependency
@diplodoc/translationbumped to 1.9.0 (adaptive code mode and theunitLocalIdsoption), so the cases skipped on 1.8.0 now run.Compatibility
yandex) runs: no change.adaptivecode handling by default: comments and mermaid labels start being translated.--code preciserestores the previous behaviour.--cache-dircaches get new keys for every unit whose ids did not already start at 1 (and for code blocks under the new default mode): one cold run after the update, then the cache is warm again. Seeds (yfm translate seed) are rebuilt from the target files on every run and are not affected.