feat(translate): code mode option and unit-local cache keys, translation 1.9.0 DOCSTOOLS-6809 - #2295
Closed
martyanovandrey wants to merge 3 commits into
Closed
martyanovandrey wants to merge 3 commits into
martyanovandrey wants to merge 3 commits into
Conversation
…STOOLS-6809 `--code <precise|adaptive>` (also `code` in the `translate` section of .yfm) selects how much of fenced code blocks goes to translation: - precise: placeholders and shell comments, as before; - adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams (@diplodoc/translation adaptive code mode, diplodoc-platform/translation#284). LLM providers default to adaptive, yandex keeps precise, so machine translation runs are not affected unless the project opts in. The seed command follows the LLM default, since seeds feed the LLM cache and the mode is part of the unit texts. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Two e2e cases run `yfm translate` against a local OpenAI-compatible mock model that translates by a dictionary and records unknown fragments: comments in yaml, python, ts, sql and bash fences and labels of mermaid diagrams are translated while commented-out code, identifiers and the code itself stay byte for byte. A third case checks that the precise mode leaves comments of other languages untouched. The e2e cases need @diplodoc/translation with the adaptive handlers and fail on 1.8.0.
Inline placeholder ids (`g-N`/`x-N`) are numbered through the document, so a unit's text - and with it the translation cache and seed key - depends on how much markup sits above it. Adding one section at the top of a file rewrote the keys of every unit below: on a YTsaurus release notes file the seed hit rate fell from 90% to 43% and 767 units went to the model instead of 229. `loadTranslationUnits` now extracts with `unitLocalIds`, so the ids restart for every unit and unchanged text keeps its key. Seeding and translation share the helper, so keys stay in parity. `translate extract` is untouched: the XLIFF handed to external tools keeps document-wide ids (verified byte-identical against the previous release). Persistent `--cache-dir` caches get new keys for units whose ids did not already start at 1: one cold run, then warm again. Seeds are rebuilt on every run and are not affected.
Brings the adaptive code mode and the `unitLocalIds` extract option, so the cases skipped on 1.8.0 now run. The eval corpus reference page `en/syntax/vars.md` repeats the sentence "Suppose the variable presets file defines:" where the source page repeats it too: with unit-local ids the pipeline serves the second copy from the run cache and never asks the model for it, so the positional capture of both sides must collapse the same way.
martyanovandrey
force-pushed
the
fix/translate-unit-local-ids
branch
from
September 22, 2026 15:07
e23d132 to
17b30de
Compare
martyanovandrey
marked this pull request as ready for review
September 22, 2026 15:07
martyanovandrey
requested review from
a team and
diplodoc-bot
as code owners
September 22, 2026 15:07
martyanovandrey
requested review from
separatrixxx and
stenin-nikita
and removed request for
a team
September 22, 2026 15:07
|
Contributor
Author
|
Folded into #2291, which already has an approval: the same two commits are on top of its branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Bumps
@diplodoc/translationto 1.9.0 and wires up both things it brings. Supersedes #2291 (the code mode part is that PR's commit, rebased here).Code mode (DOCSTOOLS-6809, diplodoc-platform/translation#284)
--code <no|all|precise|adaptive>fortranslate(andseed), alsocodein thetranslatesection of.yfm. The four values mirror thecodeoption of@diplodoc/translation:no: nothing, code blocks are copied as they are;all: the whole block, keys and values included;precise: only<placeholders>and comments of bash/shell fences, as before;adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams. Commented-out code stays as is.Defaults:
adaptivefor LLM providers (yandexgpt, openai, openrouter, anthropic),preciseforyandex, so machine translation runs are not affected unless the project opts in.seedtakes the value of thetranslatesection of.yfmand defaults toadaptive, since seeds feed the LLM cache and the mode is part of the unit texts.extractandcomposeare untouched. A single fence overrides the mode withtranslate=precise/translate=adaptivein its info string. Unknown values fail early both from the CLI and from.yfm, with theCONFIGerror code in the run report.The run report gets an optional
codefield with the mode of the run (additive,schemaVersionunchanged), documented indocs/translate-run-report.md;docs/translate-seed.mdlists--codeamong the options that must match betweenseedandtranslate.E2E: three cases in
tests/e2e/translation.spec.tsrunyfm translateagainst a local OpenAI-compatible mock model (tests/fixtures/mock-model.ts): code comments in the default (adaptive) mode, the same page with--code precise, and mermaid diagrams. Every case compares the translated page with the source line by line: only comment and label lines may differ, everything else is byte-identical. Unit tests cover config resolution per provider, the seed inheriting and defaulting the mode, the unit loader in both modes and the report field.Unit-local cache keys (diplodoc-platform/translation#285)
Inline placeholder ids (
g-N/x-N) are numbered through the document, so a unit's text - and with it the translation cache and seed key - depends on how much markup sits above it. Adding one release section at the top ofyt-server.md(1344 units) rewrote the keys of every unit below it:loadTranslationUnitsnow extracts withunitLocalIds: true, so ids restart for every unit and unchanged text keeps its key. Seeding and translation go through the same helper, so keys stay in parity.translate extractand theyandexprovider are untouched: the XLIFF handed to external tools keeps document-wide ids - checked byte-identical (XLIFF and skeleton) against 5.60.0 with translation 1.8.0.The eval corpus reference page
en/syntax/vars.mdnow repeats "Suppose the variable presets file defines:" where the source page repeats it too: with unit-local ids the pipeline serves the second copy from the run cache and never asks the model for it, so the positional capture of both sides has to collapse the same way. The harness's own guidance for this case is to align the reference with the source.Compatibility
yandex) runs: no change.adaptivecode handling by default: comments and mermaid labels start being translated.--code preciserestores the previous behaviour.--cache-dircaches get new keys for every unit whose ids did not already start at 1 (and for code blocks under the new default mode): one cold run after the update, then the cache is warm again. Seeds (yfm translate seed) are rebuilt from the target files on every run and are not affected.Verified: full unit suite (2232), translation and eval e2e (25), typecheck, lint.