feat: adaptive code mode with comments of any language and mermaid labels DOCSTOOLS-6809 - #284
Merged
Conversation
martyanovandrey
requested review from
a team and
diplodoc-bot
as code owners
September 22, 2026 08:04
martyanovandrey
changed the title
martyanovandrey
force-pushed
the
feat/code-comments-and-mermaid-labels
branch
from
September 22, 2026 08:47
4a45851 to
3efc16d
Compare
martyanovandrey
added a commit
to diplodoc-platform/cli
that referenced
this pull request
Sep 22, 2026
…STOOLS-6809 `--code <precise|adaptive>` (also `code` in the `translate` section of .yfm) selects how much of fenced code blocks goes to translation: - precise: placeholders and shell comments, as before; - adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams (@diplodoc/translation adaptive code mode, diplodoc-platform/translation#284). LLM providers default to adaptive, yandex keeps precise, so machine translation runs are not affected unless the project opts in. The seed command follows the LLM default, since seeds feed the LLM cache and the mode is part of the unit texts. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Two e2e cases run `yfm translate` against a local OpenAI-compatible mock model that translates by a dictionary and records unknown fragments: comments in yaml, python, ts, sql and bash fences and labels of mermaid diagrams are translated while commented-out code, identifiers and the code itself stay byte for byte. A third case checks that the precise mode leaves comments of other languages untouched. The e2e cases need @diplodoc/translation with the adaptive handlers and fail on 1.8.0.
martyanovandrey
force-pushed
the
feat/code-comments-and-mermaid-labels
branch
2 times, most recently
from
September 22, 2026 09:40
4f38019 to
2de657b
Compare
martyanovandrey
added a commit
to diplodoc-platform/cli
that referenced
this pull request
Sep 22, 2026
…STOOLS-6809 `--code <precise|adaptive>` (also `code` in the `translate` section of .yfm) selects how much of fenced code blocks goes to translation: - precise: placeholders and shell comments, as before; - adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams (@diplodoc/translation adaptive code mode, diplodoc-platform/translation#284). LLM providers default to adaptive, yandex keeps precise, so machine translation runs are not affected unless the project opts in. The seed command follows the LLM default, since seeds feed the LLM cache and the mode is part of the unit texts. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Two e2e cases run `yfm translate` against a local OpenAI-compatible mock model that translates by a dictionary and records unknown fragments: comments in yaml, python, ts, sql and bash fences and labels of mermaid diagrams are translated while commented-out code, identifiers and the code itself stay byte for byte. A third case checks that the precise mode leaves comments of other languages untouched. The e2e cases need @diplodoc/translation with the adaptive handlers and fail on 1.8.0.
martyanovandrey
force-pushed
the
feat/code-comments-and-mermaid-labels
branch
from
September 22, 2026 09:45
2de657b to
a9c957a
Compare
martyanovandrey
added a commit
to diplodoc-platform/cli
that referenced
this pull request
Sep 22, 2026
…STOOLS-6809 `--code <precise|adaptive>` (also `code` in the `translate` section of .yfm) selects how much of fenced code blocks goes to translation: - precise: placeholders and shell comments, as before; - adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams (@diplodoc/translation adaptive code mode, diplodoc-platform/translation#284). LLM providers default to adaptive, yandex keeps precise, so machine translation runs are not affected unless the project opts in. The seed command follows the LLM default, since seeds feed the LLM cache and the mode is part of the unit texts. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Two e2e cases run `yfm translate` against a local OpenAI-compatible mock model that translates by a dictionary and records unknown fragments: comments in yaml, python, ts, sql and bash fences and labels of mermaid diagrams are translated while commented-out code, identifiers and the code itself stay byte for byte. A third case checks that the precise mode leaves comments of other languages untouched. The e2e cases need @diplodoc/translation with the adaptive handlers and fail on 1.8.0.
martyanovandrey
force-pushed
the
feat/code-comments-and-mermaid-labels
branch
from
September 22, 2026 11:01
a9c957a to
a0a7d97
Compare
martyanovandrey
added a commit
to diplodoc-platform/cli
that referenced
this pull request
Sep 22, 2026
…STOOLS-6809 `--code <precise|adaptive>` (also `code` in the `translate` section of .yfm) selects how much of fenced code blocks goes to translation: - precise: placeholders and shell comments, as before; - adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams (@diplodoc/translation adaptive code mode, diplodoc-platform/translation#284). LLM providers default to adaptive, yandex keeps precise, so machine translation runs are not affected unless the project opts in. The seed command follows the LLM default, since seeds feed the LLM cache and the mode is part of the unit texts. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Two e2e cases run `yfm translate` against a local OpenAI-compatible mock model that translates by a dictionary and records unknown fragments: comments in yaml, python, ts, sql and bash fences and labels of mermaid diagrams are translated while commented-out code, identifiers and the code itself stay byte for byte. A third case checks that the precise mode leaves comments of other languages untouched. The e2e cases need @diplodoc/translation with the adaptive handlers and fail on 1.8.0.
martyanovandrey
force-pushed
the
feat/code-comments-and-mermaid-labels
branch
2 times, most recently
from
September 22, 2026 13:17
fe925fa to
ff62724
Compare
…bels DOCSTOOLS-6809
Fenced code stays out of translation on purpose, but two kinds of human-readable
text were lost with it: line comments in languages other than shell (yaml,
python, go, sql...) and labels of mermaid diagrams.
The precise code mode keeps its historical behaviour (placeholders and shell
comments), so existing consumers are not affected. The adaptive mode, which was
declared but identical to precise, now adds:
- Line comments of hash (#), double-slash (//) and double-dash (--) languages.
Only the comment text goes into the unit, code and indentation stay byte for
byte. Commented-out code, tool directives and separators are told apart by a
heuristic and stay untouched, exposing their <placeholders> only.
- Mermaid: notes, messages, participant aliases and block labels of sequence
diagrams; node labels, edge labels and bracketed subgraph titles of flowcharts;
the title of any diagram. Identifiers, %%{init}%% directives, comments and
styling are not touched. Other diagram types are a separate step.
- Code between labels is skipped verbatim, so a label equal to an identifier on
the same line (participant RPC as RPC) never replaces the identifier.
A single fence overrides the mode with translate=precise or translate=adaptive
in its info string. The code option is now validated with the other extract
options.
Both modes receive the liquid-unescaped code, which fixes extraction of shell
comments with {{variables}} that used to throw.
martyanovandrey
force-pushed
the
feat/code-comments-and-mermaid-labels
branch
from
September 22, 2026 14:08
ff62724 to
a5481f2
Compare
|
goldserg
approved these changes
Sep 22, 2026
martyanovandrey
added a commit
to diplodoc-platform/cli
that referenced
this pull request
Sep 22, 2026
…STOOLS-6809 `--code <precise|adaptive>` (also `code` in the `translate` section of .yfm) selects how much of fenced code blocks goes to translation: - precise: placeholders and shell comments, as before; - adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams (@diplodoc/translation adaptive code mode, diplodoc-platform/translation#284). LLM providers default to adaptive, yandex keeps precise, so machine translation runs are not affected unless the project opts in. The seed command follows the LLM default, since seeds feed the LLM cache and the mode is part of the unit texts. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Two e2e cases run `yfm translate` against a local OpenAI-compatible mock model that translates by a dictionary and records unknown fragments: comments in yaml, python, ts, sql and bash fences and labels of mermaid diagrams are translated while commented-out code, identifiers and the code itself stay byte for byte. A third case checks that the precise mode leaves comments of other languages untouched. The e2e cases need @diplodoc/translation with the adaptive handlers and fail on 1.8.0.
martyanovandrey
added a commit
to diplodoc-platform/cli
that referenced
this pull request
Sep 22, 2026
…STOOLS-6809 (#2291) * feat(translate): code mode option for comments and mermaid labels DOCSTOOLS-6809 `--code <precise|adaptive>` (also `code` in the `translate` section of .yfm) selects how much of fenced code blocks goes to translation: - precise: placeholders and shell comments, as before; - adaptive: also line comments of yaml, python, go, sql and other languages and labels of mermaid diagrams (@diplodoc/translation adaptive code mode, diplodoc-platform/translation#284). LLM providers default to adaptive, yandex keeps precise, so machine translation runs are not affected unless the project opts in. The seed command follows the LLM default, since seeds feed the LLM cache and the mode is part of the unit texts. A single fence overrides the mode with translate=precise / translate=adaptive in its info string. Two e2e cases run `yfm translate` against a local OpenAI-compatible mock model that translates by a dictionary and records unknown fragments: comments in yaml, python, ts, sql and bash fences and labels of mermaid diagrams are translated while commented-out code, identifiers and the code itself stay byte for byte. A third case checks that the precise mode leaves comments of other languages untouched. The e2e cases need @diplodoc/translation with the adaptive handlers and fail on 1.8.0. * fix(translate): key the cache on unit-local placeholder ids Inline placeholder ids (`g-N`/`x-N`) are numbered through the document, so a unit's text - and with it the translation cache and seed key - depends on how much markup sits above it. Adding one section at the top of a file rewrote the keys of every unit below: on a YTsaurus release notes file the seed hit rate fell from 90% to 43% and 767 units went to the model instead of 229. `loadTranslationUnits` now extracts with `unitLocalIds`, so the ids restart for every unit and unchanged text keeps its key. Seeding and translation share the helper, so keys stay in parity. `translate extract` is untouched: the XLIFF handed to external tools keeps document-wide ids (verified byte-identical against the previous release). Persistent `--cache-dir` caches get new keys for units whose ids did not already start at 1: one cold run, then warm again. Seeds are rebuilt on every run and are not affected. * chore(deps): bump @diplodoc/translation to 1.9.0 Brings the adaptive code mode and the `unitLocalIds` extract option, so the cases skipped on 1.8.0 now run. The eval corpus reference page `en/syntax/vars.md` repeats the sentence "Suppose the variable presets file defines:" where the source page repeats it too: with unit-local ids the pipeline serves the second copy from the run cache and never asks the model for it, so the positional capture of both sides must collapse the same way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
yfm translatekeeps fenced code out of translation on purpose, but two kinds of human-readable text were lost with it (feedback from a YT docs run, DOCSTOOLS-6809):# Клиентская частьin yaml stayed in Russian);Note over RPC: Создает protobuf сообщение,RPC->>Bus: Передает сообщение).translate=allis not an option for examples: it sends the whole block, keys and values included.What
Modes. The
precisecode mode (default withcompact) keeps its historical behaviour:<placeholders>of any fence and comments of shell fences. Existing consumers see no change. Theadaptivemode was declared inCodeProcessingbut identical topreciseand passed by nobody; it now carries the new behaviour below. A single fence overrides the mode withtranslate=precise/translate=adaptivein its info string, which works through the existingoptions.translate || state.code || defaultchain. Thecodeoption is validated together with the other extract options, and README documents the modes.Line comments (adaptive).
#(yaml, python, toml, ini, dockerfile, ...),//(js, ts, go, java, c, cpp, rust, proto, ...) and--(sql, lua). Only the comment text becomes a unit; code, indentation, markers and decorations (# --- Section ---) stay byte for byte. A marker counts only at line start or after whitespace and outside string literals, so"#fff",https://,a--band"prefix // words"stay code.String literals.
src/skeleton/rules/code/literals.tsdescribes the literal conventions of every language in the table (which quotes delimit literals, which may span lines, where a backslash escapes, triple quotes, heredocs, yaml block scalars) and scans a block with that state: template strings and tagged templates (html\...`), docstrings and text blocks, shell / sql / yaml strings spanning lines, heredocs (<<EOF,<<-'EOF',<<<EOT,<<~EOS) and block scalars (key: |) are literals on every line; doubled quotes ('it''s') and backslashes escape; a literal opens after any token (()=>"..."), while an apostrophe inside or after a word (don't,users'), an inch mark (10") or a lifetime (&'a str) does not open one. Comments of scripts and configs embedded in yaml block scalars are therefore not translated: inside a literal a#cannot be told from data. The table-driven suiteliterals.spec.ts` covers a hundred cases across javascript, typescript, java, go, rust, python, yaml, sql, yql, shell, dockerfile, php, ruby, perl, lua, toml, ini, haskell.Mermaid (adaptive). A dedicated handler extracts:
Note over/left of/right of X: text, messagesA->>B: text(all arrow kinds, activation markers),participant X as Label,loop/alt/else/opt/par/critical/... label,box;[ ],( ),([ ]),[( )],(( )),((( ))),> ],{ },{{ }},[/ /],[\ \],[[ ]], quoted labels, the v11A@{ shape: ..., label: "text" }form), edge labels-->|text|and-- text -->,subgraph id [Title];titleof any diagram (body or frontmatter).Identifiers,
%%{init}%%directives,%%comments,classDef/style/clicklines and the diagram structure are not touched. Other diagram types are a separate step.Engine details.
#/:, so a label equal to an identifier on the same line (participant RPC as RPC) never replaces the identifier. Leading indentation is never part of a skip, which keeps fences inside list items working.{{variables}}used to throwUnable to extract valid tokens; now they extract.Checks
src/skeleton/rules/code/index.spec.ts(modes and per-fence overrides),comments.spec.ts,mermaid.spec.ts(extract, skeleton, compose roundtrip, compose with translations, prose heuristic table, corner cases below).precise(the default): 0 unit changes, apart from 14 pages that used to throw on a{{variable}}in a shell comment and now extract, and one page whose fence is written as "``` bash" with a space, which is now recognized (the info string is trimmed). This is the "existing consumers see no change" claim, measured.adaptive: +360 / -106 and +2856 / -776 units. Everything added is comments and diagram labels; everything removed is either commented-out code or junk the old handlers produced: generics (List<T>,Struct<a:Int32>), html tags,<matched across brackets ((List<Int32), comparisons (a < b && c > d),#inside values. No new errors, no new roundtrip mismatches; the 52 errors left on the second corpus are the same on 1.8.0.Corner cases found by the corpora and covered by tests: a comment that starts with a placeholder (
# <worker> - address ...) is prose; list enumerators (# 1. Step) stay in the skeleton; a placeholder must read as a word (a letter first, no whitespace at the edges), in typed languages (java, c++, ts, rust, yql, ...) also as prose (non-ASCII letters or several words without:/,), and in markup languages (html, xml, jsx) angle brackets are tags;%%{init}%%directives spanning several lines;--o/--xarrow heads are not edge labels;title "Text"loses its quotes; unicode node ids;;comments of ini,--of yql,#and//of hcl.CLI side (option
--code, LLM providers default toadaptive, yandex toprecise, e2e): diplodoc-platform/cli#2291, draft until this change is released and bumped there.Follow-ups (not in this PR):
subgraph Titlewithout brackets,/* block */comments, other mermaid diagram types,clicktooltips.