Skip to content

fix(ci): vendor autodoc prompt templates so docs regeneration works again - #11

Merged
anthony-at-pieces merged 3 commits into
mainfrom
fix/autodoc-vendored-prompt-templates
Sep 2, 2026
Merged

anthony-at-pieces merged 3 commits into
mainfrom
fix/autodoc-vendored-prompt-templates

Conversation

@anthony-at-pieces

@anthony-at-pieces anthony-at-pieces commented Aug 24, 2026 •

Copy link
Copy Markdown

Summary

docs-freshness fails on main since the sampling-capability merge, and because the release workflow gates on CI success on main, releases cannot fire. The staleness itself is expected (lib/src/server and lib/src/types changed); the defect is that auto-autodoc's regeneration skips every module with Empty prompt for <type>, skipping, so the docs never regenerate and the hashes never advance. This PR makes regeneration work again by vendoring the prompt templates at the paths the config already declares.

Root Cause

.runtime_ci/autodoc.json declares its templates as consumer-repo paths (scripts/prompts/autodoc_quickstart_prompt.dart, ...) -- that is what the autodoc scaffolder writes -- but the scaffolder never writes the script files themselves. Resolution relied on a fallback in manage_cicd: when the relative path does not exist in the consumer repo, it resolves the runtime_ci_tooling package root from the consumer repo's .dart_tool/package_config.json and reads the scripts from lib/src/prompts/ there.

5e00fe5 removed runtime_ci_tooling from dev_dependencies, following the tooling's own v0.23.13 guidance ("use global activation only -- do not add as a dev_dependency"). Globally activated packages are not in the consumer repo's package_config.json, so the fallback returns null, templatePath stays scripts/prompts/..., and the prompt step runs dart run on a file that does not exist:

  [server] Pass 1: Generating QUICKSTART.md...
  [server] Empty prompt for quickstart, skipping
Generated docs for 0 modules, skipped 2 unchanged.
Autodoc failed for 3 module(s); hashes were not advanced

The sub-process failure is swallowed (runSync returns empty stdout), which is why the log says "Empty prompt" instead of "file not found". The last successful regen (00ef6ee, 2026-03-25 02:21 UTC) ran hours before 5e00fe5 (04:55 UTC) landed; every run since has this failure. The v0.23.13 guidance and the v0.23.13 template resolution contradict each other -- that tooling-side fix (resolve templates from the globally activated package, e.g. via its embedded-prompts extraction) belongs in runtime_ci_tooling.

Second layer, exposed once prompts generated: the workflows install the Gemini CLI unpinned (npm install -g @google/gemini-cli@latest). Current latest (0.56.x) rejects the --allowed-mcp-server-names= (empty) flag that runtime_ci_tooling's docsOnlyPolicyArgs passes for MCP isolation -- Invalid policy rule: mcpName is required if specified (cannot be empty) -- so every Gemini pass died at CLI startup (first commit's auto-autodoc run on this PR). The tooling's comment documents the old contract ("Gemini CLI 0.34+ behavior: empty list -> none"), which no longer holds.

Fix

Vendor the four prompt scripts from runtime_ci_tooling v0.23.13 lib/src/prompts/ -- _ci_config.dart, autodoc_quickstart_prompt.dart, autodoc_api_reference_prompt.dart, autodoc_examples_prompt.dart -- verbatim (formatted at 120 cols) into scripts/prompts/, the exact paths autodoc.json already declares. Template resolution now takes the primary consumer-repo branch (File('$repoRoot/$templatePath').existsSync()), before the broken fallback is ever consulted. The scripts are self-contained (dart:io, dart:convert, and the sibling _ci_config.dart only). scripts/** is excluded from analysis so the vendored files stay byte-diffable against upstream.

The source variants are vendored rather than the tooling's embedded (AOT) variants: the embedded quickstart prompt is trimmed (no changelog-context section), and the source variants are what produced the existing March docs.

Second commit: pin @google/gemini-cli@0.35.0 (the @latest of the last successful regen, 2026-03-25, and the version v0.23.13's flags were written against) at all 9 install sites across ci.yaml, issue-triage.yaml, and release.yaml, so one upstream CLI release cannot break every Gemini-backed job at once. These are template-generated files; the durable fix belongs in the templates (tracked with the resolver bug in open-runtime/runtime_ci_tooling#70).

Test plan

  • dart run scripts/prompts/autodoc_quickstart_prompt.dart Types <repo>/lib/src/types <repo>/lib/src/types -- exit 0, 8904 bytes of prompt (previously: file did not exist, empty stdout)
  • Same for autodoc_api_reference_prompt.dart -- exit 0, 1871 bytes
  • dart pub global activate ... --git-ref v0.23.13 + manage_cicd autodoc locally -- reaches the Gemini gate ("Gemini CLI not installed") instead of failing at prompt generation; the Gemini half only runs in CI where the CLI and GEMINI_API_KEY exist
  • dart analyze -- 71 issues, identical to main baseline (scripts/** excluded)
  • dart format --line-length 120 --set-exit-if-changed on the vendored files -- 0 changes
  • First commit's auto-autodoc run: prompt generation succeeded for all modules (Pass 1: Generating QUICKSTART.md... with no Empty prompt), proving the vendored templates resolve; Gemini then crashed at startup on the unpinned-CLI flag incompatibility above
  • This PR's auto-autodoc run on the second commit is the end-to-end proof: regenerate server / types / top_level docs, push a bot(autodocs) commit to this branch, docs-freshness green (reviewer-visible in this PR's checks)
  • Tooling bug filed: open-runtime/runtime_ci_tooling#70 (global-activation guidance vs package_config-based template resolution); once fixed there, this vendored copy can be dropped or kept as an explicit override

…gain

.runtime_ci/autodoc.json points its templates at scripts/prompts/*.dart
in this repo, but the autodoc scaffolder never wrote those files; the
tooling instead resolved them from the runtime_ci_tooling package root
via the consumer repo's package_config.json. 5e00fe5 removed
runtime_ci_tooling from dev_dependencies (per the tooling's own
"global activation only" guidance), which broke that resolution:
manage_cicd ran `dart run scripts/prompts/...` on files that do not
exist, got empty stdout, and skipped every module with "Empty prompt".
Hashes never advanced, so docs-freshness fails on every run since the
sampling-capability change touched lib/src/server and lib/src/types,
and the release workflow (gated on CI success on main) cannot fire.

Vendor the four prompt scripts from runtime_ci_tooling v0.23.13
lib/src/prompts (verbatim, formatted at 120 cols) at the exact paths
autodoc.json already declares. Template resolution now hits the
primary consumer-repo branch and prompt generation produces non-empty
output for both quickstart and api_reference locally. scripts/ is
excluded from analysis to keep the vendored files diffable against
upstream. The tooling-side fix (resolve templates from the globally
activated package) belongs in runtime_ci_tooling.
@anthony-at-pieces anthony-at-pieces added the bug Something isn't working label Aug 24, 2026
anthony-at-pieces and others added 2 commits August 24, 2026 10:45
With the vendored templates in place, autodoc got past prompt generation
and hit the next failure: `npm install -g @google/gemini-cli@latest` now
resolves to 0.56.x, whose PolicyEngine rejects the
`--allowed-mcp-server-names=` flag that runtime_ci_tooling's
docsOnlyPolicyArgs passes for MCP isolation ("Invalid policy rule:
mcpName is required if specified (cannot be empty)"), so every Gemini
pass dies at CLI startup. The last successful regen (2026-03-25) ran on
0.35.0, the @latest of that day and the version runtime_ci_tooling
v0.23.13 was written against.

Pin 0.35.0 everywhere the generated workflows install the CLI (ci,
issue-triage, release) so one upstream CLI release cannot break all
Gemini-backed jobs at once. These files are template-generated; the
durable fix (pin in the templates, drop or adapt the empty
--allowed-mcp-server-names flag) is tracked in
open-runtime/runtime_ci_tooling#70.
@anthony-at-pieces

Copy link
Copy Markdown
Author

Executive summary, since the diff looks bigger than it is.

What actually changed (556 lines, ~10 of them decisions):

  • 546 lines are 4 files copied verbatim from runtime_ci_tooling v0.23.13's own lib/src/prompts/ into scripts/prompts/. Zero original code -- review is "diff against upstream", not "read 546 lines".
  • 9 one-line version pins: gemini-cli@latest -> @0.35.0 in the generated workflows.
  • 1 analyzer-exclude line.

How it broke, and when:

  1. On 2026-03-25, 5e00fe5 removed runtime_ci_tooling from dev_dependencies -- following that tool's own v0.23.13 upgrade guidance. Side effect nobody could see: the autodoc config points at prompt scripts under scripts/prompts/ that were never in this repo; the tool silently found them through the dev_dependency. With it gone, every autodoc run since has generated empty prompts and skipped all modules ("Empty prompt for quickstart, skipping"). Invisible while no source changed.
  2. PR fix(mcp): model sampling tools and context as object capabilities #10 changed lib/src/server + lib/src/types, so the docs are now legitimately stale, regeneration is broken per (1), and docs-freshness fails on every main run.
  3. Fixing (1) exposed (2): the workflows install gemini-cli@latest, and the current latest (0.56.x) crashes on a flag the March-era tooling passes. Pinning to 0.35.0 -- the version running when autodoc last worked -- fixes that.

Why fix it at all:

Blast radius: CI/docs tooling only. No lib/ code, no runtime behavior, nothing shipped to consumers. Both root causes are also filed upstream (open-runtime/runtime_ci_tooling#70) so the vendored copy and pins can eventually go away.

@anthony-at-pieces

Copy link
Copy Markdown
Author

I'm open to different approached to fixing this, but currently I'm downgrading this in priority to focus on cost saving issues and will come back to this.

@anthony-at-pieces
anthony-at-pieces merged commit 7bc1088 into main Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants