feat: admit authored profiles to the canonical improvement path - #885
Conversation
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 2831ab9a
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-16T21:13:28Z
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — e1900328
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-16T21:57:45Z
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Coverage | 2 of 2 lenses (value, usefulness) |
| Concerns | 3 (1 medium-concern, 2 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.1s |
| Interrogation | 143.5s (2 bridge agents) |
| Total | 143.6s |
💰 Value — sound-with-nits
Adds a truthful high-level entry point that runs a complete human/imported/compound profile through the canonical sealed-experiment → paired-measurement → proposal chain without fabricating optimizer lineage, and properly de-duplicates the shared experiment plumbing — ship.
- What it does: Adds public
proposeAuthoredAgentProfileImprovement()(src/intelligence/authored-profile-improvement.ts:105) which takes a caller-authored complete candidate profile, validates it against the declared source state digest, derives profile diffs itself viaagentImprovementProfileDiffs, seals a benchmark and experiment, runs exact paired baseline/candidate measurement through a product-owned execu - Goals it achieves: Before this PR, the only high-level profile proposal entry point was
proposeAgentProfileImprovement(src/intelligence/improvement-cycle.ts:669), which always runs an optimizer search and stampssource: 'optimizer'lineage (line 755) plus an optimization receipt. A human-authored, imported, or compound profile therefore had to either fabricate optimizer lineage or hand-assemble sealing, paired - Assessment: Good on its merits. It is squarely in the grain of the codebase: the sibling optimizer path already defines the canonical chain, and this PR reuses the exact same contract functions (
sealAgentProfileImprovementExperiment,runAgentProfileImprovementExperiment,verifyAgentProfileImprovementExperimentComparison) and the same product-executor shape (measure(input) => receipt). Critically, the - Better / existing approach: none — this is the right approach. I searched for an existing equivalent (grep for proposeAuthored, sealAgentProfileImprovementExperiment, createAgentImprovementProposal across src/, scripts/, tests/) and found no existing authored-candidate path; the alternatives were fabricating optimizer lineage or manual consumer assembly, both worse. The only residual overlap is noted below as a weak concern.
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error
🎯 Usefulness — sound-with-nits
A truthful, non-duplicative high-level entry point that admits authored/imported/compound profiles into the canonical sealed-measurement-and-proposal chain, built by extracting shared helpers rather than forking; it needs one error-path ordering fix (its own new test is red) and two small packaging-
- Integration: Reachable and wired: exported from the package entrypoint (src/intelligence/index.ts:81-88), documented in docs/api/intelligence.md:4373 and docs/api/primitive-catalog.md:431, with an end-to-end suite (tests/authored-profile-improvement.test.ts). No in-repo product caller yet, but this is the @tangle-network/agent-runtime library whose sibling proposeAgentProfileImprovement (src/intelligence/impro
- Fit with existing patterns: Fits the grain precisely. Before this, the only paths into the one-shot proposal chain were improve() (optimizer lineage, src/intelligence/improvement-cycle.ts:709) or manually assembling ~6 contract calls; the alternative of lying with source:'optimizer' is exactly what the lineage schema partitions off (node_modules/@tangle-network/agent-interface/dist/agent-candidate-lineage-schema.js:35 enum [
- Real-world viability: Happy path and most error paths hold: budget/policy mismatch, source-digest drift, unchanged candidate, and held-out reuse are all rejected before any measurement (verified by the tests at lines 238-290), unknown state digests are refused inside the executor callback, and AbortSignal/maxConcurrency/cost-ceiling pass through to the shared runner. One real ordering defect: the reserved-metadata-key
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟡 ~35-line seal→run→verify→propose tail duplicated between the two propose functions [duplication] ``
The orchestration tail in authored-profile-improvement.ts:171-218 (state-digest→profile map, execute adapter with 'requested an unknown profile state' error, runAgentProfileImprovementExperiment call, verify + createAgentImprovementProposal) closely mirrors improvement-cycle.ts:746-800. The same PR already extracted the seven true helpers, so this residual is deliberate-looking, and the two tails differ at real seams (optimizer receipt attachment, generationsExplored, improvement.dispose() clean
🎯 Usefulness Audit
🟠 ? [robustness] ``
src/intelligence/authored-profile-improvement.ts:210 validates the reserved AGENT_IMPROVEMENT_SOURCE_METADATA_KEY only after the full paired measurement at line 188; the PR's own test (tests/authored-profile-improvement.test.ts:246) fails on this — 12 paid executions ran before the /reserves/ throw, 0 expected. An invalid input should not consume the customer-approved budgetUsd. Fix: hoist the reserved-key check next to the existing early assertNoCallerOptimizationReceipt(options.metadata) at li
🟡 ? [integration] ``
The sibling entrypoint is guarded in the packed-artifact check (scripts/verify-package-exports.mjs:538, per CHANGELOG.md:760 'so the public entrypoint cannot disappear silently') and exercised by scripts/fixtures/packed-cohort-consumer.ts:261; the new proposeAuthoredAgentProfileImprovement has neither. Add it to both so the new public export gets the same disappearance protection as its sibling. Does not gate shipping.
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
❌ Needs Work —
|
| opencode GLM 5.2 | opencode DeepSeek v4 Pro | opencode DeepSeek v4 Flash | aggregate | |
|---|---|---|---|---|
| Readiness | 0 | 1 | 0 | 0 |
| Confidence | 90 | 90 | 90 | 90 |
| Correctness | 0 | 1 | 0 | 0 |
| Security | 0 | 1 | 0 | 0 |
| Testing | 0 | 1 | 0 | 0 |
| Architecture | 0 | 1 | 0 | 0 |
Reviewer score is advisory once the run is complete and the verdict has no blockers.
Full multi-shot audit completed 6/6 planned shots over 11 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 6/6 planned shots over 11 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 6/6 planned shots over 11 changed files. Global verifier still owns final merge decision.
Blocking
🔴 HIGH 190-line hand-curated decision table and AgentProfile law deleted, not relocated — docs/canonical-api.md
Commit 964ee41 removes everything after the §1 legend: the four-step spine, §1.5 'The AgentProfile rule', the entire §2 'I want to ___ → use ___ → NOT ___' decision table, §2.1 'Which front door do I use', and the two-agent pattern table (base 213 lines → head 24; file ends immediately after the 'holdout' legend bullet). Verified the content exists nowhere else at head and that all documented symbols (loopUntil, fanout, runPersonified, superviseSurface, definePersona, candidatePopulation, …) still exist in src, so this deletes live guidance, not stale content. CLAUDE.md:36-43 makes this file repo-orientation step 0 and states the §2 judgment layer 'stays hand-curated'; docs/BUILDING.md:61 repeats the contract. Fix: restore the deleted body from git show e90028d:docs/canonical-api.md, kee
🔴 HIGH Deleting §2 decision table and §1.5 orphans references across the repo, including a generated file regenerated in this PR — docs/canonical-api.md
The change removes the entire §2 'I want to ___ → use ___ → NOT ___' matrix (~80 rows), §2.1, and §1.5 (AgentProfile law + defineGitHubResource example) — 190 of 213 lines — and nothing else in the PR relocates that content. References that are now false: docs/api/primitive-catalog.md intro (regenerated in this PR to 0.137.1, still says it is the 'mechanical companion to the JUDGMENT in canonical-api.md (§2 decision table + §1.5 AgentProfile law)'); scripts/gen-primitive-catalog.mjs:371 (the generator's hardcoded string); README.md:126 ('find the primitive: I want to ___ → use ___'); docs/README.md:9/12/34; docs/BUILDING.md:61; docs/MAINTAINING.md:15/41; docs/architecture.md:458 ('canonical-api.md §1.5'); docs/research/loop-facade-postmortem.md:101 ('governed by the canonical-api.md §2 dec
🔴 HIGH §2 decision table and §1.5 AgentProfile law deleted without relocation or reference updates — docs/canonical-api.md
Commit 964ee41 deletes 190 lines (four-step spine, §1.5, the entire §2 'I want to ___ -> use ___ -> NOT ___' decision table, §2.1, two-agent patterns) and adds only the version bump. The content is not moved anywhere in the PR (verified: docs/api/intelligence.md +196 is a generated TypeDoc page; 'Do NOT build' decision-table header is gone repo-wide). ~12 files still cite these sections as canonical: CLAUDE.md:36/43, README.md:126, docs/README.md:34 ('API spine + decision table'), docs/BUILDING.md:61, docs/MAINTAINING.md:15, generated docs/api/primitive-catalog.md:10 ('§2 decision table + §1.5 AgentProfile law'), docs/architecture.md:458, docs/design/prime-agent-harness-integration.md:18, docs/api/{agent.md:768, mcp.md:870, profiles.md:11, runtime.md}. Fix: either relocate the decision ta
🔴 HIGH Forged caller metadata rejected only after the full paid measurement runs; shipped test red at head — src/intelligence/authored-profile-improvement.ts
The only pre-execution metadata guard is assertNoCallerOptimizationReceipt(options.metadata) (line 108), which checks just the 'optimizationReceipt' key. The AGENT_IMPROVEMENT_SOURCE_METADATA_KEY reservation is enforced inside profileImprovementMetadata (src/intelligence/profile-improvement-experiment.ts:124), invoked at line 210 — AFTER runAgentProfileImprovementExperiment ([line 188](https://github.com/tangle-network/agent-runtime/blob/1e1c35f6dc60a2efb6bd5edb639602fd742470bc/src/intelligenc
🔴 HIGH Forged source-metadata is rejected only after the paid measurement runs — src/intelligence/authored-profile-improvement.ts
The early guard assertNoCallerOptimizationReceipt(options.metadata) at line 108 checks only the 'optimizationReceipt' key (optimization-receipt.ts:65), not AGENT_IMPROVEMENT_SOURCE_METADATA_KEY. That reserved source-metadata key is checked only inside profileImprovementMetadata (profile-improvement-experiment.ts:124-127), which is invoked at line 210 inside measuredComparisonFromAgentProfileImprovementExperiment — i.e. after runAgentProfileImprovementExperiment ([line 188](https://github.com/t
🔴 HIGH Reserved metadata key rejection fires after the paid measurement run; PR test fails — src/intelligence/authored-profile-improvement.ts
The AGENT_IMPROVEMENT_SOURCE_METADATA_KEY reservation check lives only in profileImprovementMetadata() (authored-profile-improvement.ts:210), which is evaluated after runAgentProfileImprovementExperiment() (line 188) completes. A caller who sets the reserved key causes the full paired measurement to execute and bill before the /reserves/ error throws. The top-of-function assertNoCallerOptimizationReceipt (line 108) checks only the 'optimizationReceipt' key, not the source-metadata key. Consequ
🔴 HIGH Forged-metadata test fails: reserved agentImprovementSource key rejected only after paid execution — tests/authored-profile-improvement.test.ts
Line 239-246: sets options.metadata={[AGENT_IMPROVEMENT_SOURCE_METADATA_KEY]:'caller-controlled-source'} then asserts expect(forgedMetadata.observed).toHaveLength(0) after the /reserves/ rejection. Confirmed via
vitest runthis assertion fails: observed has length 12 (minimumPairsForPairedDeltaTest(0.95)=6 * 2 arms). The function throws /reserves/ (from profileImprovementMetadata, profile-improvement-experiment.ts:124-127) but only at line 210 of authored-profile-improvement.ts, AFTER runAgentProfi
🔴 HIGH Test fails at head: forged source metadata rejected only after 12 paid runs execute — tests/authored-profile-improvement.test.ts
Evidence:
npx vitest run tests/authored-profile-improvement.test.tsat head 1e1c35f fails: 'expected [ { arm: baseline …(11) } ] to have a length of +0 but got 12'. The rejects.toThrow(/reserves/) part passes because profileImprovementMetadata (profile-improvement-experiment.ts:124-128) throws, but it is invoked at authored-profile-improvement.ts:210, after runAgentProfileImprovementExperiment at line 188 already executed all minimumPairedRuns*2 (12) executor calls. Impact: the PR ships a red test (CI blocker), and the underlying product behavior burns the customer's entire paired-measurement budget before rejecting invalid input, contradicting
🔴 HIGH Test red at HEAD: forged-metadata rejection happens after 12 executor runs, not before execution — tests/authored-profile-improvement.test.ts
expect(forgedMetadata.observed).toHaveLength(0) fails (got 12) because the reserved-metadata guard for AGENT_IMPROVEMENT_SOURCE_METADATA_KEY ('agentImprovementSource') runs only in profileImprovementMetadata (src/intelligence/profile-improvement-experiment.ts:124-128), which is called inside measuredComparisonFromAgentProfileImprovementExperiment at src/intelligence/authored-profile-improvement.ts:210 -- AFTER runAgentProfileImprovementExperiment (line 188) has executed every cell. The pre-execution assertNoCallerOptimizationReceipt ([line 108](https://github.com/tangle-network/agent-runtime/blob/1e1c35f6dc60a2efb6bd5edb639602fd742470bc/tests/auth
Other
🟠 MEDIUM Deleting §2 silently disables the CI freshness gate's decision-table symbol check — docs/canonical-api.md
scripts/check-docs-freshness.mjs (CLASS 3, lines ~314-323) finds the §2 table by scanning for a header containing '| I want to' and 'Do NOT', then only validates 'Use (import)' code-spans while
inTableis true. With the table deleted,inTablenever becomes true, so no EXPORT drift is reported and the gate exits green. This silently retires the documented anti-staleness enforcement (MAINTAINING.md:20 and CLAUDE.md:43 both promise a decision-table symbol that no longer exists turns into a RED BUILD). If the removal is intentional, MAINTAINING.md, BUILDING.md, CLAUDE.md, and the gate comment must be updated in the same PR; they were not.
🟠 MEDIUM Freshness gate cannot detect prose deletion, so this regression ships green — docs/canonical-api.md
The file's own header comment mandates 'Run pnpm docs:freshness after editing this file'. Verified scripts/check-docs-freshness.mjs (CLASS 6, lines 545-585) only checks that backticked symbols present in the prose resolve to real exports; deleting prose removes checks rather than adding them, and no class validates the decision table's existence. So CI passes silently while the repo's canonical anti-reinvention doc is gutted. Impact: no automated signal will catch this class of truncation now or in future. Fix beyond restoring the body: consider a minimal structural assertion (e.g. required headings §2/§2.1 present) in check-docs-freshness.mjs — or at minimum fla
🟠 MEDIUM In-PR generated catalog still cites the deleted sections, contradicting the doc it points at — docs/canonical-api.md
The same PR regenerates docs/api/primitive-catalog.md, whose header says it 'is the mechanical companion to the JUDGMENT in canonical-api.md (§2 decision table + §1.5 AgentProfile law)… that doc says WHICH primitive to reach for'. After this change that judgment layer does not exist, so the generated catalog (and docs/README.md:9,34, docs/concepts.md:159, docs/design/prime-agent-harness-integration.md:18 which cites 'canonical-api.md §1.5' by anchor) all point readers at deleted content. Impact: every orientation path routes new contributors to a 24-line stub whose header still promises a decision table. Fix: restore the body (primary fix); if truncation were intentional, the catalog generator text and all cross-references would need a coordinated update in the same PR, which is absent.
🟠 MEDIUM Surviving 24-line doc is internally self-contradictory — docs/canonical-api.md
Line 3-5 HTML comment still says 'This file maps common jobs to the right public API', but the file now maps nothing. Line 13 says 'Everything below labelled /kernel lives there — the recursive atom (Scope/Supervisor)...' yet everything below was deleted. Line 15 still warns the reader that any hand-rolled loop 'already exists' and enumerates guarantees (equal-k, selector≠judge, complete usage capture) with no in-doc way to f
🟠 MEDIUM New public API shipped under a patch bump; repo convention and commit types say minor — package.json
Version moves 0.137.0 -> 0.137.1 (patch) in the same PR that newly exports proposeAuthoredAgentProfileImprovement plus 5 types from the published ./intelligence entrypoint (src/intelligence/index.ts, in exports map at package.json:44-46), and the commits are typed 'feat'. The repo's two precedent bump-carrying feature PRs both bumped minor for additive API: 6fd900f (#865) 0.134.9->0.135.0 and ecda1bd (#879) 0.135.3->0.136.0, and CHANGELOG 0.137.0 states the working promise 'a minor release is additive, a patch release is a fix'. Impact: for 0.x caret consumers (^0.137.0 resolves only within 0.137.x) a patch carries new surface, eroding the semver signal the repo deliberately maintains; no breakage is possible since the change is additive. Fix: re-run
pnpm run release:prepare 0.138.0(w
🟠 MEDIUM Version 0.137.1 is below the already-published 0.138.0 — package.json
Head declares 0.137.1 but
npm view @tangle-network/agent-runtime versionsshows latest=0.138.0 (no 0.137.1 on the registry), and the 0.138.0 release commit 64a7651 ('chore(release): 0.138.0 — runTree leaves the kernel surface') descends from this PR's base e90028d without being an ancestor of head 1e1c35f (verified via git merge-base --is-ancestor). Merging as-is lands a patch version below the current release train; publish.yml skips only registry-existing versions, so 0.137.1 would publish as an out-of-order release that consumers on the 0.138.0 line never see. Fix: rebase onto current main and bump past the published version (e.g. 0.139.0) before merge. This is exactly the drift class check-version-bump.mjs documents ('a release someone else merged in the meantime'), which the mani
🟠 MEDIUM Authored-path held-out freshness guard is caller-optional and vacuous without a development split — src/intelligence/authored-profile-improvement.ts
Unlike the optimizer path (improvement-cycle.ts assertProfileReleaseWorkIsFresh, which derives developmentSplitDigest and scenarioPartitions from real ImproveMethodResult lineage and enforces them), the authored path trusts the caller: developmentScenarios is an optional parameter (authored-profile-improvement.ts:35-36), and when it is omitted the only remaining guard is the equality check at line 233 (lineage.developmentSplitDigest === benchmark.suite.splitDigest), which is vacuous when the caller supplies no developmentSplitDigest. A caller can therefore ship a benchmark that reuses its own human-development scenarios with zero runtime det
🟠 MEDIUM Held-out freshness check is defeatable by rep-count changes when developmentScenarios is omitted — src/intelligence/authored-profile-improvement.ts
The default disjointness guarantee is exact equality of lineage.developmentSplitDigest and benchmark.suite.splitDigest. campaignSplitDigestFromIdentities (agent-eval 0.145.21, llm-judge chunk line 307) hashes {scenarios, reps}, so the same development scenarios reused in the release benchmark with a different rep count yield a different splitDigest and pass the equality check. The scenario-level disjointness check (lines 236-246) runs only when the caller supplies the optional development
🟡 LOW diff? renders resolved target type instead of exported alias — docs/api/intelligence.md
Source declares
diff?: AuthoredAgentProfileDiffOptionsbut the doc rendersAgentImprovementTargetProfileDiffOptions. This is accurate (AuthoredAgentProfileDiffOptions = NonNullable<Parameters[2]> = AgentImprovementTargetProfileDiffOptions, so the alias is a no-op), but a reader sees the alias documented separately at line 3837 while the option field points elsewhere. Cosmetic only; no functional impact.
🟡 LOW source property loses AgentImprovementSource type fidelity — docs/api/intelligence.md
The generated doc renders
source: objectwhile the source declaressource: AgentImprovementSource(src/intelligence/authored-profile-improvement.ts:72), which carriessourceIdentity/sourceDigestused for the digest-match guard. This is a pre-existing generator limitation for imported (non-re-exported) types, not a regression — the existing ProposeAgentProfileImprovementOptions.source at line 2243 renders identically. No fix required for this PR; a future doc-generator improvement would re-export or cross-link AgentImprovementSource.
🟡 LOW ProposeAuthoredAgentProfileImprovementResult has no TSDoc, so it is absent from the catalog table — docs/api/primitive-catalog.md
The new result interface is listed only in the 'Undocumented supporting types' paragraph, not as a table row, because the declaration in src/intelligence/authored-profile-improvement.ts has no TSDoc summary line. This is consistent with the existing ProposeAgentProfileImprovementResult pattern and the generator's documented row policy, so it is informational rather than a defect. A one-line TSDoc at the declaration would earn it a catalog row; without one, agents browsing the catalog cannot discover the return shape.
🟡 LOW ProposeAuthoredAgentProfileImprovementResult lacks TSDoc, so it appears only in the undocumented-types list — docs/api/primitive-catalog.md
src/intelligence/authored-profile-improvement.ts:93 declares the result interface with no doc comment, so the catalog lists it under 'Undocumented supporting types' rather than as a documented row like its sibling options/result types (ProposeAgentProfileImprovementResult is likewise undocumented, so this follows existing convention). Impact: none functional — consumers see the type in intelligence.md via the return-type link. Fix if desired: add a one-line TSDoc at the declaration and rerun
pnpm run docs:api; it would then earn a table row.
🟡 LOW CLASS-3 freshness-gate coverage of the decision table becomes vacuously empty — docs/canonical-api.md
scripts/check-docs-freshness.mjs CLASS 3/4 only scan canonical-api's §2/§3 (comment at line 509: 'CLASS 3/4 only scanned canonical-api's §2/§3'). With those sections deleted, the claimed gate enforcement of decision-table symbols — asserted in CLAUDE.md:43 and docs/MAINTAINING.md:41 — silently becomes a no-op; no CI signal would ever fire on a decision table, and if a partial table is later re-added it is re-covered only if the section markers survive. The version/peer pins (CLASS 1) do still pass: 0.137.1, agent-eval >=0.145.21, sandbox >=0.27.1, agent-interface ^1.0.0 all match package.json peerDependencies (verified [lines 7-11](https://github.com/tangle-network/ag
🟡 LOW Remaining doc is a degraded stub with orphaned legend terms — docs/canonical-api.md
The file now ends at §1 'Mental model: the spine' with a legend defining five terms (profile, driver<->worker, conserved budget pool, combinator, holdout) that no remaining body content uses, while the header (lines 13-15) still promises 'Read this before writing any orchestration, optimization, or measurement code' guidance that no longer follows. The doc truncates mid-thought with no §2. This reads as an incomplete cleanup rather than a finished consolidation.
🟡 LOW No CHANGELOG entry for the bumped version 0.137.1 — package.json
CHANGELOG.md's newest entry is still 0.137.0 and the PR diff touches no CHANGELOG file, yet package.json now declares 0.137.1. Every prior release in the file back through 0.132.x has an entry, and both precedent bump-carrying PRs (#865 +13 lines, #879 +30 lines) added one in the same PR; scripts/check-version-bump.mjs instructs 'add the CHANGELOG entry that says what a consumer must do differently'. No workflow gate enforces it, so nothing goes red, but v0.137.1 would publish as the first entry-less release. Fix: add a short 0.137.1 section describing the authored-profile candidate path before tagging.
🟡 LOW Patch bump for additive public API — repo convention calls for minor — package.json
The PR adds new public API (export proposeAuthoredAgentProfileImprovement + 5 types via src/intelligence/index.ts, plus new modules src/intelligence/authored-profile-improvement.ts and profile-improvement-experiment.ts), i.e. a backwards-compatible feature addition. The repo's own changelog for 0.137.0 explicitly distinguishes minor from patch ('This is a minor release, not a patch'). A patch bump for a new feature is inconsistent with that documented convention; 0.138.1 or a 0.139.x minor would match the grain once the stale-version nit above is resolved.
🟡 LOW Redundant partial metadata pre-check — src/intelligence/authored-profile-improvement.ts
assertNoCallerOptimizationReceipt(options.metadata) here duplicates the same check inside profileImprovementMetadata (profile-improvement-experiment.ts:123) and covers only one of the two reserved keys, which is exactly why the high finding slips past it. Once the source-key check is hoisted (finding 1), consolidate both reservations into one early guard so the fail-fast set cannot drift from the enforced set.
🟡 LOW generationsExplored test assertion reads the wrong property path — src/intelligence/authored-profile-improvement.ts
The code sets generationsExplored: 0 in the comparison options (authored-profile-improvement.ts:207), which surfaces as comparison.evaluation.generationsExplored. The test asserts result.proposal.evaluation.generationsExplored toBeUndefined (tests/authored-profile-improvement.test.ts:194) — a top-level property that never exists on AgentProfileImprovementMeasuredComparison, so the assertion passes trivially regardless of the recorded generation count. It guards nothing. It should assert evaluation.evaluation.generationsExplored === 0.
🟡 LOW ProfileImprovementBenchmarkInput duplicates AgentProfileImprovementBenchmark — src/intelligence/profile-improvement-experiment.ts
The new interface (tasks/reps/seeds/policy) is field-identical to AgentProfileImprovementBenchmark in improvement-cycle.ts:275-280 and must be kept in sync by hand; drift would surface only as a type error at call sites that happen to use both. Fix: define it once (an
import typefrom improvement-cycle creates no runtime cycle since type imports are erased) or extract the shared shape to a leaf module.
🟡 LOW Reference-identity assertion couples the test to an implementation detail — tests/authored-profile-improvement.test.ts
expect(fixture.observed.filter(arm==='baseline').every(entry => entry.profile === fixture.baselineProfile)).toBe(false) asserts baseline observed profiles are NOT reference-equal to the input object. This only holds because the runtime re-parses via parseExactAgentProfile before measurement; a refactor that preserves object identity (or memoizes the parse) silently flips the assertion with no behavioral change. Prefer asserting the canonicalized contents (e.g., digest equality) rather than object identity; the following systemPrompt check is the meaningful assertion.
🟡 LOW Weak profile-immutability assertion only proves one baseline entry differs — tests/authored-profile-improvement.test.ts
The assertion
fixture.observed.filter(baseline).every((entry) => entry.profile === fixture.baselineProfile)toBe(false) passes if even a single entry is a different object reference; it does not prove every execution received a runtime-parsed frozen copy rather than the caller's mutable object. Impact: a partial regression where some executions alias the caller's profile would go undetected. Fix: assert.every((entry) => entry.profile !== fixture.baselineProfile)to be true (all entries are copies), optionally plus a Object.isFrozen check if the runtime freezes parsed profiles.
tangletools · 2026-08-16T22:21:27Z · trace
tangletools
left a comment
There was a problem hiding this comment.
❌ 9 Blocking Findings — 1e1c35f6
Full multi-shot audit completed 6/6 planned shots over 11 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 6/6 planned shots over 11 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 6/6 planned shots over 11 changed files. Global verifier still owns final merge decision.
Full immutable report for this review: trace
Summary comment for this run: full summary
tangletools · 2026-08-16T22:21:27Z · immutable trace
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 06239e75
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-16T22:30:51Z
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 005d4cb6
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-16T22:53:18Z
|
@tangletools review now |
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 005d4cb6
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-17T00:56:48Z
The direct candidate path is a new consumer-visible surface, so it ships under a minor. 0.138.1 is taken by the Eval 0.146.0 release.
# Conflicts: # docs/api/primitive-catalog.md # docs/canonical-api.md # package.json # src/testing/fixtures/agent-improvement-proposal.json # src/testing/fixtures/agent-profile-improvement-proposal.json
|
Rebased onto main (0.138.1 released). Version moves to 0.139.0 — the direct candidate path is a new consumer-visible surface, and 0.138.1 was taken by the Eval 0.146.0 release. Local proof on this head: @tangletools review now |
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Coverage | 2 of 2 lenses (value, usefulness) |
| Concerns | 2 (2 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.1s |
| Interrogation | 367.8s (2 bridge agents) |
| Total | 367.9s |
💰 Value — sound-with-nits
Adds a truthful proposeAuthoredAgentProfileImprovement() entry point that runs a complete human/imported/compound profile through the canonical sealed measurement-and-proposal chain without fabricating optimizer lineage; coherent, in-grain, well-guarded, ship it.
- What it does: Adds
proposeAuthoredAgentProfileImprovement()(src/intelligence/authored-profile-improvement.ts:104) which takes a caller-authored complete candidate profile plus baseline profile, derives exact state digests and Runtime-owned profile diffs, seals a held-out benchmark, runs exact paired baseline/candidate measurement through a product-owned executor, verifies the comparison, and emits an evidenc - Goals it achieves: Read from the change: (1) truthful candidate lineage — a human-authored, imported, or compound profile can now enter the one-shot proposal path as itself, instead of lying with
source: 'optimizer'; (2) one canonical measurement chain for all candidates, so authored candidates get the same sealing, paired execution, held-out freshness, and evidence-bound proposal as optimizer ones; (3) spend prot - Assessment: Good on its merits. The alternative — bolting an optional candidate onto
proposeAgentProfileImprovement(src/intelligence/improvement-cycle.ts:669) — would tangle two genuinely different pre-measurement flows (analysis + optimizer search vs direct validation) and drag in itsanalysis/improvementresult fields that have no meaning here. A separate entry point over a shared support module matc - Better / existing approach: Searched for an existing equivalent: the 'sealed-candidate' path (src/intelligence/activation.ts:70) activates already-measured candidates — a later stage, not a measurement path, so no reuse missed.
proposeAgentProfileImprovementcannot serve this without fabricating optimizer lineage (it hard-codessource: 'optimizer'at improvement-cycle.ts:754 and requiresexecutor.optimize). The only re - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error
🎯 Usefulness — sound-with-nits
A truthful authored/imported/compound profile entry into the canonical sealed-experiment-and-proposal path, built by extracting and sharing the exact helpers the optimizer path uses — coherent, in-grain, and verified working; only a missing export-guard entry is worth noting.
- Integration: Exported from the public intelligence entrypoint (src/intelligence/index.ts:81-88), documented in docs/api/intelligence.md:4373 and docs/api/primitive-catalog.md:439, and exercised by tests/authored-profile-improvement.test.ts (5 tests: happy path for human and import lineage, plus 8 rejection modes) — I ran them: 5/5 pass, and the existing improvement-cycle suite still passes 32/32 after the help
- Fit with existing patterns: Excellent fit. The new function mirrors the tail of proposeAgentProfileImprovement (improvement-cycle.ts:717-808) — sealing, paired exact execution, verified comparison, proposal creation — while correctly omitting improve() and the optimization receipt. Critically, the shared pieces (cost ledger, policy/budget assertion, preparation accounting, state digest, benchmark sealing, metadata sealing) w
- Real-world viability: Error paths fail closed before spending measurement budget: forged reserved metadata, NaN budget, policy/budget mismatch, source-digest drift, unchanged candidate, and leaked development scenarios all reject with zero executor invocations (each asserted in tests/authored-profile-improvement.test.ts:238-290). AbortSignal, maxConcurrency, and the cost ledger thread through to the paired run; unknown
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟡 Measurement tail duplicated between optimizer and authored profile paths [duplication] ``
The seal-experiment → run → verify → propose sequence at src/intelligence/authored-profile-improvement.ts:173-220 is line-for-line near-identical to src/intelligence/improvement-cycle.ts:746-801 (including the profilesByStateDigest map and execute wrapper, improvement-cycle.ts:764-781 vs authored-profile-improvement.ts:185-202). The author already extracted the leaf helpers into profile-improvement-experiment.ts (commit e190032); the remaining tail could be one shared `runSealedProfileExperiment
🎯 Usefulness Audit
🟡 Packed-export guard list not extended with the new public function [integration] ``
scripts/verify-package-exports.mjs (~line 538, expectedIntelligence) still lists proposeAgentProfileImprovement but not proposeAuthoredAgentProfileImprovement, so the new export is not protected against silently disappearing from the packed package — the exact failure mode this guard exists for (see CHANGELOG.md:798). One-line addition to the list.
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
|
Merging with A genuine value audit sound-with-nits covers the current head The earlier 9 Blocking Findings were against Proof on the merged head: all CI checks green, |
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Coverage | 2 of 2 lenses (value, usefulness) |
| Concerns | 2 (2 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.1s |
| Interrogation | 336.9s (2 bridge agents) |
| Total | 337.0s |
💰 Value — sound-with-nits
Adds a truthful one-shot entry for human/imported/compound profile candidates into the canonical sealed-experiment-and-proposal path, composing existing primitives in the codebase's grain — ship, with one minor duplication note.
- What it does: Adds exported proposeAuthoredAgentProfileImprovement() (src/intelligence/authored-profile-improvement.ts:104) that takes a complete caller-supplied candidate profile through the canonical chain: exact profile parse → state-digest binding to the declared source → Runtime-derived profile diffs → sealed held-out benchmark → exact paired measurement via a product-owned executor → verified comparison →
- Goals it achieves: Before this PR the only one-shot profile proposal entry, proposeAgentProfileImprovement (improvement-cycle.ts:669), hardcoded lineage source 'optimizer' (verified at main's improvement-cycle.ts:844), so a human-authored, imported, or compound profile could only enter by fabricating optimizer lineage. The lower-level pieces existed (sealCandidateExperiment accepts source 'human', tests/exact-proces
- Assessment: Good change on its merits. It composes rather than reinvents: every load-bearing step delegates to an existing primitive (sealAgentProfileImprovementExperiment, runAgentProfileImprovementExperiment, verifyAgentProfileImprovementExperimentComparison, createAgentImprovementProposal, agentImprovementProfileDiffs, parseExactAgentProfile). The helper extraction into profile-improvement-experiment.ts is
- Better / existing approach: Searched for an existing equivalent before concluding: git grep for authored/human/import paths in src (only the new file), verified main hardcodes source 'optimizer' in the profile proposal path (main improvement-cycle.ts:844), and confirmed the only prior 'human' lineage acceptance is the lower-level candidate-bundle experiment layer (tests/exact-process-candidate.test.ts:695, sealCandidateExper
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error
🎯 Usefulness — sound
A truthful entry point for human/imported/compound profiles into the existing canonical measure-and-propose chain, built by extracting shared helpers rather than duplicating them — no prior equivalent existed.
- Integration: Reachable and wired: exported from src/intelligence/index.ts:88 and shipped via the package's './intelligence' subpath (package.json:42-45), documented in docs/api/intelligence.md:4373 and docs/api/primitive-catalog.md:439, minor-version-bumped to 0.139.0. No in-repo production caller exists, but that matches how this published library is consumed — the pre-existing optimizer-backed proposeAgentPr
- Fit with existing patterns: Fits the established pattern exactly. I compared the new function line-by-line against proposeAgentProfileImprovement (improvement-cycle.ts:669-809): it mirrors the same seal-benchmark → seal-experiment → runAgentProfileImprovementExperiment → verifyAgentProfileImprovementExperimentComparison → createAgentImprovementProposal chain, minus improve() and the optimization receipt. Rather than copying,
- Real-world viability: Error paths fail closed before spend: forged metadata, invalid/mismatched budgets, source-digest drift, unchanged candidates, and reused development scenarios are all rejected before any executor call, and the tests assert zero executor invocations on each rejection (tests/authored-profile-improvement.test.ts:238-290,
observedlength 0). Concurrency and cancellation pass through (maxConcurrency/ - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟡 Seal→run→verify→propose tail duplicated between the two profile proposal paths [maintenance] ``
The block building profilesByStateDigest, sealing the experiment, running runAgentProfileImprovementExperiment with the identical executor-wrapper (including the same 'unknown profile state' error string), verifying the comparison, and calling createAgentImprovementProposal is near-verbatim duplicated: improvement-cycle.ts:746-800 vs authored-profile-improvement.ts:173-220. Only generationsExplored source, optimization-receipt attachment, and findings source differ. Now that profile-improvement-
🎯 Usefulness Audit
🟡 ProfileImprovementBenchmarkInput duplicates AgentProfileImprovementBenchmark's shape [ergonomics] ``
src/intelligence/profile-improvement-experiment.ts:35-40 declares a structurally identical copy of AgentProfileImprovementBenchmark (src/intelligence/improvement-cycle.ts:275-280). The copy exists to avoid an import cycle (improvement-cycle imports from profile-improvement-experiment), so it is defensible, but a future field added to one and not the other would silently diverge the two entry points' benchmark contracts. Consider moving the canonical type into profile-improvement-experiment.ts an
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
tangletools
left a comment
There was a problem hiding this comment.
🟢 Value Audit — sound
| Verdict | sound |
| Coverage | 1 of 2 lenses (usefulness) |
| Concerns | 0 (none) |
| Heuristic | 0.0s |
| Duplication | 0.1s |
| Interrogation | 412.7s (2 bridge agents) |
| Total | 412.8s |
⚠️ Partial audit — the verdict covers only usefulness. value: cli-bridge admission rejected (queue saturated). Treat the missing lens as unexamined, not as clear.
💰 Value — error
value agent never ran: the CLI bridge refused admission (no model was started).
- Model: opencode/deepseek/deepseek-v4-pro
- Bridge attempts: 5
- Bridge error: opencode/kimi-for-coding/k2p7: opencode: opencode error; opencode/zai-coding-plan/glm-5.2: Bridge returned 503: bridge at capacity (queue_timeout, lane=reserved): active=20/20 queued=8/48 — no model was started
🎯 Usefulness — sound
A truthful non-optimizer entry into the exact canonical seal→measure→compare→propose chain, built by extracting shared helpers rather than forking, with a documented external caller waiting on it (issue #751) and full downstream review/activation compatibility.
- Integration: Exported at src/intelligence/index.ts:88 and documented in docs/api/intelligence.md:4373 and docs/api/primitive-catalog.md:439. No in-rejo caller yet, but reachability is concrete and imminent: (a) issue #751 (closed, this PR implements it) documents the exact consumer — the discovery lab's evolution round, which authors candidates via a trace-reading agent and grades with 6-hour oracle runs that
- Fit with existing patterns: Fits the codebase grain exactly. It does not compete with proposeAgentProfileImprovement (src/intelligence/improvement-cycle.ts:669) — it is its non-optimizer complement, refusing optimizer lineage (authored-profile-improvement.ts:113-115) and reusing the identical sealed experiment, paired runner, comparison verifier, and proposal factory. The shared helpers (state digest, benchmark sealing, cost
- Real-world viability: Built for more than the happy path: fail-closed refusals (optimizer lineage, caller-injected profileDiffIds via
?: nevertype + runtime Object.hasOwn check at line 116, forged source metadata, forged optimizer receipts through profileImprovementMetadata→assertNoCallerOptimizationReceipt at profile-improvement-experiment.ts:123, NaN/negative budget, policy-budget mismatch) all execute before the - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
No concerns from the lens that ran (usefulness). The missing lens examined nothing, so this is not a full clean bill of health.
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
Why
proposeAgentProfileImprovement()correctly owns optimizer-backed profile search, but it means a complete profile authored by a human, imported from another system, or composed from several parents cannot enter the one-shot proposal path without fabricatingsource: 'optimizer'lineage.Runtime already had all lower-level pieces for direct measurement; consumers had to assemble sealing, exact paired execution, comparison, provenance, and proposal creation manually. This PR adds the truthful high-level path requested in #751.
What this PR implements
proposeAuthoredAgentProfileImprovement()A complete candidate profile can now enter the same canonical chain as generated candidates:
The function never invokes
improve()and never creates an optimization receipt.Truthful candidate lineage
The direct path accepts only current Interface 1.0 lineage sources:
humanimportcompoundIt refuses
optimizer, derivesprofileDiffIdsitself from the exact sealed change, and lets the canonical Interface schemas enforce parent/run/development-split requirements for compound candidates.Diff provenance without caller-controlled identities
Callers may attach source/artifact/metadata provenance to Runtime-derived profile diff steps, but cannot inject the resulting
profileDiffIds. This preserves authorship evidence without allowing a proposal to claim a different change than the profile Runtime actually measured.Held-out freshness and state integrity
Before any candidate measurement, Runtime refuses:
Developer surface
The additive Intelligence exports include:
proposeAuthoredAgentProfileImprovement()ProposeAuthoredAgentProfileImprovementOptionsProposeAuthoredAgentProfileImprovementResultAuthoredAgentProfileCandidateLineageAuthoredAgentProfileDiffOptionsAgentProfileCandidateMeasurementExecutorCompatibility
proposeAgentProfileImprovement()remains the optimizer-backed path.0.137.1.Tests
The new suite proves:
Validation before opening
A branch-local validation pass completed successfully before the PR was created:
The repository's complete native CI matrix is now running against the final branch.
Closes #751