Skip to content

CL-6439: prove and lock the template-block deploy freeze #180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
TheGreatAxios merged 2 commits into from
Aug 21, 2026
Merged

CL-6439: prove and lock the template-block deploy freeze #180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c43e3cc
Add tests for template-block source freezability
TheGreatAxios Aug 21, 2026
652cca5
Update docs: template-block deploy freeze closes the webhook-launch gap
TheGreatAxios Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 1 addition & 16 deletions bun.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

51 changes: 25 additions & 26 deletions packages/evals/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,39 +41,38 @@ connect card's start-reviewing step after install — so the per-repo
grant and `webhook_trigger` row mint for real and the fire-webhook step
fires an actual trigger.

| # | Scorer | Result on a scratch-hub run | Why |
| --- | ------------------------------------------ | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | `githubConnectedViaConnectionsLayer` | **PASS** | Both halves now: the MCP fake connects through the real `POST /mcp-servers` route, and the Plugins PAT proves against the fake REST origin through the real `/:connectorId/complete` (CL-6403's `probeBaseUrls`). |
| 2 | `agentDefinitionsHaveToolGrants` | **PASS** | The three reviewer definitions materialize via the real install, and the install now also deploys the `code-review` block workflow carrying the `@corbits/github-tools` pin (CL-6405's product fix). The snapshot's `name` is the stable definition handle (`displayName` carries the label), so handle matching is exact. |
| 3 | `triggerIsWebhookPerPr` | **PASS** | Install drives start-reviewing against the fake REST origin's repo list; one enabled `webhook_trigger` row mints per repo, bound to the deployed `code-review` definition. |
| 4 | `reviewCommentsAttributable` | **SKIP** (product gap) | `WorldSnapshot` has no `reviewComments` field — blocked on CL-6322 Phase 1 (`onTrigger` adoption giving each fired occurrence its own child run id). |
| 5 | `suggestedFixesStructurallyValid` | **FAIL** (two gaps below) | The delivery now reaches a real enabled trigger, but the launch 500s: `DefinitionProjectionMissingError` — see gap 1. Even once launched, a posted review needs genuine model tool calls, i.e. a live `EVAL_PROVIDER_API_KEY` run — plumbing mode's stub credential can never call `github_post_pr_review`. |
| 6 | `outwardGitHubActionsRespectGrantBoundary` | **FAIL** (two gaps below) | Same two blockers as #5. |
| 7 | `wholeRunInspectable` | **SKIP** (product gap) | `WorldSnapshot` has no `runs` field. Blocked on CL-6322 Phase 1. |
| # | Scorer | Result on a scratch-hub run | Why |
| --- | ------------------------------------------ | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 1 | `githubConnectedViaConnectionsLayer` | **PASS** | Both halves now: the MCP fake connects through the real `POST /mcp-servers` route, and the Plugins PAT proves against the fake REST origin through the real `/:connectorId/complete` (CL-6403's `probeBaseUrls`). |
| 2 | `agentDefinitionsHaveToolGrants` | **PASS** | The three reviewer definitions materialize via the real install, and the install now also deploys the `code-review` block workflow carrying the `@corbits/github-tools` pin (CL-6405's product fix). The snapshot's `name` is the stable definition handle (`displayName` carries the label), so handle matching is exact. |
| 3 | `triggerIsWebhookPerPr` | **PASS** | Install drives start-reviewing against the fake REST origin's repo list; one enabled `webhook_trigger` row mints per repo, bound to the deployed `code-review` definition. |
| 4 | `reviewCommentsAttributable` | **SKIP** (product gap) | `WorldSnapshot` has no `reviewComments` field — blocked on CL-6322 Phase 1 (`onTrigger` adoption giving each fired occurrence its own child run id). |
| 5 | `suggestedFixesStructurallyValid` | **FAIL** (gap 1 below) | The launch itself now succeeds: the template-block deploy freezes its definition through `@corbits/workflow-freeze` (CL-6439), so the fired trigger answers 202 with a real run instance instead of `DefinitionProjectionMissingError`. What remains is that a posted review needs genuine model tool calls, i.e. a live `EVAL_PROVIDER_API_KEY` run — plumbing mode's stub credential can never call `github_post_pr_review`. |
| 6 | `outwardGitHubActionsRespectGrantBoundary` | **FAIL** (gap 1 below) | Same blocker as #5. |
| 7 | `wholeRunInspectable` | **SKIP** (product gap) | `WorldSnapshot` has no `runs` field. Blocked on CL-6322 Phase 1. |

### Remaining gaps, precisely

1. **The block-workflow deploy records no frozen wire projection.** A
webhook-fired launch (`launchWebhookTrigger` ->
`readDefinitionProjection`) reads the definition's frozen inert
projection off its version row — recorded only by the sidecar
probe/approve gate (`installAndApproveWorkflowDefinition`, the path
behind `POST /workflows/deployments`). The template-block deploy
(like the agent-directory create it mirrors) materializes the asset
and `workflow_definition` row but never probes, so firing the
trigger answers `DefinitionProjectionMissingError` ("No stored
launch body for definition \"code-review\""). True in production,
not just here: the block deploy needs to ride the platform's real
probe-and-freeze deployment path (or the webhook launch needs the
same lazy-freeze a first wake performs). Next ticket, never a hack.
2. **Scorers #5/#6 need a live run.** They grade genuine
1. **Scorers #5/#6 need a live run.** They grade genuine
`github_post_pr_review` tool calls off the trace; a plumbing-mode
stub credential produces a credential-error turn with no tool calls
by design. Re-run with `EVAL_PROVIDER_API_KEY` once gap 1 falls.
3. **CL-6322 Phase 1 (`onTrigger` adoption)** — unblocks #4/#7
by design. Re-run with `EVAL_PROVIDER_API_KEY`.
2. **CL-6322 Phase 1 (`onTrigger` adoption)** — unblocks #4/#7
(per-comment and per-run ids in the snapshot). Unchanged.

Also closed in this pass: `workflows/code-review` pinned
Closed in the CL-6439 pass: the block-workflow deploy used to record no
frozen wire projection, so a webhook-fired launch
(`launchWebhookTrigger` -> `readDefinitionProjection`) answered
`DefinitionProjectionMissingError` ("No stored launch body for
definition \"code-review\""). The hub's `deployWorkflowSource` binding
now freezes the serialized definition through
`@corbits/workflow-freeze` (the hub-local counterpart of the sidecar
probe gate, shared with the Agents page create path since CL-6447), and
a plumbing-mode run confirms the fired trigger answers 202 with a real
run instance. `packages/workflow-catalog/test/block-workflow-freeze.test.ts`
locks the block source's freezability.

Also closed in the CL-6405 pass: `workflows/code-review` pinned
`@corbits/github-tools@0.0.3` while CL-6403 released 0.0.4 (the
baseUrl seam), so closure resolution would have failed at wake; the pin
now names 0.0.4.
Expand Down
Loading
Loading