Native agent_events snapshot writer for AgentForensics (EvalBench-adapter exit ramp)

[caohy1988]

Parent: #435. This is the next engineering slice after the Week 0 freeze evidence (PRs #455–#462). #455 is merged (the recordable MVP e2e demo); #456–#462 are open and agent-review-complete, and must not be merged as part of this work.

Why (team verdict, 2026-08-30)

Unanimous keep-as-adapter verdict (Claude / Codex / Kimi + Field Brief bot):

• EvalBench is a snapshot format we read (configs / results / scores by job_id) — not a runtime library (nothing EvalBench appears in pyproject.toml) and not the partner.
• The customer object is a session in agent_events, not an EvalBench scenario.

The consequence is an exit ramp, not a removal: BQAA should be able to produce its own snapshot contract directly from production ADK agent_events, with the EvalBench adapter remaining an optional on-ramp.

Goal

A native writer that starts from production ADK agent_events rows and emits the same BQAA-owned contract the evalbench-import adapter already produces:

• a pinned, immutable snapshot (versioned BQAA-owned tables + manifest, as EvalBenchRun.materialize does today),
• the failed_sessions view (the per-dataset view whose body is failed_sessions_sql pinned to the latest successful import),
• G1 taxonomy labels — for the widget-stock silence session 7e352c34 that means task/planning, finalization, tool blockers,

with no EvalBench source tables anywhere in that path. The existing evalbench-import adapter (#97) stays as an optional path — do not delete it.

Contracts the writer must honor (all frozen in the Week 0 evidence)

• Identity. EvalBench results.id / eval_id is the first eight characters of the ADK session_id (7e352c34 from 7e352c34-4c1c-4395-acd5-fb3c8f215346), falling back to the full id on collision (see the identity table in examples/evalbench_mvp_e2e.md). The native writer must preserve a joinable identity so the same session is the same object with or without the adapter.
• Pin identity (native snapshot key). Retain (job_id, import_version) — the same tuple EvalBenchRun.materialize already uses. Do not invent a different native key. Snapshot event rows, BQAA-owned score rows, and the manifest row all carry both fields. The failed_sessions view body is failed_sessions_sql with @job_id / @import_version rendered as literals for the latest successful native publication. Latest-success rule: the newest committed manifest row for that job_id (ORDER BY imported_at DESC, import_version DESC LIMIT 1). Manifest generation_id plus view_policy pin the view body the same way the adapter already does. For this slice the job is mvp-e2e-real-traces.
• Read vs write. Read production ADK agent_events only (test-project-0728-467323.bqaa_e2e_real.agent_events for the widget-stock acceptance). Write BQAA-owned snapshot events table + BQAA-owned scores table + manifest + the failed_sessions view. Never write into production agent_events. Never read EvalBench source configs / results / scores tables on this path.
• Denominator. failed_sessions is the denominator. A live/LLM judge is not the denominator. returncode == 0 means completed, not passed — only the score policy decides passed.
• Native score source and score policy. Scores live in the BQAA-owned snapshot scores table (same schema the adapter already writes: job_id, import_version, scenario_id, session_id, comparator, score, source_row), not EvalBench source scores tables. Native score rows are derived deterministically from the session (fixture-testable; no live BQ and no live/LLM judge). The gate is existing EvalScorePolicy: min_scores (comparator → threshold) plus missing_score_fails (default true), persisted as canonical JSON on the manifest view_policy column (_policy_pin). score_failed is true iff any listed comparator fails: missing/NULL score when missing_score_fails is true, or score < min_score. Empty policy ⇒ score_failed is FALSE (process_failed / missing_completion still apply). For this slice's widget-stock session the frozen rubric (examples/fixtures/week0_real_rubric.json, docs/week0_partner.md) is comparator goal_completion with min_score 1.0 (failed_session_score 0 vs gold sibling ab7535a5 score 1) and missing_score_fails: true, so 7e352c34 is score_failed and maps to task/planning.
• G1 mapping is already frozen in src/bigquery_agent_analytics/failure_taxonomy.py (taxonomy_version: 0.1.0, g1_frozen: True; see docs/week0_g1_taxonomy.md): missing_completion → finalization, process_failed → tool blockers, score_failed → task/planning. Freezing G1 did not start the clock.
• Destination. BQAA-owned snapshot tables + the failed_sessions view only. Never write into production agent_events.
• D4 fail-closed. The D4 consumer is Hai-Yuan Cao only — do not add D4 people as part of this slice.
• Clock. This issue does not start the six-week clock. The clock starts only when the first Week 1 snapshot job is kicked, which this issue does not do.

Acceptance

• Widget-stock session 7e352c34-4c1c-4395-acd5-fb3c8f215346 from test-project-0728-467323.bqaa_e2e_real.agent_events can be snapshotted without any EvalBench tables and still yields failed_sessions + the G1 names (task/planning, finalization, tool blockers) + the punchline next-action (the agent never answered / never called check_inventory).
• The snapshot is keyed by (job_id, import_version) with job_id=mvp-e2e-real-traces; the failed_sessions view is pinned to the latest successful native publication of that tuple; score rows and view_policy use comparator goal_completion / min_score 1.0 / missing_score_fails: true.
• Offline fixture tests — unit tests require no live BigQuery.
• Nothing is written into production agent_events.
• The six-week clock is not started.
• D4 fail-closed remains Hai-Yuan Cao only.

Out of scope

• Merging docs(435): record merged slices 451–455 and snapshot changelog #456–feat(evalbench): Week 0 freeze e2e team demo on the widget-stock session (#435) #462.
• The preregistration seal.
• Clock start / kicking the Week 1 snapshot job.
• Dropping the EvalBench adapter (Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 stays optional).
• SANA fork.
• A live judge as the denominator.

Related

• Parent: RFC: AgentForensics collaboration — automated failure taxonomy and span-level trace auditing over BQAA traces #435
• EvalBench adapter / import (stays optional): Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97
• Week 0 evidence: docs(evalbench): recordable MVP e2e demo of import → failed-sessions → score (#435, #97) #455 (merged MVP e2e demo), feat(evalbench): freeze Week 0 partner, D4, and G1 taxonomy v0.1 (#435) #461 (Week 0 freeze), feat(evalbench): Week 0 freeze e2e team demo on the widget-stock session (#435) #462 (post-freeze e2e)
• Demo (EvalBench adapter path, MVP e2e): https://caohy1988.github.io/evalbench/

[caohy1988]

Reviewer: Kimi
Review-round: 1

Verdict: ISSUE_OK

• P0: 0
• P1: 0
• P2: 1

Customer-object check: PASS

The issue body is unambiguous about what the customer object is. The "Why" section states it verbatim: "The customer object is a session in agent_events, not an EvalBench scenario." The goal starts from "production ADK agent_events rows," and the acceptance criterion is anchored on the widget-stock session 7e352c34-4c1c-4395-acd5-fb3c8f215346 from test-project-0728-467323.bqaa_e2e_real.agent_events. An implementer cannot read this issue and come away building against an EvalBench scenario as the object.

EvalBench-in-customer-sentences check: PASS

Checked the three sentences a PM/demo reader actually reads:

• Title: "Native agent_events snapshot writer for AgentForensics (EvalBench-adapter exit ramp)" — leads with the agent_events snapshot writer for AgentForensics. EvalBench appears only in a parenthetical engineering qualifier ("exit ramp"), not as the subject. A reader parses this as "AgentForensics gets a native writer," not "run EvalBench."
• Goal one-liner: "A native writer that starts from production ADK agent_events rows…" — leads with the session source. The evalbench-import adapter is referenced only as the contract the writer must match ("the same BQAA-owned contract the adapter already produces"), which is engineering-contract territory and explicitly allowed.
• Acceptance punchline: "Widget-stock session 7e352c34… can be snapshotted without any EvalBench tables and still yields failed_sessions + the G1 names + the punchline next-action (the agent never answered / never called check_inventory)" — leads with the session, the never-answered story, and the G1 next-action. EvalBench appears only as a negative constraint, which is the strongest possible framing that it is not the product.

Everywhere else EvalBench appears (identity first-8 join, configs/results/scores by job_id, adapter stays optional per #97) is clearly marked as engineering contract or exit-ramp rationale, not customer surface. The body also pre-empts the partner misreading directly: EvalBench is "a snapshot format we read … not a runtime library … and not the partner."

Guardrails check: PASS

• Adapter preserved: "stays as an optional path — do not delete it," repeated in Out of scope. No removal language anywhere.
• Clock: explicitly not started by this issue; starts only when the Week 1 snapshot job is kicked, which is out of scope here.
• D4: fail-closed, Hai-Yuan Cao only, in both Contracts and Acceptance.
• No merging of docs(435): record merged slices 451–455 and snapshot changelog #456–feat(evalbench): Week 0 freeze e2e team demo on the widget-stock session (#435) #462; production agent_events is never written to.

Findings

• P2 (informational): The Related section's bare "Demo: https://caohy1988.github.io/evalbench/" link has evalbench in the URL with no qualifier. A casual reader skimming only the tail could file this mentally as "the demo is EvalBench." Consider labeling it as the adapter-path MVP demo (e.g., "Demo (EvalBench adapter path, MVP e2e): …") so the label, not the URL, carries the meaning. Non-blocking; the body's framing already does the work.

What can wait

• The demo-link label tweak above — cosmetic, do it whenever the issue is next touched.
• No changes required to title, goal, or acceptance punchlines. Issue is ready for an implementer as written.

[caohy1988]

Reviewer: Codex
Review-round: 1

Verdict: NEEDS_EDIT

P0 count: 0
P1 count: 2
P2 count: 0

The issue text closes the named P0 escape hatches: it forbids writes to production agent_events, starting the six-week clock or kicking Week 1, deleting the EvalBench adapter, using a live/LLM judge as the denominator, merging #456–#462, breaking the first-8/full-on-collision eval_id rule, expanding D4 beyond Hai-Yuan Cao, and claiming a SANA fork.

P1 findings against the issue text

1. The native snapshot pin identity is not defined.

   Hole: “a pinned, immutable snapshot (versioned BQAA-owned tables + manifest, as EvalBenchRun.materialize does today)” and a view “pinned to the latest successful import.”

   “Versioned” plus “manifest” does not tell an implementer what stable native key identifies one snapshot, what all destination rows carry, or how the view selects the latest successful native publication. Add an explicit pin contract: retain (job_id, import_version) or name an equivalent native tuple, require the snapshot tables, manifest, and failed_sessions view to use it consistently, and define the latest-success rule for native runs. Without this, two valid implementations can produce incompatible manifests and view pins.

2. The native score source and score policy are not defined once EvalBench scores tables are removed.

   Hole: the path must have “no EvalBench source tables anywhere,” while “returncode == 0 means completed, not passed — only the score policy decides passed.”

   The text correctly rules out both completion-as-passage and a live/LLM-judge denominator, but it never names the deterministic native score inputs, comparator/threshold policy, missing-score behavior, or persisted policy identity. Add the contract that derives score_failed and failed_sessions from version-pinned BQAA-owned score data/policy without EvalBench source tables. Keep returncode solely as the completion signal. Otherwise an implementer must invent denominator semantics and cannot deterministically produce the frozen task/planning mapping.

What can wait

The CLI/API entry point, whether materialization is shared internally with the adapter, an explicit native-vs-adapter parity acceptance check, and the operational evidence format for the negative acceptance assertions can be planned later. They are useful follow-ups, not blockers under this round’s P0/P1 rubric.

[caohy1988]

Reviewer: Kimi
Review-round: 2

Verdict: ISSUE_OK

• P0: 0
• P1: 0
• P2: 0

Re-review of the edited body after the Codex round-1 P1 fixes. Focus: customer object, EvalBench-in-customer-sentences, and whether the new pin/score contract language regresses the product bar.

Customer-object check: PASS (unchanged)

The edited body keeps the load-bearing sentence verbatim: "The customer object is a session in agent_events, not an EvalBench scenario." The Goal still starts from "production ADK agent_events rows," and the acceptance anchor is still the widget-stock session 7e352c34-4c1c-4395-acd5-fb3c8f215346 from test-project-0728-467323.bqaa_e2e_real.agent_events. The new contract sections never redefine the object — the native score rows are "derived deterministically from the session," which reinforces, rather than dilutes, the session-as-object framing.

EvalBench-in-customer-sentences check: PASS

• Title: unchanged — leads with "Native agent_events snapshot writer for AgentForensics," EvalBench confined to the parenthetical "exit ramp" qualifier.
• "Why" section: now even more explicit than round 1 — EvalBench is "a snapshot format we read … not a runtime library (nothing EvalBench appears in pyproject.toml) and not the partner." A PM cannot read this and conclude EvalBench is the product; the sentence exists precisely to foreclose that reading.
• Goal / acceptance punchlines: unchanged from round 1 — lead with the session, failed_sessions, the G1 names, and the never-answered story; EvalBench appears only as a negative constraint ("without any EvalBench tables").

New pin/score contract language: PASS (no product-bar regression)

The two sections added for the Codex P1s were checked specifically for "does this make a PM think the product is EvalBench":

• Pin identity. "Retain (job_id, import_version) — the same tuple EvalBenchRun.materialize already uses. Do not invent a different native key." EvalBenchRun.materialize here is a code-symbol reference to the existing implementation, i.e. engineering-contract anchoring ("match what we already ship"), not a product claim. The surrounding nouns are all BQAA-owned: "BQAA-owned snapshot events table + BQAA-owned scores table + manifest." The latest-success rule (ORDER BY imported_at DESC, import_version DESC LIMIT 1) is pure BQAA mechanics with no EvalBench dependency.
• Native score source and score policy. Explicitly states scores live in the "BQAA-owned snapshot scores table … not EvalBench source scores tables," and that the gate is "existing EvalScorePolicy" — again a code reference to an already-frozen BQAA component, not a statement that scoring belongs to EvalBench. The widget-stock rubric (goal_completion / min_score 1.0 / missing_score_fails: true) is pinned to BQAA fixtures (examples/fixtures/week0_real_rubric.json) and the frozen G1 mapping in src/bigquery_agent_analytics/failure_taxonomy.py.

Both additions define the contract in terms of "the same X the adapter already does/writes," which is the correct way to pin behavior to existing implementation without elevating the adapter to product status. No regression.

Round-1 P2 resolution: CONFIRMED

The demo link is now labeled "Demo (EvalBench adapter path, MVP e2e): https://caohy1988.github.io/evalbench/" — exactly the fix suggested in round 1. The label, not the URL, now carries the meaning. Resolved.

Guardrails check: PASS (unchanged)

• Adapter preserved: "stays as an optional path — do not delete it," repeated in Out of scope.
• Clock: explicitly not started; Week 1 job kick is out of scope.
• D4: fail-closed, Hai-Yuan Cao only, in both Contracts and Acceptance.
• Read/write separation: read-only on production agent_events, writes confined to BQAA-owned tables, "Never write into production agent_events" stated twice.
• docs(435): record merged slices 451–455 and snapshot changelog #456–feat(evalbench): Week 0 freeze e2e team demo on the widget-stock session (#435) #462 not to be merged.

What can wait

Nothing new. The issue is ready for an implementer as edited.

[caohy1988]

Reviewer: Codex

Review-round: 2

Verdict: ISSUE_OK

P0: 0
P1: 0

Round-1 P1 closure check against the current issue text:

• P1-1 closed — native pin identity. The issue explicitly retains (job_id, import_version) across snapshot event rows, BQAA-owned score rows, and the manifest; pins failed_sessions_sql to literal values from the latest successful native publication; and defines latest success as the newest committed manifest row for the job, ordered by imported_at DESC, import_version DESC.
• P1-2 closed — native score source and policy. The issue explicitly sources scores from the BQAA-owned snapshot scores table, forbids EvalBench source score reads, pins canonical EvalScorePolicy JSON in manifest view_policy, freezes widget-stock to goal_completion >= 1.0 with missing_score_fails: true, and states that returncode == 0 means completion only, not pass.

No new P0/P1 holes remain in the current text. The production-write, clock-start, adapter-removal, live-judge-denominator, PR #456–#462 merge, first-eight identity, D4 expansion, and SANA-fork escape hatches are all explicitly closed.

[caohy1988]

PR is up: #464 — native agent_events snapshot writer (NativeAgentEventsRun + thin evalbench-native-import CLI), stacked on the #461 freeze HEAD 2c4804b (does NOT include #456/#457/#462; do not merge as part of review). Adapter #97 stays; clock NOT started; fixture-only tests (19 new, 331 green across the evalbench suites); widget-stock 7e352c34 yields task/planning, finalization, tool blockers.