[P1] Reliability and observability proof for the onboarding vertical slice

[Aparnap2]

Context

Follow-up to #65. Once the ACME golden slice is green, prove it is reliable and observable under failure, not just on the happy path. Rationale: per hygiene ruling, reliability/observability concerns enter as scoped follow-up criteria rather than separate runtimes (see #48/#51 closures); this issue carries them.

Scope

• Replay equivalence: same event log replays to the same outcome.
• Idempotent duplicates/retries: duplicate events and retried executions do not double-execute (domain idempotency keys).
• Temporal restart survival: worker restart mid-mission resumes via replay with no lost or duplicated side effects.
• Graceful degradation: connector timeout/failure degrades cleanly with audit record (no silent skip).
• Malformed-output rejection: malformed LLM output rejected by contract validation with no partial write.
• Full trace chain on the golden slice: event → evidence → context → mission → reasoning → intent → authorization → execution → verification → outcome, each link recorded and queryable.

Monitoring

• OTel traces across the chain; Langfuse LLM-call visibility.
• Temporal state visibility; Redpanda consumer-lag monitoring.
• Metrics: success/failure rates, policy rejections, HITL latency, context size, tool calls per mission, evidence freshness.

Acceptance

• Fault-injection tests green against the test(e2e): prove onboarding-to-verified-ROI workflow #65 harness; trace chain queryable end to end; monitoring in place for every metric above.

Verification

• uv run pytest apps/ai/tests/ -k "reliability or observability or replay or idempotency or restart or degrade" -v (from apps/ai)

Follow-up

• Follow-up to test(e2e): prove onboarding-to-verified-ROI workflow #65; blocked until the test(e2e): prove onboarding-to-verified-ROI workflow #65 golden slice is green.

[Aparnap2]

Sprint C evidence (branch feature/ontologyai-onboarding-restructure, commits 46e0860 + addf5ef): 18/18 green in tests/adversarial/ (ruff clean; 171 passed with scope regression). C1 stale checkpoint → deny:stale_context, zero writes. C2 reject → zero execution, key unmarked, no silent lower-risk retry; timeout → deny:approval_expired, late resolve cannot resurrect. C3 crash-before → key unmarked, retry executes once; crash-after → verify+audit reconcile, no duplicate. C4 replay from audit+frozen intent reconstructs equivalent state (prose never compared). C5 cross-tenant evidence rejected at assembly; trusted tenant binding pinned; executor target-tenant gap recorded for Sprint F (no new check built). C6 version gate: mismatch → deny:stale_version, refresh resubmits. FailureClass skipped with rationale (deny strings + audit suffice). Known gap carried to Sprint F: executor target-resource tenant match.