Epic: double down on durable execution (bet 3)

[mcclowes]

Epic: double down on durable execution (bet 3)

The wedge, in one line: durable execution without the determinism tax.

Temporal, Restate, and DBOS make you write durable workflows in real code, so you must obey determinism rules (no Date.now() in a workflow, careful workflow/activity split, or replay corrupts). That's the tax everyone pays for general-purpose durability. Reqon doesn't have to charge it: every side effect in a .vague mission goes through a known step type (fetch, store, map, pause, wait), so the runtime owns the entire I/O boundary and can guarantee deterministic replay, automatic checkpointing, and idempotent effects by construction. We trade generality for a domain where durability is free and safe.

This epic is the plan to make that real and provable.

Step zero — the critical bugs are the product. Every "critical" in the code review is a line item in the durable-execution spec: non-atomic writes that lose data on crash, checkpoints that advance before the store completes, retries that duplicate POSTs, in-memory timers, pauses that resume twice. Fixing those isn't pre-work for the vision, it is the vision. A durability product dies on a single data-loss report.

The architecture move that ties it together

Model a running mission as an append-only event log. Resume = replay and fold to last state. Time-travel, audit trail, and idempotent resume all fall out of the same structure, and it collapses four fragile half-built subsystems (trace/, sync/, execution/, pause/) into one. That's #137.

Phases

Phase 1 — Foundation (make durability real)

• Event-source the execution: one append-only log; replay-based resume #137 Event-source the execution (one append-only log; replay-based resume)
• Transactional state backend (Postgres + SQLite) for the execution log #138 Transactional state backend (Postgres + SQLite)
• Exactly-once effects: idempotency keys for fetch + store dedup on stable keys #139 Exactly-once effects (idempotency keys + store dedup)

Foundation bug fixes that land alongside:

• File stores lose all data on a crash mid-write (no atomic write; corruption read as empty) #115 atomic file writes · Trace snapshots are written non-atomically and can replay a wrong/partial run #116 trace atomicity · Incremental sync checkpoint advances before data is stored, silently dropping records #119 checkpoint ordering
• HTTP requests have no timeout and can hang forever #120 HTTP timeout · HTTP client retries non-idempotent requests (POST/PATCH/DELETE), causing duplicate writes #121 idempotent retries · Store dedup breaks: missing key -> random key; undefined key -> collision; shallow merge #129 stable store keys
• Paused executions can resume twice (timeout poller vs webhook race, no atomic status) #130 idempotent pause resume · Long pauses overflow setTimeout (>24.8d fires instantly); timers lost on restart #131 durable timers (no overflow, survive restart)
• Flow-control directives (skip/retry/jump/queue) crash the mission instead of running #114 flow-control signals · sql/nosql stores silently write JSON to local disk (DEVELOPMENT_MODE defaults true) #118 stop silent sql/nosql file fallback

Phase 2 — Prove it

• Crash-injection test suite: kill -9 mid-step, assert no loss / no duplicate on resume #140 Crash-injection test suite (kill -9 mid-step, assert no loss / no dup)
• Document and publish durability guarantees (delivery / effects / resume semantics) #141 Document and publish durability guarantees

Phase 3 — Differentiate on the domain

• Durable connectors: resumable pagination, incremental sync, and pausable backfill #142 Durable connectors (resumable pagination, incremental sync, pausable backfill)
• Resource-free long waits as the flagship (hardened pause/wait + restart-survival demo) #143 Resource-free long waits as the flagship (restart-survival demo)

What we are NOT doing

• Out-generalizing Temporal. Stay narrow: declarative, API-native, determinism-free.
• Adding more step types breadth-first. Go deep on guarantees, not wide on features.
• Claiming exactly-once against APIs that don't support idempotency keys. Be honest: at-least-once + store dedup is the floor.

Risks

• The lane is filling fast (DBOS, Inngest, Restate). Our only differentiated ground is the declarative + API-native + determinism-free combination — the DSL and the durability are one package.
• Durability is a trust product with a savage failure mode. The upside is real; it commits us to a quality bar most projects never meet.

Milestone: Durable execution. Source: code review CODE_REVIEW-2026-06-22.md.