api: add durable idempotent batch receipts

[ajroetker]

Summary

• add opt-in exactly-once batch execution through Idempotency-Key while preserving the header-free fast path
• bind principal, table, and key to a stable transaction ID and atomically create a canonical receipt before execution
• force keyed work through retained 2PC and allow only coordinator-proven commit or abort decisions to become terminal receipts
• isolate receipts, leases, expiry entries, and recovery indexes in versioned durable namespaces
• fence all durable receipt ownership with atomic owner leases in both clustered and standalone deployments
• supervise leases independently from recovery, retry transient heartbeat failures within the confirmed deadline, and renew only active mutable sessions in bounded batches
• persist the complete terminal API envelope so every replay returns the original code, message, retryability, outcome, and transaction ID
• preserve old persisted receipts through additive decoding and a legacy replay fallback
• keep thrown storage/coordinator errors pending and retryable; a generic DecisionConflict is no longer treated as proof of abort
• return typed retryable idempotency_receipt_pending when initial receipt publication is ambiguous
• preflight insert, delete, and transform table existence and durably terminalize deterministic pre-execution failures
• pin unresolved execution, visibility, and coordinator handoff debt against cleanup
• expose the keyed route through regenerated Go, Python, and TypeScript SDK artifacts

Durable execution model

The stable boundary is receipt-first and execution-second. Receipt lookup, comparison, forwarding, adoption, and replay do not consume execution capacity. Only the bounded backend job may atomically mark a receipt started and enter retained 2PC.

Before execution starts, deterministic validation failures can safely become terminal not_applied. After execution starts, only an explicit typed coordinator outcome can prove commit or abort. Thrown errors remain unknown, preserve the pending receipt, and are reconciled by recovery without repeating execution.

Terminal state is one atomic durable value: broad outcome plus exact error code, message, and retryability. That prevents restart replay from degrading a specific error such as table_not_found or transaction_conflict into a generic response.

Ownership and restart recovery

Every durable store receives lease defaults. Owner incarnation and lease CAS fencing apply equally to cluster-shared and node-local stores, including standalone node zero. Foreground execution and recovery share one deadline-aware heartbeat: transient store failures are retried while the last confirmed lease remains valid, and only explicit fencing loss or deadline exhaustion marks ownership uncertain.

Global renewal has its own durable-job owner and in-flight state, so slow recovery cannot suppress a due renewal pass. Renewal scans an in-memory set maintained atomically with session publication, containing only locally owned mutable sessions and terminal sessions with live recovery debt. Immutable retained receipts therefore create no periodic lease-write amplification. Large active sets are advanced in bounded fair batches.

Configured renewal intervals are clamped to one third of the lease TTL. Foreground work uses a conservative deadline captured before its durable execution marker; if that write consumes the safety window, it must establish a fresh fenced lease before invoking externally visible transaction work.

After a crash and lease expiry, a successor may atomically adopt the pending receipt and continue recovery without requiring the client to replay the request. Scope still controls whether a local miss is authoritative; it no longer disables ownership fencing or takeover for an atomic local durable store.

Compatibility

Requests without Idempotency-Key retain the existing fast path and response shape. Persisted schema changes are additive. Older receipt records that contain only the broad terminal outcome remain readable and replay with the legacy generic fallback, while newly completed receipts retain the exact response envelope.

Pre-feature binaries do not scan the versioned receipt namespaces. New binaries continue to scan both interactive-session and receipt namespaces with fair recovery, cleanup, renewal, retention, and shared capacity accounting.

Long-term regressions

• process-level E2E persists a missing-table terminal response, performs a real server restart, and requires byte-for-byte semantic replay of status, code, message, retryability, and transaction ID
• standalone recovery test kills the original owner after execution starts, waits for lease expiry, and proves successor maintenance completes the receipt without client replay
• foreground heartbeat test runs execution beyond two lease periods, injects a transient lease-store failure, and proves ownership remains live
• independent-supervisor test blocks recovery maintenance while two separate lease renewals continue to advance
• active-set test proves immutable terminal receipts leave the renewal set and bounded batches cover the remaining active sessions
• ambiguity test throws bare DecisionConflict and proves the receipt stays pending
• abort publication failure test proves recovery later publishes and replays the exact terminal conflict
• receipt creation failure test proves the first response is typed and retryable and the same key can converge

Validation

• make generate
• make zig-generated-check
• make zig-build
• zig build api-http-runtime-test --summary all — 71 passed
• zig build api-transactions-docid-test --summary all — 17 passed
• zig build lib-api-session-maintenance-test --summary all — 41 passed
• restart E2E — passed
• Go SDK tests — passed
• Python SDK focused idempotent-batch tests — passed
• TypeScript SDK build — passed
• git diff --check