Skip to content

Durable-execution guarantees overstated (non-determinism, replay, fsync) #199

Description

@mcclowes

Problem

Several durable-execution guarantees are asserted in docstrings but not delivered by the code. Grouping them because they're all the same class of gap in the execution-log layer.

H4 — non-determinism breaks exactly-once

now() (src/interpreter/evaluator.ts:252) and anyOf (:269, uses Math.random()) are non-deterministic. Effect identity for fetches hashes the request body, and store dedup keys on a positional step index. If a fetch body or a value upstream of a store uses now()/anyOf, the hash or the step count differs between the original run and the replay, so the idempotency key changes and the server sees a brand-new write. fetch post "/events" body {ts: now()}, retried after a blip, double-posts.

Fix options: reject non-deterministic calls in durable contexts, or fold their resolved values into the log so replay reuses them.

H5 — "deterministic replay" is not implemented

src/execution-log/events.ts claims recorded outputs are "read back on replay rather than re-derived, so replay is deterministic." No event carries an output payload — EffectAppliedEvent records only an effectId, and there's no fetch-response event. On resume the executor re-runs every step against a live, mutating API; prior.completedSteps is loaded but never consulted to skip anything. This is at-least-once with idempotent writes, not deterministic replay. Either implement output recording/replay or correct the claim.

H6 — the default file log isn't fsync'd or concurrency-safe

FileExecutionLog.append (src/execution-log/store.ts:140-166) uses appendFile with no fsync (survives a process crash via page cache, not OS/power loss) and assigns sequence numbers from a per-instance cache with no file locking, so two processes appending to the same execution can derive the same seq and interleave writes. This is the default backend the crash-injection proof runs against.

Why deferred

These are design-level changes to the log format and durability model, not drive-by patches. Worth deciding deliberately: either tighten the guarantees to match the docstrings, or soften the docstrings to match reality (at-least-once with idempotent writes is a perfectly respectable contract — it just needs to be stated honestly).

See CODE_REVIEW.md (H4, H5, H6).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions