Skip to content

File store rewrites the whole map per item, so large shards are quadratic #234

Description

@mcclowes

Every store on a file() store serialises the entire map and fsyncs it. Cost per item therefore scales with items already stored, and the DSL gives no way out.

Measured on this machine, ~1.3KB records:

items total per item
250 1.29s 5.18ms
500 2.69s 5.39ms
1000 5.98s 5.98ms
2000 13.55s 6.78ms

A real 2000-manager FPL run produced a 2.6MB file, so the last writes each rewrite 2.6MB — roughly 2.6GB written and 2000 fsyncs across the run. That was invisible at 120rpm, where the rate limit dominates, but it does not stay invisible: the file grows linearly and the per-item cost with it.

FileStore already supports persist: 'batch' | 'debounce', and createStore accepts fileOptions — but store-manager.ts never passes it, so a store declared in a .vague file is always immediate. There is no syntax to request otherwise.

This matters for the sharded bulk-fetch story specifically. #226 describes "a 200k-item shard against an origin enforcing a per-egress budget"; against a file store that shard would rewrite a ~260MB file 200k times. PostgREST avoids it, which is presumably how it was validated, but the shipped fpl-sharded example now defaults to a file store precisely so it runs without infrastructure.

Options, roughly in order of appeal:

  1. Expose the persist mode in the DSL, e.g. store managers: file("fpl_manager_history") { persist: batch }, with flush() already implemented for the exit path.
  2. Default to debounce for file stores, since immediate buys durability that a local JSON file does not really offer anyway.
  3. Append-only or per-key files for large stores.

Not urgent — nothing is incorrect, and the concurrency write-loss bug (#227) is separately fixed. This is purely about the ceiling.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions