Every store on a file() store serialises the entire map and fsyncs it. Cost per item therefore scales with items already stored, and the DSL gives no way out.
Measured on this machine, ~1.3KB records:
| items |
total |
per item |
| 250 |
1.29s |
5.18ms |
| 500 |
2.69s |
5.39ms |
| 1000 |
5.98s |
5.98ms |
| 2000 |
13.55s |
6.78ms |
A real 2000-manager FPL run produced a 2.6MB file, so the last writes each rewrite 2.6MB — roughly 2.6GB written and 2000 fsyncs across the run. That was invisible at 120rpm, where the rate limit dominates, but it does not stay invisible: the file grows linearly and the per-item cost with it.
FileStore already supports persist: 'batch' | 'debounce', and createStore accepts fileOptions — but store-manager.ts never passes it, so a store declared in a .vague file is always immediate. There is no syntax to request otherwise.
This matters for the sharded bulk-fetch story specifically. #226 describes "a 200k-item shard against an origin enforcing a per-egress budget"; against a file store that shard would rewrite a ~260MB file 200k times. PostgREST avoids it, which is presumably how it was validated, but the shipped fpl-sharded example now defaults to a file store precisely so it runs without infrastructure.
Options, roughly in order of appeal:
- Expose the persist mode in the DSL, e.g.
store managers: file("fpl_manager_history") { persist: batch }, with flush() already implemented for the exit path.
- Default to
debounce for file stores, since immediate buys durability that a local JSON file does not really offer anyway.
- Append-only or per-key files for large stores.
Not urgent — nothing is incorrect, and the concurrency write-loss bug (#227) is separately fixed. This is purely about the ceiling.
Every
storeon afile()store serialises the entire map and fsyncs it. Cost per item therefore scales with items already stored, and the DSL gives no way out.Measured on this machine, ~1.3KB records:
A real 2000-manager FPL run produced a 2.6MB file, so the last writes each rewrite 2.6MB — roughly 2.6GB written and 2000 fsyncs across the run. That was invisible at 120rpm, where the rate limit dominates, but it does not stay invisible: the file grows linearly and the per-item cost with it.
FileStorealready supportspersist: 'batch' | 'debounce', andcreateStoreacceptsfileOptions— butstore-manager.tsnever passes it, so a store declared in a.vaguefile is alwaysimmediate. There is no syntax to request otherwise.This matters for the sharded bulk-fetch story specifically. #226 describes "a 200k-item shard against an origin enforcing a per-egress budget"; against a file store that shard would rewrite a ~260MB file 200k times. PostgREST avoids it, which is presumably how it was validated, but the shipped
fpl-shardedexample now defaults to a file store precisely so it runs without infrastructure.Options, roughly in order of appeal:
store managers: file("fpl_manager_history") { persist: batch }, withflush()already implemented for the exit path.debouncefor file stores, sinceimmediatebuys durability that a local JSON file does not really offer anyway.Not urgent — nothing is incorrect, and the concurrency write-loss bug (#227) is separately fixed. This is purely about the ceiling.