Analysis memory grows superlinearly in window (.srw) count; 815 windows exhaust 44 GB

[jimmykirk]

Summary

The post-parse analysis phases use memory that grows sharply superlinearly in
the number of window objects (.srw). 815 windows exhaust 44 GB. Parsing
itself is fine — every file parses successfully first — and DataWindows are
not implicated despite being the bulk of the corpus and the largest files.

This is separate from the stack-overflow crash in #3; the runs below all have
an adequate stack.

Measurements

Splitting one ~5,500-file tree by file type and indexing each subset alone:

subset	files	outcome
.srd DataWindows	3695	clean
.sru user objects	654	clean
.srw windows	815	OOM-killed above 44 GB

.srd is the majority of that tree by both file count and total bytes, so
size alone does not predict this.

Growth against window count, -M40G -N4, peak RSS:

windows	time	peak RSS
100	8s	10 GB
200	15s	11 GB
400	203s	22 GB
600	287s	23 GB
815	—	OOM > 44 GB

Two things stand out beyond the ceiling itself: time rises 15s → 203s for a
2x input increase, and even 100 windows costs 10 GB, which seems a steep
baseline.

The full mixed tree behaves consistently with this — 5,473 files with an
adequate stack parsed 5,446 files and wrote a 2.0 GB database before being
OOM-killed at 707s.

Notes

• -M does not bound this. Resident memory routinely exceeds the GHC heap
  cap, presumably because pinned and foreign allocation (DuckDB buffers) sit
  outside it — so -M40G inside a 44 GB limit still gets OOM-killed.
• Lowering -N does not help either; peak tracks the -M ceiling rather than
  worker count.
• pbc --help exposes no memory controls, so there is nothing to tune from
  the outside.

Impact

This puts whole-repository indexing out of reach for large codebases on
ordinary hardware, since 815 windows already exhaust 44 GB. Partitioning
per-library would keep each run small, but it sacrifices exactly the
cross-object analysis that spans libraries.

I am not able to share the tree itself, but I can measure whatever would
help — happy to run further experiments, bisect which phase the growth
happens in, or produce a profile if you would find one useful.

Environment

• pbc built from haskell:9.10.2-bullseye, GHC 9.10.2, per the versions in
  the project's own CI workflow
• Linux x86_64, 64 GB RAM, 16 cores

[jimmykirk]

Tracked this down — root cause and a fix are in #5.

It is not the analysis phases broadly, and not really about window objects except that they generate the schema legs. cosliceClosure emits every shortest leg once per target reachable past it, which is a cross product that grows with graph density: 29,786,641 rows for ~106k (seed, target) pairs on a 300-object corpus. materializeDecompositionCoslice then sorts all of it with a window function and joins it five ways — before discarding all but one row per ordinal, which is what it wanted all along.

Picking that one row at emission time instead gives byte-identical output (md5 over every column of every row) with 48x fewer intermediate rows, and turns the 300-object case from OOM into a 49-second run.

The measurements in the original report still stand, but the framing in the title is wrong — it is a cross product in the schema coslice, not superlinear growth in .srw handling per se.