bench(plugin): a reproducible harness for the render queue's storage costs

[harper-joseph]

Why

Every scheduling decision in this package is justified by a number, and the numbers the queue design rests on cannot be reproduced — #80 cites 20-lanesim.mjs and 21-duerank.mjs, neither of which is in this repo. Worse, two figures that are in the tree disagree by 80×, and both describe a one-sided ascending range read over the same index on the same table:

source	claim	implied
util/renderSchedule.js	20 keys in 0.43 ms	~21 µs/row
util/backlogSnapshot.js	"~3.5s per 2,000 rows"	~1.75 ms/row

Which one is right decides the architecture, so I stopped designing and measured.

What it found (Harper 5.2.4, 200,000 rows, realistic key shape)

Reads are cheap and flat, and yielding is free. 7.5 µs/row at limit 20, then 2.3–2.4 µs/row from 200 to 20,000. Yielding every 200 rows: 2.375 vs 2.387 µs/row — no cost. So renderSchedule.js is right, and backlogSnapshot.js's calibration comment is off by ~760× for the read it describes (4.6 ms measured for exactly 2,000 rows). Its MAX_ON_DEMAND_CAP is documented as "~3 minutes of yielding walk"; it is ~0.24 s. Extrapolated, a full 1.6M-row sweep is ~3.8 s.

Writes are the expensive thing — 32× reads.

	µs/row
put, one indexed attribute	76.4
put, queue key split into its own column	79.1
put, two indexed attributes	86.3
patch of a single attribute	105.0

Two claims in #80 do not survive this:

• A second index costs ~13%, not ~100%. "The existing secondary index is 39–48% of reschedule wall clock, so a per-lane index or table roughly doubles the hot write" is overstated ~4×, and it is the main argument against both a per-lane table and giving the queue its own column (the latter measures +3.5%, i.e. free).
• patch is 37% worse than put, so "a lane change is an in-place numeric update" is not an advantage.

Per-lane seeks: 0.244 ms for three vs 0.133 ms for one covering the same rows — cheap either way, consistent with #80's 0.29–0.32 ms.

What it does not yet settle

The first run got both scale-dependent tests wrong, and the second commit here fixes them:

• Q6 used a window above every seeded row, so nothing was above the lower bound — and the failure mode is O(rows above the lower bound). It measured an empty seek and called it cheap.
• Q7 patched keys in key order; production claims the head repeatedly, which is what piles dead entries at the seek point. Its floored seek was also pinned inside the region the churn had just rewritten, which is why it came out slower than the unfloored one.

Re-run in progress. I'll post the corrected Q6/Q7 numbers as a comment.

Notes for whoever edits this next

Three ways the harness silently succeeded while running nothing, all now documented in the README:

• A read-only mount fails Harper's component loader with EROFS on the node_modules symlink — and the server comes up healthy having loaded nothing. It stages into a temp dir and mounts read-write.
• handleApplication is never invoked on a root component — it is the hook for a component used as an extension. Needs jsResource.
• Harper intercepts process.exit, so the harness signals itself.

Docker mode mirrors how kohls-pr's CI stands Harper up. It refuses to run against ~/hdb.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive benchmark harness under bench/queue-index to measure and analyze the storage costs of the render queue in HarperDB, comparing different schema designs and access patterns. The feedback identifies critical issues where process.exit(1) is used, which would cause the process to hang due to HarperDB intercepting it, and suggests using SIGTERM instead. Additionally, it addresses a permission issue with mktemp -d in Docker mode, recommending chmod 777 to allow the container user to access the staged directory.

[gemini-code-assist]

Since HarperDB intercepts process.exit (as documented in the README and at the end of main), calling process.exit(1) here will cause the process to silently hang instead of terminating. Consider signaling the process with SIGTERM to ensure it exits reliably on failure.

Suggested change

	if (!(await tablesReady())) {
	console.error('[bench] databases.bench never appeared — the schema did not load');
	process.exit(1);
	}
	if (!(await tablesReady())) {
	console.error('[bench] databases.bench never appeared — the schema did not load');
	process.kill(process.pid, 'SIGTERM');
	}

[gemini-code-assist]

Similar to the schema loading failure, calling process.exit(1) here will cause the process to silently hang because HarperDB intercepts process.exit. Use process.kill(process.pid, 'SIGTERM') to ensure the process terminates on unhandled exceptions.

Suggested change

	main().catch((e) => {
	console.error('[bench] failed', e);
	process.exit(1);
	});
	main().catch((e) => {
	console.error('[bench] failed', e);
	process.kill(process.pid, 'SIGTERM');
	});

[gemini-code-assist]

On Linux, mktemp -d creates a directory with 0700 permissions (readable/writable only by the host user). When running in Docker mode, the container typically runs as a non-root user (e.g., harperdb), which will not have permissions to read the staged files or write node_modules inside the mounted directory. This leads to silent failures where the component fails to load. Explicitly setting the directory permissions to 777 ensures the container user can access and write to it.

Suggested change

	STAGE="$(mktemp -d)"
	cp "$HERE"/config.yaml "$HERE"/schema.graphql "$HERE"/bench.js "$STAGE/"
	trap 'rm -rf "$STAGE"' EXIT
	STAGE="$(mktemp -d)"
	chmod 777 "$STAGE"
	cp "$HERE"/config.yaml "$HERE"/schema.graphql "$HERE"/bench.js "$STAGE/"
	trap 'rm -rf "$STAGE"' EXIT

[harper-joseph]

Corrected Q6 / Q7 — both original claims confirmed, and my first run was wrong

Both of the first run's scale-dependent tests were badly constructed, and both produced a false negative. Fixed and re-run at 200,000 rows / 40,000 reschedules.

Q7 — the seek degradation is real, and reproduces almost exactly

Head-claim-then-reschedule (the production shape), sampling both seeks as it goes:

reschedules	unfloored seek	floored seek
0	0.073 ms	0.059 ms
10,000	1.416 ms	0.070 ms
20,000	~2.9 ms	0.061 ms
30,000	4.124 ms	0.072 ms
40,000	5.600 ms	0.073 ms

Monotonic, linear, and the floored seek is flat throughout. util/renderLease.js documents 0.36 → 6.25 ms over 40,000 reschedules; measured 0.073 → 5.60 ms. Confirmed. The claim floor is load-bearing.

My first run reported no degradation because it patched keys in key order rather than repeatedly churning the head — so dead entries spread over a key range instead of piling at the seek point, which is the entire mechanism. It also pinned the "floored" seek inside the region the churn had just rewritten, which is why that one came out slower than the unfloored seek. Both were artifacts of the test, not findings.

One nuance worth keeping: the 5.6 ms is the cost of a limit-20 seek walking 40,000 dead entries. A large read pays that walk once and then reads at 2.4 µs/row, so the degradation is prohibitive for small reads and amortised in big ones.

Q6 — the two-sided range is catastrophic, confirmed

Low, narrow window (139 matching rows) with a limit of 1,390 that cannot fill, against 200,000 rows above the lower bound:

query	time	rows returned
two-sided, limit cannot fill	739.7 ms	277
one-sided, same lower bound	2.89 ms	1,390
two-sided, wide window, limit fills	0.087 ms	20

256× slower. claimSchedules documents 1,128–2,977 ms at 400k rows; 740 ms at 200k is the same effect at half the scale. Keeping the <= now half in application code is correct and stays correct.

Unchanged from the first run

Reads ~2.4 µs/row and flat, yielding free, put ~76–89 µs/row, patch 37% worse than put, a second index +13%, per-lane seeks ~0.08 ms each.

So of #80's figures: the seek degradation and the two-sided-range penalty hold; the "second index is 39–48% of reschedule wall clock" claim does not (it is ~13%), and "a lane change is a cheap in-place update" does not (patch is the most expensive write measured).

[harper-joseph]

Thanks @gemini-code-assist — both applied. Both were mine to have caught.

process.exit on the failure paths — applied

This one is embarrassing in a useful way: I'd already fixed the success path to signal itself because Harper intercepts process.exit, wrote that fact into the README, and then left both failure paths on process.exit(1). So a schema that never loaded, or any unhandled rejection, printed its message and then sat there as a running server — a failed run looking exactly like a slow one, which is the worst possible presentation for a harness whose whole job is to produce a number you trust.

Both now process.kill(process.pid, 'SIGTERM').

mktemp -d permissions — applied

Correct, and it's the third instance of the same failure shape in this harness. mktemp -d is 0700 owned by the host user; the container runs as harperdb and has to create node_modules inside the mount to symlink itself, so without a chmod the component fails and Harper comes up perfectly healthy having loaded nothing.

That's the same silent success I hit with a read-only mount, and with handleApplication on a root component. Docker Desktop on macOS remaps ownership and hid it from me; on Linux and in CI it would not have. chmod 777 is safe here for the reason worth stating: it's a throwaway directory holding three harness files, removed on exit — not a persistent path.

The README already lists the other two ways this harness silently succeeded while running nothing; this is now the third entry in that list, since the pattern is clearly the thing to warn about rather than any individual cause.

[gemini-code-assist]

Thank you for the update, @harper-joseph. It is excellent to see the harness now correctly handling the process.exit interception and the mktemp permissions. These kinds of environmental nuances are exactly what make performance benchmarking tricky, and documenting them in the README is a great service for future maintainers. The corrected methodology for Q6 and Q7 provides much more reliable data for evaluating the storage engine's behavior under load. This looks like a solid, reproducible foundation for your upcoming architectural decisions.