bench(recursion): measure the verifier at real query counts, over real blocks

[Oppen]

Summary

Stacked on #845. The recursion cycle benchmark only measured MIN_PROOF_OPTIONS (blowup=2, 1 FRI query) verifying a proof of the empty program — a diagnostic floor, not the production verifier cost (blowup=2 at 128-bit is 219 queries, and real inner proofs are multi-epoch continuation bundles of real blocks).

Presets and guests:

• Preset::Blowup2 (219 q) / Blowup4 (110 q) — the realistic base-layer regimes — alongside the existing Min/Blowup8; one guest ELF per preset.
• bench_vs/lambda/recursion's continuation guest feature: verify a whole ContinuationProof bundle in-VM (recursion-cont-<preset>.elf), built on feat(recursion): supply DECODE/global-memory-genesis roots via private input for continuation verify #844/perf(prover): verify continuation proofs in place via rkyv #845's continuation supplied-roots verify path and zero-copy guest API.

Benchmark plumbing:

• test_dump_recursion_input gains env knobs: RECURSION_DUMP_PRESET, RECURSION_DUMP_INNER_ELF/INPUT (any inner program, e.g. ethrex blocks), RECURSION_DUMP_EPOCH_LOG2 (continuation mode).
• scripts/bench_recursion_cycles.sh: preset-aware dump, --release test invocations, per-preset regime labels; /bench-verify now posts min + blowup2 + blowup4 tables.
• scripts/bench_recursion_scaling.sh: block-size × preset ladder over the committed ethrex fixtures (1/4/8/16 transfers).
• A real-block detailed profile test (blowup=4, 4 transfers) plus a make test-profile-recursion-block target, since cycle counts across every preset/block size are already covered by the scripts above.

Note: this branch currently includes a cherry-picked copy of #843's commit (fix(syscalls): raise the private-input clamp to 512 MiB) so the real-block tests are runnable now (real blocks exceed the old 64 MiB cap). That commit will disappear from this diff once #843 merges and this branch is rebased onto the updated base.

Test plan

• cargo test -p lambda-vm-prover --lib (516 passed)
• test_recursion_profile_blowup4_block run end-to-end against a real 4-tx ethrex block (not just compiled) — produced the full cycle/step/function breakdown
• cargo clippy --all-targets -- -D warnings / cargo fmt --check
• make -n test-profile-recursion-block resolves the full build graph correctly

[github-actions]

Benchmark Results for modified programs 🚀

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
head hashmap	128.4 ± 4.6	122.2	134.6	1.00

[Oppen]

/ai-review

[github-actions]

Codex Code Review

• High — memory.rs: Raising the private-input cap to 512 MiB allows store_private_inputs to expand one input into roughly 134 million HashMap<u64, [u8; 4]> entries. This can consume many gigabytes and crash the executor/prover through host-memory exhaustion. The cap should reflect the sparse-memory representation’s safe limit, or private input should use a compact/page-backed representation before accepting inputs this large.

• Medium — flamegraph.rs: The oversized-input test now allocates over 512 MiB merely to exercise rejection, making an ordinary test prone to OOM. Test the boundary without constructing such a large buffer, or move this into an explicitly ignored/resource-heavy test.

[claude]

Reviewed as a static diff review (working tree is at main, so gh pr diff 846 was the source of truth). This is largely benchmarking infrastructure — new verifier presets, a continuation-guest benchmark path, scripts, and Makefile plumbing. It is careful and well-documented. No blocking issues found.

Verified correct

• New presets Blowup2/Blowup4: all three match self arms in recursion.rs (options/artifact_stem/name) are updated exhaustively, Preset::ALL lists all four, and there are no other Preset match sites to break. The compile_error! mutual-exclusion guard in main.rs correctly enumerates all six preset pairs, and --all-features (which also enables continuation) still trips it.
• ContinuationGuestInput / encode_continuation_guest_input: pure relocation earlier in recursion.rs, no behavior change.
• Makefile continuation rules: --features "continuation $(1)" builds both the cont and non-cont bins but copies the correctly-named artifact; race-free under -j via distinct bin names.
• flamegraph.rs test now derives the oversized-input size from MAX_PRIVATE_INPUT_SIZE instead of a hardcoded 64 MiB — no drift.

FYI (not blocking, out of scope for this PR)
The MAX_PRIVATE_INPUT_SIZE 64->512 MiB bump is the cherry-picked commit you flagged. Worth calling out for whoever reviews it: beyond the 8x host allocation, it widens the verifier honest page-count bound (page::max_private_input_pages, 257->2049) that guards num_private_input_pages against a malicious prover (prover/src/lib.rs:1350, continuation.rs:933). Both copies of the constant stay in sync and the bound remains enforced and pinned by test_max_private_input_pages_is_tight, so it is sound — just a security-relevant surface that belongs to the other PR review, and drops out of this diff on rebase as noted.

Minor

• The two MAX_PRIVATE_INPUT_SIZE definitions (executor + syscalls) are kept consistent only by a doc comment ("Must match"), with no compile-time cross-check. Pre-existing, not introduced here, but the bump is a reminder it would be nice to enforce.

[Oppen]

Note: MAX_PRIVATE_INPUT_SIZE findings come from and are handled in #843

[github-actions]

Benchmark Results for unmodified programs 🚀

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base binary_search	41.1 ± 1.4	38.8	42.6	1.03 ± 0.04
head binary_search	39.8 ± 1.0	38.5	41.6	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base bitwise_ops	41.1 ± 1.4	38.8	43.1	1.05 ± 0.05
head bitwise_ops	39.0 ± 1.1	38.5	41.5	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base ecsm	2.1 ± 0.0	2.1	2.1	1.03 ± 0.03
head ecsm	2.0 ± 0.0	2.0	2.2	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base fibonacci_26	42.2 ± 1.4	40.3	44.0	1.03 ± 0.04
head fibonacci_26	41.2 ± 1.1	40.2	43.3	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base keccak	78.1 ± 2.0	75.4	80.6	1.03 ± 0.03
head keccak	75.8 ± 1.6	74.7	78.5	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base matrix_multiply	45.7 ± 2.9	41.4	48.6	1.00
head matrix_multiply	47.3 ± 3.2	41.2	54.2	1.04 ± 0.10

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base modular_exp	40.1 ± 1.1	38.9	41.8	1.01 ± 0.04
head modular_exp	39.9 ± 1.3	38.8	41.7	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base quicksort	42.5 ± 1.0	41.1	44.0	1.00
head quicksort	42.5 ± 1.3	41.4	44.3	1.00 ± 0.04

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base sieve	44.2 ± 1.2	42.3	45.8	1.02 ± 0.03
head sieve	43.2 ± 0.9	42.2	44.7	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base sum_array	49.3 ± 1.2	47.4	51.0	1.00 ± 0.03
head sum_array	49.2 ± 1.0	47.8	51.3	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base syscall_commit	59.1 ± 1.7	57.4	61.6	1.00
head syscall_commit	66.4 ± 2.2	64.7	70.4	1.12 ± 0.05

[Oppen]

/ai-review

[github-actions]

Codex Code Review

• Medium — .github/workflows/bench-verify.yml:134: Both new full-query comparisons always fail against the current origin/main, because bench_recursion_cycles.sh rejects any non-min baseline lacking RECURSION_DUMP_PRESET. The if then suppresses that failure, so CI posts only “unavailable” notes and never measures blowup2 or blowup4 for this PR. Generate the baseline blob with explicitly matched options or avoid advertising these regimes until both refs support them.

[claude]

Review — no significant issues found

Reviewed the diff (safety/security, correctness, performance, readability). This PR is almost entirely benchmark/test/build plumbing on top of the existing recursion verifier, and it's carefully done and well-documented.

Verified as correct:

• Preset expansion (recursion.rs): Blowup2/Blowup4 added consistently across options()/artifact_stem()/name(), and Preset::ALL matches the 4-variant enum (Preset is Copy, so the into_iter().find(...) name lookup is fine). with_blowup(2|4) mirrors existing with_blowup(8) usage.
• Guest compile_error! mutual-exclusion (main.rs): the pairwise all(...) guard correctly rejects any two presets enabled at once, and the not(any(...)) guard rejects none — preserving the security invariant that a preset is pinned at build time, not via private input.
• Makefile --bin fix: the added --bin $(2) correctly defeats the required-features subset-match race described in the comment (a --features "continuation min" build would otherwise also build plain recursion-min-bench into the shared target path, racing make -j). Distinct bin names + --bin pinning is race-free.
• CI full-query loop (bench-verify.yml): the if scripts/...; then ... else <note> fi wrapper correctly tolerates refs predating the preset-aware dump (the if-condition is exempt from set -e; pipefail still makes the script's exit status authoritative through the tee).
• Profile test correctness upgrade (recursion_smoke_test.rs): checking the guest's committed split_attestation against a host-side expected_program_id / continuation recompute turns the profile tests from crash-freedom into an actual identity+output oracle — a good addition. The added blob.len() <= MAX_PRIVATE_INPUT_SIZE assert in the dump test is a sensible guard.
• Scripts: bench_recursion_scaling.sh cleans up each ~100+ MB blob per iteration (no disk blowup), fails fast on missing prereqs, and tolerates per-cell failures without aborting the sweep.

Note (not a finding): the continuation verify path (verify_continuation_and_attest_blob, ContinuationGuestInput) lives in the stacked base (#844/#845), not in this diff — those functions are only moved/referenced here, so I could not re-verify their internals from this PR alone.

Nothing blocking.

[github-actions]

AI Review

PR #846 · 10 changed files

Findings

Status	Sev	Location	Finding	Found by
confirmed	critical	bench_vs/lambda/recursion/src/main.rs:100	continuation guest calls non-existent verify_continuation_and_attest_blob (compile error)	glm openrouter/z-ai/glm-5.2 moonmath zro/minimax-m3 nemotron openrouter/nvidia/nemotron-3-ultra-550b-a55b kimi openrouter/moonshotai/kimi-k2.7-code
confirmed	high	prover/src/tests/recursion_smoke_test.rs:978	test_dump_recursion_input allows blowup8 with continuation but no guest ELF exists	nemotron openrouter/nvidia/nemotron-3-ultra-550b-a55b
confirmed	medium	scripts/bench_recursion_scaling.sh:75	bench_recursion_scaling.sh writes empty inner_cycles/epochs to results on failure	nemotron openrouter/nvidia/nemotron-3-ultra-550b-a55b
confirmed	low	prover/src/recursion.rs:115	Preset::ALL includes Blowup8 but no continuation variant exists	nemotron openrouter/nvidia/nemotron-3-ultra-550b-a55b
confirmed	low	scripts/bench_recursion_scaling.sh:19	bench_recursion_scaling.sh doc claims ethrex_bench_{1,4,8,16}.bin are committed; only _4 is	moonmath zro/minimax-m3

Status column reflects the verdict from the verifier: deepseek-verifier (openrouter/deepseek/deepseek-v4-pro).

AI-001: continuation guest calls non-existent verify_continuation_and_attest_blob (compile error)

• Status: confirmed
• Severity: critical
• Location: bench_vs/lambda/recursion/src/main.rs:100
• Found by: glm:openrouter/z-ai/glm-5.2, moonmath:zro/minimax-m3, nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b, kimi:openrouter/moonshotai/kimi-k2.7-code
• Verified by: deepseek-verifier:openrouter/deepseek/deepseek-v4-pro
• Rejected by: -

Claim

The continuation guest feature build references a function lambda_vm_prover::recursion::verify_continuation_and_attest_blob that does not exist. The actual public function in prover/src/recursion.rs is named verify_continuation_and_attest (line 310). This is a hard compile error whenever the continuation feature is enabled.

Evidence

main.rs line 100 calls lambda_vm_prover::recursion::verify_continuation_and_attest_blob(blob, &options) (and the doc comment on line 19 references the same name). A repo-wide grep for verify_continuation_and_attest_blob returns only those two main.rs references — no fn definition anywhere. The continuation verify function in prover/src/recursion.rs (line 310) is pub fn verify_continuation_and_attest(blob: &[u8], proof_options: &ProofOptions) -> Result<Option<Vec<u8>>, Error> — same signature, different name. The Makefile now builds recursion-cont-<preset>.elf artifacts via --features "continuation <preset>" (RECURSION_CONT_PRESETS = min blowup2 blowup4), and compile-recursion-elfs is a prerequisite of compile-programs and thus make test. So this breaks the guest build for all continuation variants, the new test-profile-recursion-block make target, the test_recursion_profile_blowup4_block ignored test (which loads recursion-cont-blowup4.elf), scripts/bench_recursion_scaling.sh, and the blowup2/blowup4 full-query regimes of bench_recursion_cycles.sh (which dump continuation blobs fed to recursion-cont-*.elf).

Suggested fix

Rename the call (and the line-19 doc-comment reference) from verify_continuation_and_attest_blob to verify_continuation_and_attest, matching the actual function name in prover/src/recursion.rs. The signature ((&[u8], &ProofOptions) -> Result<Option<Vec<u8>>, Error>) already matches, so only the name needs to change.

AI-003: test_dump_recursion_input allows blowup8 with continuation but no guest ELF exists

• Status: confirmed
• Severity: high
• Location: prover/src/tests/recursion_smoke_test.rs:978
• Found by: nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b
• Verified by: deepseek-verifier:openrouter/deepseek/deepseek-v4-pro
• Rejected by: -

Claim

The test accepts RECURSION_DUMP_PRESET=blowup8 with RECURSION_DUMP_EPOCH_LOG2 (continuation mode) but there's no recursion-cont-blowup8.elf built (missing from Cargo.toml [[bin]] and Makefile RECURSION_CONT_PRESETS). This creates a blob that cannot be executed.

Evidence

Line 978-985: test accepts any preset from Preset::ALL (includes Blowup8). Lines 1001-1020: when RECURSION_DUMP_EPOCH_LOG2 is set, it encodes a continuation blob for that preset. Cargo.toml has no recursion-cont-blowup8-bench bin. Makefile RECURSION_CONT_PRESETS := min blowup2 blowup4 (excludes blowup8).

Suggested fix

Either add blowup8 to RECURSION_CONT_PRESETS and the corresponding Cargo.toml bin, OR reject blowup8 in test_dump_recursion_input when RECURSION_DUMP_EPOCH_LOG2 is set (since benchmarks don't measure it).

AI-005: bench_recursion_scaling.sh writes empty inner_cycles/epochs to results on failure

• Status: confirmed
• Severity: medium
• Location: scripts/bench_recursion_scaling.sh:75
• Found by: nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b
• Verified by: deepseek-verifier:openrouter/deepseek/deepseek-v4-pro
• Rejected by: -

Claim

The script uses || true when extracting inner_cycles and epochs, allowing them to be empty. These empty values are written to the results file, which could break downstream parsing.

Evidence

Line 75: ic="$(...)" || true - if CLI fails, ic is empty. Line 89: epochs="$(grep ...)" || true - if grep fails, epochs is empty. Lines 84, 100, 104 write these potentially-empty values to results.

Suggested fix

Validate ic and epochs are non-empty before writing results, or use a sentinel value like 'unknown' instead of empty string.

AI-006: Preset::ALL includes Blowup8 but no continuation variant exists

• Status: confirmed
• Severity: low
• Location: prover/src/recursion.rs:115
• Found by: nemotron:openrouter/nvidia/nemotron-3-ultra-550b-a55b
• Verified by: deepseek-verifier:openrouter/deepseek/deepseek-v4-pro
• Rejected by: -

Claim

Preset::ALL array includes Blowup8 (line 115) but RECURSION_CONT_PRESETS and Cargo.toml bins only cover min/blowup2/blowup4. This inconsistency allows test_dump_recursion_input to accept blowup8 for continuation mode (finding #2) and could confuse users.

Evidence

recursion.rs lines 112-117: Preset::ALL = [Min, Blowup2, Blowup4, Blowup8]. Makefile line 74: RECURSION_CONT_PRESETS := min blowup2 blowup4. Cargo.toml lines 48-61: no recursion-cont-blowup8-bench bin.

Suggested fix

Either add Blowup8 continuation support (bin + build rule) or remove Blowup8 from Preset::ALL and document that continuation is only for min/blowup2/blowup4. The latter matches the comment 'Kept to the presets the recursion benchmarks actually measure'.

AI-007: `bench_recursion_scaling.sh` doc claims `ethrex_bench_{1,4,8,16}.bin` are committed; only `_4` is

• Status: confirmed
• Severity: low
• Location: scripts/bench_recursion_scaling.sh:19
• Found by: moonmath:zro/minimax-m3
• Verified by: deepseek-verifier:openrouter/deepseek/deepseek-v4-pro
• Rejected by: -

Claim

The script's usage doc says fixtures are "read from executor/tests/ethrex_bench_<N>.bin (committed for 1/4/8/16) and generated via tooling/ethrex-fixtures when missing." executor/.gitignore only re-includes ethrex_bench_4.bin (line 10) — _1, _8, and _16 are NOT committed, contradicting the doc.

Evidence

executor/.gitignore ignores /tests/ethrex_bench_*.bin (line 4) then re-includes only !/tests/ethrex_bench_4.bin (line 10); the comment explicitly notes "The other ladder sizes (_1/_8/_16) and bigger ones (e.g. _20) stay ignored." glob executor/tests/ethrex_bench* confirms only ethrex_bench_4.bin is present in the repo. The script does handle missing fixtures (it invokes tooling/ethrex-fixtures to generate them), so this is purely a doc bug, not a behavior bug.

Suggested fix

Change "(committed for 1/4/8/16)" to "(committed for 4; other sizes are generated on demand via tooling/ethrex-fixtures)" so the doc matches .gitignore and reality.

Reviewer Lanes

Lane	Model	Prompt	Status	Findings
glm	openrouter/z-ai/glm-5.2	general	success	1
kimi	openrouter/moonshotai/kimi-k2.7-code	general	success	1
minimax	minimax/MiniMax-M3	general	error: opencode failed (provider/auth/runtime error) and no findings were submitted	0
moonmath	zro/minimax-m3	general	success	3
nemotron	openrouter/nvidia/nemotron-3-ultra-550b-a55b	general	success	4

Verification Lanes

Lane	Model	Status	Confirmed	Rejected	Uncertain
deepseek-verifier	openrouter/deepseek/deepseek-v4-pro	success	5	1	0

Native Codex and Claude reviews run separately and post their own comments. They are not included in this structured provenance report.

Discarded candidates (1) — rejected by the verifier

• Workflow comment understates the cost of the blowup2/blowup4 regimes (.github/workflows/bench-verify.yml:131, found by moonmath:zro/minimax-m3) — The claim is that a workflow YAML comment on line 131 understates the cost of full-query blob dumps. This is about a documentation comment being imprecise ('seconds' vs potentially longer), not about broken code behavior, incorrect logic, or any functional defect. Comments are aspirational guidance, not enforceable correctness constraints. No actual bug or misbehavior results from an inaccurate time estimate in a code comment.

Raw lane outputs, candidates, final issues, and model metrics are uploaded as workflow artifacts.