research: merkle-zero-hash-pruning - #49
Conversation
Both off-chain Merkle tree builders (scripts/src/compliance-utils.ts, frontend/src/lib/merkle-tree.ts) rebuilt the entire 2^depth-wide tree on every call, hashing every zero-padding pair for real - O(2^depth) Poseidon calls regardless of leaf count. seed-credential-tree.ts already warned this takes ~60s for a single-leaf tree; the same cost sits inside every real transfer proof via useProofGeneration.ts. Both now only hash the real leaf prefix and fill the always-zero remainder from precomputed per-level zero-subtree hashes - O(n + depth) instead, verified bit-identical to the old output by three independent checks (a dedicated benchmark, 34 new scripts unit tests, and a ~76s naive-reference cross-check in the frontend suite). Measured: naive n=1 build = 101,358ms (constant regardless of n); pruned = 43.8ms at n=1 up to 108,763ms at full occupancy (correctly no speedup left when there's no padding to skip). Also confirms EXPERIMENTS.md item 12 (chained circuits npm test hang) is already fixed on main (PRs #16/#17), and reconfirms item 1 (on-chain gas) is still blocked by network policy via a second, independent method. Full report, depth-24 numbers, and threat/privacy framing: docs/research/2026-08-31-merkle-zero-hash-pruning.md Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016m6rsxgsUNWCbZd4x9SibV
Both bun-using CI jobs pinned oven-sh/setup-bun@735343b, labelled v2.0.1 - but that commit doesn't exist in oven-sh/setup-bun at all (GitHub Actions: "unable to find version"). The real v2.0.1 tag points to 4bc047ad259df6fc24a6c9b0f9a0cb08cf17fbe5; the broken hash appears to be a near-miss for v2.0.2's real SHA (735343b667d3e6f658f44d0eca948eb6282f2b76), off by a handful of trailing characters. This has been breaking both the "Proof converter + compliance utils" and "Frontend" CI jobs since the commit that introduced them (e37ecd6) - every CI run on main since then has failed at the action-resolution step, before any real test runs. Unrelated to this PR's diff; ported here so this PR's own CI can go green rather than waiting on main to pick up the same fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016m6rsxgsUNWCbZd4x9SibV
|
CI failed on the first push, but not because of this PR's diff: both That SHA doesn't exist in Ported the fix here ( Generated by Claude Code |
…ree test circomlibjs ships no type declarations. Every other direct import of it in this package goes through src/lib/dynamicRequire.ts, which already carries a @ts-expect-error for exactly this. The new test file imported it directly instead, which bunx tsc --noEmit correctly flagged in CI (TS7016) - not caught locally since I'd typechecked before adding this file and only re-ran vitest after. Same suppression, same reasoning, matching the existing convention. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016m6rsxgsUNWCbZd4x9SibV
2 participants
Summary
Nightly research loop, 2026-08-31. Full report:
docs/research/2026-08-31-merkle-zero-hash-pruning.md.scripts/src/compliance-utils.ts,frontend/src/lib/merkle-tree.ts) rebuild the entire2^depth-wide tree on every call —O(2^depth)Poseidon calls regardless of real leaf count. Pruning the always-zero padding (fill from precomputed per-level zero-subtree hashes instead of re-hashing) should cut this toO(n + depth), bit-identical output.compliance-utils.ts(same signature, same fully-padded output shape every existing caller relies on);frontend/lib/merkle-tree.ts's private internals only, same public surface.n. Pruned build = 43.8 ms at n=1 (today's actual real-world usage in both files) up to 108,763 ms at full occupancy (correctly no speedup left — no padding to skip). ~2,315x at the real usage pattern. Also ran depth 24 (~16.8M capacity) to show the technique removes the "off-chain rebuild is impractical at greater depth" objection for a future anonymity-set increase.scriptsunit tests (every layer, 15 values ofnincluding boundary cases), and a real ~76s naive-reference cross-check in a new frontend vitest suite (merkle-tree.tshad zero test coverage before this).EXPERIMENTS.mditem 12 (chainedcircuitsnpm testhang) is already fixed onmain(PRs fix(circuits): look up compliance artifacts in build-compliance/ #16/fix(circuits): exit test runners explicitly after the last proof #17); reconfirmed item 1 (on-chain gas /suiCLI) is still blocked by network policy, now via a second independent method (JSON-RPC403, crates.io content403, git-protocol clones do work).BASELINE.md,LEDGER.md,EXPERIMENTS.md, anddocs/threat-model.md(RR5) updated accordingly.No circuit, Move module, or proof format changed — roots and proofs are identical to before, so no soundness/leakage argument applies (see the report's Threat/privacy model section for the full reasoning on why this is an availability/UX fix, not a cryptographic one).
Test plan
cd scripts && bun run src/test-compliance-utils.ts— 101/101 pass (34 new)cd scripts && bun run src/test-converter.ts— 109/109 passcd scripts && bun run src/fuzz-tests.ts— 6/6 properties, 500 cases eachcd frontend && bunx vitest run— 23/23 pass (4 new)cd circuits && npm test— 65/65 pass (hash-only mode; no circuit touched, no compiled artifacts available in this sandbox)cd contracts && sui move test— NOT RUN,suiCLI unavailable in this sandbox (pre-existing, unrelated blocker — see report)bun run scripts/bench/merkle-scale.mjs— raw benchmark output in the report, correctness gate passes before any timing is reportedCo-authored by an autonomous nightly research routine (Claude Sonnet 5) — see the linked report for full methodology and raw command output.
Generated by Claude Code