chore(scripts): #2247 add a regenerable trace-method availability probe

[edycutjong]

Issue

Refs #2247. Not a competing survey — #2273 is the survey, and I am not
duplicating it. This is only the probe that keeps its table checkable.

How this happened

@tenk-earn and I picked up #2247 within five minutes of each other and, without
knowing it, probed overlapping windows the same morning. Their survey landed
first and it is the better document on the axis I explicitly punted: the
commercial provider tier matrix (Alchemy / Infura / QuickNode / Ankr / dRPC with
actual CU and credit costs) is exactly what scope item 2 asked for, and I had
deferred it to a maintainer. Their Aetherlay check is also more thorough than
mine — they ran a recursive tree scan where I checked two paths.

So rather than file a second table, this PR is the part that does not overlap.
If you would rather fold it into #2273, or not take it at all, say so and I will
close it — the survey question is answered either way.

Why a tool rather than a second table

Our two independent runs agree on the headline (plasma and tempo serve traces,
the established EVM mainnets do not) and disagree on one number in a way that is
worth keeping:

plasma-mainnet, 5-tx block	debug_traceBlockByNumber	trace_block	ots_getBlockTransactions
#2273	63 KiB	101 KiB	7.5 KiB
this probe	348 KB	562 KB	7.5 KB

The ots_ figures match almost exactly, and that is the tell. ots_ scales with
transaction count; trace size scales with a block's internal call depth,
which varies several-fold between blocks holding the same number of
transactions.

Neither measurement is wrong. The conclusion is that a single sampled block
under-determines the cost envelope #2241 is trying to estimate — which is
precisely the number that decides whether full-block tracing is affordable.
Re-running is the cheap way to bound it, and that needs a tool.

What it does

Parses the chain list out of lib/rpc/rpc-config.ts at run time instead of
hand-copying it, then probes debug_traceBlockByNumber, trace_block and
ots_getBlockTransactions against both the publicDefault and the
publicFallback of every entry — 41 endpoints, 102 probes — recording the
verbatim response body, raw and gzipped size, fetch and parse time.

node scripts/trace-method-probe.mjs --dry-run    # resolve targets, no requests
node scripts/trace-method-probe.mjs --mainnets   # EVM mainnets only
node scripts/trace-method-probe.mjs              # everything (~4 min)
node --test scripts/trace-method-probe.test.mjs  # 23 tests

Probing fallbacks systematically turned up cases the primary alone would miss:
op-sepolia, 0g-mainnet and robinhood-testnet each answered a trace method
on their fallback while refusing it on their primary.

Classification is the load-bearing part

HTTP status alone decides nothing — two endpoints return HTTP 403 meaning
opposite things:

Endpoint	Body	Verdict
mainnet.base.org	-32601 "rpc method is unsupported"	absent; no plan lifts it
ethereum-rpc.publicnode.com	-32602 "Archive requests require a personal token"	purchasable

So a structured JSON-RPC error outranks the HTTP status, and within the body a
tier signal outranks absent-method wording. Reading "not available on the free
plan" as "never implemented" would tell #2241 a chain cannot trace when it can
be paid for — the costliest mistake available here. Every shape above was
observed live and is pinned as a regression test.

Four classifier defects were found this way, by live responses contradicting the
code, and fixed before this PR: status read before body, -32600 "not allowed"
falling through unclassified, a non-JSON 401 policy refusal reported as
malformed, and empty blocks measured as though they were trace sizes.

Placement and cost to you

scripts/ is excluded by biome.jsonc, and tsconfig.json includes
.ts/.tsx/.mts but not .mjs — so this adds nothing to either gate. No
dependencies, no app or database access, no credentials, no runtime code path
touched.

Tests run against the repository's own lib/rpc/rpc-config.ts rather than a
vendored snapshot, so config drift fails the test instead of quietly skewing a
future run. That matters here: CHAIN_CONFIG has 24 entries today, while #2239
says "22 entries / 11 mainnets" and #2240 says "10 EVM mainnets" — it has
already drifted once.

Politeness, since these are free third-party endpoints: strictly sequential,
750 ms between every request, one block sampled per endpoint, and a single retry
reserved for failures that describe themselves as transient.

How it was verified

• node --test scripts/trace-method-probe.test.mjs — 23 passed, against the
  repo's own config
• Full sweep: 24 chains, 41 endpoints, 102 probes, ~4 minutes

I have not run pnpm check or pnpm type-check. This repo needs Node 24+,
Postgres 16+ and Docker per CONTRIBUTING.md, which I have not set up. Both
files sit outside those gates by construction as described above, but I would
rather say so than tick two boxes I did not verify.

I have the full results.json (verbatim body for all 102 probes) and the
fallback-coverage rows if either is useful to #2273 — happy to post them there
instead.

[github-actions]

About the build check on this pull request

This pull request comes from a fork, so GitHub does not pass it the credentials build normally uses for our image registry cache and staging build configuration. The build still runs and still compiles the image, so a red build here is real; it just takes longer than on team branches.

Every workflow run on a pull request from a fork also waits for a maintainer to approve it, so checks can sit at "awaiting approval" for a while after each push. Nothing is needed from you for either of these.

[suisuss]

Welcome, and thanks for this - the contributing guide is in CONTRIBUTING.md, and ISSUES.md covers when a change needs an issue first.

What this changes

Two new files, nothing else touched. scripts/trace-method-probe.mjs is a dependency-free Node CLI that regex-parses PUBLIC_RPCS and CHAIN_CONFIG out of lib/rpc/rpc-config.ts, probes each endpoint's liveness and head block, walks back to find a block with transactions, then tries debug_traceBlockByNumber, trace_block and ots_getBlockTransactions, classifying each result into one of twelve buckets and writing results.json and report.md. scripts/trace-method-probe.test.mjs adds 23 node:test cases over classify and parseChainConfig.

On the questions worth answering up front: it reads no process.env at all, sends no headers beyond content-type, and writes nothing that could be a secret. The default --out ./out is gitignored, so no generated output is checked in. It runs nowhere in CI.

Does it match the description

Matches, with two consequences the description does not draw out. CONFIG_URL defaults to raw.githubusercontent.com/.../staging/lib/rpc/rpc-config.ts, so a no-argument run surveys staging's config rather than the working tree. And the POST targets are 39 third-party hosts taken verbatim from that remotely fetched file.

Blocking

• scripts/trace-method-probe.mjs:266, 268 - /not found/ and /not available/ sit in NOT_SUPPORTED_PATTERNS with no code constraint, so a block-availability error is bucketed method-not-found, which the report's own legend renders as "no plan lifts it". -> rpc.plasma.to is load-balanced; a backend a block or two behind answers -32001 "block not found" for the head-5 block. Plasma, which does implement debug_traceBlockByNumber and which the survey's headline cites as a trace success, is recorded as not supporting it, and method-not-found is not in RETRYABLE so nothing re-tries. The same applies to -32000 "header not found" and geth's "missing trie node ... state is not available". -> Drop both patterns - geth's genuine absent-method text always carries -32601, which :328 already handles - and add a block-unavailable bucket ahead of it, made retryable so the walk-back can pick another block.

• scripts/trace-method-probe.mjs:327 - the tier check precedes the rate-limit check, so a throttle worded with "plan" or "upgrade" is classified tier-gated. -> A full run makes roughly four requests per endpoint across 39 hosts at a fixed 750 ms spacing with no backoff; drpc and ankr free tiers throttle with exactly that vocabulary. A transient throttle is written into the report's most decision-relevant column as "traces are purchasable here", with no retry and nothing distinguishing it from a real gate. -> Hoist the rate-limit test above the tier test, or make shouldRetry return true for a tier-gated result whose message matches a rate-limit pattern.

• scripts/trace-method-probe.mjs:587 - meta.limits.BLOCK_SCAN_LIMIT is not in the limits object built at :787-792, which carries only REQUEST_DELAY_MS, REQUEST_TIMEOUT_MS, MAX_RESPONSE_BYTES and BLOCKS_BEHIND_HEAD. -> Every generated report reads "walked back up to undefined blocks". -> Add BLOCK_SCAN_LIMIT to that object.

• scripts/trace-method-probe.mjs:576 - the provenance line says scripts/trace-probe/probe.mjs; the file is scripts/trace-method-probe.mjs and that directory does not exist. -> A reader following the line to re-run the survey cannot find it, which is the point of a regenerable table. The test file's header comment has the same stale name. -> Use the real path in both.

• scripts/trace-method-probe.mjs:425-436 - when the first eth_getBlockByNumber is not ok the scan loop breaks with txCount at 0 and blockNum never advanced, and control falls through to probe all three trace methods against a block that was never read. -> An endpoint 429s the block fetch; the probe then traces an unverified block number and records the answer as a capability finding, which with the first item above reads as method-not-found. -> Track whether the scan confirmed a readable block, and record skipped if it did not.

Mechanical - actionable as-is

• The script embeds an emoji legend at :512-529 and writes emoji into every report.md. AGENTS.md:42 and CLAUDE.md both forbid emoji in code and generated content without exception. Not CI-enforced, but it is a stated rule. Use text markers.
• :590 claims "Transient failures are retried once", but only the trace-method loop at :470 consults shouldRetry. The eth_chainId liveness call at :391 and the eth_blockNumber call at :412 are not retried, and a blip on either discards all three method findings for that endpoint. Route both through the retry helper, or narrow the claim.
• :443 - blocksScanned: scanned + 1 records 13 when the loop exhausts a 12-block limit, and 1 when it breaks on the first unreadable block despite zero successful reads.
• The test file reads a new env var RPC_CONFIG_PATH that is not in .env.example, and .github/workflows/maintainability.yml's env-sync greps .mjs files. That produces a new warning annotation on every PR from here on. Add it to .env.example or read it differently.

Verdict

Changes requested - two classification rules turn routine node behaviour into permanent capability verdicts, which is the one thing this survey's output is used for.

One structural note. Nothing runs the test file: vitest.config.mts matches only .ts/.tsx, no package.json script runs node --test, and no workflow invokes it. biome.jsonc:27 excludes scripts/ from lint and tsconfig.json does not type-check .mjs. So the 23 cases are documentation with assertions that a human has to remember to run - worth either wiring into CI or saying explicitly in the file that it is run by hand.

[edycutjong]

Thank you for this - the plasma finding in particular. You are right that it is
the failure that matters here: the survey's whole output is capability verdicts,
and two of my rules were manufacturing them out of routine node behaviour.

All five blocking items are addressed in the new commit.

:266, 268 - not found / not available. Both patterns are gone. A new
block-unavailable bucket sits ahead of NOT_SUPPORTED_PATTERNS and is in
RETRYABLE, covering block not found, header not found, missing trie node
and state is not available. Genuine absence arrives as -32601 and is tested
before any of it, as you said.

One thing I want to flag, because writing the tests for it changed the patterns.
Every entry in the new bucket is anchored on \bblock\b rather than a bare
/block/. Without that anchor, trace_block not found and
the method trace_block does not exist both classify as a missing block -
the mirror image of the bug you found, and _ being a word character is the
only reason the boundary separates them. Two of the new cases assert exactly
that, and both failed on my first attempt.

:327 - tier before rate limit. Hoisted, and I took the second half of your
suggestion too: shouldRetry now retries a tier-gated result whose message
reads as a throttle, for the case where an HTTP 403 carries no body for the
classifier to read. Tests pin both directions, since hoisting must not start
swallowing real paywalls.

:587 - BLOCK_SCAN_LIMIT. Added to meta.limits. Verified against a
generated report: the Sampling line now reads "walked back up to 12 blocks".

:576 - provenance path. Fixed in the script and the test header. The cause
was a local copy of these files living at that path under a different name; I
have renamed mine to match the repo so it cannot drift again.

:425-436 - unread block. The scan now tracks whether it ever confirmed a
readable block and records skipped with the failing status if it did not,
instead of falling through to probe three methods against a block number that
was never advanced.

Mechanical list: emoji removed - the status name already carried the meaning,
so I dropped the marker column rather than replacing it with a second severity
vocabulary that could disagree with the first. Every call now goes through one
rpcWithRetry, so the "retried once" claim covers the liveness and head calls
it did not before. blocksScanned is now blocksRead and counts requests
actually made. RPC_CONFIG_PATH is in .env.example.

On your structural note - agreed, and it was the fair reading. pnpm test:trace-probe
now runs the file, and the header says why it sits outside vitest rather than
leaving the next reader to work it out. 29 cases, up from 23.

What I have not run: pnpm check and pnpm type-check. Worth saying that as
far as I can tell neither would reach these files anyway - biome.jsonc excludes
scripts, and tsconfig.json does not take .mjs - so please treat that as
unverified rather than clean. The 29 tests pass, and I ran the probe end-to-end
against a single chain to confirm the report renders.

Unrelated to the review, but it is evidence for your "single run is a snapshot"
point and I would rather say it than not: on that smoke run today, eth-sepolia's
public endpoint refused all three methods, where the original survey run recorded
trace_block answering with 133 transactions and 1293 KB. Same endpoint, five
days apart. Nothing in this PR ships that table, so nothing here is stale - but
it does suggest any figure #2241 takes from a single run wants a second one
behind it.