docs: #2247 survey trace-method availability across chain upstreams

[tenk-earn]

Closes #2247.

What

Research survey of trace-method availability (debug_traceBlockByNumber, trace_block, Otterscan ots_*) and pricing across every chain in the production CHAIN_CONFIG / PUBLIC_RPCS at staging. One markdown table per chain, plus a short recommendation on which chains are viable for trace triggers.

No trigger implementation. No Aetherlay changes. No secrets. Public provider docs and unauthenticated public RPCs only.

Why

#2241 cannot be scoped until we know which configured upstreams actually expose traces, at what tier, and how large the responses are. The useful result is mostly negative: the established EVM mainnets (ETH, Base, Arbitrum, Polygon, BNB, OP, Avalanche) do not expose full-block traces on the tree-configured public defaults. Trace triggers are viable today on Plasma and Tempo official public RPCs, and on paid Debug/Trace tiers of the common commercial providers (Alchemy PAYG, Infura Developer+, QuickNode paid, Ankr Premium) — none of whose free/current public plans include those APIs.

Aetherlay internal/server/server.go:410 is not in this public repository (404); the no-method-allowlist claim is flagged as maintainer-asserted rather than verified here. Production CHAIN_RPC_CONFIG is Parameter Store, also not in the tree; those cells are listed as maintainer-only.

Deliverable: docs/issue-2247-trace-upstream-survey.md.

Filed for the KeeperHub Agent Economy hackathon bounty (Best KeeperHub Feature PR). Author: tenk-earn.

[edycutjong]

We picked this up within five minutes of each other and, without either of us
knowing, probed overlapping windows the same morning. You landed first, and your
provider tier matrix (Alchemy / Infura / QuickNode / Ankr / dRPC, with CU and
credit costs) is the part I had explicitly deferred to a maintainer — that is
scope item 2 answered, and mine does not have it. Your Aetherlay check is also
more thorough than mine; I checked two paths, you ran a recursive tree scan.

Rather than file a second table I have opened #2277 with only the probe, and one
finding from the overlap that I think belongs in your document.

Our plasma-mainnet numbers disagree by ~5x, and the reason is useful. Both of
us sampled a 5-transaction block:

plasma-mainnet, 5-tx block	debug_traceBlockByNumber	trace_block	ots_getBlockTransactions
yours	63 KiB	101 KiB	7.5 KiB
mine	348 KB	562 KB	7.5 KB

The ots_ figures match almost exactly, which is the tell: ots_ scales with
transaction count, while trace size scales with the block's internal call
depth — and that varies several-fold between blocks holding the same number of
transactions.

Neither of us measured wrong. But it means a single sampled block
under-determines the cost envelope, and that envelope is the number #2241
actually needs to decide whether full-block tracing is affordable. Might be worth
a line in the doc saying the sizes are one-block samples with several-fold
block-to-block variance, so nobody plans capacity off a single figure.

Two smaller things from my run that may be worth folding in:

• Fallbacks differ from primaries more than I expected. Probing both
  systematically, op-sepolia, 0g-mainnet and robinhood-testnet each
  answered a trace method on their fallback while refusing it on their
  primary. Your doc catches the 0g and OP Sepolia cases; robinhood-testnet's
  dRPC fallback answered debug_traceBlockByNumber for me on one run and not on
  another, which fits your "chain-specific leakage, not a plan we can design
  around" reading.
• 0g-mainnet fallback trace_block is unstable rather than absent — dRPC
  returned Can't route your request to suitable provider on repeated runs,
  which is a different finding from a method being unimplemented.

I have results.json with the verbatim body for all 102 probes (24 chains, 41
endpoints, both primary and fallback) if any of it is useful to you — say the
word and I will post the rows here, or send a PR at your branch rather than a
separate one. Happy for #2277 to be closed or folded into this if the maintainers
would rather have one artefact.

[suisuss]

Welcome, and thanks for this - the contributing guide is in CONTRIBUTING.md, and ISSUES.md covers when a change needs an issue first.

What this changes

One new file, docs/issue-2247-trace-upstream-survey.md, 162 lines, no code. It surveys trace-method reachability (debug_traceBlockByNumber with callTracer, trace_block, ots_*) against the endpoints in lib/rpc/rpc-config.ts: a mainnet table, a testnet table, a commercial-provider pricing matrix, size and latency bounds, a maintainer-only checklist, and a curl reproduction.

Does it match the description

Matches on content. The chain inventory checks out exactly - CHAIN_CONFIG has 24 entries, 12 mainnet and 12 testnet, and every chain id, jsonKey, primary and fallback in both tables matches the source, including the five chains correctly marked as having no fallback.

What the description does not say is where the file lands.

Blocking

• docs/issue-2247-trace-upstream-survey.md - docs/ is the content root of the public documentation site, not an internal directory. docs-site/Dockerfile:28-29 does RUN rm -f ./content then COPY docs/ ./content/, and docs-site/app/sitemap.ts derives sitemap.xml from that tree. -> This memo publishes at docs.keeperhub.com/issue-2247-trace-upstream-survey and enters the public sitemap, carrying internal issue numbers, maintainer-asserted hedging, and a competitive read of Alchemy, Infura, QuickNode, Ankr and dRPC pricing. The five existing top-level files under docs/ are FAQ.md, _meta.ts, concepts.md, index.md and platform-reference.md, all product documentation. -> Move it to .planning/, which already holds this kind of document and publishes nothing.

• docs/issue-2247-trace-upstream-survey.md:81, 39, 40, 85, 66 - the binary-prefix figures are decimal-prefix arithmetic. Line 81 reports 1 905 937 B as "1.91 MiB"; it is 1.82 MiB. The same slip runs through the document: 62 749 / 101 311 / 7 572 B given as "63 KiB / 101 KiB / 7.5 KiB" (61.3 / 98.9 / 7.4), 778 302 B as "778 KiB" (760), 3 635 528 B as "3.6 MiB" (3.47). -> A reader sizing a response buffer off a summary line rather than the byte column provisions about 5 percent low on every chain, and "1.91 MiB" sitting beside "1 905 937 B" in one table invites them to treat one of the two as a typo. -> Relabel as kB/MB, or recompute against 1024. Either convention, applied once throughout.

• docs/issue-2247-trace-upstream-survey.md:66 - "payloads ranged from 2 KiB to 3.6 MiB" contradicts the tables below it. Line 86 records an OP Sepolia trace_block at 1 718 B, and line 148 records 36-byte empty-block responses. -> Line 148 tells a trigger implementer to tolerate 36-byte empty blocks while line 66 tells them the floor is 2 KiB; a minimum-length sanity check written off the range rejects the case the same document warns about. -> State the non-empty range from 1 718 B and note the 36-byte empty-block response separately.

Mechanical - actionable as-is

• docs/issue-2247-trace-upstream-survey.md:3, 44, 60, 138, 142 - the document is framed for #2247, then addresses #2241 five times without ever saying what #2241 is. One sentence in the preamble naming the relationship.
• docs/issue-2247-trace-upstream-survey.md:27 - "out of scope of this tree" reads as unverifiable. Aetherlay is a separate repository, not an unknowable one; say the allowlist question lives there and name it.
• docs/issue-2247-trace-upstream-survey.md:7 - scripts/seed/seed-chains.ts writes the output of getRpcUrlByChainId, which is the resolved primary and fallback, not PUBLIC_RPCS literally. For an unconfigured install the two coincide, which is the point being made - say "resolved".
• The curl at 152-162 reproduces neither the byte counts nor the timings, and no harness is committed, so the positive cells are unfalsifiable as published. Committing the probe script would fix that; the negative results are deterministic method-refusal codes and stand on their own.

Verdict

Changes requested - the file publishes to the public docs site as written, and the size figures use binary prefixes for decimal arithmetic throughout.

One thing from this survey is worth separating out from it: line 104 records rpc-amoy.polygon.technology as NXDOMAIN, and that host is the seeded primary for chain 80002. That is a live defect rather than a trace question, and I am filing it separately so it does not sit inside a survey.