fix(ws): skip gzip ResponseWriter wrap on WebSocket upgrades - #17
Open
azgms wants to merge 92 commits into
Open
Conversation
* feat: add `erpc dump` command and make ScoreMultiplier fields optional - Add `erpc dump <config-file>` command to parse TS/JS/YAML config and output the resolved configuration as YAML or JSON - Support `--with-defaults` flag to apply eRPC defaults before dumping - Support `--format yaml|json` flag for output format - Add LoadConfigRaw() for parsing config without defaults/validation - Add MarshalYAML() for Duration, DataFinalityState, CacheEmptyBehavior, CachePolicyAppliesTo, AvailbilityConfidence, RateLimitPeriod, and SelectionPolicyConfig - Fix RateLimitPeriod.UnmarshalYAML to try integer enum before string (fixes parsing of `period: 1` in generated YAML) - Make ScoreMultiplierConfig.Network and Method optional (omitempty), defaulting to wildcard "*" via SetDefaults() - Regenerate TypeScript types with optional network/method fields Co-authored-by: Cursor <cursoragent@cursor.com> * fix: SelectionPolicyConfig.MarshalYAML handles TS/JS eval functions When config is loaded from TypeScript/JS, EvalFunction is a compiled callable but evalFunctionOriginal is empty. MarshalYAML now checks EvalFunction != nil and outputs "<function>" as a placeholder, matching the existing MarshalJSON behavior. Also omits zero-value intervals. Co-authored-by: Cursor <cursoragent@cursor.com> * simplify: drop --with-defaults flag from dump command Raw dump is the primary use case (comparing TS output against prod YAML). Defaults add noise and fail when env vars are missing. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: reject unsupported --format values in dump command Co-authored-by: Cursor <cursoragent@cursor.com> * fix: bounds-check enum String() methods, add omitempty to JSON tags - DataFinalityState.String() and CacheEmptyBehavior.String() now return "invalid(N)" instead of panicking on out-of-range values - ScoreMultiplierConfig.Network/Method JSON tags now include omitempty to match YAML tag behavior Co-authored-by: Cursor <cursoragent@cursor.com> * feat: add --defaults flag to dump command When --defaults is passed, applies SetDefaults() to the parsed config before dumping. This shows the final resolved config with all eRPC defaults applied, which is what users want to see when validating their config against production behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: rename --defaults to --with-defaults in dump command The flag was incorrectly named 'defaults' but the PR description documents it as '--with-defaults', causing a mismatch between the documented CLI interface and actual implementation. * simplify: always apply defaults in dump, remove --with-defaults flag There's no use case for dumping raw config without defaults. The dump command now always applies SetDefaults before output. Co-authored-by: Cursor <cursoragent@cursor.com> * docs: add CLI commands section with validate and dump usage Co-authored-by: Cursor <cursoragent@cursor.com> * fix: add --with-defaults flag to dump command for optional defaults application * Revert "fix: add --with-defaults flag to dump command for optional defaults application" This reverts commit cd03bbc. * docs: update LoadConfigRaw comment to reflect dump always applies defaults Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: remove LoadConfigRaw, use LoadConfig everywhere There's no valid use case for loading config without defaults and validation. The dump command now goes through the same pipeline as production: load + SetDefaults + Validate. This ensures dump output reflects exactly what eRPC would accept at startup. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: simplify LoadConfig comment Co-authored-by: Cursor <cursoragent@cursor.com> * fix: redact secrets in YAML dump output, fix SelectionPolicyConfig MarshalJSON, reuse getConfig in dump - Add MarshalYAML methods for RedisConnectorConfig, PostgreSQLConnectorConfig, AwsAuthConfig, ProviderConfig, UpstreamConfig, and SecretStrategyConfig to redact sensitive fields (passwords, URIs, API keys) in YAML output, matching existing MarshalJSON redaction behavior - Fix SelectionPolicyConfig.MarshalJSON: use else-if so evalFunctionOriginal source is preserved instead of being overwritten by "<function>" - Refactor dump command to reuse getConfig() instead of duplicating config loading logic, enabling --config flag and default config path resolution - Regenerate TypeScript types after rebase onto main Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: include rateLimitBudget in SecretStrategyConfig.MarshalYAML Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Enables org members to comment /generate-diagram on any PR to get an auto-generated architecture diagram of the changes. Restricted to OWNER and MEMBER roles for security. Made-with: Cursor
…sus (erpc#825) ErrUpstreamsExhausted wraps the shared ErrorsByUpstream map via errors.Join in its Cause. When other consensus participants store execution reverts in this map, HasErrorCode traversal finds them and misclassifies the exhausted error as ResponseTypeConsensusError with a different hash, creating a phantom voting group that fragments consensus and can cause incorrect short-circuit decisions. Always classify ErrUpstreamsExhausted as infrastructure error regardless of wrapped errors, since it represents "no upstream was reachable" rather than an actual upstream response. Made-with: Cursor
* chore: replace logic-diagram with xray for PR architecture diffs Made-with: Cursor * chore: address review — pin to tag, add harden-runner, fork protection Made-with: Cursor * chore: use @main for xray during development Made-with: Cursor * chore: switch xray to OpenRouter Made-with: Cursor * fix: proper fork check via API instead of missing payload field Made-with: Cursor
* feat: add x402 nanopayment auth strategy
Add native x402 (HTTP 402 Payment Required) support as a new auth
strategy alongside existing secret/database/jwt/siwe/network strategies.
This enables pay-per-request RPC access via the x402 protocol — clients
without an API key can authenticate by paying with USDC through an x402
facilitator. The payer's wallet address becomes their eRPC user ID,
enabling per-payer rate limiting and metrics.
Inlines x402 protocol types and facilitator client (~180 lines) to avoid
heavy transitive dependencies from external x402 libraries.
Made-with: Cursor
* fix: x402 resource field, EIP-712 extra config, dead code cleanup
- Add RequestURL to AuthPayload so 402 responses include the resource
object (url, mimeType, description) required by Circle Gateway SDK
- Add Extra map to X402StrategyConfig for providing EIP-712 domain
params (name, version) when the facilitator doesn't supply them
- Merge config-level extra into payment requirements before facilitator
fetch, so facilitator values can override
- Change Resource field type to interface{} to support both string and
object formats across facilitators
- Include raw facilitator response in settle error for debugging
- Remove unused encodePaymentRequirementsHeader (dead code)
- Add TODO: settlement happens during auth before upstream forwarding;
should be deferred to post-response hook for production
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add x402 facilitator latency metrics and Grafana dashboard
Track verify/settle round-trip latency, request counts, and payment
outcomes per facilitator (circle, x402org) via Prometheus histograms
and counters. Includes a dedicated x402 Grafana dashboard with latency
percentiles, error rates, and payment success tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: expose /metrics on main HTTP port, add x402 Grafana dashboard
- Add /metrics route to the main HTTP server (port 4000) so external
Prometheus scrapers can reach metrics without a second public port
- Add x402 Grafana dashboard with facilitator latency, error rates,
and payment outcome panels
- Update monitoring Dockerfile and Prometheus config for Fly deployment
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: settle-after-response for upTo scheme, skip verify for exact
Two payment flows based on scheme:
- exact: settle during auth (skip verify per Circle guidance — verify
cannot guarantee funds due to race conditions). Single round-trip.
- upTo: defer settlement until after successful upstream response.
Payer is never charged for failed requests. Background retry with
exponential backoff if facilitator is temporarily unavailable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: remove x402 dashboard, fly.toml, and monitoring Dockerfile changes
These are deployment-specific artifacts that don't belong in the PR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: revert monitoring config changes (prometheus, grafana, dashboards)
Restore to main branch versions — these are deployment-specific.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: verify signature before serving upTo requests
The upTo scheme was extracting the payer address from the raw unverified
payload, allowing forged payments to get free RPC calls. Now calls the
facilitator verify endpoint first to validate the cryptographic signature.
Verify can't guarantee fund availability (Circle's known limitation),
but it catches forged/invalid signatures — which is the gate we need
before serving a request on credit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: settle upfront for both exact and upTo schemes
Remove deferred settlement for upTo — without a hold/lock mechanism in
the facilitator API, deferring means verify-only during auth, and verify
can't guarantee funds. An attacker with a valid signature but empty
wallet could send unlimited free requests.
Both schemes now settle during auth (skip verify, straight to submit).
Left a TODO for when facilitators add pre-auth/hold support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: test FailedVerification with VerifyOnly=true
The test was passing by accident — with VerifyOnly=false, it hit /settle
(unhandled 404) instead of the /verify invalid path it intended to test.
Now explicitly uses VerifyOnly=true and asserts /verify was called.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: deferred settlement for upto scheme (Permit2)
- upto: verify Permit2 signature during auth, settle after successful
Forward(). On upstream failure, don't settle — authorization expires
unused and user keeps their money.
- exact: unchanged, settle during auth (Circle recommended).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use detached context for upto settlement
The request context may be near expiration after a slow upstream
forward. Use a fresh 30s context so the settle HTTP call doesn't
fail with deadline exceeded.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove unauthenticated /metrics endpoint from main HTTP port
The /metrics handler was exposed on the main HTTP port without auth,
which is a security concern. Prometheus metrics should be scraped via
the dedicated admin/metrics port instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: move RequestURL from AuthPayload to X402Payload
RequestURL was on the generic AuthPayload struct and computed for every
request regardless of auth strategy. Move it into X402Payload where it
belongs, and only compute it when x402 headers are present.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: default x402Version to 2 for upto scheme
The upto scheme is a v2 feature in the x402 protocol. The server was
returning x402Version: 1 because the facilitator's /supported endpoint
doesn't list upto explicitly, causing the v2 client SDK to fail with
"No client registered for x402 version: 1".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add CDP JWT auth for x402 facilitator client
CDP facilitator (api.cdp.coinbase.com) requires Ed25519 JWT auth.
Added cdpApiKeyId and cdpApiKeySecret config fields. When set, the
facilitator client signs each request with a short-lived JWT.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: include x402Version in facilitator verify/settle requests
CDP facilitator requires x402Version in the request body. Without it,
requests fail with "property x402Version is missing".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: strip V1-only fields from V2 payment requirements for CDP
V2 x402 requirements must not include maxAmountRequired, description,
resource, or mimeType — these are V1-only fields. CDP rejects payloads
containing them. Also handle CDP's 400 responses with valid verify body
and add debug logging for verify requests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* debug: log resolved facilitator address at startup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* debug: log CDP verify response body
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: remove upto scheme and CDP auth — defer to later phase
CDP's hosted facilitator doesn't actually support upto/Permit2
verification despite advertising it in /supported. Strip all
upto-related code (deferred settlement, CDP JWT auth, V2 field
handling) to keep the PR focused on the working "exact" scheme
with Circle Gateway.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use facilitator-verified payer from settle response
settlePayment was discarding the X402SettlementResponse and
authenticateWithSettle fell back to extractPayerFromRaw — a fragile
heuristic on client-supplied data. Now settlePayment returns the
full response so we prefer the facilitator-verified Payer field.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: inject resource URL into payment payload for facilitator
Circle GatewayClient omits the resource field from payment headers
but Circle's facilitator requires paymentPayload.resource for
settlement. Inject the request URL when the client doesn't provide it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: send resource as object not string in payment payload
Circle facilitator expects paymentPayload.resource to be an object
with url/mimeType/description, not a plain URL string.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add x402 auth strategy docs, Grafana dashboard section
- Add x402 section to auth.mdx with config examples (yaml + ts),
facilitator explanation, and Grafana metrics reference
- Add "x402 Payments" row to Grafana dashboard template with three
panels: payment counts, facilitator requests, facilitator latency
- Rename "nanopayment" to "payment" in X402StrategyConfig comment
- Add upto scheme to roadmap checklist
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: restore import indentation in http_server.go
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: deep-copy Extra map in paymentRequirementsResponse
Shallow copy via copy() shares the Extra map reference with the
strategy's internal state. Deep-copy prevents downstream mutations
from corrupting config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…rpc#833) ApplyDirectiveDefaults is called twice per request: once in http_server (before EnrichFromHttp), again defensively in Network.Forward. The second call was overwriting directives the user explicitly set via query params (e.g. enforce-highest-block=false silently reverting to config default). Make it idempotent by early-returning when directives are already set. The existing three paths that create directives (ApplyDirectiveDefaults, EnrichFromHttp, SetDirectives) all run under r.Lock(), so the non-nil check is a safe proxy for "already initialized". Also avoids the symmetric footgun: if anyone ever reorders http_server to call EnrichFromHttp before ApplyDirectiveDefaults, defaults would have clobbered HTTP values; this guard makes the function order-agnostic. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: fork-safe xray with GitHub App token - pull_request_target: workflow runs from main (safe for forks) - GitHub App token: comments show as xray-pr bot, not github-actions - Removed fork check (no longer needed with pull_request_target) - Updated action ref to xray-pr/xray-pr@main Requires XRAY_APP_ID (variable) and XRAY_APP_PRIVATE_KEY (secret) to be configured in repo settings. Made-with: Cursor * ci: simplify — drop GitHub App token, keep fork-safe + GITHUB_TOKEN - pull_request_target for fork safety (workflow runs from main) - GITHUB_TOKEN for posting (no app setup needed) - xray-pr/xray-pr@main as action ref - No XRAY_APP_ID or XRAY_APP_PRIVATE_KEY required Made-with: Cursor * ci: revert to pull_request trigger — avoid pwn request vulnerability pull_request_target + checkout of PR head is a known attack vector. Revert to standard pull_request trigger which is safe by default: - Fork PRs: secrets not available, xray posts deterministic table only - Internal PRs: full output with diagram - /xray command: available for owners/members on any PR Made-with: Cursor * ci: skip auto-run on fork PRs, owners can still /xray Made-with: Cursor
* feat(metrics): config-driven label drop for histograms
Adds two optional metrics config fields:
- histogramDropLabels: list of labels to strip from every histogram
- histogramLabelOverrides: per-metric exceptions that re-add labels
Motivation: in large deployments the per-instance /metrics response can grow
past the managed scraper's sample/body-size limits due to high-cardinality
labels on histograms (buckets multiply the cross-product). Dropping a
label like "user" from histograms cuts series count significantly while
leaving per-user attribution intact on the corresponding counters
(erpc_upstream_request_total, erpc_network_request_received_total, etc.).
The overrides map lets operators keep a label on specific histograms where
dashboards depend on it — e.g. drop "user" globally but keep it on
network_request_duration_seconds for customer-facing p99 panels.
Example config:
metrics:
histogramDropLabels: [user, composite]
histogramLabelOverrides:
network_request_duration_seconds: [user]
Default behavior (no config) is unchanged: all current labels stay.
Implementation: a LabeledHistogram wrapper (telemetry/labeled_histogram.go)
holds the canonical label schema and filters positional values before
forwarding to the underlying HistogramVec. Call sites continue to pass the
full label set; the wrapper drops the filtered positions. ObserverHandle
now accepts a HistogramObservable interface so the existing caching path
works for both *prometheus.HistogramVec and *LabeledHistogram.
Three filter-aware histograms for now:
- upstream_request_duration_seconds
- network_request_duration_seconds
- network_evm_get_logs_range_requested
Other histograms (cache_*, consensus_*) remain plain HistogramVec since they
don't carry high-cardinality labels today; they can be migrated later if
needed by swapping their promauto.NewHistogramVec for NewLabeledHistogram.
* fix(metrics): address cursor-bot review — nil guard + cache dedup
1. SetHistogramBuckets: on bucket-parse failure, fall through with
DefaultHistogramBuckets so the three filter-aware histograms
(upstream_request_duration_seconds, network_request_duration_seconds,
network_evm_get_logs_range_requested) are always initialized. Previously
MetricNetworkEvmGetLogsRangeRequested was package-init'd via promauto and
thus always non-nil; moving it into SetHistogramBuckets introduced a
nil-deref risk when the caller only logs the parse error. The parse error
is still returned for logging.
2. ObserverHandle: when hv is *LabeledHistogram, build the cache key from
post-filter label values via LabeledHistogram.ActiveLabelValues.
Multiple full-label tuples that resolve to the same underlying observer
now share one cache entry instead of one per unfiltered tuple, matching
the cardinality of the underlying HistogramVec.
* test(metrics): verify label-filter cardinality and body-size reduction
With 50 users × 10 networks × 5 upstreams (2500 combos), baseline /metrics
is ~7.4 MB / 40k lines. Dropping "user" globally shrinks to 138 KB / 810
lines (-98%). Dropping "user" while overriding to keep it on
network_request_duration_seconds preserves that one metric's cardinality
and reduces the rest, landing at 3.2 MB / 17k lines (-56%).
Asserts invariants so regressions in the filter path fail the suite.
* refactor(metrics): route every histogram through the label filter
Previously only 3 histograms (upstream_request_duration_seconds,
network_request_duration_seconds, network_evm_get_logs_range_requested)
were filter-aware. The other 10 — cache_*, consensus_*, hedge_delay,
x402_facilitator_* — continued to use promauto.NewHistogramVec and
silently ignored histogramDropLabels / histogramLabelOverrides.
Consolidate so the filter applies uniformly:
- Add telemetry.RegisterOrReplaceHistogram: a one-liner that unregisters
the old vec (if any), builds a LabeledHistogram under the current filter,
and registers it with prometheus.DefaultRegisterer.
- Move every histogram declaration out of the package-init var block and
into SetHistogramBuckets, using RegisterOrReplaceHistogram for each.
- Add a package init() that calls SetHistogramBuckets("") so tests and any
code observing before the binary's explicit init see non-nil vecs.
Net effect: histogramDropLabels now affects ALL 13 histograms, not just 3.
Call sites are unchanged — the helper preserves the HistogramVec API via
LabeledHistogram.WithLabelValues.
* test(metrics): verify filter applies uniformly to all 13 histograms
Adds TestHistogramLabelFilter_AllHistogramsObeyFilter that drops a shared
label ("category") and asserts every histogram carrying it shrinks
proportionally. x402_facilitator_request_duration_seconds (which has no
"category" label) must stay identical, proving the filter is precise and
not global-string-replace.
Companion to the existing SizeAndCardinality test which covers the three
"user"-carrying histograms.
* chore(metrics): remove redundant comments
…pc#842) erpc.Init calls SetHistogramLabelFilter then SetHistogramBuckets. Under the original design the package-level init() pre-registered all histograms with the empty default filter, so when SetHistogramBuckets re-registered them with the configured (filter-applied) label set, prometheus panicked because dimHashesByName is retained across Unregister by design. Local tests didn't catch it because each test swapped prometheus.DefaultRegisterer to a fresh registry first, which clears dimHashesByName and masks the conflict. Production runs against the live default registry that init() already populated. Fix: - init() now builds LabeledHistogram wrappers WITHOUT registering them (via buildFilterAwareHistograms). Metric globals are non-nil for tests and early-startup code that might observe before erpc.Init. - SetHistogramBuckets is the single authoritative registration point. For each histogram it uses registerOrReuse, which calls prometheus.Register and, on AlreadyRegisteredError (same name + same labels), returns the existing collector. This makes SetHistogramBuckets idempotent for same-filter calls on the same registry — fixing previously-passing tests that call it more than once. - Label-set changes post-registration still panic, by design: prometheus disallows them, and silently dropping the change would be worse. Add TestProductionFlow_SetFilterThenRegister which does NOT swap the registry — exactly the regression we missed.
…pe (erpc#843) The network-scope retry predicate called IsRetryableTowardNetwork but only returned early when it was true. Explicit opt-outs via WithRetryableTowardNetwork(false) (sendRawTransaction errors, vendor error normalizers, deterministic client failures) fell through to the default "err != nil → retry" rule and were silently re-attempted on another upstream. Rather than add a parallel helper, tighten IsRetryableTowardNetwork itself so a single function gives the call site an unambiguous answer: - Multi-error wrapper causes (ErrUpstreamsExhausted, ErrConsensusDispute, ErrConsensusLowParticipants, or any errors.Join bundle) are traversed by explicit iteration — retry if ANY child is retryable. Previously only ErrUpstreamsExhausted got this treatment; the others silently fell into the DeepSearch path and could be poisoned by a single child's flag with non-deterministic outcome (child iteration via sync.Map.Range is unordered). - The top-level flag lookup no longer uses DeepSearch. It checks only the outermost error's Details, which is where all production callers of WithRetryableTowardNetwork(false) actually apply the flag. - Empty ErrUpstreamsExhausted (no cause) remains terminal — preserves the existing contract covered by TestIsRetryableTowardNetwork_EmptyUpstreamsExhausted. With these semantics the failsafe predicate can trust the function and return its result directly: no dual-function dance, no silent fallthrough. Adds three regression tests: single explicit opt-out stops after one attempt, a mixed exhausted bundle (one non-retryable child, one plain error) still retries to MaxAttempts, and an all-non-retryable exhausted bundle stops after one attempt.
Some chains deviate from client assumptions in ways that make specific
RPC requests unanswerable. For example, a chain whose genesis block is
at height 1 has no valid response for eth_getBlockByNumber("0x0", false);
clients that probe block 0 see upstream errors or divergent results, and
the upstream may be flagged as misbehaving.
Add a per-network `staticResponses` list. When an inbound request matches
a configured (method, params) entry, the stub response is returned and
no upstream is contacted. Both result-shaped and error-shaped stubs are
supported. Matching uses recursive deep equality that tolerates numeric
type divergence between YAML config and JSON request decoding and is
order-independent for map keys.
The short-circuit fires after method extraction and before the
multiplexer / cache / upstream selection in Network.Forward, and echoes
the inbound request id. Hits are counted by the new metric
erpc_network_static_response_served_total{project,network,category}.
Internal state pollers are unaffected.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…pc#846) The response-ID normalization that preserves the client's original request ID was gated by: switch n.Architecture() { case common.ArchitectureEvm: ... } That meant any non-EVM JSON-RPC architecture leaked whatever ID the upstream echoed back — including upstreams that renumber or multiplex IDs toward themselves. Clients submitting id=1 could receive a response with id=99 for any non-EVM architecture, violating the JSON-RPC 2.0 expectation that the response id matches the request id. Drop the architecture gate so the rewrite applies to every JSON-RPC architecture. EVM behavior is unchanged; non-EVM architectures now get the correct behavior they were silently missing. Adds a test asserting the rewrite fires for both EVM and a non-EVM JSON-RPC architecture, and that a nil response is a no-op.
* feat(auth): merge PaymentRequired Accepts across x402 strategies When multiple x402 strategies are configured (e.g. one per accepted chain), each returns its own ErrPaymentRequired with a single-entry Accepts list. The registry currently returns the first such error verbatim, so the 402 response only ever advertises the first strategy's chain. SDK clients then sign payments only for that chain, even though every other strategy would have accepted them. Concatenate Accepts arrays from all PaymentRequired errors into one combined response. Falls back to the first error if any payload isn't an X402PaymentRequirementsResponse. Tests cover single-error pass-through, multi-error concatenation order, and the foreign-payload fallback path. * test(auth): add registry-level integration test + Resource/Error preservation - Authenticate-level test: configures two real x402 strategies and verifies unauthenticated flow returns one merged ErrPaymentRequired with both networks in Accepts (covers the wiring, not just the helper). - Header-fields test: verifies X402Version/Error/Resource come from the first error when merging.
* fix(json-rpc): preserve verbatim request id bytes through response
The request id was parsed via interface{} (Go decodes JSON numbers as
float64) then cast to int64, which silently truncated:
- integers above 2^53 (e.g. nanosecond timestamps used by indexers)
- fractional ids (uncommon but legal per JSON-RPC spec)
Capture the original id bytes during UnmarshalJSON and round-trip them
verbatim on the response. Programmatically-built requests fall back to
the existing typed-id path. Existing small-int and string ids are
unchanged byte-for-byte.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(json-rpc): consolidate id preservation into SetIDBytes; fix Clone/SetID gaps
Addresses review feedback on PR erpc#851:
1. @aramalipoor — "Why don't we fix/improve SetIDBytes itself?"
Right call. Removed the SetIDBytesPreserving parallel API and fixed
the root cause in parseID(): it was overwriting r.idBytes with a
re-marshalled int64 after parsing, which destroyed the byte
fidelity for ids outside the int53 safe range. Drop that overwrite.
Now SetIDBytes is preservation-correct by construction; the wire
output (WriteTo uses idBytes verbatim) round-trips losslessly.
Side fix: the call chain SetIDBytes → parseID had a latent
recursive-lock deadlock on r.idMu. Split parseID into a locked
public version and parseIDLocked for already-locked callers.
2. cursorai bot — "Clone() drops new idRaw field from copy"
Real gap. JsonRpcRequest.Clone() now propagates idRaw so cloned
requests still round-trip the id byte-for-byte. Added test:
TestJsonRpcRequest_Clone_PropagatesIDRaw.
3. cursorai bot — "SetID doesn't clear stale idRaw bytes"
Real footgun. JsonRpcRequest.SetID() now clears idRaw so the new
typed id wins on the response path (which prefers IDRawBytes).
Added test: TestJsonRpcRequest_SetID_ClearsStaleIDRaw.
All existing tests still pass (6 byte-fidelity sub-cases, 7 IDRawBytes
sub-cases, common + EVM full sweep).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: add DVN-ready config preset Minimal self-hosted preset with unanimous consensus on the methods DVN verification depends on (eth_getLogs, eth_getBlockByNumber, eth_getTransactionReceipt, eth_getBlockReceipts), preferNonEmpty + preferLargerResponses, returnError on disputes, and misbehavior export. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: introduce top-level Presets section Move dvn-ready out of Config (it's a scenario preset, not a config primitive). Adds room for additional presets without bloating Config. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(presets): drop KelpDAO incident intro paragraph Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: nest Presets under Config Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(presets/dvn-ready): drop fabricated cache-disable block methods.enforceMethodCompatibility is not a real config key; cache control lives on the database side and the minimal preset doesn't configure caching at all. Removed the misleading row from the table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(presets/dvn-ready): drop specific provider names Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: flatten dvn-ready directly under config Drop the extra config/presets/ subdir; render Presets as a sidebar separator inside the Config section instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: nest dvn-ready as collapsible under erpc.yaml/ts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: restore Presets group with DVN-ready inside Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: position Presets directly under erpc.yaml/ts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: add Presets parent page to fix sidebar ordering Nextra sorts folder-only entries (no parent .mdx) at the end regardless of _meta.js order. Adding config/presets.mdx as a small landing page makes the Presets section honor its position right under erpc.yaml/ts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(presets/dvn-ready): drop run-yourself callout Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(presets/dvn-ready): rename to DVN (LayerZero) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: rename Presets section to Examples Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(presets/dvn-ready): bump maxAttempts to 3 on verification policy Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ods (erpc#867) An empty array `[]` is the legitimate response for eth_getBlockReceipts on a block with zero transactions. Including this method in the default MarkEmptyAsErrorMethods list causes the post-forward hook to convert correct empty responses into ErrEndpointMissingData, which has two unwanted consequences: 1. The retry policy's `err != nil` branch is taken instead of the empty-result branch, bypassing `emptyResultAccept` entirely. So even operators who explicitly opt in via `emptyResultAccept: ['eth_getBlockReceipts']` cannot prevent the retry storm. 2. Each retry uses `emptyResultDelay` per the delay path. With reasonable operator settings (e.g. `emptyResultDelay: 4s, maxAttempts: 3`), the total retry budget can exceed the network outer timeout floor (e.g. 6s), making the retry chain unable to exhaust before the deadline fires. The result is a `caller_abandoned` outcome on every 0-tx-block request — a deterministic correctness failure that surfaces as a timeout to the client. This mirrors the existing rationale for excluding eth_getTransactionReceipt ("pending txs correctly return null") — empty receipts are a legitimate chain state, not missing data. Operators who specifically need missing-data behavior on this method can still opt in via the network's `markEmptyAsErrorMethods` config field. Tests: - hooks_test.go updated: eth_getBlockReceipts moved from "ListedMethods" to "NonListedMethods" (no longer triggers conversion by default). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Full-stack reproduction of the 2026-06-12 zkSync incident: real eRPC HTTP server + WS upstream whose TCP connection is killed with no close handshake. Asserts eRPC re-dials, re-subscribes with a fresh upstream sub ID, resumes delivering heads to the already-connected client on its original subscription ID, and accepts new client subscriptions — all with no process restart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per review feedback (health reporter excessive vs ping/pong), drop the observability layer and keep only the fixes required to prevent the incident: - revert HealthReporter/IngressHealth/LastHead (indexer), healthcheck subscriptions block, newHeads refusal gate + ErrNoLiveSubscriptionSource, and the last-head-timestamp metric; delete their tests. Kept: client liveness, resubscribe retry loop, breaker default fix, and the erpc_upstream_websocket_connected gauge. Race fixes from review: - snapshot wsPingInterval/wsPongWait (and resub backoff bounds) into per-client/per-adapter fields at construction; goroutines no longer read package vars, fixing the -race failure where test cleanup restored them while a prior client's pingLoop was still running - pingLoop: write the ping to an explicit conn and compare-and-close via teardownConn, so a ping failure can no longer close a fresh connection that readLoop already re-dialed - subscribeNewHeads/subscribeFilter: check epoch ctx under subsMu before committing a sub ID, so a cancelled epoch's in-flight subscribe can't unregister the live handler installed by the newer epoch - Stop marks the adapter stopped under resubMu so a reconnect callback racing Stop can't start a new resubscribe epoch Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…edge) Root cause of the internal-eRPC "probe/selection wedge" (incident 2026-06-24). Breaker.Record returned early for OutcomeIgnore BEFORE the HalfOpen branch that releases the trial permit (halfOpenInflight--). A HalfOpen trial reserves a permit in TryAcquirePermit; if that trial resolves as ignorable (timeout, cancellation, soft error — precisely what a transient redis/upstream blip produces), the permit was never released. After enough such trials halfOpenInflight saturates the trial capacity, every subsequent TryAcquirePermit in HalfOpen is denied, and the breaker wedges open indefinitely — failing real traffic AND the selection-recovery probes (which are breaker-eligible). The upstream can never re-admit, so eRPC serves no healthy upstream for the chain until the pods are rollout-restarted (which resets in-memory breaker state). Evidence: during a wedge the selection-probe error RATIO sits at ~1.0 (every probe denied) sustained for tens of minutes across multiple chains at once, recovering within minutes of a restart; onset correlates with redis-haproxy churn. Fix: on OutcomeIgnore, still release a reserved HalfOpen trial permit (without counting it as success/failure — the trial was inconclusive). Minimal, behaviour- preserving for Closed/Open. Test: breaker_test.go reproduces the leak — without the fix the breaker "wedges after 0 ignored trials" (halfOpenInflight leaks to 1 and TryAcquirePermit denies); with the fix, repeated ignored trials never wedge and a later success still closes. Follow-ups (separate): mark selection-recovery probes breaker-ineligible so a genuinely-open breaker can't blind its own recovery probe; bounded HalfOpen dwell / cordon TTL as defence-in-depth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(failsafe): release half-open permit on ignored outcome (breaker wedge / probe-selection wedge root cause)
… is fatal (backport erpc#973) Backports upstream erpc commit a04655e (PR erpc#973) verbatim, adapted only for this fork's attemptRemainingTasks signature. The auto-retry loop exited permanently when State() returned Fatal, which happens when ANY single task is fatal (e.g. one misconfigured upstream with a chainId mismatch). Every transiently-failed sibling task was then abandoned until process restart. In production this meant an upstream that blipped during a daemonset rollout (robinhood-mainnet) stayed unregistered (network n/a) for two days while all its traffic escaped to a paid 3P fallback. The loop now stops only when every task is terminal (succeeded or fatal); a fatal task is terminal for itself only.
…sk defence) Upstream (erpc#973) still waits unbounded on WaitForTasks each retry round: one task hung inside its Fn (e.g. a client dial that ignores ctx and never returns) stays Running forever and blocks the retry loop for every other task. Bound each round's wait by TaskTimeout so a hung task only delays a round, never stops retries. Candidate for upstreaming.
…lock-then-wait deadlock) Stop() held tasksMu while waiting for the auto-retry goroutine, which itself acquires tasksMu inside attemptRemainingTasks. If the loop was blocked acquiring the mutex when the cancel landed, Stop waited forever. Cancel-and-wait now happens before taking the mutex. Present upstream as well; candidate for upstreaming.
Prevent HTTP "latest"/eth_blockNumber from regressing below a head already delivered on WS, which trips Chainlink MultiNode FinalizedBlockOutOfSync. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
- Reap Running tasks past TaskTimeout to TimedOut so ignore-ctx Fns cannot wedge hasPendingWork forever; completion uses attempt-ID + CAS so a late return cannot clobber a newer retry. - State(): one fatal sibling no longer maps the whole Initializer to Fatal while others succeeded/recover — mix is Partial; all-fatal stays Fatal. - waitForTasks waits in parallel and distinguishes wait-context abort from a task that already finished as TimedOut; Wait surfaces TimedOut/Fatal errors via lastErr. Co-authored-by: Cursor <cursoragent@cursor.com>
fix(initializer): keep auto-retry alive for transient tasks despite fatal or hung siblings
…test fix(ws): advance network latest tip before newHeads fan-out
TipHW alone still allowed eth_getBlockByNumber("latest") to fail-open to a
stale upstream block when re-fetch of the WS tip missed. Cache the newHeads
header before fan-out and prefer it in EnforceHighestBlock for header-only
requests so HTTP cannot regress below a tip already delivered on the pod.
Co-authored-by: Cursor <cursoragent@cursor.com>
Record the tip-source upstream id with the cached newHeads head, partition
eth_getBlockByNumber("latest") using TipHW so WS upstreams are tried first,
and pin EnforceHighestBlock re-fetch to that tip source instead of only
excluding the stale HTTP responder.
Co-authored-by: Cursor <cursoragent@cursor.com>
Tip ownership already lives on per-upstream SuggestLatestBlock/LatestBlock and partitionUpstreamsByLatestBlock. Keep TipHW partitioning for "latest" plus the header cache floor; remove the redundant upstreamId pin registry. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the newHeads header-cache / tip-source registry overbuild. Tip ownership already lives on per-upstream pollers (SuggestLatestBlock) and EvmLeaderUpstream; EnforceHighestBlock now UseUpstream-pins the concrete tip re-fetch to that leader when its LatestBlock covers TipHW. Co-authored-by: Cursor <cursoragent@cursor.com>
Availability checks and EnforceHighestBlock were calling PollLatestBlockNumber, which can reuse a debounced tip behind network TipHW (WS/Redis), falsely rejecting eth_call and fail-opening stale latest. Add PollLatestBlockNumberNow and use it on those paths; force-poll the leader before pinning the tip re-fetch. Co-authored-by: Cursor <cursoragent@cursor.com>
EnforceHighestBlock used pickHighestBlock, which fail-opened to a lagging "latest" when the concrete TipHW fetch returned null/error. That is the MultiNode FOOS trigger after WS newHeads already delivered the higher head. Re-fetch tip (leader pin, then unconstrained), accept only responses that meet the tip floor, and return an error instead of stale. Also fail-open the per-upstream availability gate when poller lags TipHW, and only skip enforcement for cached latest that already meets the tip. Co-authored-by: Cursor <cursoragent@cursor.com>
Cross-pod TipHW Redis push was async, so sibling pods could still serve a lower HTTP latest after WS newHeads advanced highestUserObservations, silently demoting MultiNode via FOOS. Publish TipHW before fan-out and refresh from Redis when local TipHW would skip EnforceHighestBlock. Co-authored-by: Cursor <cursoragent@cursor.com>
If TipHW advanced from a WS newHeads observation on this pod, HTTP
eth_getBlockByNumber("latest", false) must return that header instead of
hard-failing when concrete tip re-fetch cannot reach TipHW yet.
Co-authored-by: Cursor <cursoragent@cursor.com>
Fallback newHeads must not advance TipHW while primaries are up (same rule as poller aggregation). Also let the per-request fallback escape hatch fire when primaries return null/emptyish for a concrete tip block, so tip re-fetch can use public fallbacks instead of hard-failing. Co-authored-by: Cursor <cursoragent@cursor.com>
That path masked TipHW inflation / missing failover. Keep TipHW floor from primary WS only, refuse-stale when tip re-fetch misses, and escape to fallbacks on emptyish primary misses. Co-authored-by: Cursor <cursoragent@cursor.com>
Skipping TipHW for fallback WS while Ingest still fans those heads out caused MultiNode FOOS on matic. TipHW must cover every delivered head; tip re-fetch of a fallback-advanced TipHW relies on the emptyish escape hatch to reach public upstreams when primaries miss. Co-authored-by: Cursor <cursoragent@cursor.com>
…hed-head fix(ws): refuse stale latest below TipHW (FOOS)
Direct tip/tip+1 getBlock reads set UseUpstream to the primary leader (typically the WS ingress advanced by SuggestLatestBlock) on first forward — same idea as PR11 tip re-fetch pin, so the lagging sibling is not tried first. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop helper + dedicated HTTP test; keep only the small UseUpstream pin next to partitionUpstreamsByLatestBlock. Co-authored-by: Cursor <cursoragent@cursor.com>
Flatten nested Forward pin into early-return helper; assert lagging sibling gets zero hits on concrete tip getBlock. Co-authored-by: Cursor <cursoragent@cursor.com>
…vm-leader fix(ws): pin near-tip getBlock to EvmLeaderUpstream
Cloudflare always sends Accept-Encoding: gzip. gzipHandler wrapped the writer in conditionalGzipWriter, which does not implement http.Hijacker, so gorilla Upgrade failed with 500. Mirror TimeoutHandler and pass the raw writer through for WS handshakes. Co-authored-by: Cursor <cursoragent@cursor.com>
xray — see through AI slop with deterministic architecture PR diff reviews |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Accept-Encoding: gzip, which causedgzipHandlerto wrap theResponseWriterinconditionalGzipWriter.http.Hijacker, so gorilla WebSocketUpgradefailed with500 Internal Server Error/response does not implement http.Hijacker.TimeoutHandler). Direct origin without gzip worked; CF-proxied WS failed.Image
docker.io/azgms/erpc:ws-gzip-hijacker-fix1(digestsha256:b29dc6e0c51c90d8c8d8735f2143b3f8fb403e95907bf49e0d40ec6c0fd7ab74)9e39885/ pin-near-tip-getblock). Could not push tobashingscripts/erpcwith current Docker Hub login (azgms).Test plan
go test ./erpc/ -run TestGzipHandler_SkipsWebSocketUpgradecurlWS withAccept-Encoding: gzipvia CF → expect101Made with Cursor