chore: KEEP-1291 remove dead fee-escalation code and alert on stuck pending transactions

[suisuss]

Linear: KEEP-1291

What this does

lib/web3/gas-strategy.ts carried a complete same-nonce fee-escalation implementation that nothing called. This removes it, corrects the two places that advertised the capability as if it existed, and adds the visibility that was the actual reason to delete it rather than wire it up.

Deleted

From lib/web3/gas-strategy.ts: RetryConfig, DEFAULT_RETRY_CONFIG, TransactionStuckError, waitForConfirmation, executeWithRetry, plus the sleep and logUserError imports they were the only users of.

Verification that nothing imported them: a repo-wide grep for each of the five symbols (excluding node_modules, .next, and other worktrees) returned matches only inside lib/web3/gas-strategy.ts itself and inside tests/unit/gas-strategy.test.ts.

Two names need disambiguating, and neither is touched here:

• executeWithRetry in app/api/execute/_lib/retry.ts is a different, live function with its own RetryConfig from app/api/execute/_lib/types.ts. It is used by app/api/execute/node/route.ts and covered by tests/unit/retry.test.ts.
• waitForConfirmation in plugins/tempo/steps/tempo-tx-core.ts is an unrelated boolean parameter.

The deleted tests were the describe("executeWithRetry types and config") block. Its four cases asserted that a config object equals itself, that a class is a class, that an error subclasses Error, and that a function has type "function". There was no behaviour to preserve. The remaining 38 tests in that file pass.

Corrected

Two comments on pending_transactions in lib/db/schema-extensions.ts described automated gas bumping that does not exist: "Detect stuck transactions that may need gas bumping" on the table and "for stuck tx analysis" on gas_price. Both now describe what the table and column are actually for, and point at the gauge below as the only thing that notices a stuck transaction.

Added

keeperhub_web3_pending_transactions_stuck, a gauge counting rows in pending_transactions still status = 'pending' more than 15 minutes after submitted_at, labelled chain_id.

Nothing acts on it automatically. Deleting escalation code is only defensible if the condition it claimed to handle stops being invisible, so this gauge is the replacement: a stuck transaction remains a human decision, but the backlog now pages instead of failing silently. The Grafana alert rule lands separately in the infra repo: techops-services/infrastructure#344 (draft), which alerts on max by (chain_id) (keeperhub_web3_pending_transactions_stuck{cluster="techops-prod", namespace="keeperhub"}) > 0 for 5m at warning severity. That rule must not be applied until this PR is merged and deployed and the series is confirmed present in the grafanacloud-techopsservices-prom datasource, so the two should land in that order.

Counted in SQL alone. Establishing that a row is genuinely stuck rather than merely old means comparing its nonce against the chain's current nonce, which is one RPC call per wallet and too expensive for a scrape path; that version is KEEP-1315. This query therefore over-counts rows the reconciler has not yet reaped, which is the safe direction for an alert.

Where it is emitted, and why not the cron route

It is a DB-sourced gauge declared on dbRegistry in lib/metrics/collectors/prometheus.ts and set inside refreshDbMetricsNow(), the scrape-time refresh, from a new getStuckPendingTransactionCountsFromDb() in lib/metrics/db-metrics.ts. That is the same registry and the same function as keeperhub_workflow_queue_depth, which has live series in prod.

The alternative considered was emitting it from the execution-reconciler cron route via setGauge. That does not work here. Prometheus gauges are process-local; the app runs replicaCount: 4 in prod, and the CronJob hits one arbitrary pod per run. Three of four pods would report nothing while the fourth held a value that never refreshed, so max by (chain_id) would latch high forever and sum would multiply. METRICS_REFERENCE.md already codifies the rule this would break: DB-sourced gauges must report identical values from every pod. The DB-sourced path satisfies it because every scrape recomputes from the same database.

Consequently there is no MetricNames entry. MetricNames feeds setGauge, which this gauge does not use, and DB-sourced gauges such as keeperhub_executions_unconfirmed have no entry either. Adding one would create exactly the kind of unreferenced export this PR removes.

Verification

• tsc --noEmit: clean.
• biome check on all changed files: clean.
• vitest run over the 9 affected unit files: 153 passed.
• Three new cases in tests/unit/db-metrics-cache.test.ts cover the gauge end to end through the real prom-client registry, mocking only the DB query: one series per chain, series dropped once a chain's backlog clears (so the alert can recover), and last value held when the query returns null (so a query error does not report a misleading 0). getStuckPendingTransactionCountsFromDb was also added to that file's default mock map, which every updateDbMetrics test depends on.

Rendering the registry confirms the exposition is what the infra alert expects:

# HELP keeperhub_web3_pending_transactions_stuck Pending transactions still unconfirmed more than 15 minutes after submission, by chain_id
# TYPE keeperhub_web3_pending_transactions_stuck gauge
keeperhub_web3_pending_transactions_stuck{chain_id="1"} 4
keeperhub_web3_pending_transactions_stuck{chain_id="8453"} 2

Note for reviewers

keeperhub_executions_unconfirmed currently has zero series in prod. That is not a broken emitter: executionsUnconfirmed.set() is called in refreshDbMetricsNow() alongside the gauge added here. It was added yesterday in d4552a3 (KEEP-1282) and has not reached prod yet. The two gauges share a code path, so both should appear on the same deploy.

Index for the new query (second commit)

metrics-db-review-gate correctly flagged the first commit. The gauge query filters pending_transactions on status and submitted_at with no wallet_address, so idx_pending_tx_status - which leads on wallet_address - cannot serve it, and nothing prunes the table: there is no DELETE against pending_transactions anywhere in the codebase. Left alone the query would be a sequential scan growing with lifetime transaction volume, executed on every DB-metrics refresh. That is the regression class the gate exists to catch.

drizzle/0150_keep_1291_pending_tx_stuck_index.sql adds idx_pending_tx_stuck, partial on status = 'pending' so it stays the size of the in-flight set rather than the table, leading on submitted_at so the age predicate is a bounded range scan, with chain_id included so the GROUP BY is satisfied from the index.

Measured on Postgres 16 with 200k rows, 200 of them pending:

	Plan	Time	Buffers
With idx_pending_tx_stuck	Index Only Scan	0.39 ms	201
Without	Parallel Seq Scan	32.7 ms	3709

The gap widens as the table grows, which it does monotonically.

drizzle-kit generate produced the migration and snapshot cleanly, and a follow-up generate reports no drift. The migration was applied against a real Postgres 16 to confirm it runs.

Two gates need a human

• db-prep-check is red by design. The migration carries -- @requires-db-prep on line 1 and uses CREATE INDEX IF NOT EXISTS, because the bare statement drizzle-kit emits takes an ACCESS EXCLUSIVE lock. Before merge an operator must build the index without holding a write lock and then apply db-prepped-staging. The exact statement and its validity check are in the migration header.
• metrics-db-review-gate needs the metrics-db-reviewed label. The evidence for that attestation is the measurement above; I was not able to apply the label myself.

[joelorzet]

Four items to correct before merge. One is a defect, three are stale or wrong references.

1. The stuck-transaction gauge cannot recover on its own

lib/metrics/db-metrics.ts:432

The query is bounded only on the young side (submitted_at < now() - 15 minutes). Nothing erases rows from pending_transactions. validateAndReconcile leaves a row in pending when a different RPC endpoint answered. Reconciliation runs only at workflow start, and only for that exact wallet and chain.

As a result, one orphan row from an abandoned wallet holds keeperhub_web3_pending_transactions_stuck{chain_id} at 1 or more forever. The infra rule > 0 for 5m then fires permanently, and no automatic path clears it. The test "drops a chain's series once its backlog clears" passes only because the DB is mocked.

The same accumulation weakens the premise that the index stays the size of the in-flight set.

Add an upper bound to the query, or add a path that resolves rows that the reconciler will never revisit.

2. The schema comment describes code that no longer exists

lib/db/schema-extensions.ts:542

The parenthetical describes GasBumpOverrides, gasBumpMultiplier, and an overrides argument in app/api/execute/_lib/retry.ts. Staging removed all three after the merge base of this branch. The comment is false on merge.

3. The metrics reference repeats the same stale claim

lib/metrics/METRICS_REFERENCE.md:125

4. The migration header cites a file that is not in the repo

drizzle/0150_keep_1291_pending_tx_stuck_index.sql:14

The header cites docs/incidents/2026-05-29-db-cpu-spike.md. This path does not exist. It comes from metrics-db-review-gate.yml.

Checked and correct

• The removed gas-strategy.ts symbols (RetryConfig, DEFAULT_RETRY_CONFIG, TransactionStuckError, waitForConfirmation, executeWithRetry) have no importers left. app/api/execute/_lib/retry.ts and the tempo waitForConfirmation are unrelated. The removal of the sleep and logUserError imports is correct.
• The Promise.all order in refreshDbMetricsNow matches the destructuring. lt, count, and pendingTransactions are imported. The metricsDb pool is used.
• Migration, journal, and snapshot agree. The journal when value is monotonic. 0150_snapshot.prevId matches 0149_snapshot.id. The snapshot index entry matches the schema declaration. Staging has no schema drift since the merge base.
• I ran the DDL against Postgres 16. The qualified partial predicate WHERE "pending_transactions"."status" = 'pending' deparses to the same index as the unqualified CONCURRENTLY form in the runbook, so the IF NOT EXISTS no-op holds.
• Partial-index plan risk with postgres.js prepared statements: under plan_cache_mode=force_generic_plan the index is unusable and the planner falls back to a parallel sequential scan. With the default auto setting Postgres keeps the custom plan and does an index-only scan. This is not a defect, but it is worth knowing.
• The test mock map, the DB_METRICS_CACHE_TTL_MS=0 path, and the gauge reset() semantics are all correct.

Process

• The PR title and several new code comments carry Linear ticket IDs. The project CLAUDE.md forbids them in PR titles and in code comments. The existing metrics code already does this, so this is a pattern to stop, not a new mistake.
• Merge staging into the branch before it lands. That merge is also what surfaces items 2 and 3.