Use native backend proof for mining status

[jeremyharkness]

Summary

• makes catch-up pause/status policy honor native backend mining-safe proof
• collects Prometheus metrics even in no-log status-sampler mode
• parses live pool backend metrics labeled with node=...
• lowers default dashboard global RPC worker count from 24 to 4

Tests

• python3 -m unittest ops.tests.test_pool_efficiency_loss_ledger ops.tests.test_no_miner_collect_status ops.tests.test_chain_rpc_resilience ops.tests.test_status_sampler_mining_imperative -v

Live validation

• restarted status-sampler only
• status-sampler reported overall=ok, mode=mining, can_mine=true during soak
• 5-minute measurement post-cached-template-evm-advisory stayed in mode_values=[mining]

[jeremyharkness]

Update from live stack validation on 2026-06-21:

• repair_hold can now allow only explicitly allow-listed pool/ASIC recovery actions; node restart, node/container recreate, and config edits remain blocked.
• Added stack defaults/docs for POOL_PAYOUT_MAX_BLOCKS_PER_TICK=5 and documented that payout submission waits for native backend mining safety.
• Validation: 519 ops tests passed, stack defaults validated, git diff --check passed. Release readiness still fails correctly because getBlockTemplate returns Node is Syncing.

[jeremyharkness]

Update from live catch-up optimization on 2026-06-21:

• Added stack default POOL_STRATUM_CONNECTION_LOG_INTERVAL_SECONDS=60 and documented the Stratum retry log-flood mitigation.
• Live pool container has POOL_STRATUM_CONNECTION_LOG_INTERVAL_SECONDS=60 and POOL_PAYOUT_MAX_BLOCKS_PER_TICK=5. Node was not restarted.
• Validation: 519 ops tests passed, stack defaults validated with 176 canonical values, git diff --check passed. Live stack remains correctly fail-closed in catchup_pause with can_submit_blocks=false and Node is Syncing.

[jeremyharkness]

Deployed and verified final mining-appliance runtime hardening.\n\nChanges pushed in 6652731:\n- Default node runtime to low-noise logging: BDAG_NODE_DEBUG_LEVEL=warn and BDAG_NODE_NO_FILE_LOGGING=1, appended by the nodeworker entrypoint.\n- Quote .env values with whitespace in release/runtime writers so Compose env remains shell-sourceable where scripts need it.\n- Add preflight evidence for node logging pressure and warn on trace/debug/info or file logging.\n- Add BDAG_EVM_SYNC_BACKOFF_SECONDS=60 to compose/defaults/env example and docs for redis-dash EVM sync cooldown.\n- Document the observed installation/mining failures and mitigations.\n\nVerification:\n- bash -n docker/entrypoint-nodeworker.sh ops/release-install.sh scripts/release/installers/install-unix-common.sh\n- python3 -m py_compile ops/status_sampler.py scripts/mining-appliance-preflight.py\n- python3 -m unittest ops.tests.test_nodeworker_entrypoint ops.tests.test_mining_appliance_preflight ops.tests.test_status_sampler_mining_imperative\n- python3 -m unittest discover -s ops/tests -p 'test_*.py'\n- python3 -m unittest discover -s scripts -p '*test.py'\n- python3 scripts/validate-stack-defaults.py\n- docker compose config --quiet\n- git diff --check\n\nLive deployment:\n- Rebuilt stack-dashboard and stack-status-sampler.\n- Restarted only dashboard/status-sampler; node and pool were left running to preserve active mining.\n- Final sample: synced, can_mine=true, can_submit_blocks=true, backend mineable=true, submit_ready=true, P2P fresh=true, 4 ready miners, 11 fresh consensus peers, peer lead -1/0, accepted block submissions 241, template conversion failure window about 8.2%.

[jeremyharkness]

Live recovery update from 2026-06-21:

• Recovered the local stack by clearing a stale repair hold after proving the node was stalled (eth_syncing.currentBlock stayed fixed while highestBlock advanced), then restarting the node without deleting/recreating chain data.
• Recreated the pool with the best observed 05:40-05:57 SAST timing: POOL_TEMPLATE_TTL_REFRESH_MS=100, POOL_MAX_BLOCK_CANDIDATE_JOB_AGE_MS=1750, POOL_RECENT_STALE_BLOCK_CANDIDATE_SUBMIT_GRACE_MS=250, and multiple block candidates enabled.
• Live result after recovery: dashboard overall=ok, mode=mining, can_submit_blocks=true; job-state ok with 4/4 ready miners; accepted block submissions rose past 148 during validation.
• Added regression coverage so stack defaults, .env.example, and rendered Compose cannot drift back to the old cold-start timing values unnoticed.

Verification:

• python3 -m unittest ops.tests.test_stack_defaults
• python3 scripts/validate-stack-defaults.py
• docker compose config confirmed the rendered timing env values.