fix(replication): batch record prune audit challenges by mickvandijke · Pull Request #170 · WithAutonomi/ant-node

mickvandijke · 2026-07-09T15:26:31Z

Summary

Add reason-specific audit responder admission logging so capacity drops distinguish global-pool saturation from per-peer cap saturation.
Raise audit responder capacity caps from 16→32 global responders and 2→4 per-peer responders.
Batch record prune-confirmation AuditChallenges by target peer to avoid same-peer one-key request bursts.
Align prune-confirmation batch sizing with the normal responsible-chunk audit sender limit, and derive the receiver-side incoming limit from that same named limit with the existing 2x skew margin.

Why

Production logs showed responsible audit replies dropped with the old opaque message:

Audit challenge reply not sent: kind=responsible response=dropped source=... (audit-responder capacity reached)

The drops were concentrated in same-source, same-millisecond bursts. That shape matched the per-peer audit responder cap and can be produced by prune-confirmation audit traffic, which previously sent one single-key AuditChallenge per (peer, key) concurrently.

Key changes

admit_audit_responder now reports structured admission failures:
- global_pool_full
- per_peer_cap_full
Capacity-drop logs now include global and per-peer inflight/limit fields.
MAX_CONCURRENT_AUDIT_RESPONSES is doubled from 16 to 32.
MAX_AUDIT_RESPONSES_PER_PEER is doubled from 2 to 4.
Record prune-confirmation challenges are grouped by peer and split into chunks no larger than ReplicationConfig::responsible_audit_key_limit(local_stored_key_count).
The normal responsible audit sender now also calls responsible_audit_key_limit(...), making prune-confirmation use the same sender-side limit literally.
max_incoming_audit_keys(...) is now explicitly derived from responsible_audit_key_limit(...) with the existing 2x margin for challenger/receiver store-size skew.

Validation

Ran locally:

cargo test -q replication::pruning::tests::prune_audit_challenges_split_peer_batches_at_responsible_audit_limit -- --nocapture
cargo test -q responsible_audit_key_limit_matches_audit_sample_count -- --nocapture
cargo test -q max_incoming_audit_keys_scales_dynamically -- --nocapture
cargo test -q replication::pruning::tests -- --nocapture
cargo fmt --all
cargo test -q replication::tests::audit_responder_admission_reports -- --nocapture
cargo check -q
git diff --check

Results:

New prune split test passed.
Responsible-audit key-limit config tests passed.
Full pruning test filter passed: 21 tests.
Audit responder admission tests passed: 2 tests.
cargo fmt --all passed.
cargo check -q passed.
git diff --check passed.

Follow-up: clippy/Copilot review handling

Fixed local/CI clippy issues in src/replication/mod.rs.
Addressed Copilot review feedback for prune audit batches by:
- stopping failure-report attempts after the first successful per-peer report for a batch;
- storing expected 32-byte audit digests instead of retaining full local record bytes for batch verification.
Replied to the remaining Copilot batch-size comment: prune-confirmation now uses the same sender-side responsible-audit key limit, while receiver acceptance keeps the existing 2x skew margin.

Follow-up validation after commit d681533:

cargo clippy --all-targets --all-features -- -D warnings
cargo test -q replication::pruning::tests -- --nocapture
cargo test -q replication::tests::audit_responder_admission_reports -- --nocapture
cargo test -q responsible_audit_key_limit_matches_audit_sample_count -- --nocapture
cargo test -q max_incoming_audit_keys_scales_dynamically -- --nocapture
cargo fmt --all
cargo check -q
git diff --check

Copilot

Pull request overview

This PR reduces audit-responder load spikes during record pruning by batching prune-confirmation AuditChallenges per target peer (multi-key per peer) instead of sending one one-key challenge per (peer, key). It also improves observability and capacity behavior of the shared audit responder by reporting which admission ceiling caused drops and increasing the responder caps.

Changes:

Batch record prune-confirmation audit challenges by target peer and verify multi-key digest responses per key.
Add structured audit-responder admission failure reporting (reason + decision-time counters) and update logs accordingly.
Increase audit responder concurrency caps (global and per-peer) and add tests for the new admission reporting.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/replication/pruning.rs	Coalesce prune audits into one multi-key challenge per peer and verify digest responses per key.
src/replication/mod.rs	Return detailed admission failure reasons/counters from audit responder admission and log them; add tests.
src/replication/config.rs	Increase global/per-peer audit responder concurrency limits.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mickvandijke · 2026-07-09T16:21:01Z

+    let challenge_keys: Vec<XorName> = challenge_material.iter().map(|(key, _)| *key).collect();
    let (challenge_id, nonce) = {
        let mut rng = rand::thread_rng();
        (rng.gen::<u64>(), rng.gen::<[u8; 32]>())
    };
-    let (encoded, key_count) = encode_prune_audit_challenge(&peer, key, challenge_id, nonce)?;
+    let Some((encoded, key_count)) =
+        encode_prune_audit_challenge(&peer, &challenge_keys, challenge_id, nonce)
+    else {


Thanks. The latest version intentionally chunks prune-confirmation batches at ReplicationConfig::responsible_audit_key_limit(local_stored_key_count), i.e. the exact same sender-side limit used by normal responsible chunk audit. The receiver-side max_incoming_audit_keys is derived from that same named limit with the existing 2x skew margin. I don't think adding retry-with-smaller-batches is needed here: if a peer rejects a challenge at this size, it would also be rejecting a normal responsible-audit-sized challenge from this sender, and retrying would add more prune-audit traffic on the failure path.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

mickvandijke · 2026-07-09T16:21:02Z

+        for key in &challenge_keys {
+            report_prune_audit_failure_once(&peer, key, p2p_node, config, report_state).await;
+        }


Fixed in d681533. On timeout/malformed response, the code now stops attempting failure reports after the first successful per-peer report, avoiding repeated responsibility checks for the rest of the batch.

mickvandijke · 2026-07-09T16:21:04Z

+            PruneAuditStatus::Failed => {
+                report_prune_audit_failure_once(&peer, &key, p2p_node, config, report_state).await;
+            }


Fixed in d681533. The decoded failed-status path now tracks whether an audit failure has already been reported for this peer/batch and skips further report attempts after the first successful one.

mickvandijke · 2026-07-09T16:21:05Z

+    let mut challenge_material = Vec::new();
+    for key in keys {
+        if let Some(local_bytes) = local_record_bytes(&key, storage).await {
+            challenge_material.push((key, local_bytes));
+        }


Fixed in d681533. The batch path now computes and stores only the expected 32-byte audit digest per key before sending the challenge, rather than retaining full local record bytes for the whole batch.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

mickvandijke added 3 commits July 9, 2026 14:58

chore: improve audit responder capacity tracking and logging

2202dda

fix(replication): raise audit responder capacity caps

53171c1

fix(replication): batch record prune audit challenges

5b3eebe

Copilot AI review requested due to automatic review settings July 9, 2026 15:26

Copilot started reviewing on behalf of mickvandijke July 9, 2026 15:26 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

fix(replication): align prune audit key limits

1dec5ba

Copilot AI review requested due to automatic review settings July 9, 2026 16:08

Copilot started reviewing on behalf of mickvandijke July 9, 2026 16:08 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

fix(replication): address prune audit review feedback

d681533

Copilot AI review requested due to automatic review settings July 9, 2026 16:20

Copilot started reviewing on behalf of mickvandijke July 9, 2026 16:21 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(replication): batch record prune audit challenges#170

fix(replication): batch record prune audit challenges#170
mickvandijke wants to merge 5 commits into
mainfrom
fix-audit-timeouts-part-capacity-fix

mickvandijke commented Jul 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

mickvandijke Jul 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

mickvandijke Jul 9, 2026

Uh oh!

mickvandijke Jul 9, 2026

Uh oh!

mickvandijke Jul 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mickvandijke commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Key changes

Validation

Follow-up: clippy/Copilot review handling

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mickvandijke Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

mickvandijke Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

mickvandijke Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

mickvandijke Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mickvandijke commented Jul 9, 2026 •

edited

Loading