Skip to content

fix(iouring): correct IORING_NOTIF_USAGE_ZC_COPIED constant and treat loopback as functional probe (#465) - #489

Open
FumingPower3925 wants to merge 3 commits into
mainfrom
fix/465-iouring-sendzc-probe-constant
Open

fix(iouring): correct IORING_NOTIF_USAGE_ZC_COPIED constant and treat loopback as functional probe (#465)#489
FumingPower3925 wants to merge 3 commits into
mainfrom
fix/465-iouring-sendzc-probe-constant

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Closes #465

1. Reproduction

  • Environment: Linux x86_64 (msa2-client, kernel 7.0.0-30-generic), base commit f4749e9
  • Command: running probeSendZC() against loopback
  • Raw output before fix:
INFO SEND_ZC probe result result="true zero-copy"

The probe erroneously reported "true zero-copy" from a loopback socket, which Linux networking always copies via skb_orphan_frags_rx.

2. Root cause

In engine/iouring/probe.go:73, notifUsageZCCopied = 2 was defined as bit 1, whereas the Linux kernel UAPI defines IORING_NOTIF_USAGE_ZC_COPIED = (1U << 31). The bitwise check notifRes&notifUsageZCCopied != 0 checked bit 1 instead of bit 31 and could never evaluate to true, causing the probe on loopback to always fall through to SendZCTrueZeroCopy. Additionally, completionEntry.Res is int32, requiring an unsigned cast uint32(notifRes)&notifUsageZCCopied to avoid overflow.

3. Fix

  • Corrected constant to notifUsageZCCopied uint32 = 1 << 31 and updated comparison to uint32(notifRes)&notifUsageZCCopied != 0.
  • Factored out parseSendZCResult to evaluate probe CQE outcomes (SendZCUnsupported, SendZCBroken, SendZCNoNotification, SendZCCopyFallback, SendZCTrueZeroCopy).
  • Handled missing CQE_F_MORE on initial CQE as distinct SendZCNoNotification (opcode accepted but notification unobserved; excluded from functional).
  • Preserved distinct wait failure diagnostic: "notification CQE wait failed: <err>".
  • Added error check on rawConn.Control and verified fd > 0.
  • Split probe meaning: treated SendZCCopyFallback on loopback as confirmation that SEND_ZC opcode and notification delivery are functional in the kernel. Honest logging: functional=true loopback="copy-fallback (expected)".
  • Preserved default policy: resolveSendZCPolicy(functional, envVal) keeps profile.SendZC enabled when the functional probe passes (default unchanged). Final default decision is pending a measured cluster fabric A/B benchmark once the cluster is free (~2026-09-08). Explicit overrides CELERIS_IOURING_SEND_ZC=on|1|true and off|0|false are supported; unrecognized values log a warning and fall back to auto.
  • Documented CELERIS_IOURING_SEND_ZC, CELERIS_IOURING_MULTISHOT_RECV, CELERIS_IOURING_PBUF_COUNT, and CELERIS_MAX_IOURING_TIER in engine/iouring/doc.go, including clamping and registration-failure fallback behavior. Removed duplicate package comment in engine.go.
  • Added unit tests covering all probe outcomes against synthetic CQEs (TestParseSendZCResult), constant correctness (TestNotifUsageZCCopiedConstant), live loopback probe execution (TestProbeSendZCLiveLoopback), and policy resolution (TestResolveSendZCPolicy). TestProbeSendZCLiveLoopback skips cleanly on runners where io_uring or SEND_ZC is unsupported.

4. Proof

  • Tested on Linux x86_64 (msa2-client, kernel 7.0.0-30-generic):
=== RUN   TestNotifUsageZCCopiedConstant
--- PASS: TestNotifUsageZCCopiedConstant (0.00s)
=== RUN   TestParseSendZCResult
--- PASS: TestParseSendZCResult (0.00s)
=== RUN   TestResolveSendZCPolicy
--- PASS: TestResolveSendZCPolicy (0.00s)
=== RUN   TestProbeSendZCLiveLoopback
    probe_test.go:156: probeSendZC() live result: copy fallback (copy fallback), reason: "REPORT_USAGE notification reports IORING_NOTIF_USAGE_ZC_COPIED (kernel did the copy)"
--- PASS: TestProbeSendZCLiveLoopback (0.00s)
=== RUN   TestUseSendZC
--- PASS: TestUseSendZC (0.00s)
=== RUN   TestPrepSendSQEGatesBySize
--- PASS: TestPrepSendSQEGatesBySize (0.00s)
=== RUN   TestPrepSendSQELinkedNeverZC
--- PASS: TestPrepSendSQELinkedNeverZC (0.00s)
PASS
  • Engine logs on msa2-client now report:
INFO SEND_ZC probe result functional=true loopback="copy-fallback (expected)" reason="REPORT_USAGE notification reports IORING_NOTIF_USAGE_ZC_COPIED (kernel did the copy)"
INFO io_uring engine selected tier=high multishot_accept=true multishot_recv=true provided_buffers=true fixed_files=false send_zc=true
  • Full engine/iouring test suite passed cleanly on msa2-client.

5. Performance

  • No impact on request hot path: probeSendZC executes at most once during process startup via sync.Once.
  • Default behavior preserved; measured A/B comparison on cluster fabric will inform any future default policy change.
  • Gating invariants (sendZCMinBytes = 32768, unlinked-only) preserved.

6. Not verified

  • arm64 Linux live probe execution (msr1 pending until the cluster finishes soak/benchmark; cross-compiled cleanly for arm64).
  • Physical NIC fabric A/B measurement (SEND vs SEND_ZC on get-json-64k, ws-large-echo, streaming cell, alternating builds, >=3 trials) pending cluster availability (~2026-09-08).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

io_uring: probeSendZC can never detect copy-fallback — SEND_ZC enabled on every host regardless of NIC

1 participant