feat(build): make the SCTP transport an optional dependency - #73
Merged
Conversation
Thecave3
force-pushed
the
feat/optional-sctp
branch
from
August 26, 2026 03:46
697a2c3 to
222e353
Compare
Contributor
CI report —
|
| Workflow | Result | Time | Run |
|---|---|---|---|
| Commit policy | ✅ success | 3m36s | #95 |
| E2E dApp Integration | ✅ success | 1m57s | #110 |
| E2E Topologies (multi-dApp / multi-RAN) | ✅ success | 2m26s | #108 |
| Full-loop Latency Benchmark | ✅ success | 1m13s | #109 |
| Unit Tests | ✅ success | 4m40s | #136 |
| latrec portability | ⏭️ not triggered (paths filter) | — | — |
| MPMC Queue Benchmark | ⏭️ not triggered (paths filter) | — | — |
E2E Topologies (multi-dApp / multi-RAN)
zmq/ipc
- ✅ 1 RAN - 1 dApp: indications=
5dapppeer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=1 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
- ✅ 1 RAN - 2 dApps: dApp#1 ind=
5sub=1, dApp#2 ind=6sub=2, RAN saw2dApps - ✅ 2 RANs - 1 dApp: from ran-a ind=
5, from ran-b ind=5
zmq/tcp
- ✅ 1 RAN - 1 dApp: indications=
5dapppeer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.8 max=1 @seq=1) hist[<=1:5 2-5:0 6-10:0 >10:0]
- ✅ 1 RAN - 2 dApps: dApp#1 ind=
5sub=1, dApp#2 ind=6sub=2, RAN saw2dApps - ✅ 2 RANs - 1 dApp: from ran-a ind=
5, from ran-b ind=5
E2E dApp Integration
✅ posix/ipc
- dApp exit:
0 - Indications received:
7
✅ posix/tcp
- dApp exit:
0 - Indications received:
7
✅ zmq/ipc
- dApp exit:
0 - Indications received:
7
✅ zmq/tcp
- dApp exit:
0 - Indications received:
7
Full-loop Latency Benchmark
Full-loop latency
Full-loop latency benchmark (N=1014 after 50 warmup)
All values in microseconds (μs). Link: zmq, transport: ipc, encoding: ASN.1 APER.
| # | Description | Tags | mean | p50 | p99 | max |
|---|---|---|---|---|---|---|
| 1 | Collect indication data | RECORD_BEGIN to ENCODE_E3SM_BEGIN |
0.13 | 0.11 | 0.35 | 2.36 |
| 2 | Create & encode indication | ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE |
1.05 | 0.98 | 1.73 | 33.72 |
| 3 | Encode E3AP (indication) | EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE |
3.11 | 3.03 | 5.78 | 6.23 |
| 4 | Queuing (indication) | ENQUEUE to DEQUEUE |
5.18 | 4.48 | 14.23 | 62.68 |
| 5 | Delivery (indication) | ENCODE_E3AP_DONE to SEND_DONE |
0.34 | 0.28 | 1.70 | 7.09 |
| 6 | E3 wire (RAN -> dApp) | SEND_DONE to RECV |
52.56 | 52.75 | 65.54 | 84.70 |
| 7 | Decode E3AP (indication) | RECV to DECODE_E3AP_DONE |
1.79 | 1.50 | 2.96 | 22.74 |
| 8 | libe3 dispatch (indication) | DECODE_E3AP_DONE to DELIVER_BEGIN |
0.13 | 0.13 | 0.22 | 0.47 |
| 9 | Decode indication | DELIVER_BEGIN to DECODE_E3SM_DONE |
0.67 | 0.60 | 1.05 | 1.41 |
| 10 | Process data | DECODE_E3SM_DONE to ENCODE_E3SM_BEGIN |
0.04 | 0.04 | 0.05 | 0.13 |
| 11 | Create & encode control | ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE |
0.34 | 0.31 | 0.60 | 0.78 |
| 12 | Encode E3AP (control) | EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE |
3.79 | 3.77 | 5.43 | 6.72 |
| 13 | Queuing (control) | ENQUEUE to DEQUEUE |
16.66 | 16.50 | 23.37 | 67.97 |
| 14 | Delivery (control) | ENCODE_E3AP_DONE to SEND_DONE |
5.01 | 4.86 | 8.12 | 9.82 |
| 15 | E3 wire (dApp -> RAN) | SEND_DONE to RECV |
51.47 | 50.82 | 65.22 | 98.74 |
| 16 | Decode E3AP (control) | RECV to DECODE_E3AP_DONE |
2.76 | 2.60 | 4.95 | 15.54 |
| 17 | libe3 dispatch (control) | DECODE_E3AP_DONE to DECODE_E3SM_BEGIN |
0.27 | 0.25 | 0.52 | 0.64 |
| 18 | Decode & handle control | DECODE_E3SM_BEGIN to DECODE_E3SM_DONE |
0.42 | 0.38 | 0.70 | 0.91 |
| Total | Total round-trip | 146.23 | 147.34 | 181.33 | 224.11 |
ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.
These numbers are measured inside a GitHub Actions container and should be treated as an upper bound on E3AP's and the library's own latency, not a representative deployment measurement.
Ready to merge (fast-forward only)
A maintainer can land the reviewed commits with:
git fetch origin
git checkout main && git merge --ff-only 44d168446942dc4a2fa9ecac938ec38cbe24b3cc && git push origin mainHead: 44d168446942dc4a2fa9ecac938ec38cbe24b3cc (branch feat/optional-sctp). If --ff-only fails as non-fast-forward, the branch must be rebased on the latest main.
One comment per PR, rewritten in place once every workflow for
44d1684finished.
A downstream container image that builds libe3 without the SCTP headers fails:
src/connector/posix_connector.cpp:19:10:
fatal error: netinet/sctp.h: No such file or directory
posix_connector.cpp included <netinet/sctp.h> unconditionally, so every consumer
had to install libsctp-dev whether or not it ever used SCTP. Images that happened
to install it for another reason built fine; images that did not could not build
libe3 at all, for a transport they were never going to use.
Gate it behind LIBE3_ENABLE_SCTP, default OFF. SCTP is the only dependency that
needs a kernel module plus a distro -dev package for a single system header, while
deployments typically run the E3 link over IPC or TCP.
What does NOT change: E3TransportLayer::SCTP stays in the enum, at value 0. It is
part of the ABI and of the wire-facing value the C API and the Python bindings
mirror, so removing or renumbering it would break them. transport_layer_to_string
and its test are untouched.
What changes: with the flag off, a request for the SCTP transport fails at socket
creation with a message naming the flag, instead of falling through to another
protocol -- which would look like it worked while putting traffic on the wrong
one. The four socket() sites now go through one sctp_stream_socket() shim that
returns -1 with errno=EPROTONOSUPPORT, which every existing caller's error path
already handles.
libe3 calls no libsctp function -- only IPPROTO_SCTP and SCTP_NODELAY, both from
the header -- so there is nothing to link and no find_package to add. The
dependency is the header alone, checked at configure time so -DLIBE3_ENABLE_SCTP=ON
without it fails naming the package, rather than partway through the build.
The define is PUBLIC, so it reaches libe3.pc: a consumer can tell whether asking
for SCTP will work instead of finding out at socket().
build_libe3 grows --enable-sctp/--disable-sctp and no longer installs the SCTP
headers with --install-deps; README documents the flag, the packages per distro,
and that the enum survives either way.
Verified in a container with no libsctp-dev, which is the environment that was
failing:
* default (off): configures and builds to completion
* -DLIBE3_ENABLE_SCTP=ON: fails at configure with the package names
* preprocessing posix_connector.cpp with the flag off shows netinet/sctp.h is
not reached at all
Both arms also build on a host that does have the header.
Assisted-by: Claude Opus 5
Thecave3
force-pushed
the
feat/optional-sctp
branch
from
August 27, 2026 18:10
80cb9df to
8c1fe8c
Compare
Two tests hardcoded a literal TCP base port (26990 / 25990) with no per-process derivation, so concurrent CTest runs on this shared, multi-session machine collide on the same port triple -- confirmed by running two copies of the binary at once, which reproduced the exact intermittent connect()/bind() failure seen in CI. Adds unique_base_port(), spread by pid (or an explicit LIBE3_TEST_PORT_OFFSET env var for CI runners that want disjoint ranges per job) with *3 spacing so no two processes' port triples overlap. Also fixes a real startup race in multi_peer_slow_peer_does_not_stall_others: wait_for_setup() only confirms the SETUP handshake socket, not the RAN's publisher socket that the test's raw connect() targets directly -- added connect_with_retry() with a bounded backoff instead of a bare, unretried connect(). Verified: the specific test passed 8/8 alone and under both 2-way and 4-way concurrent stress (the exact collision scenario) after this fix.
tools/__pycache__/ has been showing up as untracked in every git status this session; .gitignore had no Python entries at all.
The posted CI report showed an "E3 wire (RAN -> dApp)" row with a mean of ~3.6e17 and a max of exactly UINT64_MAX/1000 -- a classic unsigned subtraction underflow, meaning some round trip's RECV timestamp was paired with the wrong (later) SEND_DONE. Root cause: the outbound quintuple is joined to a round trip by exact key (EMIT_ENTER.aux == business seq), but the inbound triple/pair has no such key and is joined by chronological position instead (documented in this file's own header -- the inbound side genuinely has nothing else to key on). Position-based pairing silently drifts by one for every round trip whose own inbound data didn't fully land -- most commonly at shutdown, when the SM can emit one more indication than the harness waits for a control reply to. Once drifted, every later "ii"/"ci" belongs to some other round trip, not the one it's paired with. Verified with a debug build that instrumented and printed the exact misaligned pairing, then reproduced it reliably under 20-way concurrent runs of the binary (1-4 affected round trips per run; 0/15 sequential runs hit it, matching how the original CI failure looked like a rare event). Fix: after joining both legs, check that the position-based pieces are at least causally consistent with the key-based ones (send_done <= recv, and the inbound triple's own last stamp doesn't run past the business-seq side's next stage) before accepting the round trip -- drop it otherwise, the same tolerance this file already extends to an incomplete tail. Re-ran the 20-way concurrent stress after the fix: 13/20 runs hit the drop path (1-4 round trips each), 0/20 produced a bad value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
67-68-message-id-correlation(#72) — review that first.The problem
A downstream container image that builds libe3 without the SCTP headers fails:
posix_connector.cppincluded<netinet/sctp.h>unconditionally, so every consumer had to installlibsctp-devwhether or not it ever used SCTP. Images that happened to install it for anotherreason built fine; images that did not could not build libe3 at all — for a transport they were
never going to use.
The change
LIBE3_ENABLE_SCTP, default OFF. SCTP is the only libe3 dependency that needs a kernel moduleplus a distro
-devpackage for a single system header, while deployments typically run the E3link over IPC or TCP.
E3TransportLayer::SCTPstays in the enum, at value 0. It is part of the ABI and of thewire-facing value the C API (
transport_layer: 0=SCTP) and the Python bindings mirror, so removingor renumbering it would break them.
transport_layer_to_stringand its test are untouched.With the flag off, asking for the SCTP transport fails at socket creation with a message naming
the flag — rather than falling through to another protocol, which would look like it worked while
putting traffic on the wrong one. The four
socket()sites now go through onesctp_stream_socket()shim returning
-1witherrno=EPROTONOSUPPORT, which every existing caller's error path alreadyhandles.
Worth noting: libe3 calls no libsctp function — only
IPPROTO_SCTPandSCTP_NODELAY, bothfrom the header — so there is nothing to link and no
find_packageto add. The dependency is theheader alone. It is checked at configure time, so
-DLIBE3_ENABLE_SCTP=ONwithout it fails namingthe package per distro instead of dying partway through the build.
The define is
PUBLIC, so it reacheslibe3.pc(same pattern asLIBE3_ENABLE_LATREC): a consumercan tell whether asking for SCTP will work instead of finding out at
socket().build_libe3gains--enable-sctp/--disable-sctpand no longer installs the SCTP headerswith
--install-deps.README.mddocuments the flag, the package per distro, and that the enumsurvives either way.
Verification
Run in a container with no
libsctp-dev— the environment that was failing:-DLIBE3_ENABLE_SCTP=ON<netinet/sctp.h>was not found. Install the SCTP headers (Debian/Ubuntu: libsctp-dev, …)"Also: preprocessing
posix_connector.cppwith the flag off showsnetinet/sctp.his never reached,and both arms build on a host that does have the header.
Consumer impact
None required. A consumer that already passes explicit
-DLIBE3_*flags simply gets thedefault-off behaviour; one that genuinely needs SCTP adds
-DLIBE3_ENABLE_SCTP=ONand the distropackage.