Skip to content

feat(build): make the SCTP transport an optional dependency - #73

Merged
Thecave3 merged 4 commits into
mainfrom
feat/optional-sctp
Aug 27, 2026
Merged

feat(build): make the SCTP transport an optional dependency#73
Thecave3 merged 4 commits into
mainfrom
feat/optional-sctp

Conversation

@Thecave3

@Thecave3 Thecave3 commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Stacked on 67-68-message-id-correlation (#72) — review that first.

The problem

A downstream container image that builds libe3 without the SCTP headers fails:

src/connector/posix_connector.cpp:19:10:
  fatal error: netinet/sctp.h: No such file or directory

posix_connector.cpp included <netinet/sctp.h> unconditionally, so every consumer had to install
libsctp-dev whether or not it ever used SCTP. Images that happened to install it for another
reason built fine; images that did not could not build libe3 at all — for a transport they were
never going to use.

The change

LIBE3_ENABLE_SCTP, default OFF. SCTP is the only libe3 dependency that needs a kernel module
plus a distro -dev package for a single system header, while deployments typically run the E3
link over IPC or TCP.

E3TransportLayer::SCTP stays in the enum, at value 0. It is part of the ABI and of the
wire-facing value the C API (transport_layer: 0=SCTP) and the Python bindings mirror, so removing
or renumbering it would break them. transport_layer_to_string and its test are untouched.

With the flag off, asking for the SCTP transport fails at socket creation with a message naming
the flag — rather than falling through to another protocol, which would look like it worked while
putting traffic on the wrong one. The four socket() sites now go through one sctp_stream_socket()
shim returning -1 with errno=EPROTONOSUPPORT, which every existing caller's error path already
handles.

Worth noting: libe3 calls no libsctp function — only IPPROTO_SCTP and SCTP_NODELAY, both
from the header — so there is nothing to link and no find_package to add. The dependency is the
header alone. It is checked at configure time, so -DLIBE3_ENABLE_SCTP=ON without it fails naming
the package per distro instead of dying partway through the build.

The define is PUBLIC, so it reaches libe3.pc (same pattern as LIBE3_ENABLE_LATREC): a consumer
can tell whether asking for SCTP will work instead of finding out at socket().

build_libe3 gains --enable-sctp / --disable-sctp and no longer installs the SCTP headers
with --install-deps
. README.md documents the flag, the package per distro, and that the enum
survives either way.

Verification

Run in a container with no libsctp-dev — the environment that was failing:

arm result
default (SCTP off) configures and builds to completion
-DLIBE3_ENABLE_SCTP=ON fails at configure: "LIBE3_ENABLE_SCTP=ON but <netinet/sctp.h> was not found. Install the SCTP headers (Debian/Ubuntu: libsctp-dev, …)"

Also: preprocessing posix_connector.cpp with the flag off shows netinet/sctp.h is never reached,
and both arms build on a host that does have the header.

Consumer impact

None required. A consumer that already passes explicit -DLIBE3_* flags simply gets the
default-off behaviour; one that genuinely needs SCTP adds -DLIBE3_ENABLE_SCTP=ON and the distro
package.

@Thecave3
Thecave3 force-pushed the feat/optional-sctp branch from 697a2c3 to 222e353 Compare August 26, 2026 03:46
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

CI report — 44d1684 — ✅ all checks passed

Workflow Result Time Run
Commit policy ✅ success 3m36s #95
E2E dApp Integration ✅ success 1m57s #110
E2E Topologies (multi-dApp / multi-RAN) ✅ success 2m26s #108
Full-loop Latency Benchmark ✅ success 1m13s #109
Unit Tests ✅ success 4m40s #136
latrec portability ⏭️ not triggered (paths filter)
MPMC Queue Benchmark ⏭️ not triggered (paths filter)
E2E Topologies (multi-dApp / multi-RAN)

zmq/ipc

  • 1 RAN - 1 dApp: indications=5
    • dapp peer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=1 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=t12 ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=t12 ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=t2a ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=t2b ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

zmq/tcp

  • 1 RAN - 1 dApp: indications=5
    • dapp peer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.8 max=1 @seq=1) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=default ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.2 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=default ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0.166667 max=1 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=default ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.4 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=off100 ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
E2E dApp Integration

✅ posix/ipc

  • dApp exit: 0
  • Indications received: 7

✅ posix/tcp

  • dApp exit: 0
  • Indications received: 7

✅ zmq/ipc

  • dApp exit: 0
  • Indications received: 7

✅ zmq/tcp

  • dApp exit: 0
  • Indications received: 7
Full-loop Latency Benchmark

Full-loop latency

Full-loop latency benchmark (N=1014 after 50 warmup)

All values in microseconds (μs). Link: zmq, transport: ipc, encoding: ASN.1 APER.

# Description Tags mean p50 p99 max
1 Collect indication data RECORD_BEGIN to ENCODE_E3SM_BEGIN 0.13 0.11 0.35 2.36
2 Create & encode indication ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE 1.05 0.98 1.73 33.72
3 Encode E3AP (indication) EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE 3.11 3.03 5.78 6.23
4 Queuing (indication) ENQUEUE to DEQUEUE 5.18 4.48 14.23 62.68
5 Delivery (indication) ENCODE_E3AP_DONE to SEND_DONE 0.34 0.28 1.70 7.09
6 E3 wire (RAN -> dApp) SEND_DONE to RECV 52.56 52.75 65.54 84.70
7 Decode E3AP (indication) RECV to DECODE_E3AP_DONE 1.79 1.50 2.96 22.74
8 libe3 dispatch (indication) DECODE_E3AP_DONE to DELIVER_BEGIN 0.13 0.13 0.22 0.47
9 Decode indication DELIVER_BEGIN to DECODE_E3SM_DONE 0.67 0.60 1.05 1.41
10 Process data DECODE_E3SM_DONE to ENCODE_E3SM_BEGIN 0.04 0.04 0.05 0.13
11 Create & encode control ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE 0.34 0.31 0.60 0.78
12 Encode E3AP (control) EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE 3.79 3.77 5.43 6.72
13 Queuing (control) ENQUEUE to DEQUEUE 16.66 16.50 23.37 67.97
14 Delivery (control) ENCODE_E3AP_DONE to SEND_DONE 5.01 4.86 8.12 9.82
15 E3 wire (dApp -> RAN) SEND_DONE to RECV 51.47 50.82 65.22 98.74
16 Decode E3AP (control) RECV to DECODE_E3AP_DONE 2.76 2.60 4.95 15.54
17 libe3 dispatch (control) DECODE_E3AP_DONE to DECODE_E3SM_BEGIN 0.27 0.25 0.52 0.64
18 Decode & handle control DECODE_E3SM_BEGIN to DECODE_E3SM_DONE 0.42 0.38 0.70 0.91
Total Total round-trip 146.23 147.34 181.33 224.11

ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.

These numbers are measured inside a GitHub Actions container and should be treated as an upper bound on E3AP's and the library's own latency, not a representative deployment measurement.

Ready to merge (fast-forward only)

A maintainer can land the reviewed commits with:

git fetch origin
git checkout main && git merge --ff-only 44d168446942dc4a2fa9ecac938ec38cbe24b3cc && git push origin main

Head: 44d168446942dc4a2fa9ecac938ec38cbe24b3cc (branch feat/optional-sctp). If --ff-only fails as non-fast-forward, the branch must be rebased on the latest main.

One comment per PR, rewritten in place once every workflow for 44d1684 finished.

A downstream container image that builds libe3 without the SCTP headers fails:

    src/connector/posix_connector.cpp:19:10:
      fatal error: netinet/sctp.h: No such file or directory

posix_connector.cpp included <netinet/sctp.h> unconditionally, so every consumer
had to install libsctp-dev whether or not it ever used SCTP. Images that happened
to install it for another reason built fine; images that did not could not build
libe3 at all, for a transport they were never going to use.

Gate it behind LIBE3_ENABLE_SCTP, default OFF. SCTP is the only dependency that
needs a kernel module plus a distro -dev package for a single system header, while
deployments typically run the E3 link over IPC or TCP.

What does NOT change: E3TransportLayer::SCTP stays in the enum, at value 0. It is
part of the ABI and of the wire-facing value the C API and the Python bindings
mirror, so removing or renumbering it would break them. transport_layer_to_string
and its test are untouched.

What changes: with the flag off, a request for the SCTP transport fails at socket
creation with a message naming the flag, instead of falling through to another
protocol -- which would look like it worked while putting traffic on the wrong
one. The four socket() sites now go through one sctp_stream_socket() shim that
returns -1 with errno=EPROTONOSUPPORT, which every existing caller's error path
already handles.

libe3 calls no libsctp function -- only IPPROTO_SCTP and SCTP_NODELAY, both from
the header -- so there is nothing to link and no find_package to add. The
dependency is the header alone, checked at configure time so -DLIBE3_ENABLE_SCTP=ON
without it fails naming the package, rather than partway through the build.

The define is PUBLIC, so it reaches libe3.pc: a consumer can tell whether asking
for SCTP will work instead of finding out at socket().

build_libe3 grows --enable-sctp/--disable-sctp and no longer installs the SCTP
headers with --install-deps; README documents the flag, the packages per distro,
and that the enum survives either way.

Verified in a container with no libsctp-dev, which is the environment that was
failing:
  * default (off): configures and builds to completion
  * -DLIBE3_ENABLE_SCTP=ON: fails at configure with the package names
  * preprocessing posix_connector.cpp with the flag off shows netinet/sctp.h is
    not reached at all
Both arms also build on a host that does have the header.

Assisted-by: Claude Opus 5
@Thecave3
Thecave3 force-pushed the feat/optional-sctp branch from 80cb9df to 8c1fe8c Compare August 27, 2026 18:10
Two tests hardcoded a literal TCP base port (26990 / 25990) with no
per-process derivation, so concurrent CTest runs on this shared,
multi-session machine collide on the same port triple -- confirmed by
running two copies of the binary at once, which reproduced the exact
intermittent connect()/bind() failure seen in CI.

Adds unique_base_port(), spread by pid (or an explicit
LIBE3_TEST_PORT_OFFSET env var for CI runners that want disjoint ranges
per job) with *3 spacing so no two processes' port triples overlap.

Also fixes a real startup race in multi_peer_slow_peer_does_not_stall_others:
wait_for_setup() only confirms the SETUP handshake socket, not the RAN's
publisher socket that the test's raw connect() targets directly -- added
connect_with_retry() with a bounded backoff instead of a bare, unretried
connect().

Verified: the specific test passed 8/8 alone and under both 2-way and
4-way concurrent stress (the exact collision scenario) after this fix.
tools/__pycache__/ has been showing up as untracked in every git status
this session; .gitignore had no Python entries at all.
The posted CI report showed an "E3 wire (RAN -> dApp)" row with a mean of
~3.6e17 and a max of exactly UINT64_MAX/1000 -- a classic unsigned
subtraction underflow, meaning some round trip's RECV timestamp was
paired with the wrong (later) SEND_DONE.

Root cause: the outbound quintuple is joined to a round trip by exact key
(EMIT_ENTER.aux == business seq), but the inbound triple/pair has no such
key and is joined by chronological position instead (documented in this
file's own header -- the inbound side genuinely has nothing else to key
on). Position-based pairing silently drifts by one for every round trip
whose own inbound data didn't fully land -- most commonly at shutdown,
when the SM can emit one more indication than the harness waits for a
control reply to. Once drifted, every later "ii"/"ci" belongs to some
other round trip, not the one it's paired with.

Verified with a debug build that instrumented and printed the exact
misaligned pairing, then reproduced it reliably under 20-way concurrent
runs of the binary (1-4 affected round trips per run; 0/15 sequential runs
hit it, matching how the original CI failure looked like a rare event).

Fix: after joining both legs, check that the position-based pieces are at
least causally consistent with the key-based ones (send_done <= recv, and
the inbound triple's own last stamp doesn't run past the business-seq
side's next stage) before accepting the round trip -- drop it otherwise,
the same tolerance this file already extends to an incomplete tail.
Re-ran the 20-way concurrent stress after the fix: 13/20 runs hit the
drop path (1-4 round trips each), 0/20 produced a bad value.
@Thecave3
Thecave3 marked this pull request as ready for review August 27, 2026 19:25
Base automatically changed from 67-68-message-id-correlation to main August 27, 2026 19:31
@Thecave3
Thecave3 merged commit 44d1684 into main Aug 27, 2026
24 checks passed
@Thecave3
Thecave3 deleted the feat/optional-sctp branch August 27, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant