Sendspin Conformance

Capability-aware conformance harness for local Sendspin implementations.

Current scenarios:

• client-initiated-pcm (Client initiates connection and client wants PCM): start the server first, let the client discover/connect to it, advertise PCM as the only supported audio format, and compare canonical PCM hashes
• server-initiated-pcm (Server initiates connection and client wants PCM): start the server first, let the client advertise a listener and PCM as its only supported audio format, let the server connect in, and compare canonical PCM hashes
• server-initiated-metadata (Server initiates connection and client wants Metadata): start the server first, let the client advertise a listener, let the server connect in, receive a metadata snapshot, and compare normalized metadata fields
• server-initiated-artwork (Server initiates connection and client wants Artwork): start the server first, let the client advertise a listener, let the server connect in, receive album artwork bytes, and compare the encoded image hash
• server-initiated-controller (Server initiates connection and client wants Controller): start the server first, let the client advertise a listener, let the server connect in, observe controller state, send a control command, and verify the server recorded it
• server-initiated-flac (Server initiates connection and client wants FLAC): start the server first with PCM audio decoded from almost_silent.flac, let the client advertise a listener and FLAC as its only supported audio format, let the server connect in, encode the PCM to FLAC using the SDK, stream it to the client, and compare the transported FLAC bytes
• server-initiated-opus (Server initiates connection and client wants OPUS): start the server first with PCM audio decoded from almost_silent.flac, let the client advertise a listener and OPUS as its only supported audio format, let the server connect in, encode the PCM to OPUS using the SDK, stream it to the client, and compare the transported OPUS bytes
• server-initiated-pcm-24bit (Server initiates connection and client wants 24-bit PCM): start the server first, let the client advertise a listener and 24-bit PCM as its only supported audio format, let the server connect in, negotiate the 24-bit packed wire format, stream it to the client, and compare canonical PCM hashes — a client SDK with no 24-bit decode path misreads the bytes and produces a hash mismatch
• client-initiated-request-format-pcm (Client renegotiates PCM bit depth): start the server first, let the client connect advertising two PCM formats (24-bit then 16-bit), let the server stream the preferred 24-bit format, have the client emit stream/request-format for the 16-bit format, and verify the client observed the server re-emit stream/start with the renegotiated 16-bit format
• client-initiated-request-format-flac (Client renegotiates PCM to FLAC): start the server first, let the client connect advertising PCM then FLAC, let the server stream the preferred PCM format, have the client emit stream/request-format for FLAC, and verify the client observed the server re-emit stream/start with the FLAC format

Current coverage

• aiosendspin: real server adapter, including OPUS encoding, and real client adapter; the client SDK decodes PCM and FLAC only, so it does not advertise OPUS
• sendspin-dotnet: real client adapter for client-initiated PCM plus the server-initiated PCM, metadata, artwork, controller, and FLAC scenarios; server placeholder
• SendspinKit: real Swift client adapter for client-initiated PCM, Flac, and Opus plus the server-initiated PCM, Flac, Opus, metadata, artwork, and controller scenarios; server placeholer
• sendspin-cpp: real C++ client adapter for client-initiated PCM plus the server-initiated PCM, metadata, artwork, controller, and FLAC scenarios; server placeholder
• sendspin-go: real Go client adapter and real Go server adapter across the PCM, FLAC, metadata, artwork, and controller scenarios (no OPUS yet)
• sendspin-js: real Node.js client adapter driving the public SendspinCore SDK over an adapter-owned WebSocket; covers client-initiated PCM plus the server-initiated PCM, metadata, and controller scenarios (FLAC/OPUS decode and artwork@v1 are not exposed by the public SDK in a headless Node environment); server placeholder
• sendspin-jvm: real JVM client adapter written in Kotlin, for client-initiated PCM plus the server-initiated PCM, metadata, artwork, controller, and FLAC scenarios; server placeholder
• sendspin-rs: real Rust client adapter for client-initiated PCM plus the server-initiated PCM, metadata, artwork, controller, and FLAC scenarios, and the client-initiated stream/request-format renegotiation scenarios; server placeholder

The OPUS scenario is currently exercised only by the SendspinKit and sendspin-cpp clients against the aiosendspin server until other implementations opt in via supports_opus.

The client-initiated-request-format-* scenarios are currently exercised only by the sendspin-rs client against the aiosendspin server until other implementations opt in via supports_request_format.

Unsupported client roles use fail-fast adapters that emit a summary and exit non-zero. Unsupported server roles are filtered out before case creation, so the matrix only shows server rows that can actually run a scenario.

Quick start

python scripts/setup_workspace.py --clone
. .venv/bin/activate
python scripts/run_all.py

That flow:

• clones the required repositories
• installs the Python harness and aiosendspin
• builds the adapter sources that are available locally
• runs the current matrix for the selected host environment
• generates the static HTML report

Useful commands

Run the full harness:

python scripts/run_all.py --results-dir results --build-report-path artifacts/build-report.json

Run the full harness for an explicit host label:

python scripts/run_all.py \
  --results-dir results \
  --build-report-path artifacts/build-report.json \
  --environment-id linux \
  --environment-name Linux

Run a subset of the matrix:

conformance run --from aiosendspin,sendspin-rs --to SendspinKit

Run the matrix with parallel case execution:

conformance run --jobs 4
python scripts/run_all.py --jobs 4

The runner assigns a dedicated server port and client-listener port to each case, so parallel cases do not fight over 8927/8928.

Build the adapter sources only:

conformance build --report-path artifacts/build-report.json

Generate the static site from existing results:

conformance report --results-dir results

Merge multiple host result sets into one combined report:

python scripts/merge_results.py \
  --output-dir artifacts/results \
  artifacts/linux-results \
  artifacts/macos-results

Report site

The generated site includes:

• a global matrix overview with one section per test scenario
• a single matrix view per scenario without host-specific grouping in the UI
• the greener PCM scenarios listed first on the index page
• a separate static HTML page per test under results/scenarios/
• a dedicated static HTML page per pairing under results/cases/
• a dedicated static HTML page per implementation under results/implementations/ for linkable filtered overviews
• the audio formats each client declared in client/hello, marked exercised or not verified, on that client's implementation page
• per-case status and reason with explicit server/client labeling
• summary, server, client, and build tabs on each dedicated case page when build data exists
• run artifacts under results/data/, including declared-formats.json for the declared-versus-exercised data
• the static report at results/index.html
• linked case artifacts for drill-down: result.json, client/server summaries, and logs

Repository layout

• src/conformance/: runner, adapters, fixture decoding, report generation
• adapters/sendspin-go/: Go client/server adapter source
• adapters/sendspin-dotnet/: .NET client adapter source
• adapters/README.md: CLI contract for adapters
• scripts/setup_repositories.py: clones implementation repositories
• scripts/setup_workspace.py: bootstraps a local Python environment
• scripts/run_all.py: build + run + report orchestration
• scripts/merge_results.py: merges multiple host result directories into one report
• .github/workflows/publish.yml: macOS GitHub Pages publishing