WP15: head-tracked HOA to binaural monitoring + JUCE 9.0.1 - #5
Merged
Merged
Conversation
…ing) What to take as-is from spatcore (orientation sources, wfs_headtrack plugin ABI, HeadFrame math, SOFA bake), copy-and-adapt from WFS-DIY (webcam source, tracker manager, fast-path pattern), and build new (SH rotation, MagLS-style decode); load-bearing conventions and milestones. Prerequisite: bump spatcore pin to >= c8fd7bf. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…) + WP15 stage 0 Vendors ThirdParty/libmysofa + ThirdParty/zlib (trimmed trees from WFS-DIY) and wires spatcore-mysofa; bundles the SADIE II KU100 set at assets/SOFA (default binaural monitoring set + smoke fixture). Adds the fixture-gated XoaSofaSmokeTests (SofaLoader -> HrirDatabase, ITD/alignment sanity). Docs: WP15 promoted from the post-v1 backlog with decision records D51-D54 (HOA-native path, post-dual-band tap, RT orientation pull, no OSC yet); CLAUDE.md pin + stale next-D-number corrected; third-party notices for libmysofa / zlib subset / SADIE II. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…15 stage 1) The orientation half of binaural monitoring, with no audio-path change: the offline-render harness is byte-identical to the previous commit (verified by building both and diffing --check output). - AmbiHeadMapping.h: the spatcore<->XOA convention boundary. spatcore's HeadOrientation (+yaw right / +pitch up / +roll right-ear-down, radians) becomes an XOA soundfield rotation, and compensation is the matrix TRANSPOSE, never three negated angles. AmbiRotation forbids flips inside the DSP, so they live here and only here. - XoaCameraHeadTrackerSource / HeadTrackerManager: ported from WFS-DIY. The wfs_headtrack plugin ABI, exeDir-first load, ABI-check-first, plugin-thread zero calibration by matrix composition, 1-Euro filtering and the 300 ms staleness fallback all carry over verbatim; XOA has no logger and no settings file, so failures surface as a status string and the camera index comes from the store. - XoaMonitoringEngine: message-thread controller owning the trackers. A tracker that fails to start resolves to manual WITHOUT clearing the persisted id, so the selection survives an unplugged camera. - Monitoring section: binaural* ids route there (new scope + undo domain), persist as monitoring.xml, and a project now carries a sofa/ folder. Defaults keep the monitor OFF. No OSC bindings by design (D54). - MonitoringTab: enable/gain, tracker combo that rescans on popup, Set Zero shown only while a tracker is active, manual attitude dials and a live orientation readout. Tests: convention goldens anchored on the encoder (turn right -> source moves left, look up -> source drops, right-ear-down -> left source drops), the transpose-is-inverse and composition-is-not-per-angle-negation traps, and the Monitoring section's routing, undo isolation and project round-trip (including a pre-WP15 project with no monitoring.xml). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
designShFilters() projects a baked HRIR grid onto the SN3D/ACN basis:
h_ear[c][n] = (2l+1)/(4pi) * sum_g w_g * Y_c(d_g) * hrir_ear(d_g)[n]
the same law as XOA's SAD speaker decode with the virtual speakers taken to
the whole grid. Two conventions are load-bearing and documented where they
are applied: spatcore's HRIR grid measures azimuth positive to the listener's
RIGHT while XOA's SH basis is +Y-left, so the basis is evaluated at the
NEGATED azimuth; and quadrature uses exact zone areas (summing to 4pi) rather
than a cos(el) Riemann weight, which would misweight the rings and give the
poles zero.
The ITD is put back before projecting. SofaLoader hands over time-aligned
HRIRs with the onsets stripped into relDelaySec; a decoder that ignored them
would have no interaural time difference at all. Re-applied with a
Blackman-windowed sinc.
Every filter carries a constant 16-sample lead. The sinc kernel rings
BACKWARD as well as forward, and without the lead any ITD smaller than the
kernel half-width had its pre-ring clipped at tap 0 — losing energy
asymmetrically between the near and far ear, i.e. corrupting the very ILD the
bank exists to carry. The lead is identical for both ears, so it is only
0.33 ms of common monitor latency. The W-channel DC test below is what caught
this.
Tests: the full 2x121 bank against an mpmath golden built from the spec (not
transcribed from the C++) over a synthetic HRIR fixture; an encoder-anchored
reconstruction check; interaural polarity (ipsilateral ear louder AND
earlier, symmetric dead ahead); and the exact W-channel anchor — for SN3D the
W basis is identically 1, so that filter must be the solid-angle-weighted
mean HRIR, which pins the (2l+1)/(4pi) factor and the quadrature weights with
no SH truncation involved.
Measured and documented: the sampling decode reproduces arrival TIME within
~1 sample but smears peak AMPLITUDE (5% ipsilateral, 20%+ on a delayed
contralateral ear), because a direction-dependent delay is not band-limited
in the SH domain. That is inherent, and is what MagLS attacks in stage 7.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The RT half. 121 SH channels in, a binaural pair out on a reserved hardware output pair, with the loudspeaker render provably untouched. - AmbiBinauralRenderer: uniform-partitioned overlap-save with a SHARED input transform. Each SH channel is transformed once per block and both ears accumulate from the same frequency-delay line, so a block costs 121 forward + 2 inverse transforms rather than 242 of each. A short or varying device block is staged through a FIFO; at the steady n == P case the step runs in the same call and adds no latency. Nothing allocates after prepare(). - AmbiBinauralFilterBank: FFT-bakes a design into partition spectra behind the same double-buffer + RtSnapshot hot-swap DecoderMatrixBuilder uses, so the pointer the RT side holds stays valid until the next publish and nothing is ever freed on the audio thread. - Tap seam (D52): one call in AmbiBusAlgorithm between the dual-band stage and the decode GEMM. The renderer only reads the bus. The bail-out path marks the monitor idle so a stale block can never reach the pair. - Output pair (D52): the Monitoring section doubles as the shared patch matrix's binaural tree, so the matrix greys the pair out and no speaker can be patched over it. Written inside the scatter phase, after every hardware input has been read (the aliasing invariant) and before the clear loop, which still silences the rows whenever the monitor is idle. - BinauralDesignWorker + XoaMonitoringEngine: SOFA load and design run on a background thread (latest-wins, generation-stamped, AsyncUpdater hop back), never on the message thread. The bundled SADIE II set is staged next to the binary and used when no project HRTF file is chosen; a repeat request for the same key is dropped so a failed load does not retry forever. - Enabling fades in over a block. Headphones are on someone's head. Tests: partitioned convolution against a direct time-domain reference for K=1/K=2/K=4; silence (not a null deref) with no bank; refusal of a bank cooked for another block size; the enable edge; short/varying blocks; and the D52 acceptance test — the same input rendered with and without a live monitor gives BIT-identical speaker output, with both paths asserted non-silent so the comparison cannot pass vacuously. Plus an end-to-end pass over the real bundled SADIE set: load, design, cook, render, finite and bounded. The offline-render harness is byte-identical to stage 2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The monitor now rotates the SOUNDFIELD by the inverse of the head attitude, so the static SH->ear bank keeps pointing where the listener's ears actually are. This is why the bank can be static at all (D51): head motion costs one block-diagonal multiply, not a new HRIR selection. - The attitude is read fresh from the active orientation source every block (D53), bypassing the damped control path; a tracker that has gone stale or lost the face reports invalid and the manual parameters in the snapshot take over in the same branch. - The rotation matrix is rebuilt only when the attitude actually moves, and a change crossfades over one block with the same idiom the scene rotation uses — a tracker stepping between blocks would otherwise click. - A zero attitude with no rotation history short-circuits to the bus itself, so a monitor with no tracker and no manual offset pays nothing. Reuses AmbiRotation (the handoff listed the SH rotation as "build new"; XOA has had it since WP4, golden-tested) through AmbiHeadMapping's convention boundary. Test: rendering a field with attitude A equals rendering the SAME field pre-rotated by the compensation for A with the head at zero — the definition of "rotate the field, not the listener" — plus a guard that the rotation does change the output, so the equivalence cannot pass for a renderer that ignored the head. Speaker output remains byte-identical in the offline harness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Monitoring tab gains the SOFA row: a rescan-on-popup combo listing the bundled SADIE II set first (stored as an empty filename) then every .sofa in <project>/sofa, plus an Import entry that COPIES the chosen file into that folder so a project stays self-contained and only a bare filename is ever persisted. A persisted set whose file has gone missing is listed as missing rather than silently dropped — the stored name survives so the selection returns when the file does, the same rule the tracker list follows. Load and design status (including libmysofa's own message) is shown next to the combo, which is XOA's substitute for the logger WFS-DIY had. Verified with the GUI smoke run (app launches, cycles every tab including the rebuilt Monitoring surface, exits clean). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lifts tools/headtrack/ from WFS-DIY. The plugin depends on nothing from
either app — it speaks the versioned C ABI in spatcore, so one built
wfs_headtrack.{dll,so,dylib} serves both, and the name is kept deliberately
rather than forking the artefact.
Only the app-shaped bits changed: the default staging directory now points at
XOA's CMake output (build/XOA_artefacts/<Config>) instead of WFS-DIY's
Projucer layout, and the README documents XOA's route to it (Monitoring tab
-> Head tracker -> Webcam, camera index as a project parameter, failures
reported in the UI since XOA has no log file).
The build gotchas already encoded in the scripts are carried over intact:
OpenCV's prebuilt config needs -DOpenCV_RUNTIME=vc16 pinned under VS2026, and
the staged runtime is opencv_world + the MSMF videoio DLL, NOT the 25 MB
ffmpeg one (that is for video files, not cameras).
Third-party notices updated for OpenCV and YuNet, both plugin-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sampling decode's measured weakness is that restoring the ITD makes the HRIR field direction-dependently delayed, and a delay is not band-limited in the SH domain — order 10 reproduces arrival TIME within a sample but smears the peak amplitude of a delayed ear by 20%+. alignedHf attacks exactly that: true phase (and thus the ITD) BELOW a 1.5 kHz crossover where the ear reads interaural time, and above it the time-ALIGNED HRIRs spatcore already provides — removing the very delay the truncation cannot represent. The two are combined with complementary real per-bin gains summing to exactly 1, the frequency-domain sibling of the loudspeaker path's dual-band decode. NAMING, deliberately: this shares MagLS's intent and crossover rationale but is NOT MagLS — there is no iterative magnitude fit with phase continuation. The plan and D51 called for MagLS; implementing something else under that name would have been the wrong kind of convenient, so the mode says what the code does and true MagLS stays recorded as future work. Two bugs the tests caught while building it: - The blend needs taps of its own. A one-octave crossover at 1.5 kHz has an impulse response ~1 ms long; without reserved headroom the truncation rang. - The crossover's BACKWARD smear wrapped around the transform and was cut, costing 2.3% of broadband DC gain. Both banks are now placed at a linear-phase lead so the smear lands inside the filter. Tests: DC gain survives the crossover (its gains sum to 1 and bin 0 is pure low band); the interaural time ordering survives in the blended bank; and the contralateral peak error is strictly SMALLER than the sampling decode's — the actual quality claim, not a proxy for it. The golden tests now pin DecoderMode::sampling explicitly, since the default moved. binauralDecoderMode is exposed in the Monitoring tab so sampling stays reachable as the golden-anchored reference. Speaker DSP byte-identical; GUI smoke clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measures what the monitor actually costs on the audio thread at order 10 rather than asserting a guess: Release, dev machine, 9.5-15% of real time with the head still and 12-18% while it is moving (the rotation matrix rebuilds per block), across 64/128/256-sample blocks. Printed every run, with a loose 35% tripwire that only applies to optimized builds — a Debug run measures roughly ten times this and says nothing about shipped code, so gating on it would just break the Debug suite. The dense-bank detail matters: the perf fixture loads all 121 SH channels, because a sparse one would flatter the very inner loop being measured. The obvious next lever is SIMD on the complex multiply-accumulate, which is essentially the whole cost. Docs: DEVPLAN WP15 records the status, the measured numbers, and the three deliberate deviations (alignedHf is not MagLS; sampling stays selectable as the reference decode; no --binaural offline baseline, since the unit suite and the bit-identical speaker assertion already pin correctness without a second hashed artefact to maintain). The original handoff doc keeps its intent but now opens with the corrections implementation found in it — the Wigner-D rotation already existed, PartitionedFirConvolver.h does not, and XOA has neither a logger nor a settings file. CLAUDE.md points at the DEVPLAN as authoritative and states the opposite-azimuth-sense trap between XOA's +Y-left soundfield frame and spatcore's SOFA-style head frame, which is the single easiest thing to get silently wrong here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion The head-track plugin ABI hands over RAW attitude, and a degenerate face box makes the geometric estimator emit inf/NaN. That reached rot::buildFromCartesian on the AUDIO thread, where it built a non-orthonormal rotation: the determinant jassert in Debug, and in Release NaN that then lived in the convolution FDL forever, silently killing the headphone bus. Guard at both ends. The camera source drops non-finite poses at the ABI boundary and never publishes a non-finite filter output; the RT stage treats any non-finite attitude as identity, because it does not get to trust its sources (a corrupt project file reaches the same code through the manual parameters). The regression test drives a source reporting NaN with valid=true and asserts the output stays finite across several blocks — it fails without either guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
spatcore bumped its own JUCE pin to 9.0.1; this matches it on the app side. JUCE 9.0.0 -> 9.0.1 is 56 commits of bugfixes (unresponsive Linux GUIs, Freetype on older distros, CoreAudio compilation, malformed-AudioFormat hardening). Both documented breaking changes are non-issues here: - The WebView interop package moved to native/typescript/webview-interop. Neither XOA nor spatcore uses WebBrowserComponent (juce_simpleweb is an unrelated HTTP/WS module). - zlib/libpng/libjpeg/libflac are now documented as built in C language mode, which would put JUCE's unnamespaced zlib symbols next to the vendored inflate subset that spatcore-mysofa links into the same binary. Checked: modules/juce_core/juce_core_zlib.c is byte-identical at 9.0.0 and 9.0.1, so that build mode was already in effect at the version we shipped. The doc entry is catching up, not a new hazard. spatcore d011103 -> 47b1ae5 (v0.2.0) is CI-only: minimal-app pinned to JUCE 9.0.1 plus libxi-dev on its Linux lane. No engine source changed. libxi-dev (XInput2, required since JUCE 9) is now listed explicitly in our Linux CI and in the documented apt lines. It arrives transitively via libgtk-3-dev today, which is why CI was green without it; the build no longer depends on that. Verified on Windows (VS 18 2026), Release and Debug: build clean (no new warnings), ctest 2/2, --gui-smoke exit 0, OSC control-replay 12/12 against the golden. macOS/Linux left to CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oling The head-tracker plugin sources (tools/headtrack/src, models) are byte-identical between the two apps; only the build tooling had drifted. - .gitignore: the Windows build script caches a ~180 MB OpenCV prebuilt under tools/headtrack/.opencv, which was showing up as untracked here. (WFS-DIY has ignored it since the tooling landed; its build/ tree is already covered by our generic build/ rule.) - build-headtrack-plugin.sh: cap `cmake --build --parallel` at the core count. Unbounded `make -j` OOM-killed WFS-DIY's 16 GB Linux runner (exit 143). Still deliberately NOT ported: the BUNDLED_OPENCV static-OpenCV path in WFS-DIY's tools/headtrack/CMakeLists.txt. It exists for release packaging, and XOA has no release workflow yet — when one lands, port it then. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tools/headtrack builds OUT of the app tree against its own OpenCV — that is what keeps OpenCV out of the app build — so nothing else in CI would catch a break there. WFS-DIY has gated it since the tooling landed; this matches. Windows caches the 4.10.0 prebuilt under tools/headtrack/.opencv (key opencv-win-4.10.0, ~180 MB, downloaded once); macOS uses brew opencv, Linux libopencv-dev. Release in every lane even though CI builds the app Debug: the plugin ships Release regardless (C ABI, no shared CRT state). Each step sits last in its job so the app build and tests fail first. Compile check only — runtime behaviour needs a camera and stays a dev-machine gate. The scripts stage next to the app only if that output dir exists, so a Debug-app / Release-plugin run just skips the copy. Verified the Windows lane locally end to end from a cold cache: download, configure against the prebuilt, build, dumpbin shows all 7 wfs_headtrack_* exports. macOS/Linux follow WFS-DIY's identical, green steps. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
WP15 — head-tracked HOA→binaural monitoring — plus the JUCE 9.0.1 / spatcore
v0.2.0 toolchain move that landed on top of it.
The monitor taps the SH bus post-dual-band / pre-decode, applies a
head-compensation SH rotation (reusing
AmbiRotation), convolves it with astatic 2×121 partitioned-FIR bank designed from a SOFA set on a background
worker, and lands L/R on a patch-reserved hardware pair. Head orientation comes
from manual dials or the webcam tracker via the shared
wfs_headtrackplugin.Decision records D51–D54;
Documentation/XOA-DEVPLAN.md§WP15 is authoritative.Commits
09a3324…1560330— WP15 stages 0–8 and the non-finite head-attitude fix1e24b66— JUCE 9.0.1 + spatcore v0.2.06676582— head-tracker tooling parity with WFS-DIY524f17a— head-tracker plugin compile-check in CIThe diff reads large (+49k) but is dominated by vendored and generated content:
ThirdParty/zlib,ThirdParty/libmysofa, the 18.6k-line binaural referencegolden, the bundled SADIE II KU100 SOFA set and the YuNet model.
What the tests prove
exactly, both paths checked non-silent)
gates
without either guard)
Measured monitor cost on the audio thread at order 10 (Release, dev machine):
9.5–15% of real time head-still, 12–18% head-moving, across 64/128/256-sample
blocks.
Verified locally on Windows (VS 18 2026), Release and Debug: build clean,
ctest 2/2,
--gui-smokeexit 0, OSC control-replay 12/12 against the golden,head-tracker plugin builds from a cold OpenCV cache with all 7 ABI exports.
Outstanding
§WP15) calls for live head-tracked monitoring on hardware; that waits on the
RME AoX Dante lab session, together with the spatcore-io stages. Merging
ahead of it is deliberate — the debugging session is expected to be long, and
parking the JUCE 9.0.1 move behind it would leave
mainoff-step withspatcore and WFS-DIY.
mainand PRs into it), so the macOS and Linux lanes — including the newlibxi-devdependency and the head-tracker compile-checks — are beingexercised for the first time in this PR.
Deliberate deviations, all recorded in the DEVPLAN
alignedHf(true phase below 1.5 kHz, time-aligned above),not MagLS. True MagLS remains open; the mode enum has room for it.
samplingstays selectable as the golden-anchored reference decode.--binauraloffline-render baseline (D54: no OSC surface either).🤖 Generated with Claude Code