Feat/tts benchmark - #281
Merged
Merged
Conversation
Eighteen TTS configurations speaking one 84-phrase corpus, transcribed back by Whisper so word error rate measures intelligibility end to end. Every row is playable, which is the point: word error is blind to naturalness and expressiveness, the axes the LM-backed engines are actually sold on, so the table needs an ear alongside it. Tabs are real links to each benchmark's own URL rather than client-side state, so both stay deep-linkable, keep their own social card, and render exactly one post per page, which is also what keeps heading ids unique. /benchmarks now shows the default benchmark instead of stacking every one into a single document. Playback goes through Web Audio rather than an <audio> element, following the original export. canplaythrough is a heuristic about download rate, so playback could start mid-decode and swallow the first word, which for a speech benchmark is disqualifying; decodeAudioData resolves only once the whole clip is PCM in memory. A silent buffer on first interaction wakes the macOS output device, which otherwise eats the first couple of hundred ms. Data is extracted from the export's own table rather than results.txt: it carries eight more columns and precomputed sort keys. A first extraction pass silently dropped the four NeuTTS rows because they use <tr class="neu"> for the shaded band, so row count is asserted against the 18 in results.txt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The export uses lowercase run ids (nt-2e-q4-metal, kittentts, speecht5), which are fine as identifiers but wrong as the thing a reader sees. Adds a display name per row and uses it in the table, the play button labels and the prose: NeuTTS-2E, KittenTTS, Qwen3-TTS, Chatterbox, Kokoro, VibeVoice, StyleTTS2, MeloTTS, CosyVoice3, CSM, SpeechT5, Bark, Piper. Quantisation and device suffixes are preserved verbatim after the base name, so NeuTTS-2E-q4-metal still says which artefact it is. The raw id is kept as the row key and the audio filename, so nothing about the data or the samples moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Word error rate is blind to naturalness and expressiveness, which is what the LM-backed engines are actually sold on, so the table needs a human axis. This is the backend for it. GET /tts/ballot issues a blind four-item comparison set. The set is chosen server-side so a client cannot nominate what it is voting on, and the anchor is never returned so the attention check is not detectable. POST /tts/ballot records a ranking, requires it to be a permutation of exactly the items issued, and marks the ballot rejected rather than deleting it when the seeded weak engine is placed first, so the rejection rate stays auditable. GET /tts/scores aggregates with Bradley-Terry rather than mean rank: engines do not all face the same opponents, and mean rank rewards whoever drew weak company. Verified against synthetic noisy ballots with a known ground truth, where it recovers the exact ordering. Ratings map to 0-100 through a logistic of log-strength. A linear map is unbounded and produced ratings above 130 and below zero on a well-separated field, which reads as broken in a table column; the logistic cannot leave the range and preserves the ordering. Nothing is shown until a configuration has 8 ballots, so early noise never renders as fact. The table is defined in the SCHEMA constant that initializeDatabase executes, with a comment recording why: blog_subscribers was appended to a conditional migration function that returns early on any migrated database, so it can never be created by /init-db. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Takes the refreshed export: adds chatterbox-turbo and fastpitch, and restricts the table to configurations at or under 2.0x RTF. A model that cannot keep up with its own speech is not a candidate for reading anything aloud as it happens, which is what this benchmark is for. Seven configurations fall out: chatterbox and its q4/q8 builds, both NeuTTS-2E fp32 builds, csm and bark. Note the two sources disagree slightly, and the HTML is the newer one: chatterbox-q4 is 1.97x in results.txt but 2.03x in benchmark.html, so it misses the cut by 0.03 on the numbers actually used. The prose is rewritten around the surviving field rather than left pointing at engines the filter removed. Chatterbox-turbo is now the headline arrival: 4% word error at 0.98x, the first LM-backed engine here to match Piper's accuracy and still keep up with its own speech. The quantization argument is recast, since every NeuTTS-2E build that survives the filter is quantized and the fp32 builds are excluded outright. Also fixes an entity-decoding bug that rendered the not-yet-scored PESQ marker as a literal ·, by decoding with a real HTML entity decoder rather than a hand-rolled replace list. The ballot pool is narrowed to the 13 surviving configurations. This weakens the attention check: bark at 55% word error was unmistakably bad to any listener but runs at 4.26x and is now excluded, leaving speecht5 as the only anchor. Recorded in the handler so the limitation is not silently inherited. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Chatterbox-q4 measures 2.03x in the export and was falling out by 0.03. The cutoff moves to 2.1x, which keeps 14 of 20 configurations and drops six: chatterbox and its q8 build, both NeuTTS-2E fp32 builds, csm and bark. The article records why the line sits where it does, and flags that q4 is the one row here that is only nominally real time: RTF counts silence, and against speech alone that configuration is 2.59x. Its arrival also sharpens the quantization argument, which now runs across a single family: the full Chatterbox build is 4.41x and excluded, q4 scrapes in at 2.03x, and turbo reaches 0.98x while improving on both. Ballot pool widened to the same 14. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The two NeuTTS-2E Q4 rows differed by device, not by weights. Device
changes how fast the same model runs, not how well it speaks, so CPU and
Metal are now one row: word error is the mean of both builds over 336
samples, and speed, size and memory are Metal's. The case for treating
the gap as noise is that the stronger transcriber puts the two 0.09
points apart and only the weaker one separates them at all. PESQ and
lead-in stay unpooled because both are scored on a single clip, and only
Metal's is published.
Display names now come from a variant map, so "nt-2e-q4-metal" reads
"NeuTTS-2E Q4 Metal" and "chatterbox-q4" reads "Chatterbox Q4".
The table also gets the width it needs. The article column is sized for
prose and cut roughly half the columns off, so the figure now measures
the empty gutter to its right and extends into it, up to the ~1756px the
full table wants. Three things this took, each of which failed on its
own:
- Not the 50%/50vw negative-margin trick. It centres on the viewport,
but this column is pushed right by the TOC sidebar, so the figure
hung 83-288px off the right edge.
- Not an explicit width either. A fixed width raises the article grid
track's min-content size, which grows the whole two-column layout;
the figure's own box fit fine while the page scrolled sideways.
A negative right margin buys the same pixels and lowers the
intrinsic contribution instead of raising it.
- overflow-x: clip on the figure. Without it the page still scrolled a
constant 1288px at every viewport: the inner scroll container clips
what you see, but the track kept sizing itself to the table
unclipped. `clip` rather than `hidden` so the vertical axis is not
forced to scroll too.
Verified at 430/800/1000/1280/1440/1600/1920/2200px: no horizontal page
scroll at any width, and the table gets 1280px at 1920 where it had 960.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Word error rate saturates once speech is merely intelligible, so the
table cannot separate the engines that are actually sold on naturalness
— which is most of the expensive ones. The Human column is that missing
axis, and it is the one number here not measured on my machine: readers
rank four blind clips, the control plane aggregates the pairwise
outcomes with Bradley-Terry, and the column updates live.
The dialog opens on the reader's first play, which is the moment they
have opted into listening and the one point where asking is not an
interruption. Asked once per browser, skippable, and re-openable from
the caption. Three things it is strict about, because the data is
worthless otherwise: clips are blind until the ranking is submitted;
reordering is buttons rather than drag, which is unusable on touch and
absent for keyboard users; and submit stays disabled until all four have
played to the end.
Verified against a local D1 with 240 simulated ballots, of which 3 were
junk voters who ranked the seeded weak engine first: all 3 were rejected
by the anchor check, and the aggregator recovered the planted quality
order exactly, with every rating inside 0-100.
Playback moves into useSamplePlayer so the table and the dialog share one
AudioContext, one buffer cache, and one "currently sounding" clip.
Three bugs found and fixed while testing this:
- Clip letters were derived from list position, so reordering renamed
every clip mid-ranking and moved the "already played" tick onto
different audio. Letters are now fixed when the ballot is issued.
- The row the reader originally clicked never played after the dialog
closed: starting it in the close handler raced the dialog's own
teardown, which stops whatever is sounding. It now waits for unmount.
- Chart fences render a React component but stayed wrapped in the
markdown <pre>, so every chart inherited white-space: pre. The TTS
caption became one 2643px line running off the page. Both benchmark
pages now unwrap a chart fence's <pre>; this also quietly fixes the
two agent-skills charts.
The run caption claimed Bark and full Chatterbox were "still on one
pass", but the real-time filter drops both. Rewritten against the Passed
column: two passes for autoregressive, one for feed-forward, one for
Chatterbox Turbo, and 336 for the pooled NeuTTS device builds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The article's headline claim — that nothing beats Piper on all three
axes at once — was only ever in the prose. The chart puts every
real-time engine on compute-per-phrase against word error, sizes each
marker by disk, and lets the reader see Piper alone in the corner with
the expensive LM-backed cluster off to the right buying no
intelligibility for twenty to fifty times the compute.
Two axis choices worth stating, both argued in the article:
- x is compute per phrase, not the conventional RTF. RTF is compute
per second of audio, so it flatters any engine that pads with
silence; every engine here speaks the same 84 phrases, so per-phrase
compute compares directly and cannot be gamed.
- y is WER base rather than the stronger WER med, only because
Chatterbox Turbo and FastPitch have no medium-model score and
dropping the headline arrival from the headline chart would be
perverse. base.en understates good engines, so the real spread is
wider than drawn.
Colour follows the architecture class in a fixed order, so isolating a
class never repaints the ones that remain. The skill's palette validator
was not on disk, so the six checks were reimplemented and run rather
than eyeballed: the three hues separate by at least 14.8 in OKLab under
the worst of protan/deutan/tritan against a floor of 8, all clear the
0.45-0.85 lightness band and the chroma floor, and every hue is at least
4.3:1 on the chart surface. Two candidate third hues were rejected on
those numbers before this one passed with no warnings.
The social card is generated from the live chart by scripts/build-og-card.mjs
rather than drawn separately, so it cannot drift from the data — re-run
it when a run lands. It scales the figure to fit 1200x630 instead of
cropping, which in the first attempt lost the title off the top and the
x-axis off the bottom.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Word error is the wrong y-axis for the question the chart was asking. It saturates once speech is merely intelligible, so it cannot separate engines that are all comprehensible and differ only in how they sound; two of the thirteen have no medium-model score, so the stronger transcriber cannot be used without dropping them; and the article already spends a section warning that base.en understates good engines more than bad ones. Ranking engines on a measure the page tells you not to trust is not a chart worth publishing. So the chart now plots reader preference against compute per phrase, and waits: it renders nothing at all until at least six engines have a human rating, then appears on its own with no redeploy. It sits after the table rather than before it, since it is now downstream of the ballots the table collects. It carries no headline claim, deliberately. Whether compute buys preference is precisely what is being measured, and asserting the finding before any ballots exist would be inventing it. The title says what the axes are; sharpen it once the shape is known. The table now opens sorted by compute per phrase, cheapest first, for the same reason word error is a poor way to meet it. Verified both states against a local D1: empty, the page renders one figure and no empty axis box; seeded with 240 ballots, the chart appears below the table with all thirteen engines and every label inside the plot box. That last part needed a fix — labels sit above their marker, which puts the best-rated engine's name through the top border, so labels near the ceiling now flip underneath. The social card is dropped for now rather than left showing a chart the page no longer has. scripts/build-og-card.mjs still generates it; re-run it against the preference chart once there are enough ballots to draw. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The index was a landing page that listed one thing and then asked you to
click again. /benchmarks now redirects to the default benchmark, and the
tabs on each benchmark do the navigating.
The redirect is temporary (307), unlike the /labs ones. /benchmarks is
the URL people share and the one in the site nav, and which benchmark is
default will change as more land; a 308 would be cached by browsers and
CDNs and strand readers on whichever benchmark happened to be first
today.
Two consequences of the change, both handled:
- /labs pointed at /benchmarks, which would now bounce again. It goes
straight to the default instead, so a link someone else published
still costs one hop. Verified: /benchmarks, /labs and the old Labs
post URL are all num_redirects=1, and /labs/og-do-skills.png still
resolves for social cards already cached against it.
- Every benchmark had an "All benchmarks" back link to a page that no
longer exists. It either did nothing or silently moved you to a
different benchmark, so it is gone; the tabs sit in its place.
The 286-line index page is deleted rather than left unreachable, along
with DEFAULT_BENCHMARK, whose only consumer it was. The default now lives
in one place: the first entry of BENCHMARK_TABS, mirrored in next.config
with a comment on each side, since next.config cannot import it.
Checked the thing a redirect most easily breaks: a crawler following
/benchmarks reads the destination's tags, so the shared link still
previews as a chart (og-agent-skills.png, confirmed 200 and image/png)
rather than falling back to the generic site card.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Passed was the denominator behind the word error rates (83/84, 162/168), which is methodology rather than a result — it never distinguished one engine from another, and the sample counts it carried are already stated in the caption. Frame rate was blank for nine of the thirteen engines, because it only means anything for the token-based ones, so it was mostly a column of dashes. Sixteen data columns now instead of eighteen, so the table needs 1662px rather than 1756 and the breakout cap comes down to match; leaving it at 1760 would have padded dead space onto the right on wide screens. Nothing referenced either column by index — the chart looks its columns up by name — and the header/body cell counts were checked to stay aligned after the splice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The markdown is the source but the pages render from src/data/benchmarks-posts.json, which scripts/build-content.mjs generates, because Workers have no runtime fs. The rewritten intro and description were edited in the markdown without that regenerating, so the live page and the social description were still the old text. Verified after rebuilding: the new opening renders and the old one is gone from the HTML, and <meta name="description"> matches the frontmatter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The table listed only what could keep up with real time, which meant the answer to "how far off is it" was missing and the exclusion could not be checked. Every configuration measured is now listed, and a Real time only control hides the nine above 2.1x for readers who only care about what can be spoken live. Default is everything. Fourteen configurations arrive with this export: the full Chatterbox and its q8 build, both NeuTTS-2E fp32 builds and Q4 on CPU, CosyVoice3 RL, VibeVoice 1.5B, F5-TTS, OmniVoice, BananaMind TTS, Zonos, CSM, Parler-TTS and Bark. The export also adds a Licence column, which is kept - for open weights it is often the deciding column, and it flags the four that are restricted rather than permissive. The import is now a script rather than done by hand, because this is the second export and hand-transcribing 27 rows of 19 columns is how rows go missing. It parses benchmark.html, which unlike results.txt carries the sort keys, tone classes and row grouping, and it asserts its row count against the README's own prose count. Two parsing traps are handled explicitly, both of which have silently eaten data here before: attributes are read as a set, since `title` can precede `data-sort` and a positional pattern then yields empty sort keys for exactly the numeric columns; and rows match `<tr...>`, since the grouped NeuTTS rows carry a class and a bare `<tr>` drops all four. It also prunes audio for configurations that are no longer listed. My earlier hand-pooling of the NeuTTS Q4 device builds is dropped: this run measures them separately and afresh (9% on CPU, 11% on Metal), so pooling would now mean overwriting measured values with my own arithmetic. The four-row band is shaded instead, which is what it is for. Ballots draw from all 27, and Bark returns as an attention check. It was excluded before only because the table was real-time-only, and at 55% word error it is the engine nobody listening could rank first in good faith. VibeVoice 1.5B joins it at 37% and the lowest PESQ measured, so there are two anchors rather than one - with a single known-bad engine a determined submitter can just learn never to rank it first. Three claims in the article were true only of the 13-row table and are corrected against the new numbers: SpeechT5's PESQ is no longer the only score under 3 (eight are), it is no longer the worst engine (Bark, Parler-TTS and VibeVoice 1.5B are worse), and Piper is no longer smallest on disk (BananaMind TTS is 38 MB against its 60 MB) - though nothing still beats it on either accuracy or compute per phrase. Verified: filter toggles 27 <-> 18 with a max shown RTF of 2.03, sort and the shaded band survive it, all 27 clips resolve, and no viewport from 430 to 1920px scrolls the page sideways. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
A third click on any column header dropped the table to "unsorted", which rendered the export's own row order. That is neither the order the reader arrived at nor one anything on the page explains, so the table silently rearranged itself into something arbitrary and there was no way back to the default short of a reload. The cycle is now asc -> desc -> default (compute per phrase, fastest first), so every state the reader can reach is one the page accounts for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Four engines arrive: MMS-TTS, Dots-TTS, and the Tada 1B/3B pair. The comparison is 31 rows, of which 20 keep up with real time. Two things the importer caught that would otherwise have shipped. The row-count assertion fired: the README counts 33 configurations but the table holds 31. That turned out to be correct rather than a parse failure - kugelaudio failed outright (0 of 34 phrases, no audio) and xtts produced no PESQ and no sample, so the exporter lists their measurements without putting them in the comparison. The check was comparing a parse against a prose count of everything *measured*, which cannot tell "the exporter left it out" from "my pattern missed it", so it is replaced by two checks that can: the matched rows must equal the count of <tr> tags in the tbody, and every configuration parsed must appear in results.txt. The reverse direction is now reported rather than fatal. The real find: the export gives unmeasured cells sentinel sort keys, -1 for PESQ and Params, 99 for WER med and Lead-in, so they pile up at one end of a sort. That is fine in a static table and wrong in a sortable one - sorting by PESQ ranked the three engines that have no PESQ as the worst-sounding in the table, above VibeVoice 1.5B, and thirteen with no medium-model score sorted as the least accurate. 53 cells across seven columns were affected. The key is now blanked whenever the cell reads as missing, so the table sinks them at both ends, which is what "not measured" should do. Licence "?" is deliberately left sortable: there it is a real category meaning the card could not be verified. Verified: PESQ ascending now leads with VibeVoice 1.5B at 1.60 and descending with Piper at 4.10, with the three unmeasured rows last in both directions; the filter toggles 31 <-> 20. Ballots draw from all 31. Article counts corrected: thirty-one configurations, eleven above the cutoff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Libs is a packaging detail already folded into Disk. Lead-in is measured on a single clip so it was never a mean, and in a table this wide it was costing more room than it earned - the caveat it existed to explain (that Whisper transcribes leading silence as a word, so padding inflates word error) moves into Scoring, where it applies to every row rather than one column. "Total disk" becomes "Disk" and NeuTTS's licence sentence becomes "NeuTTS, <$5m", which was the widest cell in its column. Fifteen data columns now, and the table measures 1603px rather than 1806, so the breakout cap comes down to match. The rename broke the page, which is worth recording. The preference chart resolves its columns by name, and COL["Total disk"] silently became undefined, indexed into nothing and threw at module load - taking the whole route to a 500. tsc cannot catch it because the lookup is typed Record<string, number>, so any string compiles. Both components now resolve columns through a checked helper that throws naming the missing column and listing what the run actually has, instead of failing several frames later with a stack pointing at React. The licence rewrite uses a replacement function rather than the string "<$5m": as a string, "$5" is a capture-group reference, and would quietly start substituting the day that pattern gains a fifth group. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Every row is now plain. The article paragraph that pointed at "the shaded band" refers to the four NeuTTS-2E rows instead, since there is no longer a band to see. The group field stays on the row data - the export still marks those rows, and dropping the highlight is a presentation choice, not a reason to discard what the run recorded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Samples were silent on iPhone in both Chrome and Firefox, which is one bug rather than two: every iOS browser is WebKit underneath. Dev serves the files correctly (200, audio/mpeg, right sizes), so this was playback, not delivery. iOS puts Web Audio in the "ambient" audio session, which the hardware ring/silent switch mutes. <audio> elements and native apps sit in "playback" and ignore it - which is exactly why Spotify stayed audible while the page was not, and why that comparison does not rule the switch out. The context now declares navigator.audioSession.type = "playback", the same category a music app uses. Safari 16.4+, ignored elsewhere. Second fix, same symptom: playback goes prime() -> await decode() -> start(), and decode() is a network fetch, so by the time start() runs the user gesture is long over. A context still suspended at that point plays nothing at all and the source is spent. The context is now re-asserted immediately before start, and if it will not run the attempt stops rather than consuming the source on silence. Both failures previously looked identical to a dead button, and a phone has no console to check, so the state is now visible: the table and the ranking dialog say that playback was blocked and to check the ring/silent switch. The dialog needed it most - it opens before the table, so a blocked context there is four dead buttons and a submit that can never enable, with nothing on screen explaining why. Desktop playback re-verified unchanged. The iOS half cannot be verified from here; if it is still silent, the on-screen message now distinguishes "blocked" from "failed to load" rather than leaving it a guess. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Two things the dialog did not communicate: that a reorder had taken effect, and how to reorder in the first place. The list used to teleport into its new order, so on a phone the press and the result were impossible to connect. Rows now animate between positions with a FLIP: React reorders them, then each row is put back where it was with a transform and transitioned to zero. Respects prefers-reduced-motion, where the swap stays instant. The move is also announced in a live region, since an animation says nothing to a screen reader. "Put them in order, best first" described the goal but never the mechanism. The instructions now name the arrows and say what they do, and promise the order can be checked before anything is sent - which it now can: submitting opens a confirmation listing the ranking best first, with Go back and Confirm. Clips stay playable there, because reviewing an order you cannot re-hear is not much of a review, and they stay blind, because revealing the engines before the ballot is recorded is exactly what the blind ranking exists to prevent. Verified: mid-swap the moved row carries a 60px inverted transform with transition none and settles to none; under reduced motion there is no transform at all; Go back preserves the reordered list; and the full path still records a ballot and reveals the names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The dialog handed you the clips already in an order, so the cheapest
path through it was play 1, 2, 3, 4 and press submit — which records the
order the server happened to shuffle them into, not an opinion. The slots
now start empty. There is no default to accept: a ballot is four
deliberate placements or it does not exist.
Four clips sit in a tray above four slots labelled BEST, 2nd, 3rd,
WORST. The ends are named rather than numbered because "BEST" states the
task where "1st" only states a position.
Placing works three ways, because any single way excludes someone. Drag
is built on pointer events and pointer capture, NOT HTML5 drag and drop,
which does not fire on iOS at all — and iOS is where this was reported.
Tapping a clip then a slot costs the same two taps and is what actually
works with assistive touch. Keyboard is focus, Enter, focus, Enter.
Three bugs found while building it, all of which would have shipped:
- Binding move/up in an effect keyed on drag state attached the
listeners one render after pointerdown, so a quick tap — the common
case on a phone — sent its pointerup before anything was listening
and the press did nothing. Pointer capture on the chip removes the
gap entirely.
- "Has it moved far enough to be a drag" measured from the previous
pointer position rather than the press, so a slow drag never crossed
the threshold and stayed a tap forever.
- A tap on a placed clip both selected it and removed it, the two
cancelling out. A tap now means one thing per context: pick up in the
tray, put back in a slot.
On a phone the copy pushed the WORST slot off screen, which defeats the
point of naming the ends, so the text is shorter and the boxes tighter:
BEST through WORST and both buttons now fit an iPhone SE without
scrolling. The tray is a grid, not wrapping flex — flex put three clips
on one row and the fourth alone below, which reads as a mistake. Four
across on desktop, 2x2 on phones.
Verified on desktop mouse, simulated touch, and a CDP touch drag: all
three placement paths land, the ghost follows the finger, and submit
stays shut until every clip has both been played and placed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
"What the numbers say" was the interpretation — which engine wins, what quantization buys, which four are not really in the running. Thirty-one rows that can each be played and re-sorted make that case without being told it, and the interpretation was also the part that went stale fastest: three of its claims had already had to be rewritten across the last two exports because they were only true of a smaller table. The intro line naming Piper and SpeechT5 as the ends of the scale went with it, being the same interpretation in miniature; it now just says the rows can be played. What stays is everything that says how the numbers were produced or how to read them: the column guide, which exists because RTF and WER base actively mislead taken at face value, and Methodology with its exclusions and limitations. TOC and anchors re-checked: five headings, no dead links. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Thirty-four configurations compared, twenty-two of them real time. New
arrivals: XTTS, which the previous export measured but could not show,
and the NeuTTS-2E Q8 pair.
Every one of the 32 clips is new audio under its old filename. That is
the case worth guarding: the bytes change, the URL does not, and browsers
and the CDN go on serving last week's audio against this week's numbers -
wrong in the way that is hardest to notice, because everything still
plays. Each sample URL now carries an 8-character hash of its own
contents, so changed audio is a different URL while unchanged audio keeps
its own and stays cached. The importer reports what it replaced, and
re-running it is a no-op, which is how the 32-of-32 replacement was
confirmed rather than assumed.
Two bugs found doing it, both of which defeated the point:
- The run JSON was serialised before the copy loop that appends the
hashes, so the audio was correctly replaced and every URL in the data
still pointed at the unversioned name. The write moved after the copy.
- The Q8 pair has no published clip, and the ranking dialog drops what
it cannot play - but submitBallot requires the ranking to be a
permutation of the items it issued, so seeding a clip-less
configuration produces a ballot the reader can complete and the
server is then obliged to reject. The ballot pool is now built from
configurations that actually have audio: 32 of the 34.
Verified in the browser: 34 rows, the two clip-less rows have their play
control disabled rather than failing on click, and the audio that loads
is 200 on a versioned URL.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The zip was re-cut with the two NeuTTS-2E Q8 clips that were missing, so every row in the table is playable and the ballot pool is the full 34 rather than the 32 that had audio. Only those two files changed; the importer's own comparison said so and a by-filename diff of the directory agreed. Worth noting the first check I wrote to confirm that was wrong, not the data: it piped two md5 lists sorted with `sort -k2` into `comm`, which compares whole lines and so needs whole-line ordering, and it duly reported nine changed files that had not changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
WER med is dropped and WER base becomes simply WER. With one column there is nothing to qualify, and base.en is the right one to keep: it scored every configuration, where medium.en did not finish for sixteen of the thirty-four. The column guide is rewritten to match. It keeps the part that matters - base.en is the weaker recogniser and hallucinates words onto trailing silence, which understates good engines more than bad ones, so the real spread between engines is wider than the column shows. That caveat was the point of the original entry and it survives the column going; what goes is the comparison between two columns, only one of which is now there. Fourteen data columns, table down to 1530px, breakout cap follows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The NeuTTS band was one backbone measured on two devices at three precisions, six rows of a thirty-four row table. The CPU builds go, and with them gone the remaining rows need no device suffix: NeuTTS-2E FP32, Q4 and Q8. Thirty-one rows, twenty of them real time. Done in the importer rather than by hand, so the next export inherits it, and the three orphaned clips were pruned from public/ automatically. The methodology paragraph that existed to explain a four-row band is rewritten for a three-row one, and both counts it depends on were recomputed against what is actually shown: eleven configurations above the cutoff, and fifteen of thirty-one with no medium.en score. That paragraph originally ended with "quantizing is worth roughly four times what the device is", which I then checked and it is not: on the GPU path quantizing fp32 to Q4 or Q8 is a 5.0-5.4x speed-up, while moving off the CPU is 1.31x once quantized and 0.99x - nothing - at fp32. The sentence now says that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The export re-cut the audio for FastPitch, BananaMind TTS, Parler-TTS and Bark. Exactly those four files changed and nothing else, confirmed by diffing the directory by filename before and after; their URL fingerprints rotated with them, so returning readers get the new audio rather than whatever their browser cached. The chart is back, plotting word error against compute per phrase. It is a new component rather than the old one revived, because two things have changed since: the column is now called WER, and the table is no longer real-time-only. That second one forced the axis. Compute per phrase now runs from Piper at 0.12s to Dots-TTS at 41.7s, a factor of 338, and on a linear axis seventeen of the thirty-one engines pile into the first eighth of the width. So x is logarithmic, with decade-anchored ticks labelled in seconds and the log scale called out in the caption - a log axis that is not announced is a way of understating differences without saying so. The run also has a fourth Class value, "?", for the one engine whose architecture could not be established. It is drawn in muted ink rather than a fourth hue: it is a missing value, not a category, and a colour alongside the three real ones would say otherwise. Verified: 31 markers, none outside the plot box, no label overlapping a marker after right-hand labels were lifted clear of the dense band. The preference chart stays in place below it and still renders nothing until enough engines have a human rating. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Ten gaps filled: PESQ for NeuTTS-2E Q8 and Tada 1B, Peak RSS for eight including Chatterbox Turbo, MMS-TTS and VibeVoice 1.5B. Nothing else moved - no existing value changed, none was lost, and no audio differs, all checked by diffing the two columns and the sample directory against snapshots taken before the import rather than trusting the run to be additive. Still blank: PESQ for Dots-TTS and Tada 3B, Peak RSS for seven, mostly the partial runs. They sort to the bottom in both directions, re-verified here, because the export gives unmeasured cells sentinel keys that would otherwise rank them as the best or worst in the table. Row count and both counts the article quotes are unchanged: 31 configurations, 11 above the cutoff, 15 without a medium.en score - the last re-checked against the export rather than assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Q8 gains Params (0.24B), Weights (645 MB) and a Class, which was the last "?" in the run - so the chart's Unclassified legend entry retires itself and the field is three architectures again. This export was not purely additive, which is worth recording because "missing values filled in" suggests it would be. Five existing values were revised: Q8's Disk 1.4 -> 2.0 GB and Peak RSS 2.0 -> 1.5 GB, and two Peak RSS figures that the previous export had only just filled - BananaMind TTS 358 -> 543 MB and FastPitch 527 -> 397 MB. Caught by diffing every cell against a snapshot rather than only the columns the change was announced for. Audio is untouched, all 31 rows still render, and the chart still plots 31 points. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Dropping onto an occupied slot sent the occupant back up to the tray. So the most ordinary correction there is - deciding two clips are the wrong way round - cost two moves and threw away work already done: drag one over the other, watch the other fly to the top, place it again. The occupant now takes the slot the dragged clip just left. A clip dragged in from the tray still displaces the occupant to the tray, because there is no slot to swap it into. The swap turned out to be only half the bug. It worked immediately, and was then partly undone a moment later: pointer capture sends the click that follows a drag to the chip that was grabbed, which sits in the slot dragged *away from*, so it bubbled to that slot and read as a tap - and a tap on a placed clip means "put it back". The result looked exactly like the eviction being fixed. The click a completed drag produces is now swallowed, with the listener removed on a timer so it cannot eat a later real one. Verified with mouse and with a CDP touch drag: swapping across slots and between adjacent ones leaves the tray empty, tapping a placed clip still returns it to the tray, and a tray clip onto an occupied slot still displaces as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The cutoff is 2x rather than 2.1x. It was set at 2.1 to keep Chatterbox Q4 in at 2.03x, which was always a line drawn around one row; at 2x nothing sits awkwardly against it, since the slowest engine that keeps up is OmniVoice at 1.51x and the fastest that does not is Chatterbox Q4 at 2.03x. Twenty-two configurations keep pace, twelve do not. With everything listed, the table now rules a line at the cutoff when it is sorted by RTF or compute per phrase - the two orderings where the two groups are contiguous and a line is therefore a place rather than an arbitrary row. Verified contiguous under both before relying on it: one crossing each, not several. The label says which side is which, because that flips with the sort direction: ascending it reads "slower than real time below", descending "real time below". No line under any other sort, where fast and slow interleave, and none with the filter on, where there is nothing on the far side to divide from. One trap worth recording: the boundary test reads RTF from the *rendered* columns, not the run's. Human is spliced in ahead of it, so the run's index points one column to the left and the test would have compared the wrong numbers while looking perfectly reasonable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The divider was drawn in the accent - the same orange the table uses for an active play button and a pinned note - which on a full-width rule read as an error about the rows beneath it. A change of pace is not a fault. It is a band now rather than a rule, in the header's own surface with the same border colour, so it belongs to the table's structure. The label is muted ink at small caps, and the graphic is three descending blocks: "the pace drops here" carried by shape rather than by colour, which also survives being read by anyone who cannot separate the orange from the red error text two columns over. Wording follows: "below: slower than real time, over 2x", still flipping with the sort direction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
One cell moved: Dots-TTS's PESQ, 3.53 to 3.57. No row was added or removed, no audio changed, and nothing was filled - PESQ had no blanks left to fill, having been completed by the export before this one. Recording that because the change was described as more PESQ filling in. The three PESQ values this export does add belong to the NeuTTS CPU builds, which the table stopped showing when the CPU rows were dropped; they sit in the raw export rather than on the page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Peak is the only kind of RSS reported here, so the word said nothing the column did not already imply. The definition moves into the column guide, which now opens "RSS is peak resident memory" - the detail is worth keeping, just not in a header. Sorting still reads the numeric key, so 246 MB through 8.4 GB order by size rather than by the string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
… word
Two things, both real.
The empty row was .legal-content's `table { margin-bottom: 1.5rem }`
landing inside this component's own bordered wrapper - 24px of margin
drawn as if it were a row, between the last engine and the border. The
table sets margin-bottom: 0. The gap under it is 1px now rather than 25.
While there, the same stylesheet switches tables to display:block with
their own overflow-x under 640px, which would have given this a second
scroll container nested inside the one it already has. Pinned to
display:table; the wrapper still scrolls on a phone, 1097px of table in
340px of viewport, with no page-level scroll.
The band said "below: real time" when the sort was reversed. Accurate -
that is what was below it - but it changes the subject as well as the
direction, so the sentence has to be re-read after every sort. The
subject is fixed now and the direction moves: "below: slower than real
time" ascending, "above: slower than real time" descending. Checked in
all four states across both boundary sorts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
A button under the table invites another ranking, where a link buried in
the caption asked too quietly - the Human column only fills if people
rank, and the moment to ask is when someone has just finished reading the
table.
Two rules, both enforced in the control plane because the client cannot
be trusted to count its own votes:
- Three ballots per voter, keyed on the same IP-and-user-agent hash the
table already stores. Enough that a willing listener adds real signal,
few enough that no single pair of ears can move a rating alone, which
is the failure mode of an open unauthenticated vote.
- Nothing already ranked comes back. A second opinion on the same clip
from the same ears is not a second data point, and Bradley-Terry would
count it as one. The anchor is drawn from unheard engines too,
otherwise the attention check degrades into a memory test.
The cap is checked at submit as well as at issue. Issuing is cheap and
unauthenticated, so a client can hold several open ballots and submit
them all - verified: four hoarded ballots submit 200, 200, 200, 409.
Only submitted ballots count toward either rule. An issued-then-abandoned
one must not burn an attempt or retire four clips, or closing the dialog
would quietly cost the reader a vote.
Exhaustion comes back as a 200 with a reason rather than an error status,
so the dialog can say which wall was hit instead of showing a generic
failure. Verified end to end in the browser: three rankings, twelve
distinct engines, no repeat, then "that is your lot".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Both Hume models were listed "?", which is the export saying it could not read a licence off the card - a statement about the card, not about the model. Both cards do declare one: `license: llama3.2`, and Hume's Space says plainly that "the model is licensed under the Llama 3.2 Community License Agreement". The cells link to each model repo's own LICENSE file rather than to the Space that pointed me at it, on the same rule as the rest of the column: link where the project states its licence, and prefer the weights' own statement over anything about them. Both verified 200. Overrides live in the importer with the evidence written next to them, so the next export cannot silently revert them to "?" and so it is clear they were established rather than assumed. The override sets the sort key as well as the text. Left alone it would have sorted under "?" while displaying "Llama 3.2 Community" - wrong in a way nothing on screen would reveal. Two rows are still genuinely unverified: Dots-TTS, whose repo declares Apache-2.0 while the table says "?", and OmniVoice, which has no upstream card at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Both are Apache-2.0, both verified at the source. OmniVoice gains a home as well - k2-fsa/OmniVoice - so the only row still without a model link is BananaMind TTS. No cell in the licence column says "?" any more. Dots-TTS closes the loop on the mismatch the licence cross-check turned up when these links were first added: its repo declared Apache-2.0 while the table said unknown. The repo was right. Its URL also moves to studio-dots-ai, which is where rednote-hilab had been redirecting, in both the table and the vendor strip. k2-fsa joins the vendor strip, which is nineteen organisations now, still alphabetical and still two rows. All 54 distinct model and licence URLs re-checked end to end: every one returns 200. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
0.8B, April 2026, from arXiv:2604.00688 - "OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models", submitted 1 Apr 2026. Checked the identifier resolves and the title and date match before taking the numbers from it. Both sort correctly rather than under the blank they replace: Released sorts on YYYYMM and Params on millions, so text-only overrides would have kept sorting as missing while displaying a value - wrong in a way the page would not show. Verified in place: OmniVoice lands between 2026-02 and 2026-07 by date, and between 0.64B and 0.88B by size. Its note said the licence was unverified, which the row now contradicts two columns to its left. Rewritten to say where the size and date came from instead. The three licence overrides are folded into one CELL_OVERRIDES table keyed by configuration and column, rather than growing a separate map per column, with the evidence for each written beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
BananaMind TTS was the last row missing anything: 9.5M parameters and a home at Banaxi-Tech/BananaMind-TTS-V2, whose card states both the size and the apache-2.0 the table already carried. Every row in the table now links to its model and to where that project states its licence, and every licence cell is filled. The licence links to the model repo's own LICENSE rather than the Space README, which was the source offered - the repo has one, and the weights' own statement beats anything written about them. Same rule the rest of the column follows. Banaxi-Tech joins the vendor strip. That took it to twenty and pushed it onto a third row, so the chips are tighter: 11px of horizontal padding rather than 14, 16px logos rather than 18, 6px gaps rather than 8. Ten and ten from 1280px up, where before nineteen already needed three rows at 1024. Nothing below 1024 changes shape - a phone still wraps to whatever fits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
…data Confirmed against the 13:40 export: both lose "(partial)". Dots-TTS finished 80 of 84 phrases where it had managed 68 of 72, and Zonos 68 of 84 where it had managed 63 of 78. Parler-TTS is still partial at 10 of 17, so one row keeps the marker. Finishing the corpus cost them time - the phrases they had been skipping are the long ones. Dots-TTS goes 41.68s to 51.01s per phrase and 13.02x to 14.22x; Zonos 31.60s to 34.78s and 10.59x to 11.07x. Word error is unchanged to the nearest point in both cases. That broke the chart, quietly. Its x axis ended at a hardcoded 50s, fitted to the range of an earlier export, so Dots-TTS at 51.01s drew outside the plot with nothing to indicate it. Both axes now derive from the data with headroom, and the ticks are filtered to what the axis spans, so the next slow arrival cannot walk off the edge. Verified: 34 markers, none outside the plot box. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
All five land as described: Chatterbox 441 MB to 3.5 GB, and four small corrections to Chatterbox q8/q4 and CosyVoice3/RL. The big one reads as a fix rather than drift - 441 MB was less than Chatterbox's own 3.0 GB of weights, which was never physically possible, and 3.5 GB puts it where its siblings sit (Turbo 1.7, q8 1.4, q4 1.2, against 980 MB to 3.0 GB of weights). The export also moved three Parler-TTS figures that were not in the list: WER 39% to 42%, RTF 3.90x to 2.84x, and 31.23s to 25.29s per phrase. It is not a re-measurement so much as a different sample - the run went from 10 of 17 phrases to 5 of 9, so a shorter partial over different material. It is the one row still marked partial, and this is what that marker is warning about. Nothing else changed: no audio, no other row, no column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Three chips beside the real-time toggle: ar-lm, det-ff, st-ff, ordered by how much of the table each accounts for. They are additive - none pressed shows everything - so there is no "all" chip whose state has to be kept consistent with the others, and clearing the last one returns to the full table on its own. They carry the chart's class colours, because filtering here and isolating there are the same grouping and should not look like two different schemes. The hover description is the same one the Class cells give, so the codes are readable from the filter as well as from the row. Filters combine: ar-lm plus det-ff is 29 rows, and adding real time only takes it to 18. The count now derives from what is actually rendered rather than from a precomputed real-time total, which only described one of the filters and would have gone quietly wrong beside a second. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
…mport The table and the charts each had their own class control, so pressing one did nothing to the other - which is why the filters read as being on the chart and not the table. There is one selection now: filtering to ar-lm in the table narrows the plot below it, and pressing a class in the chart's legend filters the table. They are separate ```chart fences with no common React parent to hang a provider from, so the selection lives in a module-level store read through useSyncExternalStore. Both components import it and see the same Set; no provider, no shared ancestor. Rows are removed from the table and dimmed rather than removed in the charts, because a scatter with points taken out loses its scale. Verified both directions: det-ff pressed in the table gives 7 rows and 7 lit markers with both chips showing pressed, ar-lm added from the chart's legend gives 29, and clear from the table resets both. Separately, the import now rewrites the control plane's CONFIGS list. It has to match the table - a ballot offering a clip the page no longer shows is one the server then refuses, and a row missing from the pool can never be rated - and it was being regenerated by hand after every import, which is precisely the step that gets forgotten. Proved by breaking the pool to two stale entries and re-importing: back to 34, none stale, none missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Readers reported that samples stop playing after a while on the page, on both iOS and macOS Safari. WebKit suspends an AudioContext that has been silent for a while, and interrupts one whose tab is backgrounded or whose audio session another app takes. Coming back needs resume() from inside a user gesture, and a gesture does not survive an await. Both of our resume() calls were on the wrong side of that line. prime() does resume, but latched primedRef on the first play and early-returned forever after; ensureRunning() runs after the clip has been fetched and decoded, so its resume has no activation behind it and Safari rejects it. The first plays worked, then the context went down and every later click was refused - which looked like a dead play button. prime() now attempts a resume whenever the context is not running, and is called synchronously at the top of the click so the activation is still live. ensureRunning() awaits that attempt rather than starting a second, gesture-less one. A statechange listener clears the playing state when the context leaves "running", since a clip cut off that way never fires onended and would otherwise leave the button stuck showing "stop"; a late onended from the abandoned source no longer counts as heard. Returning to a backgrounded tab also tries a resume off sticky activation, so the next click is instant when it works and no worse when it does not. Six tests around a fake context that enforces the gesture rule. The four covering suspension and interruption fail against the previous player. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
…ays down
The previous commit made Safari worse, not better: samples stopped playing
there entirely while Chrome was fine. The cause is in the code it added.
ensureRunning() awaited the resume() promise that prime() had started, on
the reasoning that this was the attempt with a gesture behind it. But
WebKit does not settle a resume it will not grant - it leaves the promise
pending for the life of the page, where Chrome rejects it. So the await was
a deadlock: setLoading stayed set, the finally never ran, and the button sat
there forever. The deployed bundle showed it plainly:
x = async () => { let e = f(); if (g.current) try { await g.current } catch {} ... }
So: no resume promise is awaited anywhere now. ensureRunning() asks again
and then waits on the "statechange" event with a 600ms cap, which WebKit
does report. Hover no longer asks for a resume at all - it has no
activation to spend, cannot be granted, and is what left those pending
promises lying around; it only wakes the output device, and the click does
the resuming.
Two fallbacks behind that, because a benchmark whose samples will not play
is not worth reading. If the context cannot be brought up, the clip plays
through an <audio> element instead: WebKit governs media elements far more
lightly, and the bytes are already local, so the clip still starts whole -
the reason Web Audio was picked in the first place. And if the context
claims to be running while its clock stands still, which is what an
interrupted audio session looks like from script, the clip is moved to the
element mid-play rather than left silently not sounding.
Verified in WebKit itself, via Playwright, against a local production
build - normal playback, a context that never runs with a resume that never
answers, a context whose clock is frozen, and suspend-then-play. The first
two paths use Web Audio and never touch the element; the last two switch
and are audible. Chromium unchanged throughout. 11 unit tests, including
one that hangs against the previous commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Thirty-three configurations now. The importer removed the stale clip and synced the ballot pool on its own; the rest is what a removal touches that a table diff does not show. Hugging Face leaves the vendor strip - Parler-TTS was the only model of theirs measured, and the strip says these are the teams behind the models in the table. Nineteen chips, still two rows. Their logo goes with it. Two counts in the prose are recomputed rather than decremented, since dropping a row moves them independently: medium.en now fails to finish for eighteen of the thirty-three (it was nineteen of thirty-four - Parler was one of the failures), and eleven configurations cannot keep up with their own speech, down from twelve.
Safari still does not play on dev, and headless WebKit - the same engine,
driven by Playwright against that same deployed page - plays it fine, both
as a first-time reader and a returning one. So the difference is something
only a real Safari has: an audio device, a real audio session, the per-site
auto-play setting. I have guessed twice from here and been wrong twice, and
the third guess would cost another deploy to disprove.
So the page can now answer the question itself. A "No sound?" control under
the table opens a report - revision, context state, sample rate, whether the
audio clock is advancing, the audio session type, which path last played,
last error - and two probes that isolate the layers:
- a test tone: a quarter-second sine started and stopped entirely inside
the click, with no fetch, no decode and no buffer. If this is silent the
browser is refusing Web Audio outright and nothing about our clip
handling is implicated.
- a clip through the element route, which answers to a different policy in
WebKit than the context does.
Between them the four outcomes are distinguishable: both sound (the problem
is in the clip path), only the element sounds (Web Audio is refused), only
the tone sounds (fetch or decode), neither (the browser is refusing
everything). A phone has no console, so this had to be in the page.
Verified in WebKit against a local production build: panel opens, tone
reports the clock advancing 0.39s, the element probe reports position
0.35s, and the report fills in - audioSession moves from "auto" to
"playback" once the context exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
It was the last two words of the caption under the table, in the same grey as the prose either side of it, distinguished only by a dotted underline. It rendered and it worked; nobody would ever see it, which is the same as not shipping it - and I asked for a reading from a control that looked like a sentence. It is now an outlined chip in the row beside "Rank four clips", right- aligned, and it turns accent-coloured while its panel is open. A reader who has heard nothing is looking for a button, so it looks like one. The panel moved up with it, directly under the control that opens it rather than stranded past the caption. Checked at 1400px and at 390px in WebKit: the chip sits at the end of the ballot row on desktop and wraps onto its own line on a phone, and the panel opens with both probes visible in each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
… here The reader ran the two probes: the test tone was silent, the same clip through an <audio> element played. That is as clean a result as this could have given. Web Audio is not recoverable on that browser by any amount of resume handling - which is what the last three attempts were - so it is no longer what anyone listens to. Web Audio was chosen for a real reason: "canplaythrough" is a heuristic about download rate, so an element could start while the MP3 was still arriving and swallow the first word, whereas decodeAudioData only resolves with the whole clip in memory. That guarantee survives without it. Every clip is fetched to completion and played from a blob URL, so the bytes are all local before play() is called; the element is never asked to guess. navigator.audioSession.type = "playback" goes too. It existed to drag Web Audio out of the "ambient" session that the iOS ring/silent switch mutes - into the session a media element is in natively. With clips on an element that is free, and the one Safari-only line in the file, which is the shape this failure always had, is gone with it. What follows from element playback rather than a context: one element for the page, reused by changing src, because iOS grants playback to the element the reader started rather than to the page. Fetches are shared and cached per clip. Hovering a row now prefetches it, which is worth more than the silent buffer it replaces - the click's only remaining wait is the download. A refused play() is told apart from a clip that will not load, so "blocked" and "could not be loaded" mean what they say. The context now exists solely for the tone probe, created only if that probe is run. Verified in both engines against a production build: the clip plays and its position advances, stop stops, switching clips leaves exactly one sounding, and no AudioContext is constructed at all. 12 unit tests rewritten around the element. Not verified: the row that plays itself after the ranking dialog closes, which needs the ballot API and so cannot run locally - it plays through the element the dialog already unlocked, but a reader who skips the dialog without playing anything may see it refused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
They existed to answer one question I could not answer from here, and the reader answered it in a single round: tone silent, element audible. Keeping a debugging panel on a published page past that point is clutter on the one control row a reader actually uses. Out with it goes the last AudioContext in the file - it was only still there to power the tone probe - along with the resume-and-wait helper written for Safari's never-settling resume promise. The player is now an element, a fetch cache and nothing else: 207 lines against the 300 it peaked at, with no Web Audio surface left to go wrong. A failed play still logs with its revision, so the console says what the panel used to, and the reader still gets the message on the page telling them whether the browser refused or the clip would not load. The test that replaces the probe tests asserts the point that now matters: playing, stopping, switching and prefetching must construct no AudioContext at all. Its stub is a tripwire, not a fake. Ten tests, and both engines re-checked against a production build - clip plays, stop stops, zero contexts, no panel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Four findings from a review, all real, though one is milder than billed. The play() that matters was on the wrong side of the fetch. The comment claimed playback started inside the click; the code awaited the download first and only then called play(), which is the thing Safari and iOS refuse. It survived testing because a hover prefetches and because WebKit forwarded the activation this time - neither of which holds on a phone, where there is no hover and the first tap is always uncached. The first play of an uncached clip now plays a silent inline frame synchronously, claiming the grant while the activation is live; the element keeps it, so the real clip inherits it when it lands. Cached clips skip this - there is nothing to wait for. Rapid switching could play the earlier click. The stale-load guard asked whether something was already playing, which is false for both clips while both are still loading, so whichever resolved last took the element. It is now a generation, bumped on every toggle and every stop, so a load that lands after the reader has moved on is dropped, and a clip stopped mid-fetch does not start. A ballot the server rejected was shown as "Thank you - that is one ballot", and the parent recorded it as submitted. So a 429, a 409 or a dropped connection lost the vote, told the reader it had counted, and marked them as already asked. It now retries once - these failures are mostly transient - and if that fails too it still reveals the names, which the listening earned, under an honest heading, and reports back that nothing was counted. Both ballot routes ran the IP rate limiter a second time, after the global unauthenticated check had already run, so public readers spent two tokens per request and authenticated ones paid for a check that does not apply to them. Removed. The reviewer put this at five requests to the limit; the configured limit is 200/minute, not the 10 they assumed, so the real effect was halving a generous budget rather than breaking the third ballot. Both engines re-checked against a production build: the silent frame is played before the blob on a first play, three rapid clicks leave exactly one clip sounding and it is the last one clicked. 38 tests; three of the four new ones fail against the previous player. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The deferred play was the one path I flagged as unverified when clips moved to an element, and it is the one that was broken. First click on a row opens the ranking dialog instead of playing; the row the reader asked for is played afterwards, from an effect. An effect has no user activation, so the silent frame that claims Safari's permission could not be spent there - and a reader who skips the dialog without playing anything in it has nothing else that could have claimed it. They ask to hear a row, decline a survey, and get silence. Closing the dialog is itself a gesture, so that is where the grant is taken. unlock() is now part of the player's surface rather than a step buried in the first play, and closeBallot calls it on every close - skip, submit, and backdrop alike. The effect that honours the original click then plays into an element that already has permission. The whole first-time path could not be exercised locally before, because it needs the ballot API. Serving the ballot from the test makes it runnable, and it now passes in both engines by both exits: the close writes the silent frame and then the clip, and the row the reader originally clicked sounds. Chromium looked like a failure at first and was not - Piper is 2.1s and the assertion was 2.5s after the click. Escape is not a close path here; the dialog has no key handler, so every exit runs through a click. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Thirty-four configurations. Supertonic measures 3.61% word error, ahead of Piper's 3.75% - both print as 4%, so the table shows a tie the underlying numbers do not - at 0.84s per phrase and 0.22x real time. Nothing else here is that accurate that cheaply. The identification took some care, because the export carries only the id "supertonic". 99M parameters across four ONNX graphs at native 44.1 kHz is the v3 checkpoint specifically; v1 was 66M and v2 covered five languages. So it is listed as Supertonic 3, linked to supertone-inc/supertonic. Its licence proof is the weights' own LICENSE on Hugging Face rather than the repository's, because the two differ and the table states the weights: the sample code is MIT, the model is OpenRAIL-M. The repo LICENSE would have "proved" the wrong one - the same trap F5-TTS and FastPitch set. Supertone joins the vendor strip, twenty chips now. No existing note is invalidated: Zonos keeps second place on PESQ, and Chatterbox Turbo's "joint-best at 4%" survives because at the table's precision it now describes a three-way tie. The prose count moves to nineteen of the thirty-four for the medium.en gaps, recomputed from the export rather than incremented - Supertonic is one of the configurations it did not finish. The real-time boundary is untouched: OmniVoice at 1.51x and Chatterbox Q4 at 2.03x still sit either side of it. Verified in a build: the row renders and plays, both links resolve 200, the chart carries 102 markers over 34 rows, and the logo loads. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The export dates it 2026-03. The configuration measured is the 99M v3 checkpoint - four ONNX graphs, native 44.1 kHz - and that shipped at the end of April 2026; March belongs to an earlier one. v1 was 66M and v2, from January, covered five languages. An override rather than an edit to the data file, so it survives the next import like every other correction here, and it carries the sort key as well as the text - Released sorts on 202604, not on the string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Thirty-six configurations, and a second vendor whose speaker comes from a written description rather than a reference clip: VoiceDesign is 1.7B at 0.87x real time and 6% word error, effectively its 0.6B sibling's accuracy for four times the compute per phrase. Orpheus is the largest LM backbone here alongside TaDA 3B, and the slowest thing in the table at 8.09x. VoiceDesign points at its own checkpoint rather than the family repository, since it is a distinct set of weights. Both licence links are model cards rather than repository LICENSE files: each declares apache-2.0 for the weights, which is what the column states, and Supertonic is the standing reminder that a repo can license the code differently. Canopy Labs joins the vendor strip - twenty-one chips - with their GitHub avatar, as with every other logo here. Both prose counts move, recomputed rather than incremented: medium.en now fails for twenty-one of the thirty-six, and twelve configurations cannot keep up with their own speech. The real-time boundary is unmoved; Orpheus is far above it and VoiceDesign comfortably below. Verified in a build: both rows render, link and play, all three new links resolve 200, the chart carries 108 markers, and no asset 404s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
The column has been empty since it shipped: at a minimum of eight, with thirty ballots spread over the field, the best-covered configuration had seven. So the threshold was not protecting readers from a noisy number, it was withholding every number. Simulating the estimator settles what the cutoff is worth. Same MM fit, ballots drawn from the Plackett-Luce model Bradley-Terry assumes: at thirty ballots a shown rating carries about twenty points either way whether the cutoff is three or five, because precision comes from the total number of ballots and not from the cutoff. Moving 5 -> 3 costs about three points of error and two points of pairwise accuracy, and triples the rows shown. What three still earns is excluding a configuration seen once or twice, where the rating is barely distinguishable from noise; correlation with the truth is about 0.70 there today, and reaches 0.89 by a hundred and twenty ballots. Two presentation changes follow from that, because the honest problem with the column was never the cutoff. It rendered 72.4 - a tenth of a point on a measurement worth twenty either way - so it now rounds to whole points, with sorting still on the full value. And the caption says what the numbers are: early, worth about twenty points either way, a rough ordering rather than a measurement. Checked against a payload of the shape the control plane will now return: ratings render as 72, 64, 50, 28, a configuration below the threshold still shows a dash, and the caption reads the threshold from the API rather than restating it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
Twenty-one of them wrapped to three, with a single chip stranded on the last line. The column they sit in is a fixed 1104px whatever the viewport, so the row count is set purely by chip width - which is why this is the second trim, and will be needed again around the twenty-third vendor. Horizontal padding 11px to 9px, the gap between chips 6px to 5px, the gap between logo and name 8px to 7px, and the label from 0.80rem to 0.78rem. Height stays at 32px and the logos stay at 16px: both are visible if shrunk, where two pixels of side padding are not. That leaves 11 + 10 with about 30px to spare. Measured rather than guessed - candidate sizings were applied in the page and the resulting row counts read back, then confirmed against a build at 1280, 1440 and 1920, where it is two rows, and at 1024, 768 and 390, where it wraps further as it should. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WebP1vdtbLxbLjfWY9YGj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.