A2 MUSHRA Listening Test Raw Data

This repository contains only the raw data. The listening-test code lives in a separate repo: tone-3000/t3k-mushra.

For background and early analysis, see the companion guide: NAM A2: The Complete Guide.

TL;DR

• 105,842 ratings from 1,184 participants, collected 2026-04-18 to 2026-05-29.
• Formal, large-scale blind MUSHRA test ("Multiple Stimuli with Hidden Reference and Anchor").
• 37 tones were rated in the listening test, drawn from our 39-tone evaluation set (2 tones were excluded from the listening test because of rendering issues with the commercial modelers; they were used only in the quantitative metrics).
• Each rating is a single score (0-100) of how closely one model matched the original recording for one tone, in one trial.
• One CSV: data/a2-mushra-data.csv.

How was the listening test run?

We used the MUSHRA methodology ("Multiple Stimuli with Hidden Reference and Anchor"), a standard for evaluating perceived audio quality used by broadcasters like the BBC and EBU.

In each trial a participant heard the real recorded gear first as a reference, then rated several subsequent versions of that same tone, modeled with A2-Full, A2-Lite, A1-Standard, A1-Nano, Neural DSP's Neural Capture V2, IK Multimedia's TONEX, and Line 6's Proxy, without knowing which was which. Participants scored each version 0-100 on how closely it matched the original recording.

Per MUSHRA methodology, two clips in each trial were quality controls:

• A hidden reference, a duplicate of the original recording, expected to score high.
• A low-passed anchor, a deliberately degraded clip, expected to score low.

These controls allow for post-screening: in our published analysis we dropped results from participants who rated the hidden reference too low or the anchor too high, so bad data didn't skew the result. This repository contains the raw, unscreened ratings. Screening is left to you, so you can apply (or change) the criteria and see their effect.

The data

data/a2-mushra-data.csv. One row per rating.

Column	Type	Description
id	integer	Unique rating ID.
user_id	string	Anonymized participant ID. Stable across that participant's ratings.
tone	string	The source tone being matched (e.g. GA-1, FRG-12). See Tones.
figure	integer	Which audio sample (e.g. a particular guitar or bass riff) of the tone was played. The same tone was recorded or rendered with several different riffs/passages; figure identifies which one this rating refers to.
model_id	string	Internal blind codename for the model under test. Maps 1:1 to model_label.
model_label	string	Human-readable model identity. See Models.
model_letter	string	The randomized label (A-I) shown to the participant in that trial. Different participants saw different letters/order for the same model, which is the blinding.
score	integer	The participant's rating, 0-100 (higher = closer to the original recording).
created_at	timestamp	When the rating was created (UTC).
updated_at	timestamp	When the rating was last updated (UTC).
version	string	Test build identifier. All rows are a2-listening-15.

Models

model_label identifies what was being rated. The two MUSHRA controls (recorded, anchor) are included alongside the seven models under test.

model_label	What it is
recorded	The original recording, presented again as the hidden reference (QC).
anchor	A low-passed version of the original, the degraded anchor (QC).
a2-full	NAM A2, full model.
a2-lite	NAM A2, slimmed model.
a1-standard	NAM A1, legacy standard model.
a1-nano	NAM A1, legacy nano model.
qc-v2	Neural DSP Neural Capture V2.
tonex	IK Multimedia TONEX.
proxy	Line 6 Proxy.

Note

Not every model appears in every trial. proxy in particular was only evaluated on a subset of tones, and a few tones are missing tonex / qc-v2. Account for this when comparing models. Compute per-model statistics over the trials in which each model actually appeared, rather than assuming a balanced design.

Tones

There are 37 tones in the listening test. The prefix groups them by gear type:

Prefix	Tones	Gear category
GA	8	Guitar amps
BA	4	Bass amps
P	6	Pedals / outboard
PA	5	Pedal + amp chains
FRG	10	Full rigs (guitar)
FRB	2	Full rigs (bass)
PB	2	Pedals / outboard (bass)

Quick start

The file is a standard UTF-8 CSV. With pandas:

import pandas as pd

df = pd.read_csv("data/a2-mushra-data.csv", parse_dates=["created_at", "updated_at"])

# Mean score per model, across all tones
print(df.groupby("model_label")["score"].agg(["mean", "median", "count"]))

# Post-screening, per the MUSHRA spec. An item is one audio sample (tone + figure).
# Exclude a participant if, across the items they rated, they scored:
#   - the hidden reference (recorded) below 90 on more than 15% of items, or
#   - the anchor above 90 on more than 15% of items.
df["item_key"] = df["tone"] + "|" + df["figure"].astype(str)

def bad_item_fraction(label, is_bad):
    items = df[df.model_label == label]
    present = items.groupby("user_id")["item_key"].nunique()
    bad = items[is_bad(items["score"])].groupby("user_id")["item_key"].nunique()
    return bad.reindex(present.index, fill_value=0) / present

users = df["user_id"].unique()
recorded_bad = bad_item_fraction("recorded", lambda s: s < 90).reindex(users, fill_value=0)
anchor_bad = bad_item_fraction("anchor", lambda s: s > 90).reindex(users, fill_value=0)

eligible = users[(recorded_bad < 0.15) & (anchor_bad < 0.15)]
screened = df[df.user_id.isin(eligible)]

Data preparation

The published CSV is produced from the raw database export by a small cleaning step that applies the following transformations:

1. Normalizes Windows-style CRLF line endings to LF.

Citation

If you use this data, please cite it (see CITATION.cff) and link back to this repository.

License

The data in this repository is released under Creative Commons Attribution 4.0 International (CC BY 4.0). You're free to share and adapt it, including commercially, as long as you give appropriate credit.

Links

• Test code: github.com/tone-3000/t3k-mushra
• Guide & peliminary analysis: NAM A2: The Complete Guide