feat: add av bench — cost/accuracy frontier harness and DeepSeek-V4.1-Flash provider

[seanphan]

Posted by Claude (Anthropic) on Sean's behalf.

Verdict: the harness is built and honest, but it has not been pointed at DeepSeek-V4.1-Flash — so this PR contains zero measured claims about that model.

Adds av bench, a cost/accuracy frontier harness, plus a DeepSeek-V4.1-Flash provider. Headline axes are tokens per query and accuracy, with a third axis a per-token API vendor cannot report: dollars per query on hardware you own.

What's here

• src/av/bench/ — cost models, deterministic ffmpeg fixtures, pinned frame extraction, receipts with labelled evidence, noise floor, dataset adapters
• src/av/cli/bench.py — probe, gate, plan, prepare, run, sweep, noise, cost
• src/av/providers/deepseek.py — endpoint resolution from config/env, capability probe, and the vision token model
• 82 new tests (143 total, all passing)
• samples/bench-receipts/ — real receipts from the runs quoted below

Two arms, not one

dense samples the whole window at a fixed rate and asks once. agentic takes a cheap coarse look, decides which moments it needs, fetches only those — and is charged for both requests. Same questions, same prompt, same scorer; the only difference is which frames the model got.

Is tokens-per-frame tunable?

Yes, but not via detail. detail is parsed by SGLang and no multimodal processor reads it back, so it is inert against a self-hosted server. The real lever is the resolution you upload, and the client owns it. av bench plan predicts the cost offline from the published preprocessor algorithm — it reproduces every published worked example below the token ceiling — and shows two walls worth knowing: an upscale floor below which shrinking frames buys nothing, and a ceiling above which extra resolution is discarded.

Measured against the endpoint that was actually available (Gemini 2.5 Flash, OpenAI-compat), the verdict came back fixed: 258 tokens/frame across every detail setting and every width from 256 to 1536. That is a capability result, recorded as such rather than faked.

The gate

Runs first, costs cents, and can save the whole exercise: a model that cannot order eight flat colours is not being measured on temporal understanding. Scores exact order and, crucially, correct_prefix — which is what distinguishes "wrong order" from "reported only the first frame".

Real frontier, on real footage

av bench sweep over the shipped CCTV sample, 8 windows per probe, zero unparseable replies:

interval	door_activity	person_enters	tokens/query (door)	video-hours per $
1 fps	0.63	0.88	7,195	3.5
1/2s	0.50	0.63	3,615	7.0
1/5s	0.13	0.50	1,454	17.1
1/10s	0.13	0.50	745	32.7
1/30s	0.13	0.25	256	87.8

Door activity falls off a cliff between 1/2s and 1/5s. Person-entry holds 0.50 all the way out to 1/10s — a 9x cost reduction for that task at the same accuracy. That per-task difference is the whole point: there is no single safe sampling rate, and picking one globally either overpays or misses events.

Dollar figures use --cost token:0.30:2.50 as an illustrative rate, not a quote.

Caveats, stated in the receipt and in the metric's name: the reference windows were produced by a vision model, not by humans, so this is reference_recall — agreement with a dense reference run — never recall against ground truth. n is 8. Precision was not measured at all.

Honesty machinery

• Claims are labelled measured / derived / documented / community-reported / untested; a non-measured claim without a source raises.
• $/hr and $/token are separate model classes and are never conflated.
• Noise floor refuses to report a saturated cell's zero spread as a floor.
• Receipts record hostnames only; private and tunnelled hosts redact to <private>.
• No endpoint in source — the DeepSeek preset points at SGLang's local default.
• No benchmark data vendored. Adapters read files you fetched, under upstream terms. MINERVA is CC BY 4.0; LVBench is CC BY-NC-SA with an explicit commercial-use ban.

Not verified

• Nothing measured against DeepSeek-V4.1-Flash. The endpoint is stopped and starting it costs real money. Provider and token model are unit-tested; no measured claim about that model appears anywhere.
• No benchmark videos downloaded, so no MINERVA or LVBench accuracy figures. av bench prepare is tested against both annotation formats.
• Published agentic-video results for other models were produced on different hardware with different methodology. The receipts say so, and nothing here puts their number and ours in one cell as a ratio.

Test plan: uv run pytest tests/ -v (143 pass), uv build, and each subcommand exercised end to end — the committed receipts are the output.