Your agent says "Done. All tests pass."
This reverts the fix, runs the new test again, and shows you the ones that still pass.
A test that has never failed has never tested anything.
npx skills add BOTIROFF-D/alibi
Your agent now runs the check before it says "done". Claude Code, Cursor, Codex, Copilot, Gemini, Windsurf, Cline, Zed and 70 others.
Nothing to install for that — the skill uses git and your own test command.
For your terminal and CI:
npx -y github:BOTIROFF-D/alibi
- Put the source back the way it was.
- Run the test your agent just wrote.
- It should fail. If it passes, it never tested the fix.
That is the whole idea. No model, no API key, no network.
| the new test passes on the old code | it tested nothing |
| the failing test was deleted | green because there is less of it |
the failing test was skipped |
one word, and it never runs again |
| the test asserts nothing | it can only fail by crashing |
| the test mocks the thing that changed | a photo of a bridge holds no weight |
| the assertion was softened | it checked contents, now it checks length |
Each of these is a real repository in examples/museum,
built and checked by CI. If one stops being caught, the build breaks.
- 75.8% of failing coding-agent runs still report success (paper)
- Asking another model to check is a coin flip: best AUROC 0.65, same paper
- Agent pull requests that touch code and tests improve coverage in 35.9% of cases in Java and 22.5% in Python (paper)
- 45% of developers say the top problem with AI is "almost right, but not quite" (Stack Overflow 2025, 49k respondents)
The thing that wrote the code is the only thing being asked whether it works. An exit code has no tone of voice.
0 |
nothing false |
1 |
a test passed on the old code, or a test was deleted or skipped |
2 |
something is only unproven, with --strict |
3 |
could not answer |
- uses: actions/checkout@v4
with: { fetch-depth: 0 }
- uses: BOTIROFF-D/alibi@v0Characterization tests lock in behaviour that already exists. Say so:
// alibi: characterization
it('still renders the legacy header', () => { ... });It stays in the report, marked, and stops counting against you. There is no way to silence a finding invisibly, and there will not be one.
- It cannot prove your change is correct. It finds tests that are not evidence for it. An empty report means nothing was caught.
- It says nothing about tests your diff did not touch.
- Rust unit tests live inside the file they test, so reverting would revert the test too. Those are reported as unexamined.
- One process per test. On a suite with heavy setup this is slow. There is no fast mode that trusts parsed output, because parsed output is exactly what this refuses to trust.
- Editable Python installs can shadow the reverted source. The worktree goes
first on
PYTHONPATH, which covers the usual layouts and not all of them. If a Python project reports implausibly many failures, suspect this first.
More in DESIGN.md — including why a regex is never allowed to
call something a lie, and why "it never ran" is a third verdict rather than a
failure.
vitest · jest · mocha · node --test · pytest · go test · cargo test ·
rspec · phpunit · or any shell command you give it.
Most projects need none. alibi.json if yours does:
{ "runner": "pytest", "link": [".venv"], "timeout": 180 }| key | meaning |
|---|---|
runner |
force one instead of detecting it |
command |
a shell template: {file}, {name}, {fullName} |
link |
directories to link into the throwaway worktree |
timeout |
per-test seconds, default 120 |
mutate |
also damage the changed lines and see if anything objects |
Mutation testing (PIT, Stryker, mutmut) asks: if I injected a bug, would the suite notice? Hundreds of runs to answer.
This asks a cheaper one: did it notice the bug that was actually there? Your change is the mutant. One run per new test.
The most useful thing you can send is a green suite this doesn't catch. Open an issue and it becomes exhibit number seven.
Doniyor Botirov, founder of dbit.one. Also: unflake · bulwark · adya · pnueli · sable
MIT.