A falsification bench for systematic trading strategies.
You hand it a set of trades, or a signal and a set of prices. It runs a protocol designed to reject the hypothesis that the strategy has an edge, and it is allowed to answer no. It is also allowed to answer "this cannot be decided", which is the answer most backtest reports quietly convert into a yes.
Pure numpy / scipy / pandas. No broker, no data vendor, no model, no I/O.
It comes out of a private research programme — ninety-two logged iterations of
reinforcement-learning research on crypto and metals — in which it adjudicated
40 hypotheses and approved none of them. Five of those adjudications
are written up in case-studies/, including the two
that ended lines of work I had spent weeks on.
A backtest is a measurement taken by the person who wants a particular result. There are four distinct ways it lies, and they are independent, so fixing one does nothing for the other three.
1. It leaks. A threshold, a quantile, or a normalisation constant is computed on a window that includes the bar it gates. The equity curve looks slightly too good, out of sample it collapses, and nothing in the code looks wrong.
2. It is one draw. A single train/test split answers "did this work on that slice?". It cannot answer "how badly does it work on the slice I have not seen?", because there is only one observation of that question. The average across folds hides the fold where the strategy broke, and live trading does not sample folds at random -- it starts in whichever regime comes next.
3. It is the best of N. The reported strategy is the survivor of a search. The maximum of N noisy estimates is high by construction, even when none of the N has any skill. A t-statistic of 2.1 on the winner of a 200-configuration sweep is what pure noise looks like, not what an edge looks like.
4. It confuses exposure with skill. A policy that stays long through a rising window earns money without predicting anything. The profit is real; the skill is not; and the drift belongs to the window, not to the strategy, so it does not transfer to the next one.
Seven gates, in a fixed order, five of them hard. A hard gate that fails ends the
evaluation at NO_GO. A gate that cannot be evaluated returns INCONCLUSIVE,
never PASS, and an inconclusive hard gate caps the verdict at ADVISORY_ONLY.
| # | gate | kind | asks |
|---|---|---|---|
| 1 | sample |
hard | is there enough out-of-sample material to decide anything? |
| 2 | worst_fold |
hard | does the worst cross-validation fold still make money? |
| 3 | net_edge |
hard | does a block-bootstrap interval of the pooled net result exclude zero? |
| 4 | beta_control |
hard | does the policy beat the best constant directional bet on the same bars? |
| 5 | deflation |
hard | does the Sharpe ratio survive the size of the search that produced it? |
| 6 | overfitting |
soft | does the in-sample winner keep its rank out of sample? |
| 7 | baseline |
soft | is the improvement over an incumbent bigger than estimation noise? |
Verdicts: STRONG_GO, PASS_WITH_CAUTION, NO_GO, ADVISORY_ONLY.
Four ideas do most of the work.
- The deflated Sharpe ratio refuses to compute when the number of trials is not
supplied. Deflating against a collapsed or guessed family under-corrects, which
is worse than no correction at all because the result is quoted as if it were
one.
deflated_sharpe_ratioraisesUnknownSelectionFamilyErrorby default. - Concatenated cross-validation paths are not independent observations. In
C(k, m)combinatorial splits each observation is replicatedC(k-1, m-1)times.effective_n_obsdivides that out before any statistic is computed; skipping it inflates the sample size fivefold atk=6, m=2. - The beta control benchmarks against the best constant directional bet, with the side chosen in hindsight. It is deliberately the hardest benchmark of its kind: a strategy that cannot beat a bet that was handed the answer has shown no evidence of direction or timing.
- The stop is tested before the target on the same bar. OHLC data does not record the order of the high and the low. Resolving that ambiguity in the trade's favour manufactures an edge that no execution can reproduce, silently, in proportion to volatility.
python examples/no_edge.py -- 200 random signals are screened on a driftless
random walk and the best one is kept, which is exactly what a research loop does:
=== what the backtest report says ===
trades 2498
mean net per trade +3.05 bps
t-statistic +2.14
win rate 51.4%
That is the whole report, and it would pass most reviews. It omits that this
is the best of 200 signals drawn at random on a market with no structure.
Corrected for that search, the winner's family-wise p-value is 0.63.
=== what the bench says ===
trades 12490 (effective sample 2498)
mean net per trade +3.05 bps
worst fold +0.10 bps
bootstrap CI [+1.78; +4.29] bps
per-trade Sharpe +0.0427
alpha vs constant bet +1.63 bps at t=+0.92
deflated Sharpe 0.2646 (family of 200 trials)
gate kind status detail
----------------------------------------------------------------------------------------------------
sample hard PASS 12490 trades over 15 folds
worst_fold hard PASS worst fold mean +0.0972 against a floor of +0.0000
net_edge hard PASS bootstrap CI [+1.7771; +4.2885] is above zero
beta_control hard FAIL alpha +1.6258 at t=+0.92 over 2498 trades (floor 2.0, long share 0.48) -- the edge is the window's drift, not direction
deflation hard FAIL deflated Sharpe ratio 0.2646 against a floor of 0.95 over 200 trials
overfitting soft FAIL PBO 0.686 (cscv, 70 splits, 200 configurations) against a ceiling of 0.50
baseline soft INCONCLUSIVE no comparable baseline folds supplied; the improvement is unmeasured
----------------------------------------------------------------------------------------------------
VERDICT: NO_GO
Note what does not save it. The sample is large, the worst fold is positive and
the bootstrap interval excludes zero: the two checks a careful backtest already
runs are passed. The three controls that price the search are the ones that
bite -- the policy does not beat a constant directional bet, a Sharpe ratio
picked as the best of 200 deflates to a quarter, and the in-sample winner does
not keep its rank out of sample. The naive t-statistic was not wrong; it was
answering a question nobody should have asked.
The second example, python examples/costly_edge.py, generates a market where a
signal genuinely anticipates part of the next eight bars, and shows the same
trades reaching three different verdicts depending on the round-trip cost and on
how many configurations were screened.
git clone <this repository>
cd falsify
python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"Requires Python 3.12 or newer.
import numpy as np
from falsify import (Evidence, SelectionFamily, adjudicate, cpcv_splits,
directional_alpha, folds_from_splits, triple_barrier)
rng = np.random.default_rng(0)
n = 20_000
close = 100.0 * np.exp(np.cumsum(rng.normal(0.0, 25e-4, n)))
wiggle = np.abs(rng.normal(0.0, 25e-4, n)) * close
high, low, unit = close + wiggle, close - wiggle, np.full(n, wiggle.mean() * 3)
hold = 8
entries = np.arange(hold, n - hold - 1, hold, dtype=np.int64)
side = np.where(rng.normal(size=entries.size) > 0.0, 1.0, -1.0)
trades = triple_barrier(entries, side, close, high, low, unit, hold,
stop_mult=1.5, target_mult=1.5, cost_bps=2.0)
move = (close[trades.exit_idx] - close[entries]) / close[entries] * 1e4
report = adjudicate(Evidence(
folds=folds_from_splits(entries, trades.net_bps, cpcv_splits(n, k=6, n_test_groups=2)),
family=SelectionFamily(n_configurations=200, selections_per_configuration=12),
directional=directional_alpha(move, side),
k=6,
))
print(report.summary())AFML is the reference for most of the machinery here, and this package deliberately implements a small subset of it. The honest comparison:
| topic | AFML | here |
|---|---|---|
| purged k-fold with embargo | ch. 7 | cpcv.cpcv_splits, same construction |
| combinatorial purged CV | ch. 12 | C(k, m) paths, same path algebra |
| triple-barrier labelling | ch. 3 | barriers.triple_barrier |
| deflated / probabilistic Sharpe | Bailey and Lopez de Prado (2014) | dsr, including the Mertens-Lo standard error |
| PBO via CSCV | Bailey, Borwein, Lopez de Prado, Zhu (2017) | pbo.pbo_cscv, full C(S, S/2) enumeration, not an approximation |
| what | why it matters |
|---|---|
| the selection family is a required, typed argument | AFML defines the trial count and leaves it to the user; in practice it is defaulted to something convenient, which turns the whole correction into decoration. Here it is SelectionFamily(n_configurations, selections_per_configuration) or an exception. |
| effective sample size after CPCV concatenation | AFML treats label overlap through sample weights; the deflated Sharpe is nonetheless routinely fed the raw concatenated path length. effective_n_obs removes the C(k-1, m-1) replication. |
| beta control against the best constant bet | AFML has no equivalent benchmark. Without it, exposure to a trending window reads as skill. |
| stop tested before target on the ambiguous bar | AFML specifies the barriers but not the intrabar tie-break. The convention is worth more than most of the rest of a backtest. |
| a verdict protocol, not just statistics | hard/soft gates, an explicit INCONCLUSIVE state, worst-fold adjudication and a serialisable report. AFML supplies the numbers; the decision rule is left to the reader. |
| a false-positive test in CI | 400 synthetic noise worlds must produce zero passes, on every commit. |
| gap | why |
|---|---|
| no meta-labelling, sample weights, fractional differentiation, structural-break tests, feature importance (MDI/MDA/SFI) | AFML is a book about building strategies. This package only refutes them. Out of scope by design. |
| no sequential bootstrap or average-uniqueness weights (AFML ch. 4) | a real gap. Overlapping labels are handled coarsely here: the caller is asked to thin entries with nonoverlapping_entries, and CPCV replication is divided out. Genuinely overlapping labels are not down-weighted. |
purging uses a scalar label_horizon, not per-observation label end times |
AFML's PurgedKFold purges with each observation's own t1. With a variable holding period the scalar version is strictly weaker; pass the maximum horizon to stay safe. |
| the embargo is symmetric around the test block | AFML embargoes forward only. Symmetric is more conservative but removes more training data than necessary, and it is a deviation from the reference. |
| CSCV ranks configurations by Sharpe ratio only | the paper allows any performance functional. The hook is not exposed. |
| the deflated Sharpe assumes independent trials | correlated trials make the correction conservative in one direction only. Not corrected. |
| no bet sizing, no ML pipeline, no HPC layer (ch. 10, 20-22) | out of scope. |
max_t_pvalues exists but adjudicate does not consume it |
the family-wise correction is available for a parameter sweep, but it is not yet one of the seven gates. |
Stated plainly, because a bench that hides its own limits is the thing it was built to detect.
- Validation in this repository is synthetic. The suite proves the bench
rejects noise. It does not prove the bench accepts every real edge. The
protocol has been run against real strategies — see
case-studies/— but that was the private implementation on licensed data, and none of it is reproducible from this repository. - The block bootstrap is mildly anti-conservative at small samples. Measured
on driftless noise: about 6-7 % of 95 % intervals exclude zero at 400
observations with 20-bar blocks, against a nominal 5 %. It converges to nominal
as the number of blocks grows.
tests/test_noise_never_passes.pypins this. - The thresholds are conventions, not laws. A 0.95 deflated-Sharpe floor, a
0.5 PBO ceiling and a t of 2 on the beta control are defaults in
GateConfig. They are arguments, and changing them changes the answer. - One candidate at a time.
adjudicatehas no notion of a portfolio, of capacity, of correlation between strategies, or of regime labels. - Costs are a scalar per trade. There is no spread model, no slippage model, no market impact and no funding.
- A pass is necessary, never sufficient.
STRONG_GOmeans the bench failed to refute the edge with the evidence it was given. It is not a recommendation. - The bench cannot see what happened before it. If the market, the period, or the instrument was chosen after seeing results, no statistic in this package can recover that. The selection family only covers the searching you declare.
- No point-in-time or survivorship handling. Input data is taken as given.
Enforced by tests/test_api_contract.py, not by convention:
- every public callable is fully type-annotated and has a numpy-style docstring;
- no function body exceeds 60 lines, docstrings excluded;
- no module-level mutable state, no global random state, no I/O, no
print; - no
torch,sklearn,tensorflow,requestsormatplotlibanywhere; - everything a module exports is re-exported from the package root.
.venv/bin/python -m pytest -q # tests, doctests, and both examples
.venv/bin/ruff check falsify tests examples
.venv/bin/ruff format --check falsify tests examples
.venv/bin/mypy # strict- Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley.
- Bailey, D. and Lopez de Prado, M. (2014). The Deflated Sharpe Ratio. Journal of Portfolio Management 40(5). SSRN 2460551.
- Bailey, D., Borwein, J., Lopez de Prado, M. and Zhu, Q. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance 20(4). SSRN 2326253.
- Politis, D. and Romano, J. (1994). The Stationary Bootstrap. JASA 89(428).
- Westfall, P. and Young, S. (1993). Resampling-Based Multiple Testing. Wiley.
MIT. See LICENSE.