A backtest answers a narrow question: how would this exact rule have performed on this exact history? It does not tell you how many other rules were tried before this one was chosen. That omission is the single largest source of false discoveries in quantitative research.
§01Selection bias in one example
Suppose a researcher tests 100 independent strategy variants on three years of daily data, and none of them has any true edge. Each variant's measured Sharpe ratio is noise around zero. For an annualised Sharpe ratio estimated over T years, the standard error near zero is roughly 1/√T — about 0.58 for three years.
The researcher then keeps the best variant. Bailey and López de Prado (2014) give an approximation for the expected maximum of N such trials:
With N = 100 and σ_SR ≈ 0.58, the bracket evaluates to about 2.53, so the expected best Sharpe ratio is roughly 1.46 — from strategies that are, by construction, worthless. A Sharpe ratio that would look respectable in isolation is exactly what pure selection produces.
§02Non-normal returns make it worse
The standard error of the Sharpe ratio also depends on the shape of the return distribution. Lo (2002) derived its sampling distribution and showed that serial correlation can distort annualised Sharpe ratios substantially. Negative skew and fat tails — common in strategies that sell insurance-like payoffs — widen the uncertainty further. The Deflated Sharpe Ratio of Bailey and López de Prado combines both corrections: it asks whether an observed Sharpe ratio is significantly above the maximum expected from the number of trials, given the skewness and kurtosis of the returns.
§03Higher hurdles
The same problem appears in academic finance. Harvey, Liu and Zhu (2016) catalogued hundreds of published return factors and argued that, after accounting for multiple testing, a new factor should clear a t-statistic of roughly 3.0 rather than the conventional 2.0. Bailey, Borwein, López de Prado and Zhu (2014) showed more generally how easily backtests can be overfitted, and how overfitted backtests tend to underperform out of sample.
§04Practices that keep research honest
- Record every trial. The number of variants tested is a parameter of the result; it cannot be reconstructed afterwards.
- Require an economic or mechanical rationale before testing, so the hypothesis space stays small and meaningful.
- Hold out data that is never used for selection, and use walk-forward evaluation that respects time ordering.
- Purge and embargo overlapping observations in cross-validation so that labels do not leak across folds (López de Prado, 2018).
- Model transaction costs and capacity before evaluating a signal, not after selecting it.
- Compare results against the expected maximum of the trial count, or report a deflated statistic alongside the raw one.
A research process should be judged by how reliably it rejects bad ideas, not by how impressive its best backtest looks.
