Multiple testing & the Deflated Sharpe Ratio

Your optimiser tried 50 variants of an NQ opening-drive model on two years of daily returns. The winner shows an annualised Sharpe ratio of 2.0 — and a standard test says it is 99.7% confident that its true Sharpe is above zero. That 99.7% answers the wrong question. It treats the winner as if it were the only thing you tested. Once you account for the 49 losers you threw away, the same backtest gives only about 60% confidence that the edge is real. That is barely better than a coin flip, from a result that looked like a sure thing.

The Sharpe ratio (tỷ số Sharpe) is mean return divided by the standard deviation of returns over the same period. The Deflated Sharpe Ratio (DSR, tỷ số Sharpe đã khử lạm phát), introduced by Bailey & López de Prado (2014) in the Journal of Portfolio Management, is a confidence level (one minus a p-value in the frequentist sense, not a Bayesian probability) that a strategy’s true Sharpe ratio exceeds what selection alone would produce, after correcting for two things that inflate backtests: selection bias under multiple testing (you picked the best of N) and non-normal returns (skew and fat tails make a Sharpe estimate less reliable).

Chart first: the best of N is never zero

Imagine N strategies that all have a true Sharpe of exactly zero. Their estimated Sharpe ratios still scatter around zero because of noise. Take the maximum and it will be positive — and it grows as N grows:

Expected maximum of N standard normal draws (approximation used in the DSR paper)

 N      E[max]  (in units of the cross-trial std of Sharpe estimates)
 5      1.19   ████████████
 10     1.57   ████████████████
 20     1.90   ███████████████████
 50     2.28   ███████████████████████
 100    2.53   █████████████████████████
 1000   3.26   █████████████████████████████████

So the right benchmark for “is the winner skilled?” is not zero. It is SR₀, the Sharpe ratio you would expect the best of N zero-skill trials to show:

SR₀ = √V[SR̂ₙ] · ( (1 − γ) · Φ⁻¹(1 − 1/N)  +  γ · Φ⁻¹(1 − 1/(N·e)) )

• V[SR̂ₙ] — the variance of the estimated Sharpe ratios across your N trials (measure it from your own optimisation log);
• N — the number of independent trials (correlated variants count as fewer; estimating the effective N is itself part of the method);
• Φ⁻¹ — the inverse of the standard normal CDF;
• γ ≈ 0.5772 — the Euler–Mascheroni constant; e ≈ 2.718.

This formula is an approximation that works well for large N. At N = 2 it slightly understates the true expected maximum; from N ≈ 3 upward it slightly overstates it (by about 0.01–0.03), which errs on the conservative side.

The DSR formula

The DSR then asks: how many standard errors is the observed Sharpe above SR₀, after adjusting the standard error for skew and fat tails?

            ⎛  (SR̂ − SR₀) · √(T − 1)                ⎞
DSR  =  Φ  ⎜ ───────────────────────────────────── ⎟
            ⎝  √( 1 − γ̂₃·SR̂ + ((γ̂₄ − 1)/4)·SR̂² )  ⎠

• SR̂ — the observed Sharpe ratio of the selected strategy, per period (not annualised);
• T — number of return observations;
• γ̂₃ — skewness of returns; γ̂₄ — raw kurtosis (3 for a normal distribution, not the “excess” kurtosis of 0 many tools print);
• Φ — the standard normal CDF, so DSR is a probability between 0 and 1.

With SR₀ = 0 the formula is the Probabilistic Sharpe Ratio (Bailey & López de Prado, Journal of Risk, 2012). The denominator is the non-normal standard error of the Sharpe ratio. With normal returns (γ̂₃ = 0, γ̂₄ = 3) it becomes √(1 + SR̂²/2), the familiar result from Lo (2002). Negative skew — many small wins, occasional big losses, typical of fading strategies — makes the denominator larger and the DSR smaller.

Worked example: deflating the opening-drive model

Inputs from the research log:

Step 1 — SR₀. Φ⁻¹(1 − 1/50) = Φ⁻¹(0.98) ≈ 2.054. Φ⁻¹(1 − 1/(50e)) = Φ⁻¹(0.99264) ≈ 2.440. E[max] ≈ (1 − 0.5772)(2.054) + 0.5772(2.440) ≈ 0.868 + 1.408 = 2.276. SR₀ = 0.05 × 2.276 ≈ 0.114 per day (≈ 1.81 annualised). The best of 50 zero-edge variants would be expected to show an annualised Sharpe near 1.8.

Step 2 — denominator. 1 − (−0.5)(0.126) + ((6 − 1)/4)(0.126²) = 1 + 0.063 + 0.020 = 1.083 → √ ≈ 1.041.

Step 3 — DSR. z = (0.126 − 0.114) × √499 / 1.041 ≈ 0.0122 × 22.34 / 1.041 ≈ 0.26. Φ(0.26) ≈ 0.60.

Compare the undeflated test (SR₀ = 0): z = 0.126 × 22.34 / 1.041 ≈ 2.70 → Φ ≈ 0.997. Same strategy, same data. The only difference is honesty about N. And watch how fast it decays:

A common reading is to require DSR ≥ 0.95 before trusting a result. That is a convention, like the 5% significance level, not a law. By that standard this backtest is not evidence yet. Two years of data are not enough to defend a search of 50 variants, and the next step is fresh out-of-sample data (M10.03) or a lockbox test (M10.06).

Common mistake: Plugging an annualised Sharpe into a formula that uses the daily number of observations T, or using excess kurtosis where the formula expects raw kurtosis. With an annualised SR̂ of 2.0 and T = 500, √(T−1)·SR̂ would be about 45 standard errors — a “certain” result produced by a units error. Keep SR̂, SR₀ and V[SR̂ₙ] in the same per-period units as T, and check whether your software reports kurtosis as 3 or 0 for a normal distribution.

Recap:

• The best of N zero-skill strategies has a positive expected Sharpe, SR₀ ≈ √V · ((1−γ)Φ⁻¹(1−1/N) + γΦ⁻¹(1−1/(Ne))), which rises with N.
• DSR = Φ( (SR̂ − SR₀)√(T−1) / √(1 − γ̂₃SR̂ + (γ̂₄−1)/4·SR̂²) ) is the probability the true Sharpe is positive after selection bias and non-normality.
• Negative skew and fat tails lower the DSR; more trials raise SR₀ and lower the DSR; more data (larger T) raises it.
• Log every trial. Without an honest N and V[SR̂ₙ] you cannot deflate anything.

Quiz: A strategy has per-period SR̂ = 0.10 over T = 401 observations with normal returns (skew 0, raw kurtosis 3). You tried only one pre-registered variant. What is the DSR, and what happens if you had actually tried 100 variants whose Sharpe estimates had a cross-trial standard deviation of 0.05? Answer: With N = 1, SR₀ = 0. The denominator is √(1 + 0.01/2) ≈ 1.0025, z = 0.10 × 20 / 1.0025 ≈ 2.0, DSR ≈ 0.977. With N = 100, SR₀ = 0.05 × 2.53 ≈ 0.127, larger than SR̂, so z ≈ (0.10 − 0.127) × 20 / 1.0025 ≈ −0.53 and DSR ≈ 0.30. Explanation: the same Sharpe ratio is convincing as a single pre-planned test and unconvincing as the winner of a 100-variant search. The evidence depends on the search, not just the result.

Further reading:

• Bailey & López de Prado (2014), “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality”, Journal of Portfolio Management 40(5), 94–107. DOI 10.3905/jpm.2014.40.5.094.
• Bailey & López de Prado (2012), “The Sharpe Ratio Efficient Frontier”, Journal of Risk 15(2). Introduces the Probabilistic Sharpe Ratio.
• Lo (2002), “The Statistics of Sharpe Ratios”, Financial Analysts Journal 58(4), 36–52. Standard errors, and why √12 annualisation can mislead.

Prerequisite: M10.01 Why most backtests lie, M10.03 In-sample/out-of-sample · Next: M10.05 Probability of Backtest Overfitting (CSCV) · Connects to: M7.10 (validating a method), M10.06 (lockbox discipline)

Educational content only — not financial advice. Futures trading involves substantial risk of loss.