In-sample, out-of-sample & walk-forward analysis
Learning objectives
- Split a data set into in-sample (fitting) and out-of-sample (judging) periods, and explain why the out-of-sample period must never feed back into the rules.
- Run a walk-forward analysis and tell anchored windows apart from rolling windows.
- Read walk-forward efficiency (out-of-sample vs in-sample performance) and recognise when a strategy is only fitting its training window.
- Handle the order-flow-specific traps — regime changes, contract rolls and session boundaries.
M7.10 — BACKTEST OVERFITTING & DEFLATED SHARPE VISUALIZER
Multiple-Testing Haircut: Reported Sharpe vs Deflated Sharpe Across N = 1 → 1000 Trials
0%
M10.03 — In-sample, out-of-sample & walk-forward analysis · Quant Validation
This lesson is in production. The outline below shows what it will cover. The full lesson will ship with a worked NQ example and a quiz with answer explanations.
Hook
- A strategy optimised on 2021–2023 NQ data, then traded through a different volatility regime: why a single split can mislead in both directions.
Chart first: the three windows
- ASCII timeline: in-sample (IS) → out-of-sample (OOS) → lockbox (M10.06).
- How OOS data used for selection quietly becomes IS data (the “peek once, tweak, peek again” loop).
Walk-forward analysis (Pardo)
- Optimise on window k, trade the next untouched window, step forward, and stitch together only the OOS segments.
- Anchored (start date fixed, window expanding) vs rolling (fixed-length window): more data versus faster adaptation to a new regime.
- Choosing window lengths by number of trades per window, not calendar time, with minimum trade counts.
- Walk-forward efficiency (annualised OOS return ÷ annualised IS return) as a diagnostic. Any “good” threshold that practitioners quote is a rule of thumb and will be labelled as one.
Order-flow-specific traps
- Contract rolls and back-adjusted series, session boundaries (ETH vs RTH), and data-vendor changes in tick or footprint aggregation.
- Volatility regime as a confounder: compare OOS performance within each regime (links to M8).
Worked example (planned)
- Five-fold rolling walk-forward on a simplified absorption entry (M3.02), with costs from M10.02 and a table of IS vs OOS expectancy in ticks per fold.
Common mistake (planned)
- Re-optimising after seeing OOS results and still calling the next run “out-of-sample”.
Sources
- Robert Pardo, The Evaluation and Optimization of Trading Strategies, 2nd ed., John Wiley & Sons, 2008 (ISBN 978-0-470-12801-5), ch. 11 “Walk-Forward Analysis”. Publisher listing
- Bailey, Borwein, López de Prado & Zhu (2014), “Pseudo-Mathematics and Financial Charlatanism”, Notices of the AMS 61(5). Link
Prerequisite: M10.02 Costs & slippage in ticks · Next: M10.04 Multiple testing & the Deflated Sharpe Ratio
Educational content only — not financial advice. Futures trading involves substantial risk of loss.
Saved in this browser (account sync comes at launch).
Not financial advice. Educational content only. Futures trading involves substantial risk of loss.