A reproducible test starts before the result
Specify the universe, data, signal, observation time, execution assumption, position sizing and exit rules. Explain how distributions, corporate actions and missing observations are handled. A chart without these details is difficult to reproduce or evaluate.
Imagine a rule selects stocks using the quarter’s final earnings report but assumes purchases at prices from before the release. It has given the strategy information it did not have. This look-ahead bias can create a persuasive result from an impossible decision process.
Survivors and repeated experiments can flatter history
A test using only companies that exist today omits some that failed, delisted or were acquired during the period. That current universe may not represent what could have been selected at the historical date. Point-in-time membership and data help address the problem.
Trying hundreds of rule variations and presenting only the best also changes the evidence. Some attractive patterns arise by chance. Separate the period used to design a rule from later evaluation, and keep a record of the research alternatives rather than silently forgetting failed tests.
The trading assumptions can change the conclusion
Suppose a fictional strategy’s gross edge is 0.2 percentage points per rebalance, while its assumed all-in implementation cost is 0.3 points. That difference turns a small gross advantage into a negative one before other uncertainties. The point is not the invented estimate but the need to model the actual frictions.
Quoted prices may not support the position size or timing assumed. Spreads, slippage, fees, borrowing and financing costs can matter. A test that ignores them should not be described as an achievable net result.
Ask what remains after a fair comparison
Compare with a suitable benchmark over matching windows and return conventions. Examine drawdowns, turnover, exposure and dependence on a few winners. Test sensitivity to modest rule changes and to less favorable market conditions.
A successful backtest can justify further investigation. It does not certify future profitability, and independent forward observation adds different evidence from reusing the same historical sample. Keep hypothetical, reconstructed and actual results clearly labeled.
A rule was selected because it produced the best result among 500 historical variations. Is that equivalent to one pre-specified successful test?
Show the explanation
No. Selecting the best of many attempts introduces a multiple-testing problem. Fresh evaluation and a transparent research record are needed to assess whether the apparent edge persists.
Sources and editorial notes
Original explanations and hypothetical examples prepared with AI assistance. Primary references checked October 16, 2026. Numerical examples are hypothetical unless explicitly identified otherwise.
- Fidelity: About Strategy Testing
Historical strategy simulation, test assumptions and limitations.