How much history is enough, and why more tuning makes it worse

Overfitting scales with how widely you searched, not with how hard you worked. That one sentence changes what a backtest is for — and it is why I now make narrow changes on long samples instead of clever ones on short samples.

The uncomfortable result

The finding that reorganised how I work comes from David Bailey and Marcos López de Prado and colleagues, in The Probability of Backtest Overfitting (2015) and the earlier Pseudo-Mathematics and Financial Charlatanism (2014).

Their argument is not that backtests are unreliable in some vague way. It is sharper: given enough trials on a finite sample, a strategy that looks excellent in-sample is guaranteed to appear, whether or not anything real is there. The more configurations you evaluate, the more certain that outcome becomes. Overfitting is not a risk you might run into; past a point it is the expected result of the procedure.

The corollary is the part that stung. If you do not record how many variants you tried, you cannot say what your best result means — and almost nobody records it, because trying things does not feel like consuming a budget.

Overfitting scales with search, not effort

This is the sentence worth keeping: the enemy is the size of the search, not the amount of work.

Two people can spend the same month on the same data. One tests four carefully-reasoned variants; the other sweeps six parameters across a grid. The second has a far higher chance of producing something that looks good and is not, and no amount of care in the sweep fixes it, because the problem is the count of trials rather than the quality of any one.

I spent about two years tuning inside a platform backtest that was not deterministic — the same rule on the same data did not reproduce. Every adjustment was a response to variance, and the variance was inexhaustible. There was no shortage of improvements available. That was exactly the problem.

Why narrow changes beat broad ones

If trials are the budget, then the shape of a change matters as much as whether it helps.

A broad cut — “skip trades below this score” — is one knob with a wide range, and every value in the range is a trial you are implicitly spending. It will find a threshold that flatters the sample. A narrow, exactly-specified veto covers less ground, is far harder to accidentally tune, and fails visibly if it is wrong.

So my rule is to prefer changes that are specific enough to be falsified and small enough that they cannot absorb noise. It also means resisting the most natural move in trading development: when results disappoint, widening the filter until they do not.

Holding data back is not optional

Robert Pardo’s Evaluation and Optimization of Trading Strategies is the standard reference for the mechanical answer: walk forward. Optimise on one window, test on the next window you have never touched, roll, repeat.

The number that comes out is walk-forward efficiency — out-of-sample performance as a share of in-sample. Pardo’s working threshold is around 70%. Below that, the in-sample result was substantially a description of that window rather than of the market.

The discipline this enforces is unglamorous and hard: the held-back window is only held back until you look at it. Once you have seen it and adjusted, it is in-sample and you need a new one.

Plateau or cliff

A robustness check I now run before believing any parameter, drawn from the standard sensitivity tests: look at the neighbours.

If a setting works at 12 but collapses at 11 and 13, the 12 is not a finding. It is a coordinate in a noise field. If it works reasonably across 10 to 14 and degrades gently outside, there is a plateau, and a plateau is the shape a real effect makes.

The single best number is almost always the worst one to ship, because being the best requires having beaten its neighbours — which is exactly what noise does.

What I require now

A minimum span before a run counts at all. Mine is 2,160 hours — ninety days — and shorter runs are labelled invalid rather than weighed lightly. A run that is not long enough is not a weak result; it is not a result.
At least a hundred trades for promotion, and no verdict at all below thirty.
Neighbours checked before a parameter is trusted.
Criteria written before the run, and not loosened afterwards.

None of this makes a strategy work. It makes the failures arrive earlier and cost less, which is the only durable advantage I have found in this part of the process.

Honest limits

These are the authors’ arguments, summarised. The papers are worth reading directly; the framing here is mine and any imprecision is mine too.
Long samples do not confer validity. Ninety days of a regime that never repeats is ninety days of one thing. Length is a floor, not a proof.
Nothing here is a claim about returns. Everything above is about how to avoid believing results, not about producing them.