Five gates I set before looking at a backtest result

I spent two years tuning a strategy on a backtest I could not reproduce. These are the rules I wrote afterwards — before running anything again.

What went wrong first

For about two years I tuned an order-flow strategy inside a platform backtest. That backtest was not deterministic: the same rule over the same data did not always produce the same trades. I knew it and kept going, because every round of tuning made the numbers look better.

They looked better because I was fitting the noise. Each parameter I moved was a response to variance I could not reproduce. Two years of work produced a strategy that existed only in the runs I happened to see.

The lesson is not “backtests lie.” It is narrower and more useful: if you cannot reproduce a result you cannot learn from it, and the improvement you feel while tuning is not evidence of anything.

What follows is what I wrote after that, before running the next set of tests. They are gates, not goals. A setup clears them or it does not, and I do not get to move them afterwards.

Gate 1 — Decide the cost before you see the result

Round-trip cost on my instrument measures at about 2.4 ticks: one tick of front-month spread, commission near 0.4, and a tick of slippage.

I set the pass/fail gate at 4.0 ticks anyway.

Not because 4.0 is more accurate — 2.4 is the honest measurement. I set it higher so that I could not quietly relax it later, when results were disappointing. A cost assumption chosen after you see the P&L is not an assumption; it is a dial.

Write the number down first. If a setup only survives at your best-case cost, it does not survive.

Gate 2 — Compute the break-even win rate your R:R demands

The cheapest test there is, and it killed my setup faster than anything else.

Measured reward-to-risk came out at 0.17. That implies a break-even win rate of about 85.5% — right roughly six times out of seven, just to pay for the losses.

Actual win rate was 81.2%.

Eighty-one percent sounds excellent, and on a win-rate-only view the setup looked healthy. It was negative, and no amount of target tuning was going to close a four-point gap against an 85% requirement.

Do this before you build anything elaborate. One division, one comparison. It costs nothing and it ends a lot of conversations early.

Do not read a high win rate as a good sign on its own. A high win rate with non-positive expectancy is picking up pennies: the losses simply arrive less often than they are large. Judge on expectancy and profit factor. The win rate is a component, not a verdict.

Gate 3 — How often entry and exit land in the same bar

I gate this at under 30%. The best I measured was 31.9%.

A high same-bar rate means the setup is not giving the trade room to be right or wrong. It enters and exits inside one bar’s noise, which makes the result a statement about your bar size rather than about the market.

It is also a warning that the exits are doing the work rather than the entries — which was true in my case, and took a long time to see because I was looking at the entry side.

Gate 4 — Below thirty trades there is no verdict

Under thirty trades I record no verdict. Not a pass, not a failure.

This matters more than it sounds. The temptation with a small sample is to call a marginal result “promising” and keep it alive. A hypothesis that is neither confirmed nor rejected has to be labelled that way, or it quietly becomes a belief.

For promotion to anything real I require at least a hundred trades. Between thirty and a hundred I can read a direction. Below thirty I cannot.

Gate 5 — Write the criteria down before you look

Everything above works only if it is written before the result exists.

So I pre-register: one hypothesis per run, gates fixed in advance, run design specified before any of it executes. The rule I find hardest is the simplest — the criteria do not get loosened. If a result misses the gate then the result failed; the gate did not turn out to be too strict.

• One hypothesis per run, and it does not change mid-run.
• Failures stay in the record, along with hypotheses I abandoned. The value of the log is that it stops me re-running something I already killed.

What this cost me

The first pre-registered run tested a belief I had held for a while: that shortening the bar would fix the same-bar problem.

Same rule, same data, only the bar size changed. Reward-to-risk stayed pinned at 0.17 across every bar size I tried. It did not move.

FAIL. Bar size was not the binding constraint, and a couple of years of intuition about it was wrong. The run took an afternoon. Believing it cost considerably more.

Honest limits

• One instrument, one feed. CL through a single data path. The numbers are mine, not yours — the gates are the transferable part, not the decimals.
• These gates reject; they do not confirm. Clearing all five does not make a setup profitable. It only means it has not yet failed in the ways I know how to check.
• This is a record of what I measured, not advice. Nothing here is a prediction, a recommendation, or a claim about what any setup will do next.

These measurements come out of OrderVane, the order-flow tooling I build for Quantower. It is the instrument here, not the subject.