Trading strategy generators compete on one number: how many strategies they can build. Hundreds of thousands per hour, millions per night. And at the end, on the screen, a list of candidates with beautiful equity curves, a profit factor of 2 and six-figure profits.
There is one question almost nobody asks: what would that same program find if we gave it data with nothing to find?
We tried it with ours. The result made us uncomfortable, which is exactly why we are sharing it.
The trap: with enough attempts, luck always wins
Toss a coin ten times and getting ten heads is very rare. Have a million people try, and about a thousand will do it. None of them has a gift: there were simply a lot of them.
A strategy generator does exactly that. It combines indicators, thresholds, exits and filters, and tests millions of combinations on the same history. Some will come out as winners just because: they happened to fit the ups and downs of that particular period. It is called overfitting, and it is why so many strategies that shine in the backtest sink as soon as they start trading.
Everybody knows this. Almost nobody measures it.
The experiment: an ES with nothing to win
To measure it we needed a control group: a market that looked like the real one in every way but one. It had to be impossible to predict.
We built it from the S&P 500 future (the ES), using its real daily bars since 2000:
- Every day keeps its real shape: how much it opened, rose, fell and closed relative to the previous day.
- But the days are shuffled at random. Trends, streaks, cycles and any exploitable pattern disappear.
- And we removed the index's average rise, so that "buy and stay in" cannot win from the market's upward drift alone. The series ends exactly where it started.
| Real ES | Made-up ES | |
|---|---|---|
| Typical daily move (volatility) | 0.84% | 0.84% |
| Memory from one day to the next | −0.08 | 0.00 |
| Streaks of calm and panic | 0.28 | 0.00 |
| Price at the start → at the end | 2,326 → 7,836 | 2,326 → 2,326 |
It moves just as hard as the ES, day by day. But it contains nothing, absolutely nothing, that can be anticipated. Any strategy that wins here wins by luck.
Same program, same effort
We used AQ Genesis, AniQuant's autonomous search engine, with the same settings on both series: long only, the same seed, 30 minutes of search, a $3 commission and one tick of slippage per fill.
70% of the history (2000-2018) was used for the search. The remaining 30% (2018-2026) was kept under lock and key: Genesis never saw it.
Along the way, every candidate had to clear a wall of seven filters: not a near-clone of another, a minimum number of trades, a minimum profit factor, an edge over buy and hold, the probability of backtest overfitting (PBO), Walk-Forward, and a survival test that stresses it with changes in parameters, costs and period. These are serious filters: the ones the academic literature recommends.
On both series the filters rejected around 93% of what was generated. And on both, a hundred strategies cleared everything.
The hundred from luck look as good as the real ones
This is the uncomfortable part. Compare the hundred strategies sealed on the made-up ES with the hundred from the real ES and, over the search period, they are practically indistinguishable:
In fact, the lucky ones made more: a median of $91,052 against $58,911. Their confidence scores ranged from 83 to 92 out of 100. Their probability of overfitting was the same. They passed Walk-Forward just as well.
How can chance make more than a market that goes up? For two reasons. On the real ES, Genesis demands beating buy and hold, and that discarded 1,782 strategies that were merely riding the trend; on the made-up one there was no trend to beat, and only 27 fell. And what is left is the best of 17 million attempts: on made-up data that is pure luck, and the best luck out of 17 million is a lot of luck. That they shine brighter is precisely the warning sign.
The best of them, found on made-up data, read like this: buy when, ten days ago, the candle had a moderate body and price sat in the bottom 20% of its three-week range; 4% target and 1% stop. It sounds reasonable: buy cheap after a pause. It has 216 trades, $93,000 of profit and a confidence of 91.9. Anyone would trade it.
The only test that was not fooled
Then we opened the drawer: the eight years no strategy had seen.
The hundred from the made-up ES behaved like what they were: coin tosses. Their typical profit factor was 1.05. 62 won and 38 lost, with no pattern at all. None of them knew anything.
The hundred from the real ES made money in 99 cases out of 100, with a typical profit factor of 1.65.
The held-out period did in one minute what seventeen million backtests and seven filters had not managed: separate luck from what might be an edge.
But careful: the real ones are not proven either
It would be easy to end here with "ours work". It would not be honest.
Those eight years were very bullish for the S&P 500. Buy and hold would have made $163,401, twice the typical real-ES strategy ($80,969). A long-only strategy has the wind at its back in a rising market, and "99 out of 100 positive" is partly explained by that. Passing the held-out period is a necessary condition, not proof that it will keep working.
What to do with your strategies
Whatever software you use, the experiment leaves five lessons:
- Keep a period under lock and key and do not look at it until the very end. If you use it to choose, it is no longer held out.
- Run a control group. Repeat the search on the same market, shuffled. If you find equally pretty strategies there, the real ones prove nothing on their own.
- Count the attempts. A strategy picked among ten is not the same as one picked among seventy thousand. Metrics such as the deflated Sharpe ratio correct exactly that.
- Always compare with buy and hold over the same period. Making money in a rising market is no achievement.
- Distrust the number of strategies generated. Generating millions is not an advantage: it is the fastest way to find coincidences.
What we built in AniQuant
This experiment led us to create AQ Market Shuffle: one click generates the shuffled version of any market, and you can run it through Genesis, the Lab or any other module with the same effort. If you find equally pretty strategies there, you know what the ones you found in the real market are worth.
We are not telling you this to sell anything. We are telling you because, without that control, any automatically generated strategy is suspect. Ours too.
Next time a generator shows you a list of spectacular strategies, ask yourself one question: how many would it have found on made-up data?
Want backtests without this bias?
AniQuant works with universes that include the companies that disappeared. Try it free for 14 days.
Download AniQuantFounder of AniSoft (software since 1995) and creator of AniQuant. Writes about honest backtesting, strategy validation and the mistakes that cost money.