1,192 indicator combinations: the winner was no better than luck

· 1 min read

Failure type: overfit. Verdict: nothing promoted.

The idea

Combine up to three indicators into a ranking score, hold the top 5 or 8 ETFs, rebalance every 5 or 21 sessions. Search everything.

What we did

  • 12 indicators, all 1–3 indicator subsets, 2 rebalance speeds, 2 portfolio sizes: 1,192 configurations.
  • Choose on training years only (2006–2016). Validation (2017–2021) was reported, not used to choose.
  • A null: run 200 complete searches on time-shifted indicators and see what the best of each looks like.
  • Deflated Sharpe and probability of backtest overfitting on the whole set.

What happened

Result
Best on training years Sharpe 0.744 (a three-indicator combo, rebalanced every 5 sessions)
Typical config on training median Sharpe 0.50; 90th percentile 0.61
Best of a scrambled-data search median 0.83; 95th percentile 1.03
How often scrambled data beat our winner empirical p = 0.73
Deflated Sharpe (1,192 tries) 0.910 (bar 0.95)
Probability of backtest overfitting 0.78 (bar 0.20)

Rebuilt in a realistic engine (next-open fills, settled cash, costs), the finalist got Sharpe 0.70 on training and 0.77 on 2017–2021, about 9.2% a year against SPY's 18.2% in that window. It traded about 41–45 times its capital a year.

Why it failed

Picking the training winner added nothing out of sample: the top ten on training averaged 0.77 on validation, against a 0.76 median for all configurations. Searching more than a thousand combinations on 23 funds finds something that looks good, and that is all it did.

Receipts: 1,192 distinct configurations were recorded in our trial registry.

Research and education only. These are paper results from historical data on a limited set of funds. Not investment advice, not a recommendation, not a prediction. See Legal.