How we test
What is overfitting?
Try enough ideas and one will look brilliant by accident. Overfitting is when a strategy is shaped (by you, or by a search) to fit the past so well that it is mostly fitting noise, so it falls apart on data it hasn't seen.
The cure isn't cleverness. It is bookkeeping: write the rule down first, count every try, and judge the result on years you didn't peek at.
What we do
1. Pre-register
The rule, the windows and the pass criteria are written into a file and committed in version control before the run. The runner refuses to start on a file that has uncommitted changes. If we change the rule after seeing results, that is a different idea and the page says so.
2. Split the years
- Training: 2005–2016. Searching and tuning are allowed here.
- Validation: 2017–2021, a strongly trending market.
- Second window: 2022–2023, when stocks and bonds fell together.
- Reserved: from 2 January 2024 on. Never used for any published result. (Honest footnote: a few early demo runs, before we enforced this, did see some of it.)
3. Count every try
Each backtest, parameter combination and variant goes into a trial registry. A result with no trial count is just a pretty line.
4. Judge against what chance would do
- Deflated Sharpe ratio (DSR) (Bailey and López de Prado, 2014): given how many things we tried, how likely is the best Sharpe ratio to be real? Our bar is 0.95.
- Probability of backtest overfitting (PBO) (Bailey, Borwein, López de Prado and Zhu, 2017): how often does the in-sample winner land below the median out of sample? Our general bar is 0.20; for one family we set a looser 0.50 in advance and it still failed.
- False-discovery control (Benjamini and Hochberg, 1995) when running many tests at once.
- Null comparisons: shift the indicator in time, pick random entry dates, or train on shuffled answers. If scrambled data does as well, the "edge" isn't one.
5. Compare fairly
A strategy that is invested 50% of the time looks safer than SPY for free. So we also compare against SPY held at the same invested level (rest in T-bills) and at the same volatility, plus an equal-weight basket of the same funds.
6. Trade it realistically
Decide at the close, fill at the next open. Spreads, impact and fees included. Cash settles the next day, like a retirement account, with no leverage. Idle cash earns the 3-month T-bill rate.
How things die
| Type | Meaning |
|---|---|
| Overfit | The winner doesn't survive an honest count of tries, or is no better than a shuffled null. |
| Not tradable | The effect may exist but can't be captured with our realistic execution. |
| Edge decayed | It worked in one window and failed a later one we fixed in advance. |
| Works but not worth it | It does something real (often smaller drops) but fails its bar or gives up the return. |
| Passed, unproven live | It cleared its pre-registered tests. That is not the same as working with real money. |
| Untestable | Not enough data to test it honestly. We say so instead of inventing a p-value. |
What this can't tell you
- We used 23 liquid ETFs and some other free-data baskets. Results may not carry to single stocks.
- The data is free-tier and not survivorship-free.
- Everything is paper. Simulated costs are not a brokerage statement.
- A failed test says "we couldn't find it with this data and rule", not "can never work".
- Sharpe ratio = average excess return per unit of volatility. It's a ratio, and it has its own failure modes.
Research and education only. Not investment advice. See Legal.