Concept 1 of 61 min
Overfitting and how to fight it
In sample against out of sample
| Sharpe in sample | 3 |
|---|---|
| Out of sample | 0.3 |
- Kept = 0.3 ÷ 3 = 10%
Only 10% of the apparent quality survives on new data, the signature of overfitting.
- Training
- Validation
The other defences are a stronger penalty such as ridge's λ, early stopping on a validation set, dropout in neural networks and a sensible learning rate. Testing many variants on the same data makes one look brilliant by chance.
A larger ridge penalty shrinks coefficients towards zero, more bias, less variance; exact zeros come from Lasso. Early stopping halts training when the validation error starts to rise.
Dropout switches off random units during training so they cannot rely on each other, and is switched off at prediction. A learning rate that is too large makes the loss jump around and grow.
Every choice made with validation data leaks a little information, so only untouched data gives an honest final score.
A great backtest that loses money live is the classic sign: the strategy was fitted to noise in the history, which the untouched test period would have exposed.
If training error falls from 0.5 at epoch 10 to 0.3 at epoch 20 and 0.1 at epoch 40, while validation error goes from 0.55 to 0.4 and then back up to 0.5, training should stop at epoch 20.
- Blaming costs or market efficiency for a collapse that overfitting explains.
- Expecting ridge to set coefficients to zero.
- Tuning on the final test set.
Your model has a great backtest but loses money live. What is the most likely explanation, and how would you have caught it?
You have tried 200 versions of a trading signal and kept the best. How would you know whether it is real?
A complete answer covers
- Expect the best of 200 to look good by chance: adjust for multiple testing.
- Test it on data never used in the search, walk-forward.
- Check that it is stable across periods and parameters.
- Prefer a simple signal with an economic reason.
Also in our technical interviews, on camera, with a score.
Concept 2 of 61 min
Leakage and validation in time
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 3 of 61 min
Random forests and boosting
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 4 of 61 min
Rare events and the right metrics
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 5 of 61 min
Distances, explanations and drift
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 6 of 61 min
AdvancedWhy more trees stop helping: the correlation floor
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inRecap
- A big gap between in-sample and out-of-sample results means overfitting; regularise, stop early, and keep a final test set untouched.
- Leakage hides everywhere: scaling on the whole dataset, labels that overlap the test period, features known only afterwards, today's index members.
- Validate financial models walk-forward, with purging and an embargo around each test fold.
- Random forests cut variance by averaging decorrelated trees; boosting fits the residuals one tree at a time and can overfit if it runs too long.
- On rare events accuracy misleads: use precision, recall and AUC; small edges can still pay at scale.
25 questions on Machine learning for finance
Timed, like the real one, drawn from a bank of 33. Every answer explained.