Concept 1 of 71 min
Precision, recall, F1, AUC and calibration
F1 against the average
| Precision | 0.6 |
|---|---|
| Recall | 0.9 |
- Top = 2 × 0.6 × 0.9 = 1.08
- F1 = 1.08 ÷ (0.6 + 0.9) = 0.72
- Average = (0.6 + 0.9) ÷ 2 = 0.75
F1 sits below the average because it punishes the weaker of the 2.
Recall from precision
| Fraud | 3% |
|---|---|
| Flagged | 2% |
| Precision | 75% |
- True positives = 0.75 × 2% = 1.5%
- Recall = 1.5 ÷ 3 = 50%
Most flags are right, yet half the fraud goes unflagged.
A reversed AUC
| AUC | 0.35 |
|---|
- Reversed = 1 − 0.35 = 0.65
An AUC below 0.5 means the scores point the wrong way.
The false positive rate is the share of actual negatives flagged. An AUC below 0.5 means the ranking is inverted, and calibration fixes probabilities without changing the order of scores.
Recall can be derived from the flag rate and precision: true positives are precision times the share flagged. A precise model can still miss most positives when it flags too few.
The harmonic mean punishes a weak side, which a simple average hides. The false positive rate divides by actual negatives, while 1 − precision divides by flags: 2 different questions.
Below 0.5, reversing the scores helps, but first look for a label flipped somewhere in the pipeline; the threshold does not change the AUC. Calibration maps scores to observed frequencies, so a predicted 80% happens about 80% of the time.
- Taking the simple average for F1.
- Dividing false positives by the flags for the false positive rate.
- Discarding a model with an AUC below 0.5 without looking at the labels.
Your fraud model has 95% precision. The head of operations says it is excellent. What would you ask before agreeing?
A complete answer covers
- What its recall is: how many frauds it misses.
- What share of transactions it flags.
- Whether the numbers come from a clean, recent test set.
- What a missed fraud and a false alarm each cost.
Also in our technical interviews, on camera, with a score.
Concept 2 of 71 min
Imbalanced classes and the decision threshold
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 3 of 71 min
Bias, variance and honest validation
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 4 of 71 min
Leaks, group splits and drift
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 5 of 71 min
Features: scaling, encoding, selection and odds
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 6 of 71 min
Choosing the model: clusters, boosting and recommendations
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 7 of 71 min
AdvancedPrecision falls with the base rate
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inRecap
- Precision = right flags ÷ all flags, recall = frauds caught ÷ all frauds, F1 = their harmonic mean, false positive rate = false alarms ÷ actual negatives.
- On rare events accuracy flatters; set the threshold from the cost of each error and reweight the training set, never the test set.
- Training error low and validation high is variance; both high and close is bias; stop at the best validation epoch and beat a simple baseline.
- Scores that look too good usually leak: features that follow the outcome, the same patient in train and test, an ID.
- Scale before distances and PCA, not before trees; L1 selects features; a logistic coefficient multiplies the odds by eβ.
25 questions on Machine learning in practice
Timed, like the real one, drawn from a bank of 33. Every answer explained.