Tech

Machine learning in practice

Read a model's scores like a reviewer: precision, recall, F1 and AUC (area under the ROC curve), bias against variance, leaks that make results too good, features that behave, and the A/B test that decides.

7 concepts8 min read25-question test, from a bank of 33Updated

Concept 1 of 71 min

Precision, recall, F1, AUC and calibration

The rule
Precision is the share of flags that are right, recall the share of positives caught, and F1 their harmonic mean 2PR ÷ (P + R).
Worked examples

F1 against the average

Given
Precision0.6
Recall0.9
  1. Top = 2 × 0.6 × 0.9 = 1.08
  2. F1 = 1.08 ÷ (0.6 + 0.9) = 0.72
  3. Average = (0.6 + 0.9) ÷ 2 = 0.75

F1 sits below the average because it punishes the weaker of the 2.

Recall from precision

Given
Fraud3%
Flagged2%
Precision75%
  1. True positives = 0.75 × 2% = 1.5%
  2. Recall = 1.5 ÷ 3 = 50%

Most flags are right, yet half the fraud goes unflagged.

A reversed AUC

Given
AUC0.35
  1. Reversed = 1 − 0.35 = 0.65

An AUC below 0.5 means the scores point the wrong way.

The picture
Formula
Precision=TP ÷ (TP + FP)
recall=TP ÷ (TP + FN)
F1=2PR ÷ (P + R)
false positive rate=FP ÷ (FP + TN)
flipped AUC=1−AUC
Why it holds

The false positive rate is the share of actual negatives flagged. An AUC below 0.5 means the ranking is inverted, and calibration fixes probabilities without changing the order of scores.

Recall can be derived from the flag rate and precision: true positives are precision times the share flagged. A precise model can still miss most positives when it flags too few.

The harmonic mean punishes a weak side, which a simple average hides. The false positive rate divides by actual negatives, while 1 − precision divides by flags: 2 different questions.

Below 0.5, reversing the scores helps, but first look for a label flipped somewhere in the pipeline; the threshold does not change the AUC. Calibration maps scores to observed frequencies, so a predicted 80% happens about 80% of the time.

Watch out
  • Taking the simple average for F1.
  • Dividing false positives by the flags for the false positive rate.
  • Discarding a model with an AUC below 0.5 without looking at the labels.
In the interview
  1. Your fraud model has 95% precision. The head of operations says it is excellent. What would you ask before agreeing?

    A complete answer covers

    • What its recall is: how many frauds it misses.
    • What share of transactions it flags.
    • Whether the numbers come from a clean, recent test set.
    • What a missed fraud and a false alarm each cost.

Also in our technical interviews, on camera, with a score.

Concept 2 of 71 min

Imbalanced classes and the decision threshold

The rule
When positives are rare, a model that predicts the majority class scores high accuracy and catches nothing.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 3 of 71 min

Bias, variance and honest validation

The rule
Low training error with high validation error is variance, cured by more data, regularisation or a simpler model; high, close errors are bias.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 4 of 71 min

Leaks, group splits and drift

The rule
Scores that look too good usually come from leakage, so split by group and check when each feature becomes known.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 5 of 71 min

Features: scaling, encoding, selection and odds

The rule
Trees ignore a feature's scale while distance-based and penalised models depend on it, and PCA (principal component analysis) on raw data follows the largest variance.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 6 of 71 min

Choosing the model: clusters, boosting and recommendations

The rule
k-means assumes compact, round clusters, so it cuts rings in the wrong place where spectral clustering or DBSCAN (density-based spatial clustering of applications with noise) do not.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 7 of 71 min

AdvancedPrecision falls with the base rate

The rule
A detector's recall and false positive rate do not depend on how common positives are, but its precision does.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Recap

  • Precision = right flags ÷ all flags, recall = frauds caught ÷ all frauds, F1 = their harmonic mean, false positive rate = false alarms ÷ actual negatives.
  • On rare events accuracy flatters; set the threshold from the cost of each error and reweight the training set, never the test set.
  • Training error low and validation high is variance; both high and close is bias; stop at the best validation epoch and beat a simple baseline.
  • Scores that look too good usually leak: features that follow the outcome, the same patient in train and test, an ID.
  • Scale before distances and PCA, not before trees; L1 selects features; a logistic coefficient multiplies the odds by eβ.
Now test it

25 questions on Machine learning in practice

Timed, like the real one, drawn from a bank of 33. Every answer explained.

Test yourself →