Tech

Product analytics and A/B tests

Run experiments a sceptical reviewer would accept: size them, read p-values and intervals honestly, spot broken tests, multiply funnels, and keep seasonality and luck out of conclusions.

Read first: Statistics and machine learning

7 concepts8 min read25-question test, from a bank of 33Updated

Concept 1 of 71 min

Sample size, power and variance reduction

The rule
At 5% significance and 80% power, each variant needs about 16 × p(1 − p) ÷ δ² users, δ being the absolute lift.
Worked examples

Users for a 2-point lift

Given
Base rate20%
Lift2 points
Significance5%
Power80%
Factor at 5%, 80%16
  1. p × (1 − p) = 0.2 × 0.8 = 0.16
  2. Users = 16 × 0.16 ÷ 0.022 = 6,400

About 6,400 users per variant for a 2-point lift.

Halving or doubling the lift

Given
At 2 points6,400 users
Lift1 or 4 points
  1. 1 point = 6,400 × 22 = 25,600
  2. 4 points = 6,400 ÷ 22 = 1,600

Half the lift needs 4 times the users, because the sample grows with the square of the precision.

The picture
Formula
n per variant ≈ 16 × p(1 − p) ÷ δ²
n ∝ 1 ÷ δ²
n ∝ variance
Why it holds

So halving the effect quadruples the sample, and a low-powered test that finds nothing proves little. Short traffic calls for more users, a longer test or variance reduction such as CUPED (controlled experiment using pre-experiment data), never a stricter significance level.

Small absolute lifts on low rates need large samples, which is why tiny effects are costly to prove.

With 30% power, a real effect of the planned size is missed 70% of the time: absence of evidence is not evidence of absence. CUPED uses each user's pre-test behaviour to remove noise, and the sample needed falls in line with the variance.

Lowering the significance level to 1% raises the sample needed further.

Watch out
  • Doubling the sample to detect half the effect.
  • Reading an underpowered test's null result as proof of no effect.
  • Tightening the significance level when traffic is short.
In the interview
  1. A product manager wants to test a new button colour on a page with 2,000 visitors a week. What would you tell them?

    A complete answer covers

    • Estimate the smallest lift worth detecting and the sample it needs.
    • With so little traffic, the test may take months to detect anything modest.
    • Test bigger changes, pool pages, or use variance reduction.
    • Or decide without a test if the change is cheap and safe.

Also in our technical interviews, on camera, with a score.

Concept 2 of 71 min

z-statistics, intervals, peeking and many metrics

The rule
The z-statistic of 2 conversion rates is their difference ÷ the pooled standard error √(p(1 − p) × (1/n₁ + 1/n₂)).

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 3 of 71 min

Designing tests that can be trusted

The rule
Before reading a test, check that the split matches the plan, since a sample ratio mismatch means users are lost.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 4 of 71 min

Funnels, revenue per visitor and proxy metrics

The rule
Funnel conversion is the product of its step rates, so a step's relative gain lifts the total by the same share.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 5 of 71 min

Product metrics: stickiness, retention, lifetime value, outliers

The rule
Stickiness is daily ÷ monthly active users, retention is measured on the starting cohort, and lifetime value is about monthly margin ÷ monthly churn.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 6 of 71 min

Correlation, seasonality, survivors and regression to the mean

The rule
A feature used by retained users may only mark engaged users, so an experiment is needed to show it causes retention.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 7 of 71 min

AdvancedCUPED: shorter tests from pre-test data

The rule
CUPED subtracts from each user's metric the part predicted by their own pre-test behaviour, removing a share ρ² of the variance.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Recap

  • Users per variant ≈ 16 × p(1 − p) ÷ δ²: half the effect needs 4 times the users; low power proves little.
  • z = difference ÷ standard error, 1.96 for 5% two-sided; peeking and many metrics manufacture false positives.
  • Before reading results, check the split, novelty, spillover and guardrails; keep a long holdout for long-term effects.
  • Funnel steps multiply; judge revenue per visitor, not a rate alone; a proxy pushed hard gets gamed.
  • Cohort retention divides by the starting cohort, lifetime value ≈ margin ÷ churn, and medians resist bots.
Now test it

25 questions on Product analytics and A/B tests

Timed, like the real one, drawn from a bank of 33. Every answer explained.

Test yourself →