Concept 1 of 71 min
Sample size, power and variance reduction
Users for a 2-point lift
| Base rate | 20% |
|---|---|
| Lift | 2 points |
| Significance | 5% |
| Power | 80% |
| Factor at 5%, 80% | 16 |
- p × (1 − p) = 0.2 × 0.8 = 0.16
- Users = 16 × 0.16 ÷ 0.022 = 6,400
About 6,400 users per variant for a 2-point lift.
Halving or doubling the lift
| At 2 points | 6,400 users |
|---|---|
| Lift | 1 or 4 points |
- 1 point = 6,400 × 22 = 25,600
- 4 points = 6,400 ÷ 22 = 1,600
Half the lift needs 4 times the users, because the sample grows with the square of the precision.
- 5% significance, 80% power
So halving the effect quadruples the sample, and a low-powered test that finds nothing proves little. Short traffic calls for more users, a longer test or variance reduction such as CUPED (controlled experiment using pre-experiment data), never a stricter significance level.
Small absolute lifts on low rates need large samples, which is why tiny effects are costly to prove.
With 30% power, a real effect of the planned size is missed 70% of the time: absence of evidence is not evidence of absence. CUPED uses each user's pre-test behaviour to remove noise, and the sample needed falls in line with the variance.
Lowering the significance level to 1% raises the sample needed further.
- Doubling the sample to detect half the effect.
- Reading an underpowered test's null result as proof of no effect.
- Tightening the significance level when traffic is short.
A product manager wants to test a new button colour on a page with 2,000 visitors a week. What would you tell them?
A complete answer covers
- Estimate the smallest lift worth detecting and the sample it needs.
- With so little traffic, the test may take months to detect anything modest.
- Test bigger changes, pool pages, or use variance reduction.
- Or decide without a test if the change is cheap and safe.
Also in our technical interviews, on camera, with a score.
Concept 2 of 71 min
z-statistics, intervals, peeking and many metrics
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 3 of 71 min
Designing tests that can be trusted
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 4 of 71 min
Funnels, revenue per visitor and proxy metrics
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 5 of 71 min
Product metrics: stickiness, retention, lifetime value, outliers
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 6 of 71 min
Correlation, seasonality, survivors and regression to the mean
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 7 of 71 min
AdvancedCUPED: shorter tests from pre-test data
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inRecap
- Users per variant ≈ 16 × p(1 − p) ÷ δ²: half the effect needs 4 times the users; low power proves little.
- z = difference ÷ standard error, 1.96 for 5% two-sided; peeking and many metrics manufacture false positives.
- Before reading results, check the split, novelty, spillover and guardrails; keep a long holdout for long-term effects.
- Funnel steps multiply; judge revenue per visitor, not a rate alone; a proxy pushed hard gets gamed.
- Cohort retention divides by the starting cohort, lifetime value ≈ margin ÷ churn, and medians resist bots.
25 questions on Product analytics and A/B tests
Timed, like the real one, drawn from a bank of 33. Every answer explained.