A/B test significance: is the lift real?
Enter visitors and conversions for each variant and get the lift, the p-value and confidence from a two-proportion z-test, and a warning when there aren't enough conversions to trust it.
When the p-value from a two-proportion test falls below your threshold, usually 0.05 for 95% significance, after the sample size you planned in advance. Get there with fewer than about 100 conversions per variant, or by stopping early on a good day, and the result is unreliable even if the number says otherwise.
- Conversion rate = conversions ÷ visitors, per variant.
- z = (p_B − p_A) ÷ √(p̄(1 − p̄)(1/n_A + 1/n_B)), with p̄ the pooled rate.
- p-value = 2 × (1 − Φ(|z|)); confidence = 1 − p.
How the number is calculated
The calculator runs a two-sided two-proportion z-test on the pooled conversion rate, then converts z to a p-value. Confidence is 1 minus the p-value. My own reading rule, from the posts linked here, adds two conditions the maths doesn't check: at least 100 conversions per variant, and a run length fixed before the test started.
A conversion win that lowers AOV can still lose money, so check the result in the AOV calculator as revenue per visitor. Plan the next one in the sample size calculator. Every calculator lives on the free tools hub.