Free calculator · instant

A/B test significance: is the lift real?

Enter visitors and conversions for each variant and get the lift, the p-value and confidence from a two-proportion z-test, and a warning when there aren't enough conversions to trust it.

5 inputsInstantNo signup
Go deeper with the same numbers
When is an A/B test result significant?

When the p-value from a two-proportion test falls below your threshold, usually 0.05 for 95% significance, after the sample size you planned in advance. Get there with fewer than about 100 conversions per variant, or by stopping early on a good day, and the result is unreliable even if the number says otherwise.

  • Conversion rate = conversions ÷ visitors, per variant.
  • z = (p_B − p_A) ÷ √(p̄(1 − p̄)(1/n_A + 1/n_B)), with p̄ the pooled rate.
  • p-value = 2 × (1 − Φ(|z|)); confidence = 1 − p.
Worked example. Control: 525 of 21,000 (2.50%). Variant: 600 of 21,000 (2.86%). That is a 14.3% lift with z ≈ 2.27 and p ≈ 0.023, significant at 95%.
Source: Taylor Sicard, Taylor Sicard Consulting · Benchmarks reviewed July 2026 · Method and benchmarks: AOV vs conversion rate: running the test
Method

How the number is calculated

The calculator runs a two-sided two-proportion z-test on the pooled conversion rate, then converts z to a p-value. Confidence is 1 minus the p-value. My own reading rule, from the posts linked here, adds two conditions the maths doesn't check: at least 100 conversions per variant, and a run length fixed before the test started.

p below threshold, 100+ conversions eachA real difference. Confirm on revenue per visitor before shipping.
p below threshold, under 100 conversionsToo early. Small samples throw up false winners.
p above thresholdNo detectable difference, so don't count it as a small win.

A conversion win that lowers AOV can still lose money, so check the result in the AOV calculator as revenue per visitor. Plan the next one in the sample size calculator. Every calculator lives on the free tools hub.

Questions

Common questions

How do you know if an A/B test is statistically significant?
Run a two-proportion z-test on the two conversion rates. If the p-value is below 0.05 you have 95% significance, provided you reached the sample size you planned and didn't stop early.
What does 95% confidence mean in A/B testing?
That if there were no real difference, a result this large would turn up less than 5% of the time by chance. It is not a 95% chance that B is better.
How many conversions do I need per variant?
At least about 100 per variant, and the full sample from your sample size calculation. Below that, random swings produce winners that vanish when you ship them.
Can I stop a test as soon as it hits significance?
No. Checking daily and stopping on the first significant day inflates false positives badly. Fix the sample size or run length before you start and read it once.