A/B test sample size: can your traffic actually run this test?
Most stores don't have the traffic to detect a small lift. Enter your conversion rate, the smallest lift you care about and your daily visitors, and you get visitors per variant and days to run, and whether your traffic can finish the test at all.
At a 2.5% conversion rate, 95% significance and 80% power, a 10% relative lift takes about 64,200 visitors per variant to detect. A 20% lift takes about 16,800 and a 5% lift about 250,800. The practical floor is roughly 100 or more conversions per variant over at least two full weeks.
- n per variant = (z(1−α/2) × √(2p̄(1−p̄)) + z(1−β) × √(p₁(1−p₁) + p₂(1−p₂)))² ÷ (p₂ − p₁)².
- p₁ is the baseline rate, p₂ the rate after the lift, p̄ their average.
- Days to run = 2n ÷ daily visitors in the test.
How the number is calculated
This is the standard two-proportion sample size for a two-sided test with a 50/50 split. It reproduces the sessions-per-variant table in my product page post: 64,200 at a 2.5% baseline and a 10% lift. Days to run divides both variants' visitors by your daily traffic.
With low traffic, test big swings and ship small fixes on judgement. When the test is done, read it with the significance calculator. Every calculator lives on the free tools hub.