A/B test calculator: significance and sample size
Enter the visitors and conversions for each version and find out whether the difference is real or chance, or calculate how many visitors you need before you start.
Confidence level
Version B wins with 98.8% confidence
It is above the 95% you chose: the difference is unlikely to be chance.
Relative uplift of B over A
+16.0%
Likely range of the difference
0.17% to 1.43%
Confidence level: 95%
p-value
0.0123
Calculated with a two-proportion z-test (two-tailed). Do not stop the test as soon as you see a result: set the sample size beforehand and stick to it.
Frequently asked questions about the A/B test calculator
What is an A/B test calculator?
An A/B test calculator is a tool that compares the results of two versions of a page, ad or email and tells you whether the difference between them is real or could just be chance. It also calculates how many visitors you need before launching the test so the result is reliable.
How do I use the A/B test calculator?
In the “Is it significant?” tab, enter the visitors and conversions for version A (the current one) and version B (the new one). The A/B test calculator instantly shows each conversion rate, the uplift of B over A and the confidence level. If confidence is above the level you chose (usually 95%), the difference is statistically significant.
What does statistically significant mean?
It means the difference between A and B is unlikely to be due to chance. With a 95% confidence level, you accept a 5% chance of seeing a difference that does not really exist. It does not mean the uplift is large: a small difference can be significant if you have a lot of traffic.
What is the p-value?
The p-value is the probability of seeing a difference as large as yours, or larger, if A and B actually converted the same. The lower it is, the stronger the evidence that the difference is real. At 95% confidence, the result is significant when the p-value is below 0.05.
How do I calculate the sample size for an A/B test?
In the “Sample size” tab of the A/B test calculator, enter your current conversion rate and the minimum uplift you want to detect (for example, 10% relative: from 5% to 5.5%). The tool calculates how many visitors each version needs. If you add the page’s daily visitors, it also estimates how many days the test should run.
How long should an A/B test run?
Until it reaches the sample size you calculated, in full weeks: behavior changes between weekdays and weekends. Stopping the test as soon as you see a winner is one of the most common mistakes, because checking the data many times increases the chance of finding a false difference.
What can I test with an A/B test?
Headlines, buttons, images, prices, forms, email subject lines or ad copy. Test one change at a time so you know what caused the difference. To track where each campaign’s visitors come from, use our UTM link generator.
Which method does this calculator use?
The A/B test calculator uses a two-tailed two-proportion z-test, the standard method for comparing conversion rates. For sample size it uses the classic formula with the confidence level and statistical power you choose (80% or 90%). With few conversions (fewer than 30 per version) the calculation loses precision and the tool warns you.
Is the A/B test calculator free?
Yes. The A/B test calculator is free, needs no sign-up and runs in your browser: your data is not sent to any server. Find more guides on data-driven decisions in the Data & Analytics section.