A/B test significance calculator

This is a two-proportion z-test, two-sided. First the conversion rates and the pooled rate:

Then the z-score, the gap divided by its standard error:

The result is significant at 95% when p is below 0.05 (|z| above 1.960), at 90% below 0.10 (1.645) and at 99% below 0.01 (2.576). Relative uplift is (pB − pA) / pA.

Example: A has 200 conversions from 4,000 visitors (5.00%), B has 248 from 4,000 (6.20%). The pooled rate is 5.6%, z = 2.334 and p = 0.0196: significant at 90% and 95%, not at 99%. B's uplift is +24%.

Pick the sample size first. Small uplifts need a lot of traffic. Decide how many visitors each version gets before you start, then read the result once.

Split traffic at random. Send visitors to A or B by chance, at the same time, so weekdays, campaigns and seasons hit both.

Change one thing. If B has a new headline and a new button, you won't know which one moved the number.

What does the p-value mean here?

If A and B truly converted at the same rate, the p-value is the chance of seeing a gap at least this big by luck. A p-value of 0.02 means a 2% chance. It is not the chance that B is better.

Can I stop the test as soon as it turns significant?

Better not. Checking every day and stopping at the first significant result makes false wins much more likely. Decide the sample size before you start and read the result once you reach it.

Is this one-sided or two-sided?

Two-sided. It asks whether B differs from A in either direction, which is the safer default when B could also be worse.