How A/B testing works — and the three ways it lies

An A/B test is the closest thing marketing has to a controlled experiment: show half your visitors one version and half another, count what each half does, compare. The arithmetic is a century old and a calculator does it in a millisecond — this one, for instance. What goes wrong is not the arithmetic but the procedure around it, and this guide is about the procedure.

The setup: random split, one change, one metric

Visitors are assigned to A (the control) or B (the variant) at random, so the two groups differ only in the change being tested. One change at a time, or you won't know which change did it. One primary metric decided in advance — conversion rate, sign-ups, revenue per visitor — because if you look at ten metrics, one will look significant by chance. Both versions run over the same period, so weekday and campaign effects hit both equally. Break any of these and the numbers that come out are just numbers.

What the p-value actually says

Suppose A converts 4.0% of 5,000 visitors and B 4.8% of 5,000. The question is not "is B better?" but "if A and B were truly identical, how often would random variation produce a gap this big?" A two-proportion z-test answers it: the p-value is that probability. p = 0.03 means a gap this large would arise by chance 3% of the time under no true difference; by convention, under 5% is called significant. Note what it does not say: it is not "a 97% chance B is better", and it says nothing about how much better. The relative lift (+20% here) and its uncertainty are separate numbers.

Sample size, and why most small tests are inconclusive

Detecting a small difference on a low conversion rate needs a lot of data. To reliably spot a 10% relative lift (4.0% → 4.4%) at 95% confidence with 80% power — the standard settings — each variant needs roughly 39,000 visitors. A 5% lift needs about four times that. A site with 500 visitors a day would run that test for five months. This is why the honest result of most small-site tests is "we can't tell", and why the useful ones test big changes (a new page, a different offer) rather than button colours. Compute the size before starting; How to calculate percentages covers the relative-lift arithmetic.

Peeking: the most common way tests lie

Here is the trap. You check the test daily, and stop the day p drops below 0.05. The p-value fluctuates as data arrives, and if you keep checking, it will dip below 0.05 at some point in a large fraction of tests where nothing is different — stopping at that moment turns a 5% false-positive rate into 20–30%. Optional stopping is the same error as flipping a coin until heads and declaring it biased. The cure is procedural: fix the sample size in advance, don't stop early on a good number, and treat the final reading as the reading. (Sequential testing methods exist that allow peeking with adjusted thresholds; the calculators most people use are not those.)

Reading the confidence interval

A significant result with a 95% interval on the lift of +2% to +38% says: B is probably better, and we have almost no idea by how much. That is a real but weak finding. An interval of +14% to +26% is a strong one. An interval that crosses zero (−3% to +21%) means B could be worse, whatever the point estimate says. Ship on significance, but plan on the low end of the interval — the lift you'll actually see over the following months regresses toward it.

A practical protocol

1. Write down the hypothesis, the single metric and the minimum lift worth detecting.
2. Compute the sample size per variant and the expected duration; if it's months, test a bigger change.
3. Run to the sample size across whole weeks. Don't peek for decisions.
4. Read p-value, lift and interval together. Ship if significant and the interval's low end is still worth having.
5. Log the result either way — negative and inconclusive results are how you learn what your users don't care about.

The ad cost calculator turns a lift into money; the percentage calculator handles the rest.