The A/B Test Is Not the Truth Machine You Think

An A/B test feels like proof. You shipped B, the numbers went up, the test said significant, so B is better. That chain has a broken link in almost every step, and the tool is trusted enough that nobody checks. A/B testing is the most rigorous method most product teams own. It is also the one they misread most often, because the maths is counterintuitive and the dashboard hides that from you on purpose.

Start with the number everyone reads wrong. A p value does not tell you the chance that your idea works. It answers a narrower, stranger question. If there were truly no difference between A and B, how often would random noise alone produce a gap at least this big? That is all. A p value of 0.05 means such a gap would show up one time in twenty by pure chance. It is a statement about a world where you are wrong, not a measure of how likely you are to be right.

P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

The professional body of statisticians felt the need to put that in writing in 2016, which tells you how common the misreading is. Think of a smoke alarm. It is tuned to go off rarely when there is no fire. Hearing it does not tell you how big the fire is, or even that there is one. It tells you the reading crossed a line you set. A p value is a smoke alarm for noise. It was never a measure of importance, and treating it as one is where most of the trouble begins.

Most tests cannot see what you are hoping for

Before you run a test, it has a property called power. Power is the chance the test will notice a real effect if one is truly there. A test with 50 percent power is a coin flip. Even when your change genuinely helps, half the time the test shrugs and reports nothing. Underpowered tests are not the exception in product work. They are the norm, and the reason is simple arithmetic that teams skip.

Small effects need enormous samples. Detecting a change from a 5 percent conversion rate to 5.25 percent, a relative lift of 5 percent, takes on the order of a hundred thousand users per variant to see reliably. Most teams do not have that traffic on most surfaces. So they run the test for two weeks, get a few thousand users, and read the flat result as proof the change did nothing. It is not proof. The test was never able to detect an effect that size. Absence of evidence got reported as evidence of absence.

Peeking, and why it wrecks the result

Here is the mistake that undoes more tests than any other. You check the results every morning. The day it crosses significance, you stop and declare victory. It feels responsible. It is statistically ruinous. The p value assumes you fixed the sample size in advance and looked once. Every extra peek is another roll of the dice, another chance for noise to wander across the line, and the reported significance stops meaning what it says.

Evan Miller laid this out with a number that should be printed on every dashboard. Take a test where the truth is that nothing changed. Check after every observation and stop the moment you see 5 percent significance, or give up at 150 observations. You would expect to be fooled about 5 percent of the time. The real rate is 26.1 percent. More than five times what the label promised, purely from looking too often.

There are two honest ways out. The first is the fixed horizon test. Decide the sample size up front, do not look at significance until you reach it, then read it once. Dull, and it works. The second is sequential testing, a family of methods built to let you look continuously while keeping the error rate honest, by raising the bar as you peek. Both are real fixes. What does not work is the default habit of watching a live graph and stopping when it pleases you.

The same trap has a wider mouth. Test twenty variants at once, or slice one result by twenty segments after the fact, and you have run twenty tests. At 5 percent each, the chance that at least one throws a false positive is well over 60 percent. So you find that the button helped left handed users in Belgium on Tuesdays, and you believe it. This is the multiple comparisons problem, and after the fact slicing is its most seductive form, because the story you build around the fluke always sounds plausible.

Significant is not important

Now the argument worth having. A result can be statistically significant and commercially meaningless, and our whole vocabulary pushes us to confuse the two. Significance is about confidence that an effect is not zero. Importance is about whether the effect is big enough to matter. With enough traffic, a 0.2 percent lift will clear significance easily. It is still 0.2 percent. Teams ship it, put it in the deck, and celebrate, because significant sounds like important. It is not the same word.

Even the companies that are best at this find that most ideas fail. That is not a knock on them, it is the point. Kohavi and Thomke, who built and ran experimentation at Microsoft, reported the honest numbers in Harvard Business Review. At Google and Bing, only 10 to 20 percent of experiments produce a positive result. Across Microsoft, roughly a third help, a third do nothing, and a third actively hurt. If your win rate looks a lot higher than that, the likeliest explanation is not that your team is brilliant. It is that you are peeking, slicing, or shipping noise.

The hill you are standing on

A/B testing has a shape problem too. It only ever compares nearby options. You test a blue button against a green one, a shorter form against a longer one, and you climb, step by step, toward the top of the hill you happen to be standing on. It will never tell you there is a taller hill across the valley, because getting there means walking downhill first, and every step down looks like a losing test. This is the local maximum, and pure optimisation cannot escape it.

Two effects distort early readings on top of that. Novelty makes a new thing test well simply because it is new, and people click anything different. Primacy runs the other way, where a change tests badly at first because regular users are thrown by it, then recover once they learn it. Both fade. A test that runs a few days can catch the spike or the dip and miss the truth entirely. Longer runs and returning user cohorts are how you tell a real effect from a first impression.

So know when not to test. Small samples where you will never reach power. Strategic bets that reshape the product, where the honest tool is judgement, not a two week readout. Brand, which compounds over years and cannot be seen in a conversion window. And accessibility, which is the clearest case of all. Whether a blind user can complete a purchase is not an optimisation question with a winning variant. It is a floor. You do not A/B test whether to have a wheelchair ramp.

None of this is a reason to test less. It is a reason to test with your eyes open, and the best account of how to do that comes from the people who ran these systems at scale.

On Monday, do four things. Calculate power before you run anything, and refuse tests that cannot reach it. Fix your sample size or adopt a sequential method, and stop peeking at a naive test. Decide the smallest effect worth shipping before you look, so significance cannot pose as importance. And keep a short list of the questions you will not hand to a test at all. The tool is genuinely excellent. It just is not the truth machine the dashboard pretends it is.