A/B Testing

Probability · Axiom Academy

How tech companies make data-driven decisions worth millions In 2008, a simple button color change increased Google's ad revenue by 200 million per year . They tested 41 shades of blue to find the winner. This is A/B testing in action. Every day, companies like Amazon, Netflix, and Facebook run thousands of A/B tests to optimize everything from button colors to recommendation algorithms. But how do they know if a change is actually better or just random luck? Imagine you're a product manager at a major e-commerce site: Version B looks better... but is it significantly better? That's where probability comes in. Each visitor to your website is like a coin flip. They either convert (success) or don't (failure). With thousands of visitors, this follows a binomial distribution . Number of conversions out of n visitors with probability p: where X is the number of conversions Just because Version B performed better in your test doesn't guarantee it's actually superior. Random chance can create differences even when the true conversion rates are identical. You run a test with 100 visitors to each version. Version A gets 10 conversions (10%) and Version B gets 15 conversions (15%). Should you roll out Version B to all users? In A/B testing, we calculate a p-value to measure statistical significance. The p-value tells us: "If the two versions were truly identical, what's the probability of seeing a difference this large or larger just by chance?"

This is the written version of the interactive lesson above. See the full Probability course.