A/B testing (also called split testing or randomized controlled experiments) is the gold standard for measuring causal impact of product changes in digital environments. By randomly assigning users to control and treatment groups, teams can isolate the effect of a single feature or variation on key metrics like conversion rate, revenue, or engagement. This methodology underpins data-driven decision-making at scale, enabling companies to ship changes confidently while minimizing risk and maximizing learning velocity. Proper experimental design, statistical rigor, and awareness of common pitfalls are essential for trustworthy results.
What This Cheat Sheet Covers
This topic spans 19 focused tables and 169 indexed concepts. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
Experiment Design Fundamentals
| Concept | Example | Description |
|---|---|---|
user_id % 100 < 50 | Random assignment of units to control or treatment groups to ensure groups are statistically equivalent and eliminate selection bias | |
Baseline variant A | • The original experience serving as the comparison baseline • receives no experimental treatment | |
New variant B | • The modified experience being tested • receives the experimental intervention | |
Control vs. Treatment | Randomized experiment comparing exactly two variants to measure causal effect on outcomes | |
Test 3 headlines × 2 CTAs | • Testing multiple variables simultaneously with factorial design • requires larger sample size than A/B tests | |
No network effects | • Stable Unit Treatment Value Assumption: treatment of one unit doesn't affect outcomes of other units • violated by social networks |