A/B testing (also called split testing or randomized controlled experiments) is the gold standard for measuring causal impact of product changes in digital environments. By randomly assigning users to control and treatment groups, teams can isolate the effect of a single feature or variation on key metrics like conversion rate, revenue, or engagement. This methodology underpins data-driven decision-making at scale, enabling companies to ship changes confidently while minimizing risk and maximizing learning velocity. Proper experimental design, statistical rigor, and awareness of common pitfalls — from peeking to spillover effects to long-term metric proxy challenges — are essential for trustworthy results.
What This Cheat Sheet Covers
This topic spans 19 focused tables and 182 indexed concepts, 150 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: Experiment Design Fundamentals
Understanding the core concepts of causal experimentation is the foundation before running any test. These terms define the structure of a randomized controlled experiment and the assumptions that must hold for results to be valid and interpretable.
| Concept | Example | Description | |
|---|---|---|---|
user_id % 100 < 50 | Random assignment of units to control or treatment groups to ensure groups are statistically equivalent and eliminate selection bias | ||
Control vs. Treatment | Randomized experiment comparing exactly two variants to measure causal effect on outcomes | ||
Baseline variant A | • The original experience serving as the comparison baseline • receives no experimental treatment | ||
New variant B | • The modified experience being tested • receives the experimental intervention | ||
Clinical drug trials | • The gold standard for causal inference • randomly assigns subjects to treatment to eliminate confounding | ||
What if user saw B? | • The unobserved outcome under alternative treatment • causal effect is difference between observed and counterfactual | ||
$Y_i(0), Y_i(1)$ | • Rubin Causal Model: each unit has potential outcomes under control $Y(0)$ and treatment $Y(1)$ • observe only one | ||
Control vs. Control | • Sanity check where both groups are identical • validates randomization and detects instrumentation bugs • expect no significant difference | ||
No network effects | • Stable Unit Treatment Value Assumption: treatment of one unit doesn't affect outcomes of other units • violated by social networks | ||
Test 3 headlines × 2 CTAs | • Testing multiple variables simultaneously with factorial design • requires larger sample size than A/B tests | ||
Hash(user_id) → A or B | • Mechanism for allocating units to experimental groups • must be independent of potential outcomes for valid inference |