A/B testing (also called split testing or randomized controlled experiments) is the gold standard for measuring causal impact of product changes in digital environments. By randomly assigning users to control and treatment groups, teams can isolate the effect of a single feature or variation on key metrics like conversion rate, revenue, or engagement. This methodology underpins data-driven decision-making at scale, enabling companies to ship changes confidently while minimizing risk and maximizing learning velocity. Proper experimental design, statistical rigor, and awareness of common pitfalls — from peeking to spillover effects to long-term metric proxy challenges — are essential for trustworthy results.
What This Cheat Sheet Covers
This topic spans 19 focused tables and 182 indexed concepts. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
Table 1: Experiment Design Fundamentals
Understanding the core concepts of causal experimentation is the foundation before running any test. These terms define the structure of a randomized controlled experiment and the assumptions that must hold for results to be valid and interpretable.
| Concept | Example | Description |
|---|---|---|
user_id % 100 < 50 | Random assignment of units to control or treatment groups to ensure groups are statistically equivalent and eliminate selection bias | |
Control vs. Treatment | Randomized experiment comparing exactly two variants to measure causal effect on outcomes | |
Baseline variant A | • The original experience serving as the comparison baseline • receives no experimental treatment | |
New variant B | • The modified experience being tested • receives the experimental intervention | |
Clinical drug trials | • The gold standard for causal inference • randomly assigns subjects to treatment to eliminate confounding | |
What if user saw B? | • The unobserved outcome under alternative treatment • causal effect is difference between observed and counterfactual |