New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
The new button design is 'winning' after two days. Is that real, or would it flip back tomorrow?
What you'll have at the end
A go/no-go call on the test, backed by the planned sample size and a significance check instead of a lucky-looking morning's numbers
You need
A live A/B test with a visitor and conversion count you can pull for both variants right now, plus the sample size target you set before the test launched.
Not covered
How big a sample the test needed before it launched, that calculation happens earlier and this recipe assumes it already ran.
Leans on
Work out how big a sample an A/B test needs
for the pre-launch sample-size math this recipe assumes already happened.
An A/B test isn't significant, and the team still needs a call
once the full planned sample is in and the honest answer is a flat, not-significant result rather than an early apparent win.
Compare two groups' averages with a t-test
when the metric under test is a continuous number like order value rather than a conversion rate.
Checked 26 Aug 2026
Part of the Data Analytics cookbook