New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
One version of my prompt looked better on ten test runs, and I have no idea if that was real or luck.
What you'll have at the end
A written verdict on whether the new prompt beat the old one, with the number behind it
You need
A prompt-based feature already live in front of real users, a challenger version of that prompt you want to test, and a way to route real traffic into two buckets and log a single win-condition outcome for each conversation.
Not covered
This does not cover offline evaluation against a fixed test set before anything reaches real users, only the statistics once both prompt versions are already live and taking real traffic.
Leans on
Build a golden test set instead of eyeballing prompt changes
for the offline question-and-answer file to check a challenger prompt against before you ever spend real production traffic testing it.
Wire a regression check that runs before every prompt edit ships
for a fast go/no-go check right after editing a prompt, before committing to the slower online test this recipe runs.
Checked 15 Aug 2026
Part of the AI-LLM App Evaluation cookbook