New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
Support keeps forwarding the same three complaints and I never turn them into anything I can test against.
What you'll have at the end
A regression file holding real, previously failed cases pulled straight from actual complaints, each one carrying the exact broken reply, a checked correct answer, and a rule for telling a future rerun pass from fail.
You need
A live LLM-powered app already used by real people, a channel where those people already report a bad reply (a support ticket queue, a feedback button, a Slack channel), and a way to look up the conversation or trace behind a given report.
Not covered
This does not cover wiring these cases into an automatic check that blocks a deploy, or building the script that grades them at scale, only building the trusted file of real failures those tasks would run against.
Leans on
Build a golden test set instead of eyeballing prompt changes
for building a broader test set of everyday questions across failure types, not only the ones users already complained about
Wire a regression check that runs before every prompt edit ships
for turning this saved file into an automatic check that blocks a prompt change before it ships
Checked 15 Aug 2026
Part of the AI-LLM App Evaluation cookbook