New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
Same question, reworded. One time correct, one time wrong.
What you'll have at the end
A list of every question where a reworded version got a different answer
You need
A set of test questions with a written correct answer beside each one, and a working way to send any single question to your chatbot and capture its full raw answer.
Not covered
Whether a retrieval step returns different source chunks for different phrasings is a separate check; this treats your chatbot as one black box and only checks its final answer.
Leans on
Build a golden test set instead of eyeballing prompt changes
start there first if you don't yet have a written set of questions with a correct answer beside each one to run this check against
Write your first LLM-as-a-judge grading script
go there for a fuller grading script if comparing a raw answer to the golden answer by eye stops being practical at your batch size
Checked 15 Aug 2026
Part of the AI-LLM App Evaluation cookbook