AI-LLM App Evaluation cookbook

15 recipes · 105 steps · about 91 minutes of reading

Build a golden test set instead of eyeballing prompt changes
You keep saying a prompt looks better without ever checking it against the same real questions twice.
BeginnerRead 5 minDoing: two to three hours, mostly gathering questions and writing answers6 steps
Write your first LLM-as-a-judge grading script
I have fifty test answers and no time to read every single one by hand.
Beginner to intermediateRead 5 minDoing: about an hour6 steps
Catch a RAG answer hallucinating despite good context
My RAG system retrieves the right paragraph and still tells the user something the document never said.
Beginner to intermediateRead 5 minDoing: about an hour6 steps
Swap a hand-rolled RAG judge for RAGAS or DeepEval's built-in metrics
I wrote my own faithfulness check by hand, and now every new metric I want means another script to write and debug.
Beginner to intermediateRead 6 minDoing: a couple of hours6 steps
Add execution-based tests instead of reading your model's generated code
Reading the code my model wrote, it looks fine, and then it crashes the first time somebody feeds it a real input.
Beginner to intermediateRead 5 minDoing: an hour or two, mostly writing test cases and rerunning the harness as you widen it6 steps
Does your fully automated LLM judge agree with a human reviewer?
My automated judge says every answer passes, but a teammate reading the same batch disagrees on a third of them.
IntermediateRead 6 minDoing: an afternoon, mostly the human review pass6 steps
A/B testing a prompt change in production without fooling yourself with random noise
One version of my prompt looked better on ten test runs, and I have no idea if that was real or luck.
IntermediateRead 6 minDoing: half a day to set it up, then however long real traffic takes to reach the number from the sample-size step7 steps
Turn user complaints into regression tests
Support keeps forwarding the same three complaints and I never turn them into anything I can test against.
IntermediateRead 5 minDoing: two to three hours, most of it pulling traces and writing corrected answers7 steps
Score your RAG's retrieval and generation as two separate problems
My RAG system gave a wrong answer, and I can't tell if it retrieved the wrong chunk or the model just ignored a good one.
IntermediateRead 6 minDoing: an afternoon8 steps
Wire a regression check that runs before every prompt edit ships
How do you know your last prompt fix didn't just break something else?
Intermediate to advancedRead 6 minDoing: half a day, most of it writing the check functions and wiring the merge step8 steps
Judge your agent's tool calls, not its final answer alone
My agent got the right answer, but it called the search tool four times when one call was enough.
Intermediate to advancedRead 7 minDoing: a couple of hours7 steps
Measure your RAG's noise sensitivity before you raise top-k
Retrieval recall went up when I widened top-k, and the RAG answers came out worse instead of better.
Intermediate to advancedRead 7 minDoing: an afternoon8 steps
Trace a failing eval score back to the exact prompt version that produced it
Your eval dashboard shows a score drop, and you have no way to tell which prompt change caused it.
Intermediate to advancedRead 8 minDoing: half a day, mostly wiring the version stamp once and then walking one flagged case through its trace end to end8 steps
Your chatbot nails one phrasing of a question and fumbles a paraphrase
Same question, reworded. One time correct, one time wrong.
AdvancedRead 6 minDoing: an afternoon7 steps
Check whether your chatbot survives ten turns
Every single reply in the conversation looks fine on its own, and the user still leaves the chat unresolved.
AdvancedRead 8 minDoing: a couple of hours9 steps

Cookbook badge

Finish all 15 recipes to earn this.

AI-LLM App Evaluation cookbook

Build a golden test set instead of eyeballing prompt changes

Write your first LLM-as-a-judge grading script

Catch a RAG answer hallucinating despite good context

Swap a hand-rolled RAG judge for RAGAS or DeepEval's built-in metrics

Add execution-based tests instead of reading your model's generated code

Does your fully automated LLM judge agree with a human reviewer?

A/B testing a prompt change in production without fooling yourself with random noise

Turn user complaints into regression tests

Score your RAG's retrieval and generation as two separate problems

Wire a regression check that runs before every prompt edit ships

Judge your agent's tool calls, not its final answer alone

Measure your RAG's noise sensitivity before you raise top-k

Trace a failing eval score back to the exact prompt version that produced it

Your chatbot nails one phrasing of a question and fumbles a paraphrase

Check whether your chatbot survives ten turns

In these roadmaps

AI-LLM App Evaluation cookbook

Build a golden test set instead of eyeballing prompt changes

Write your first LLM-as-a-judge grading script

Catch a RAG answer hallucinating despite good context

Swap a hand-rolled RAG judge for RAGAS or DeepEval's built-in metrics

Add execution-based tests instead of reading your model's generated code

Does your fully automated LLM judge agree with a human reviewer?

A/B testing a prompt change in production without fooling yourself with random noise

Turn user complaints into regression tests

Score your RAG's retrieval and generation as two separate problems

Wire a regression check that runs before every prompt edit ships

Judge your agent's tool calls, not its final answer alone

Measure your RAG's noise sensitivity before you raise top-k

Trace a failing eval score back to the exact prompt version that produced it

Your chatbot nails one phrasing of a question and fumbles a paraphrase

Check whether your chatbot survives ten turns

In these roadmaps