Score your RAG's retrieval and generation as two separate problems

My RAG system gave a wrong answer, and I can't tell if it retrieved the wrong chunk or the model just ignored a good one.

IntermediateRead 6 minDoing: an afternoon8 steps

What you'll have at the end

A short report naming whether retrieval or generation caused the failure, with a metric score for each.

Before you start

You need

A RAG pipeline where you can call the retriever on its own, separate from generation, plus a short list of real questions your users actually ask it.

Not covered

How to actually fix a broken retriever or a hallucinating prompt once you know which one it is; this only tells you which one to fix.

Leans on

8 steps

Build a golden test set instead of eyeballing prompt changes

start there first if you need more than a couple of test questions before running this diagnosis

Catch a RAG answer hallucinating despite good context

once a question comes back Evidence Override, this walks through a stricter, entailment-based version of the same claim check

Swap a hand-rolled RAG judge for RAGAS or DeepEval's built-in metrics

once you trust the diagnosis, wire both scores into a library metric instead of hand-checking every claim

Checked 18 Aug 2026