Make your RAG system say it does not know instead of guessing

When nothing relevant turns up, my RAG system still makes something up instead of saying so.

Beginner to intermediateRead 5 minDoing: an hour or two, once you already have a RAG pipeline that retrieves and answers from your own documents6 steps

▶ Start

What you'll have at the end

A RAG prompt that replies with a fixed no-match message on 5 questions your documents don't cover

Before you start

You need

A RAG pipeline that already turns a question into a vector, scores it against your stored chunks, and sends the top matches to a chat model to answer from.

Not covered

Improving what retrieval finds in the first place, such as better chunking, hybrid search, or reranking; this only covers what happens once retrieval already ran and still came back empty or off-topic.

Leans on

Find where real matches and empty ones actually split

Collect questions with and without answers
Score every calibration question for real
Set the floor inside the real gap

Make the pipeline act on that number

Block the model call below that floor
Give the model permission to say no

Confirm it catches what it's built to catch

Confirm the gate on every test question

Where to go next

Add a reranker so RAG stops burying the right chunk

when a chunk that really answers the question keeps landing just under your floor, because raw similarity is a weak proxy for true relevance

Your RAG's retrieval hit rate has never been measured

when you want ongoing proof that a floor you set once isn't quietly blocking questions your documents can actually answer

Checked 15 Aug 2026

Part of the RAG (Retrieval Augmented Generation) cookbook