Check whether your chatbot survives ten turns

Every single reply in the conversation looks fine on its own, and the user still leaves the chat unresolved.

AdvancedRead 8 minDoing: a couple of hours9 steps

What you'll have at the end

A turn-by-turn score sheet showing exactly where a long conversation went wrong

Before you start

You need

A chatbot you can call turn by turn through code, and enough comfort writing a one-off judge prompt to grade a single reply against a stated fact.

Not covered

Fixing the prompt or memory bug the score sheet turns up, or wiring this check into the automated gate that blocks a bad prompt edit before it ships.

Leans on

9 steps

Write your first LLM-as-a-judge grading script

start there first if you have never written a judge prompt before; this recipe assumes you can write one for the context-lost check.

Your chatbot nails one phrasing of a question and fumbles a paraphrase

go there instead when the failure is one wrong reply to one rephrased question, not a thread that unravels over many turns.

Wire a regression check that runs before every prompt edit ships

go there once this score sheet finds a real break and you want it checked automatically on every future prompt edit.

Checked 15 Aug 2026