Trace a failing eval score back to the exact prompt version that produced it

Your eval dashboard shows a score drop, and you have no way to tell which prompt change caused it.

Intermediate to advancedRead 8 minDoing: half a day, mostly wiring the version stamp once and then walking one flagged case through its trace end to end8 steps

▶ Start

What you'll have at the end

A trace linking one low eval score to the exact prompt version and model call that produced it.

Before you start

You need

A nightly or periodic eval run that already writes a score per case to a dashboard, and a way to send that same case's exact input through your app again on demand.

Not covered

This does not cover fixing or rolling back the prompt once you have found it, only proving which version and which call produced the failing score.

Leans on

8 steps

Where to go next

Build a golden test set instead of eyeballing prompt changes

for building the per-case golden set and pass rule this recipe assumes already exists, before there is a score to trace

Wire a regression check that runs before every prompt edit ships

for turning the exact case you just found into a gate that blocks the same regression from shipping quietly again

Checked 18 Aug 2026

Part of the AI-LLM App Evaluation cookbook