Wire a regression check that runs before every prompt edit ships

How do you know your last prompt fix didn't just break something else?

Intermediate to advancedRead 6 minDoing: half a day, most of it writing the check functions and wiring the merge step8 steps

▶ Start

What you'll have at the end

A script that reruns your test set after every prompt edit and prints which cases dropped

Before you start

You need

A saved file of test cases, each pairing an input with a plain pass or fail rule stating what the reply must and must not say, plus a live prompt or app you can actually run each input against.

Not covered

This does not cover deciding whether a case's verdict should come from a keyword rule or an LLM judge, only turning verdicts that already exist into an automatic gate that blocks a change.

Leans on

8 steps

Where to go next

Turn user complaints into regression tests

for building the case file of real failures and pass rules this check runs, before there is anything to wire up

Build a golden test set instead of eyeballing prompt changes

for a broader everyday-question case file instead of only complaint-sourced cases

Write your first LLM-as-a-judge grading script

for grading a case whose verdict needs real judgment instead of a plain keyword rule

Checked 15 Aug 2026

Part of the AI-LLM App Evaluation cookbook