Add execution-based tests instead of reading your model's generated code

Reading the code my model wrote, it looks fine, and then it crashes the first time somebody feeds it a real input.

Beginner to intermediateRead 5 minDoing: an hour or two, mostly writing test cases and rerunning the harness as you widen it6 steps

▶ Start

What you'll have at the end

A test harness that runs generated code against real inputs and prints a pass rate.

Before you start

You need

A function your model already generated for some task, plus a handful of real inputs for that function where you can say what the correct output should be.

Not covered

This does not cover code style, security scanning, or how fast the generated code runs, only whether it returns the right answer on inputs you can check by hand.

Leans on

6 steps

Where to go next

Write your first LLM-as-a-judge grading script

for output with no single correct answer to assert against, only a judgment call a person or another model has to make

Wire a regression check that runs before every prompt edit ships

for turning this one-off harness into a check that runs automatically on every future change instead of by hand

Checked 18 Aug 2026

Part of the AI-LLM App Evaluation cookbook