What you'll have at the end

A script that grades a batch of saved answers pass or fail and writes out a spreadsheet

Before you start

You need

A batch of saved question-and-answer pairs sitting in one file: each row needs the question, a reference or expected answer, and the answer your system actually produced.

Not covered

Measuring how often this judge's verdicts actually match a human reviewer's, with a real agreement number, is a separate, deeper check.

Leans on

AI-LLM App Evaluation Cheat SheetLists the LLM-as-a-judge techniques this script's binary, reasoning-first design is one instance of.
LLM Evaluation Cheat SheetCovers the rubric-based and pointwise judging methods the rubric in this recipe follows.
Structured Output Generation with LLMs Cheat SheetCovers the JSON field-ordering rule the judge prompt's reasoning-before-verdict shape follows.

Where to go next

Build a golden test set instead of eyeballing prompt changes

start there first if you don't yet have a saved batch of questions with reference answers to grade

Does your fully automated LLM judge agree with a human reviewer?

go there once you want a measured agreement rate against a human reviewer instead of a hand spot-check

Write your first LLM-as-a-judge grading script

Before you start

Where to go next

Write your first LLM-as-a-judge grading script

Before you start

Where to go next