Does your fully automated LLM judge agree with a human reviewer?

My automated judge says every answer passes, but a teammate reading the same batch disagrees on a third of them.

IntermediateRead 6 minDoing: an afternoon, mostly the human review pass6 steps

What you'll have at the end

A written comparison of your judge's verdicts against a human reviewer's blind re-grade of a stratified sample: a confusion matrix, a raw agreement percentage, a Cohen's kappa score, and every mismatched answer listed with the human's stated reason.

Before you start

You need

An automated judge script that already grades a batch of saved answers pass or fail against a written rubric, with each verdict saved alongside its answer.

Not covered

Writing or tuning the judge script's rubric and prompt from scratch, and recalibrating it once a problem is found: this stops at measuring whether the current judge can be trusted, and naming exactly where it can't.

Leans on

6 steps

Where to go next

Write your first LLM-as-a-judge grading script

you don't have a working pass/fail judge script yet: write one first, before checking whether it agrees with a human.

Checked 15 Aug 2026

Part of the AI-LLM App Evaluation cookbook