AI Systems Atlas

E-04Human-calibrated Judge

/evals/human-calibrated-judge

Label a gold subset with humans, then iterate the judge prompt until it agrees with them.

Step-by-step walkthrough5 steps
  1. Humans label a gold subset once.
  2. Judge v1 grades the same items: only 62% agreement.
  3. Every disagreement is a bug report against the judge prompt.
  4. Revise, re-run: 81%.
  5. 91% — now the judge can scale where humans can’t. Recalibrate when the task drifts.
Key insight

Treat judge–human disagreement as a bug report against the judge prompt. Agreement is a metric you climb, then monitor.

Failure mode

Calibrating once and trusting forever — task drift silently decouples the judge from what humans would say now.

Further reading
RelatedLLM-as-judgeOutcome vs process

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →Tools →Ops →