AI Systems Atlas

E-03LLM-as-Judge

/evals/llm-as-judge

A model grades outputs against a rubric, standing in for human review at scale.

Step-by-step walkthrough2 steps
  1. A model grades another model’s work against an explicit rubric.
  2. Scales to thousands of outputs per hour — but the judge is a model too, with its own biases.
Key insight

Judges make evaluation continuous instead of quarterly. The rubric — not the judge model — is usually what needs the engineering.

Failure mode

Uncalibrated judges drift: position bias, length bias, self-preference. A judge you haven’t measured against humans is a random number generator with confidence.

Further reading
RelatedHuman calibrationGenerator–Critic

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →Tools →Ops →