Agent Evals

You cannot improve what you cannot measure, and agents are miserable to measure: outcomes are sparse, trajectories are long, and “correct” is often a judgment call. This collection maps the evaluation stack from binary outcome checks to calibrated model judges.

data flowcontrol flowactive executionuntrusted / riskverified / allowedpermission gatedecisionretry / iterationparallelinactive branch

E-01Outcome Eval

/evals/outcome-eval

Run the task end to end; check only the final result.

Key insight

Ground truth where you can get it — a passing test, a correct SQL result, a booked meeting — is worth a thousand vibes.

Failure mode

Outcome-only metrics reward lucky trajectories and hide near-misses; they tell you that it failed, never why.

E-02Process / Trajectory Eval

/evals/trajectory-eval

Inspect every step of the run, not just where it landed.

Key insight

Trajectory review is where you catch reward hacking, wasted loops, and the cheats that outcome metrics score as wins.

Failure mode

Judging steps against “what I would have done” instead of “did this step make progress” — many valid trajectories look nothing like yours.

E-03LLM-as-Judge

/evals/llm-as-judge

A model grades outputs against a rubric, standing in for human review at scale.

Key insight

Judges make evaluation continuous instead of quarterly. The rubric — not the judge model — is usually what needs the engineering.

Failure mode

Uncalibrated judges drift: position bias, length bias, self-preference. A judge you haven’t measured against humans is a random number generator with confidence.

E-04Human-calibrated Judge

/evals/human-calibrated-judge

Label a gold subset with humans, then iterate the judge prompt until it agrees with them.

Key insight

Treat judge–human disagreement as a bug report against the judge prompt. Agreement is a metric you climb, then monitor.

Failure mode

Calibrating once and trusting forever — task drift silently decouples the judge from what humans would say now.

E-05Regression Evals

/evals/regression-evals

Run version A and version B over the same benchmark set; look at per-task deltas, not just the mean.

Key insight

Averages hide regressions. A change that lifts the mean 9 points and silently breaks one capability is how agents get worse while dashboards get greener.

Failure mode

A benchmark set too small or too easy to move: every change looks neutral, so every change ships.

E-06Failure Taxonomy

/evals/failure-taxonomy

Classify every failure by where in the loop it originated — perception, reasoning, planning, tool use, recovery, verification, or the final answer.

Key insight

The distribution tells you what to fix next. A recovery-heavy histogram wants a better retry loop, not a bigger model.

Failure mode

Buckets so broad (“reasoning”) that every failure fits and none suggests an intervention.

E-07Outcome vs Process Mismatch

/evals/outcome-vs-process

An agent can succeed through a bad trajectory, and fail despite a mostly sound one.

Key insight

Score both axes. Lucky passes are future failures you haven’t met yet; principled near-misses are one small fix from reliable wins.

Failure mode

Optimizing pass-rate alone quietly selects for whatever cheats and shortcuts happen to pass — you train luck.

E-08Adversarial Evals

/evals/adversarial-evals

Mix attack tasks into the benchmark and score refusals as wins.

Key insight

Capability and resistance are different axes. An agent you’ve only measured on cooperative tasks is unmeasured where it matters.

Failure mode

Attack tasks the agent can fingerprint: if adversarial items look different, you’re measuring detection, not resistance.

E-09Cross-Model Evals

/evals/cross-model-evals

Same tasks, same harness, models swapped — so differences are attributable.

Key insight

This is how you separate “the model got smarter” from “the harness got better”, and how you know when a new model release should change your stack.

Failure mode

Prompts silently tuned to one model’s quirks: the comparison measures prompt fit, not model capability.

E-10Offline vs Online Evals

/evals/offline-vs-online

Static benchmarks are controlled but stale; production monitoring is live but noisy. Run both.

Key insight

The gap between offline and online numbers is itself the metric: it measures how far your benchmark has drifted from reality.

Failure mode

Trusting the offline number when they disagree. Production is noisy, but it is not wrong.

Agent Evals

E-01Outcome Eval

E-02Process / Trajectory Eval

E-03LLM-as-Judge

E-04Human-calibrated Judge

E-05Regression Evals

E-06Failure Taxonomy

E-07Outcome vs Process Mismatch

E-08Adversarial Evals

E-09Cross-Model Evals

E-10Offline vs Online Evals

Elsewhere in the atlas