AI Systems Atlas

H-21Multi-agent Debate

/harnesses/multi-agent-debate

Several agents answer the same question, then read and critique each other’s answers over rounds until a judge or vote settles it.

Step-by-step walkthrough7 steps
  1. One question goes to three agents, each answering on its own.
  2. Round 1: independent answers, and they disagree. The disagreement is the raw material.
  3. Round 2: each agent reads the others’ answers and reasoning, and argues back.
  4. C switches to 42 — because B found a real slip, or just because two agents disagreed with it?
  5. But agreement is not evidence: copies of one model can argue each other into the same wrong answer.
  6. A judge — or a plain majority vote — aggregates the final round.
  7. One answer, at agents × rounds the cost. Worth it when a critique can change a mind; otherwise sample and vote.
Key insight

Debate pays off only when the agents’ errors are independent and an argument can actually expose one. Otherwise it is best-of-N with extra rounds: several times the tokens for the same majority.

Failure mode

Conformity. Copies of one model share one set of blind spots, and a confident majority can argue a correct minority out of its answer.

Further reading
RelatedBest-of-NGenerator–CriticLLM-as-judge

Elsewhere in the atlas

Security →Evals →Context →Coding Agents →Tools →Ops →