AI Systems Atlas

E-17User Simulators

/evals/user-simulators

A second model plays the user — persona, goal and hidden facts — so a multi-turn agent is tested on whole conversations and graded on the end state.

Step-by-step walkthrough8 steps
  1. A persona and a goal: who the user is, what they want, and what they will only reveal if asked.
  2. A second model plays that user, one turn at a time.
  3. The agent asks, clarifies and replies — the whole conversation is the test input.
  4. Along the way it acts on real state through its tools.
  5. The grader checks the end state against the goal, not how pleasant the transcript sounded.
  6. Too helpful: the simulator hands over everything up front, so the agent never had to ask a single question.
  7. Drift: ten turns in, the “terse” user is chatty and has forgotten its hidden facts — you are now testing a different user.
  8. Audit the simulator too: pin the persona, check each run stayed in character, and discard the runs that did not.
Key insight

Conversational agents fail between turns: asking the wrong question, missing a correction, acting before confirming. Only a user that talks back exercises that, and grading the end state keeps the verdict objective.

Failure mode

A simulator that is easier than real users — it volunteers every fact, never changes its mind, and quietly drifts out of persona. The agent aces the eval by never meeting the conversation it will actually have.

Further reading
RelatedOffline vs onlineOutcome evalLLM-as-judge

Elsewhere in the atlas

Harnesses →Security →Context →Coding Agents →Tools →Ops →