A Practical Framework for Evaluating LLM Agents

Unit tests assume determinism: same input, same output. Agents built on LLMs break that assumption at every step — the same prompt can produce a different tool call, a different plan, or a different final answer on two consecutive runs. That doesn't mean agents are untestable. It means the test design has to change.

Test the trajectory, not just the outcome

A common mistake is checking only the final answer. Two agent runs can reach the same correct answer through very different paths — one efficient, one that burns ten unnecessary tool calls and gets lucky. If you only score the final output, you can't tell those apart, and you'll ship the inefficient path without knowing it.

A better evaluation captures:

{
  "final_answer_correct": bool,
  "tool_calls_made": int,
  "unnecessary_tool_calls": int,
  "recovered_from_error": bool,
  "total_latency_ms": int,
}

Build a fixed regression set early

Before optimizing anything, freeze a set of representative tasks — real user queries if you have them, otherwise carefully constructed synthetic ones — and run every change against the full set. This is the only way to know whether a prompt change or a new tool definition made things better or just moved the failure mode somewhere else.

LLM-as-judge, with guardrails

Using a model to grade another model's output is useful but easy to get wrong. What works reliably:

• Grade against a rubric, not a vague "is this good?" prompt.
• Have the judge cite the specific part of the transcript that supports its score.
• Spot-check judge scores against human review regularly — judges drift as underlying models change.

Treat tool failures as a first-class test case

Real tools fail: timeouts, rate limits, malformed responses. An agent that only ever sees successful tool calls in testing will handle failures badly in production. Inject failures deliberately into the evaluation set and score how gracefully the agent recovers, not just whether it eventually succeeds.

The goal of agent evaluation isn't a single pass/fail number — it's a dashboard that tells you which failure modes are getting better or worse as you ship changes.