Did the agent finish the task? How many steps and dollars did it take? Did anything go wrong along the way? Three questions, three numbers — outcome, trajectory, safety. Without them you have vibes; with them you have a regression test that catches when next week's prompt edit silently halves your success rate. SWE-bench, GAIA, AgentBench, and tau-bench are the benchmarks that matter in 2025.
Learning Objectives
After this lesson, you will be able to:
Design evaluation frameworks that measure three things at once: Did the agent finish the task? How many steps and dollars did it take? Did anything go wrong?
Know the major agent benchmarks: SWE-bench (can it fix real bugs?), GAIA (can it browse the web and reason?), and AgentBench (general agent skills)
Understand the difference between trajectory evaluation (was the *process* efficient?) and outcome evaluation (was the *result* correct?)
Build tests for AI agents that give different answers every time, using statistical assertions ('passes 85% of 20 runs') instead of exact matches
Set up regression testing and CI pipelines so you know immediately when a prompt change breaks your agent
Evaluation is the skill that turns "I think my agent works" into "I know my agent works, and I have the numbers to prove it." It is less glamorous than building agents, but it is what separates hobbyists from professionals. You will thank yourself later for learning this.
Try it! Take any AI chatbot and ask it the same question 5 times. Notice how the answers vary slightly each time -- different wording, different structure, sometimes different content. This non-determinism is exactly why you cannot test agents with simple assert output == expected. Statistical testing is the answer, and you are about to learn how.
Agent evaluation is the hardest problem in agentic AI. Traditional software testing gives you deterministic assertions: assertEqual(add(2, 3), 5). Agents are non-deterministic -- run the same task twice and you might get different tool call sequences, different reasoning traces, and slightly different outputs. The path to the answer changes every time. So how do you test something that never does the same thing twice?
The most basic question: did the agent accomplish the goal? This sounds simple, but defining "success" is surprisingly nuanced.
Binary tasks have clear success criteria. "Did the code compile?" "Did the test pass?" "Was the email sent?" These are easy to evaluate automatically.
Quality tasks require judgment. "Write a blog post about transformers." How do you score it? Automated rubrics (check for technical accuracy, readability, completeness) help, but ultimately writing quality is subjective. LLM-as-judge -- using another LLM to evaluate the output -- has become the standard approach, though it has its own biases and limitations.
Multi-step tasks complicate matters further. An agent might complete 4 out of 5 subtasks perfectly but fail the last one. Is that a failure? A partial success? How do you credit partial progress?
Two agents that both complete the task are not equal if one took 8 steps and the other took 45. Efficiency metrics include:
Step count. How many tool calls and reasoning steps did the agent use? Fewer is generally better, but too few might mean the agent cut corners.
Token usage. Total input and output tokens consumed. This directly maps to cost. An agent that generates verbose reasoning traces might be more accurate but 3x more expensive.
Wall-clock time. How long did it take from start to finish? Users have patience limits. An agent that takes 5 minutes to do what a human could do in 30 seconds is not useful, even if the result is perfect.
Cost in dollars. The ultimate metric. Token usage times price per token, plus any tool API costs. Track this per task, per category, and per time period.
#3. Safety and Reliability (Did anything go wrong?)
Even correct, efficient agents can fail on safety dimensions:
Error rate. How often does the agent crash, enter an infinite loop, or produce malformed output?
Guardrail triggers. How often do safety layers activate? High trigger rates might indicate the agent is testing boundaries or the guardrails are too sensitive.
Hallucination rate. For tasks grounded in data, how often does the agent make claims not supported by its sources?
Recovery rate. When something goes wrong (tool failure, unexpected data), how often does the agent recover versus getting stuck?
What Do You Think?
Two code agents are evaluated on 100 programming tasks. Agent A completes 92 tasks correctly with an average cost of $2.10 per task. Agent B completes 88 tasks correctly with an average cost of $0.35 per task. Which is better for production use?
The answer depends entirely on context. If each failed task costs the company $1,000 (e.g., a critical bug reaching production), Agent A's 4% higher success rate saves $40,000 per 1,000 tasks while costing an extra $1,750 in compute -- a clear win. But if failures are cheap to catch and retry (e.g., automated code review catches mistakes), Agent B saves $175 per 100 tasks with only a small accuracy gap. Always compare the cost of failure against the cost of compute.
The field has developed standardized benchmarks to compare agents across implementations. Understanding what they measure -- and what they miss -- is critical for interpreting results.
What it tests: Real-world software engineering. Agents are given a GitHub issue (bug report or feature request) from a real open-source project and must produce a pull request that resolves the issue. The PR is evaluated by running the project's test suite.
Why it matters: This is the gold standard for coding agents because it uses real codebases, real issues, and real tests. There is no ambiguity in evaluation -- either the tests pass or they do not. The full SWE-bench dataset contains 2,294 issues from 12 popular Python repositories (Django, Flask, scikit-learn, etc.).
Key metrics: Resolve rate (percentage of issues where the agent's PR passes all tests). Current state of the art for SWE-bench Verified (a 500-issue curated subset) exceeds 60%, up from 4% when the benchmark was introduced in 2023. This represents remarkably rapid progress.
Limitations: Only tests Python projects. Only tests bug fixes and feature requests with existing test suites. Does not evaluate code quality, documentation, or architectural decisions -- only whether the tests pass.
What it tests: General AI assistants on real-world tasks that require web browsing, file processing, and multi-step reasoning. Example tasks: "What is the population of the country whose flag has a dragon holding a golden orb?" (requires: identify the flag, identify the country, look up population data).
Why it matters: GAIA tests the agent's ability to compose multiple capabilities -- search, reasoning, tool use, knowledge integration -- on tasks designed to be easy for humans but hard for AI. The tasks have unambiguous ground-truth answers, making evaluation simple and reliable.
Key metrics: Accuracy across three difficulty levels. Level 1 tasks require simple tool use. Level 2 requires multi-step reasoning. Level 3 requires complex research and synthesis. Even top agents struggle on Level 3, where human performance exceeds 90%.
What it tests: Agents across 8 different environments: operating system commands, database operations, knowledge graphs, game environments, web browsing, lateral thinking puzzles, card games, and household tasks.
Why it matters: Breadth. While SWE-bench only tests coding and GAIA tests information retrieval, AgentBench evaluates agent competence across diverse task types. This reveals whether an agent's performance is broadly capable or narrowly specialized.
Key metrics: Success rate per environment and overall. Reveals large performance gaps between environments -- an agent that excels at database operations might struggle with web browsing, exposing fundamental capability differences.
The agent receives GitHub issue #4821 from the Django repository: "QuerySet.annotate() crashes when using a subquery with OuterRef." The issue includes a stack trace, a minimal reproduction case, and a description of the expected behavior.
The agent reads the stack trace, locates the relevant source files (django/db/models/sql/query.py), and examines the annotate() method. It identifies the root cause: an OuterRef is not being resolved correctly during subquery compilation.
The agent modifies 12 lines across 2 files. It adds a check to resolve OuterRef references before compiling the subquery annotation. It also adds a regression test that reproduces the original issue.
The benchmark runs Django's full test suite (12,000+ tests). The agent's fix makes the previously-failing test pass without breaking any existing tests. The agent also added a regression test, which is good practice. Result: PASS. The agent resolved the issue in 14 tool calls using 45,000 tokens ($0.68).
Beyond pass/fail, trajectory evaluation examines how the agent worked. Did it read the right files first? Did it waste steps on dead ends? Did it modify the minimal set of files? This agent explored 3 irrelevant files before finding the right one -- a more efficient agent would have gone straight to the source. The fix is correct, but the process could be tighter.
The evaluation begins with a curated suite of test cases. Each case has an input (a task description or user request), expected behavior criteria (not exact output), and metadata: difficulty level, category, and cost baseline. Example: "Fix Django issue #4821 (QuerySet.annotate crash). Expected: tests pass. Baseline: 8 tool calls, $0.45."
The agent executes each test case independently. To account for non-determinism, each case runs 3-5 times. The agent operates normally -- reading files, reasoning, calling tools, producing output -- with full instrumentation recording every step. No special "test mode" -- the agent runs exactly as it would in production.
Every detail of the agent's execution is logged: each reasoning trace, every tool call with arguments and results, timing per step, token usage per step, errors encountered, and recovery actions. This trajectory is the raw data for evaluation -- it tells you not just what the agent produced, but how it got there.
The first evaluation dimension: did the agent accomplish the goal? For code tasks, run the test suite. For research tasks, check factual accuracy. For writing tasks, use an LLM-as-judge with a scoring rubric. Binary pass/fail for clear criteria, 1-5 rubric scores for quality assessments. Record the pass rate across all runs of each case.
The second dimension: how much did it cost? Count tool calls (14 vs baseline of 8 -- the agent wandered). Sum tokens (45,000 vs baseline 30,000 -- verbose reasoning). Calculate dollars ($0.68 vs baseline $0.45). Measure wall-clock time (47 seconds vs baseline 25 seconds). Efficiency grades catch agents that get the right answer by brute force.
The third dimension: did anything go wrong? Check for guardrail triggers (did the agent try to access unauthorized resources?), PII exposure (did any response contain sensitive data?), hallucinations (did the agent fabricate facts not in its sources?), and loop incidents (did the agent repeat itself?). A correct, efficient agent that leaks data is still a failing agent.
Scores are aggregated across all test cases and all runs. Completion rate: 91% (pass on at least 3 of 5 runs). Average cost: $0.52/task. Safety violations: 0. Results are broken down by difficulty, by category, and by error type. Statistical tests confirm whether differences from the baseline are significant or just noise from non-determinism.
The final step: compare against the previous version. Did the new prompt improve completion rate? Did the model upgrade change cost? Did the tool refactor introduce regressions? Head-to-head comparison on the same test cases controls for external factors. If completion rate dropped more than 5% or cost increased more than 20%, the change is blocked. Data drives the decision, not gut feeling.
This is a fundamental distinction in agent evaluation:
Outcome evaluation asks: "What did the agent produce?" It looks only at the final result. Did the code work? Was the report accurate? Was the email sent? This is simple, objective, and the most commonly used approach. But it misses everything about the process.
Figure
Two runs on the same task, ending the same way. The first takes eight focused steps: read the error, locate the bug, fix it, run the test, done. The second takes forty-five scattered steps, searching unrelated files and re-reading the same code repeatedly before stumbling into the fix. Both pass. Judged only on final output they are identical — which is exactly why outcome-only evaluation misses the difference that determines cost, latency, and whether you can trust the agent unattended.
Trajectory evaluation: grade the process, not just the result
Trajectory evaluation asks: "How did the agent get there?" It examines the full sequence of reasoning traces, tool calls, and decisions. Did the agent reason correctly? Did it use the right tools? Did it recover gracefully from errors? Was the path efficient?
What Do You Think?
Two agents both produce correct code that passes all tests. Agent A used 8 focused tool calls -- it read the error, located the bug, and fixed it. Agent B used 45 tool calls -- it searched random files, tried multiple wrong fixes, and eventually stumbled on the answer. Which is more reliable for future tasks?
Why does trajectory matter? Because two agents with identical outcomes can have very different reliability profiles. Agent A might have reasoned systematically to the correct answer. Agent B might have gotten lucky -- it stumbled into the right answer through a series of fortunate tool calls. On the next task, Agent B is far more likely to fail again because its process is unsound. Outcome evaluation cannot distinguish between skill and luck. Trajectory evaluation can.
Practical trajectory evaluation examines several dimensions:
Tool selection accuracy. Did the agent choose the right tool for each step? Calling web_search when the information is in the local database is a trajectory error even if the agent eventually finds the answer. Track this as a dedicated agent-specific metric: tool accuracy = (tool calls that selected the correct tool AND passed correct arguments) / (total tool calls). A tool accuracy below 80% indicates the agent is regularly misusing its toolset — usually caused by poor tool descriptions or too many similar tools.
Reasoning quality. Are the reasoning traces logical and well-structured? Do they show genuine understanding or pattern matching? Do they correctly identify what information is needed before acting?
Error handling. When a tool call fails, does the agent reason about why and adapt, or does it blindly retry? Good error handling in the trajectory is a strong predictor of reliability.
Efficiency. Was the path the shortest reasonable one, or did the agent wander through unnecessary steps? Count the ratio of "productive" steps (that moved toward the goal) to "wasted" steps (dead ends, redundant searches, unnecessary reasoning).
Traditional software testing relies on determinism: same input, same output. Agents break this. The same prompt can produce different tool call sequences, different reasoning, and slightly different outputs on every run. How do you test something that is inherently stochastic?
An agent passes 19 out of 20 evaluation runs (95% pass rate). Is this statistically significant enough to ship to production?
The issue is sample size. With only 20 runs, a 95% observed pass rate has a 95% confidence interval of roughly 75-100%. The true pass rate could easily be 80%, which might be below your production threshold. Increasing to 100 runs (95/100) narrows the confidence interval to about 89-98%, giving you much more certainty. For production gates, use at least 50-100 runs per evaluation to get meaningful statistical power.
Replace exact assertions with statistical ones. Instead of "the agent must produce exactly this output," assert "the agent must produce a correct output at least 90% of the time across 20 runs."
pythonreference · read-only
1
2
3
4
5
6
7
# Traditional test (brittle for agents)
assert agent.run("Fix the login bug") == expected_patch # Fails on minor differences
# Statistical test (robust for agents)
results = [agent.run("Fix the login bug") for _ in range(20)]
pass_rate = sum(1 for r in results if tests_pass(r)) / len(results)
assert pass_rate >= 0.85, f"Pass rate {pass_rate} below threshold 0.85"
Test properties of the output rather than exact content:
Structural tests. "The response must contain a code block." "The plan must have between 3 and 10 steps." "The tool calls must include at least one database query."
Constraint tests. "The response must not contain PII." "The total cost must be under $2.00." "The agent must not call the email tool without prior approval."
Semantic tests. "The response must address the user's question" (use an LLM judge to assess). "The generated code must handle the edge case described in the issue."
Even though agent outputs vary, you can still catch regressions:
Golden trajectory comparisons. Save the tool call sequence from a known-good run. On future runs, compare: are the same tools being called? In roughly the same order? With similar arguments? Significant deviations (calling 30 tools instead of 8, or skipping a critical tool entirely) indicate a regression.
Output quality scoring. Use an LLM judge to score outputs on a rubric (accuracy: 1-5, completeness: 1-5, clarity: 1-5). Track average scores over time. A drop in average score across multiple tasks signals a regression, even if individual outputs vary.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
Tests · Add regression tracking across evaluation runs, difficulty-based breakdown, and outlier detection for cost and time.
Figure
Pass rate is tracked across releases. It holds near 90% through several versions, then drops sharply to around 72% immediately after a model upgrade, before recovering as prompts are re-tuned. Nothing in the agent's own code changed at the point of the drop. This is the argument for continuous evaluation: without a regression suite running on every change, a swap in the underlying model silently degrades your system and the first person to notice is a user.
Agent pass rate over time: tracking how evaluation scores change across model versions, prompt updates, and tool changes reveals regressions before they reach production.
Agent quality must be tested continuously, not just at development time. A CI pipeline for agents ensures that changes to prompts, tools, or infrastructure do not degrade performance.
Step 1: Eval suite. Maintain a curated set of 50-200 evaluation tasks that cover your agent's key use cases. Each task has an input, expected behavior (not exact output), and evaluation criteria. Update the suite as you add new capabilities.
Step 2: Run on every change. When a developer changes the system prompt, modifies a tool, or updates the model version, the CI pipeline runs the full eval suite. Each task runs 3-5 times to account for non-determinism.
Step 3: Statistical comparison. Compare the new results against the baseline. Use statistical tests (not just averages) to determine if the change is a significant improvement, regression, or neutral. A 2% drop in pass rate across 100 tasks might be noise. A 10% drop is almost certainly a real regression.
Step 4: Gate deployment. If the eval suite shows regression beyond a threshold (e.g., pass rate drops more than 5% or cost increases more than 20%), block the deployment. Require the developer to investigate and fix before merging.
Happy path tasks. The most common, expected use cases. These should always pass.
Edge cases. Unusual inputs, ambiguous requests, tasks at the boundary of the agent's capabilities. These test robustness.
Adversarial tasks. Prompt injection attempts, requests for out-of-scope actions, attempts to exceed budget limits. These test safety.
Regression tasks. Tasks that previously failed and were fixed. Include them forever to prevent re-introduction of old bugs.
Cost benchmarks. Tasks with known cost profiles. If the average cost per task increases significantly, something changed in the agent's behavior (more verbose reasoning, more tool calls, different model routing).
Write evals before writing the agent: define acceptance criteria as test cases first — this prevents optimizing for the model's strengths rather than actual requirements
Benchmark score ≠ production quality: published benchmarks measure memorized patterns; your eval suite on your tasks is the only honest measure of your agent's real performance
Trajectory evaluation + outcome evaluation are both required: outcome alone misses fragile agents that luck into the right answer; trajectory alone misses agents that find novel correct paths
Non-determinism demands statistical testing: run each eval task 5–10 times and report pass rate; a single pass/fail is statistically meaningless for probabilistic systems
CI for agents should gate on pass rate and cost: block deployment if the pass rate drops >5% from baseline OR cost-per-task increases >20% — quality and economics both matter
Knowing which benchmark to cite (and which to discount) is half of agent eval literacy. A current map:
SWE-bench Verified (Anthropic, 2024). The 500-issue curated subset of SWE-bench. Frontier models cleared 70%+ in 2025 (Claude 3.7/4 Sonnet, Claude Code, OpenAI o3, GPT-5). The headline benchmark for coding agents. Beware: large training-data contamination concerns on the broader SWE-bench Full.
OSWorld (Xie et al., 2024) — 369 real desktop tasks across Ubuntu/macOS. Human baseline ~72%; frontier computer-use models in 2025 sit in the 15-25% range. The honesty stat for computer-use agents.
WebArena (Zhou et al., 2023) and Visual WebArena (2024) — self-hostable web environments. Realistic plateau around 35-50% in 2025.
GAIA (Mialon et al., 2023) — three-level general-assistant benchmark with unambiguous answers. Level-3 questions remain hard; top agents in 2025 are around 60-70% L1, 30-40% L3 -- humans clear 92%.
TheAgentCompany (CMU, 2024). A simulated software company with multi-day workflows. Top scores in the low double digits.
AgentBench, MLAgentBench, AppWorld (2024). Breadth and tool-rich environments for stress-testing generalists.
τ-bench / TauBench (Sierra, 2024). Real-customer-service-style tool-use scenarios with multi-turn user simulations. Closer to many real production use cases.
HELM Agents (Stanford, 2024-2025). A holistic battery covering helpfulness, safety, and efficiency. Useful for cross-cutting comparison.
Cite the Verified and curated subsets when comparing models, and -- always -- ship your own task suite as the production gate. Public benchmarks tell you about a model family; only your private eval tells you about your product.
Congratulations -- you have completed the AI Agents track. From the fundamental observe-think-plan-act loop to tool use and function calling, from memory systems to planning and reasoning, from safety guardrails to multi-agent collaboration, and finally to rigorous evaluation -- you now understand every layer of building, deploying, and measuring agentic AI systems. The agents are ready. The question is what you will build with them.