What Are Agent Evals? API, CLI and MCP Evals for Developer Tools
Jun Liang Lee8 min readDirect answer
Agent evals are tests in which real coding agents, such as Claude Code or Codex, attempt a real task with your API, CLI or MCP server and are judged on whether the result works. API, CLI and MCP evals name the interface under test. LLM evals score your own AI app's outputs, and unit tests check your code.
Why agent evals matter now
Coding agents now integrate developer tools unattended. As of October 2026, Claude Code runs unattended with claude -p and Codex with codex exec. When an agent writes the integration, your docs, errors and login flow have to work for the agent first.
The same product also scores differently from one agent to the next. Devtool Arena is Lightsage's public leaderboard at devtoolarena.com, scoring how well coding agents use 150+ developer tools. On it, Stripe's API scored 89 with Claude Code (Sonnet 5) and 76 with Codex (GPT-6 Sol), on the same task (Devtool Arena, Claude Code and Codex API boards, October 2026). A test suite you run against your own code can't show that gap.
How agent evals work
An agent eval has three parts: a task, a real coding agent running it in a sandbox, and a check on the outcome. Together they measure agent usability, which is whether an agent can complete a real task with your product: sign up, authenticate, integrate and get a working result.
A real task, worded the way developers ask
The task is a request a developer would type, with a result that can be checked. The agent gets the goal and credentials, and finding the package, endpoint or command is its job.
A shipped agent in a clean sandbox
The agent is one your customers run, such as Claude Code or Codex, started in a fresh sandbox. It installs packages, fetches docs, runs commands and retries after errors. You don't write its prompts or pick its model, the same as when your customers use it.
Judged on the outcome, with a trace
A run passes when the result exists, such as a payment link that resolves or an email that arrives. Scores add how the agent got there. Devtool Arena's eval score covers runtime, cost, tool calls and errors, and its API score is 70% eval and 30% discovery, a checklist of public assets such as llms.txt and OpenAPI specs. The trace of every tool call is what you fix from.
What are API evals, CLI evals and MCP evals?
They're the same test pointed at different interfaces. The task and judgment stay fixed, and the agent's effort moves. For one Stripe task followed through all three, see agent-led growth for APIs, CLIs and MCP servers compared.
What are API evals?
An API eval gives the agent your REST API or SDK. It has to pick the right package, find the right reference page, write code and run it. For a payments API, that might mean creating a test-mode payment and returning its checkout link. The trace shows where it stalled, such as an outdated doc page, the wrong SDK version or a vague error body.
What are CLI evals?
A CLI eval gives the agent your command-line tool. The agent installs it, authenticates without a browser and parses its output in a non-interactive shell. The Command Line Interface Guidelines state the rule an agent depends on: "Never require a prompt." A CLI eval also catches failures that exit 0. Our roundup of CLI usability scoring platforms covers the tooling.
What are MCP evals for developer tools?
An MCP eval connects your MCP server to the agent before the task starts. The agent can only call the tools the server exposes, so the eval tests tool selection: whether the agent finds the right tool from its name and description, sends valid arguments and recovers from errors. The MCP Inspector tests a server with calls you choose. An MCP eval checks whether an agent chooses them itself. Our MCP compatibility guide goes further.
What is coding agent testing compared with LLM evals and unit tests?
Coding agent testing is another name for agent evals: a coding agent you don't control uses your product, and you judge the result.
| Term | What it measures | Where it happens | Who owns it |
|---|---|---|---|
| API evals | Whether a coding agent completes a task with your REST API or SDK | Agent sandbox, against your API | API platform team, DevRel |
| CLI evals | Whether a coding agent installs, authenticates and drives your CLI unattended | Agent sandbox shell | CLI or developer experience team |
| MCP evals | Whether a coding agent picks the right tools on your MCP server and finishes the task | Agent session with your server connected | MCP or AI platform team |
| LLM evals | Whether your own AI app or agent gives good outputs for a dataset of inputs | Your app, before and after release | The team building the AI app |
| Unit and contract tests | Whether your code and API behave as specified | CI, against your code and spec | Engineering |
LLM-eval frameworks test an AI product you build. Promptfoo is an open-source CLI and library "for evaluating and red-teaming LLM apps". DeepEval plugs into Pytest and treats a test case as "a single unit of LLM app interaction". Braintrust and LangSmith trace your app or agent and score its outputs against datasets. They're the right tools when the model and prompts are yours. When the agent is Claude Code and the product is your API, you can change your docs, errors and interface, and an agent eval measures those.
Unit and contract tests confirm your API returns what its spec promises, starting from a correct request. An agent has to reach that request first: choose a package, find the credential, get past the login. A green CI run and a failing agent eval can describe the same API on the same day.
Two October 2026 results that show why you test each agent and interface
Same API, different agents. Both agents finished the Stripe task, and both runs passed verification. Stripe scored 89 with Claude Code (Sonnet 5) and 76 with Codex (GPT-6 Sol), and its discovery score was 86 on both boards, so the gap came from how each agent worked through the task (Devtool Arena, Claude Code and Codex API boards, October 2026). The Claude Code run took 6 tool calls and about 36 seconds. A pass-or-fail check would have scored them the same.
Same agent, different interfaces. Resend scored 81 on Claude Code's MCP board and 91 on its CLI board (Devtool Arena, Claude Code MCP and CLI boards, October 2026). The CLI run took 9 tool calls with no errors. Each board uses its own checks, so the spread says to test every interface you ship. It doesn't rank the interfaces.
Common misconceptions about agent evals
"Our LLM evals already cover agents." LLM evals cover an agent you build. Agent evals for devtools cover someone else's agent using your product, and the fix goes into your docs or interface instead of a prompt.
"Passing contract tests means agents can use the API." Contract tests start from a well-formed request. The agent's hard part comes before it: the package, the credential and the login.
"One agent's result stands for all of them." Stripe's API scored 13 points apart on Claude Code and Codex in October 2026. Test the agents your customers use.
"Testing with MCP Inspector is an MCP eval." The Inspector shows your server answers the calls you make. An MCP eval shows whether an agent makes them.
How to start running agent evals
- Pick one task per interface. Use the first thing a new customer does, worded as a developer would ask, with a result you can check, such as an object created.
- Run it on the agents your customers use, more than once. Start with Claude Code and Codex, then add the agents your users name. We'd run each task at least three times per agent, because the same agent can take a different path next time.
- Read the traces and fix the first failure. Change the doc page, error, flag or tool description the agent tripped on, then rerun.
If you're on Devtool Arena, start with your row on the Claude Code API board. The full walkthrough is in How to Run Agent Evals on Your API, CLI or MCP Server, with MCP specifics in How to Test Your MCP Server With Coding Agents.
Lightsage Agent Usability runs your task with real coding agents in sandboxes against your API, MCP server or CLI, and returns pass or fail, full traces, errors, and per-agent and per-endpoint scores. Plans start at $100/month after a 7-day trial (pricing). For other options, see Best Agent Testing Tools for Developer Tools in 2026. If you're scoring your own LLM app's outputs, Promptfoo or Braintrust fits better.
Frequently asked questions
What are agent evals for devtools?
Agent evals for devtools are tests in which a coding agent such as Claude Code, Codex or Cursor completes a task with your developer tool, judged on the result. They show whether agents can adopt your product with no person stepping in.
What do agent evals mean for an API?
For an API, an agent eval gives a coding agent a goal and a key, then checks whether the code it writes against your endpoints works. The trace shows which doc page, SDK call or error slowed it down.
What does coding agent testing mean for an MCP server?
For an MCP server, coding agent testing means connecting the server to an agent and checking whether it picks the right tools, sends valid arguments and finishes the task. It tests your tool names, descriptions and error messages as an agent reads them.
What is coding agent testing for API companies?
For API companies, coding agent testing is a recurring check that developers' agents can still integrate the API after docs, SDK or auth changes. Results differ by agent, so test each one your customers use.
What is a CLI eval?
A CLI eval runs a coding agent against your command-line tool in a non-interactive shell and checks that the task's result exists. It catches browser-only logins, interactive prompts and failures that exit with status 0.