What API Performance evaluates
Unlike Visibility Analytics, which tracks whether AI recommends you, API Performance measures whether agents can successfully use your API once they’ve chosen it. This covers the full flow:- Can the agent find and understand your documentation?
- Can it write correct code to call your endpoints?
- Does that code actually work when executed?
- Can it recover when something goes wrong?
The six metrics
Every API Performance run scores six metrics for each endpoint and agent combination:Eval Score and Discovery Score are composite indexes. Eval Score factors in success rate, error count, and recovery capability. Discovery Score measures whether the agent correctly identifies your endpoint, authentication method, and request shape without hallucinating parameters.
Letter grades
Lightsage assigns a letter grade to each endpoint and to your API overall based on the aggregated Eval Score across all runs:Leaderboard benchmarks
Lightsage’s public Devtool Arena provides reference points for how your scores compare to other APIs in your category. Sample data from the Payments category:
These scores reflect real evaluation runs across multiple coding agents on the Devtool Arena leaderboard. A score of 82 (Stripe’s position) means agents succeed on most tasks but still encounter some friction — there’s measurable room for improvement even for a top performer in the category.
Configuring evaluations
1
Add your endpoints
Go to API Performance in your dashboard and select Add Endpoints. Paste your endpoint URLs or import from an OpenAPI spec. Add the authentication method and any required headers.
2
Choose coding agents
Select which agents to run evaluations against. Each agent costs 5 credits per run. Start with Claude Code and Codex — they’re the most widely used and provide the clearest signal. Add more agents as your credit budget allows.
3
Set evaluation frequency
Choose daily or weekly evaluation runs. Daily is recommended when you’re actively improving documentation or error messages. Weekly works well for stable APIs where you want trend data without high credit consumption.
4
Review baseline results
After the first run, review your scores in the endpoint breakdown. The baseline tells you where to focus — start with D- and C-grade endpoints, since improvements there have the highest impact on overall grade.
Endpoint-level breakdown
The endpoint breakdown shows pass rates for each of your endpoints across all selected coding agents. Example from the Stripe Payments API evaluation:
Pass rate is the percentage of agent runs where the task completed successfully — meaning the agent called the endpoint, received a valid response, and the overall task finished without switching to a competitor.
Endpoints with low pass rates are your highest-priority fixes. A pass rate below 70% means a meaningful percentage of developers who ask a coding agent to use that endpoint will hit a dead end.
Coding agent leaderboard
The agent leaderboard shows which coding agents work best with your API — useful for understanding where to focus optimization effort and for communicating compatibility to your users. Sample leaderboard data across five endpoint categories:
A large gap between Claude Code and other agents usually indicates that your documentation is written in a style that Claude’s model understands well, but that other agent architectures find ambiguous. Consistent response shapes and descriptive errors narrow those gaps.
The Agent Journey flow
Lightsage records each agent’s navigation path as it works through a task. The Agent Journey flow visualizes how agents move from your documentation to your API endpoints. Each successful path shows:- Steps taken — how many pages or tool calls the agent made before reaching a successful API call
- Starting point — where the agent began (usually a docs page or search result)
- Entry point — which documentation page the agent read before calling the endpoint
Interpreting low scores and what to fix
Low Discovery Score
Low Discovery Score
The agent can’t reliably find or understand your endpoint. Common causes:
- AI crawlers are blocked in
robots.txt— ClaudeBot or GPTBot can’t read your docs - Documentation is client-side rendered JavaScript and doesn’t work without a browser
- Endpoint names are ambiguous — agents can’t determine which endpoint handles a given task
- No
llms.txtfile — agents have no machine-readable summary of your API’s capabilities
robots.txt, verify server-side rendering on your docs, and add an llms.txt file at your domain root.High error count
High error count
Agents frequently hit errors when calling your endpoint. Common causes:
- Authentication setup is unclear — agents guess at header names or token formats
- Error messages are generic (
Error 500,Bad Request) with no guidance on how to fix them - Required parameters are poorly documented — agents omit them or use wrong types
- Breaking API changes aren’t reflected in current documentation
High tool call count
High tool call count
Agents are making many API calls to complete a single task — a sign they’re exploring, recovering from failures, or confused about the right approach.Fix: Add a clear quick-start example for the endpoint that shows the exact code an agent should generate. Agents with a concrete example to pattern-match against make fewer exploratory calls.
Low completion rate / agents switching
Low completion rate / agents switching
Agents start with your API but switch to a competitor before finishing the task. This is the most damaging failure mode — it generates a negative experience at scale.Fix: Review the Agent Journey flow for abandoned runs. Identify the last endpoint called before the agent switched — that’s your highest-priority fix. Usually this is an unclear error message or a missing response field.