How Devtool Arena scores are calculated

Every tool on Devtool Arena receives an overall score from 0 to 100. That score combines two components — Eval Score and Discovery Score — captured during a live, end-to-end evaluation run by Claude Code using a real production API key. Understanding how those two components interact helps you identify exactly which levers to pull to improve your ranking.

​

The two score components

​

Eval Score

Eval Score measures whether the AI coding agent successfully completed the assigned task end-to-end. For each tool, Lightsage defines a realistic developer task — for example, “send a transactional email,” “perform a vector similarity search,” or “run an inference request” — and the agent is given a real API key and a fresh environment to work in. The Eval Score reflects the agent’s task completion rate across multiple runs. A score of 100 means the agent completed the task without getting stuck. A lower score indicates the agent failed part or all of the task: it may have generated code that returned errors it could not recover from, called the wrong endpoints, or failed to authenticate. What drives a high Eval Score:

• A quick-start guide that works end-to-end without modification
• Descriptive, specific error messages that tell agents what went wrong and how to fix it
• Consistent, typed API responses (TypeScript definitions, OpenAPI specs, JSON Schema)
• Minimal required setup before the first successful call

​

Discovery Score

Discovery Score measures how easily the agent found and understood your API without hand-holding. When a task involves your API, does the agent know your tool exists, locate the right documentation, and correctly understand what your API does — all on its own? Discovery is influenced by signals that help agents orient themselves before they write a single line of code: your llms.txt file, entries in Context7, whether AI crawlers can reach your docs, and how clearly your documentation explains what your API does (and does not do). What drives a high Discovery Score:

• A published llms.txt file with “when to use” and “when NOT to use” sections
• AI crawlers allowed in robots.txt
• Documentation indexed in Context7
• Clear, scannable API reference pages that agents can parse quickly
• Comparison pages that help agents understand your positioning

​

The six metrics collected per run

In addition to the two component scores, Devtool Arena records four efficiency metrics on every run:

​

The grade system

The overall score maps to a letter grade shown in the Grade column: As of May 2026, the majority of ranked tools score in the B–C range. A grades are rare — achieving one requires both near-perfect task completion and high discoverability.

​

Real leaderboard examples

The following table shows how score components correlate with real outcomes (API leaderboard, May 2026): Notice that a Discovery Score of 100 does not guarantee a high overall score — Deepgram’s perfect discoverability is undermined by a low Eval Score of 25. Conversely, Jina AI achieves a strong Eval Score of 90 despite a lower Discovery Score of 71. Both components matter.

​

The checklist columns

The seven checklist columns (C7, llms, MCP, SDK, API, Skills, CLI) are binary presence checks, not scored metrics. They record whether each asset exists in the tool’s publicly available resources at evaluation time. Checklist completeness correlates strongly with higher scores. In our analysis, tools with llms.txt and CLI support score an average of 18 points higher than those without. Tools with working MCP servers dominate their categories in the MCP leaderboard. The relationship is not causal by default — a broken MCP server helps no one — but high-quality presence across multiple channels consistently separates top performers from the rest.

​

Skipped entries

Tools listed as Skipped (shown with — in score columns) were not evaluated because Lightsage could not obtain a working API key during the evaluation window. Common reasons:

• The API requires a paid plan before issuing credentials
• Signup requires a credit card
• Access requires official company certification or business verification
• The tool is open-source and self-hosted with no hosted API available
• The MCP server requires browser-based OAuth that cannot be automated

Skipped tools are still listed on the leaderboard so the community has a complete picture of the tool landscape. If your tool was skipped and you want to be evaluated, submit your API with a note about how to obtain credentials.

---