> ## Documentation Index
> Fetch the complete documentation index at: https://lightsage.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# How Devtool Arena scores are calculated
> Learn how Eval Score and Discovery Score combine into an overall score, what the A–D grade thresholds mean, and what the checklist columns measure.
Every tool on Devtool Arena receives an overall score from 0 to 100. That score combines two components — Eval Score and Discovery Score — captured during a live, end-to-end evaluation run by Claude Code using a real production API key. Understanding how those two components interact helps you identify exactly which levers to pull to improve your ranking.
## The two score components
### Eval Score
Eval Score measures whether the AI coding agent successfully completed the assigned task end-to-end. For each tool, Lightsage defines a realistic developer task — for example, "send a transactional email," "perform a vector similarity search," or "run an inference request" — and the agent is given a real API key and a fresh environment to work in.
The Eval Score reflects the agent's task completion rate across multiple runs. A score of 100 means the agent completed the task without getting stuck. A lower score indicates the agent failed part or all of the task: it may have generated code that returned errors it could not recover from, called the wrong endpoints, or failed to authenticate.
**What drives a high Eval Score:**
* A quick-start guide that works end-to-end without modification
* Descriptive, specific error messages that tell agents what went wrong and how to fix it
* Consistent, typed API responses (TypeScript definitions, OpenAPI specs, JSON Schema)
* Minimal required setup before the first successful call
### Discovery Score
Discovery Score measures how easily the agent found and understood your API without hand-holding. When a task involves your API, does the agent know your tool exists, locate the right documentation, and correctly understand what your API does — all on its own?
Discovery is influenced by signals that help agents orient themselves before they write a single line of code: your `llms.txt` file, entries in Context7, whether AI crawlers can reach your docs, and how clearly your documentation explains what your API does (and does not do).
**What drives a high Discovery Score:**
* A published `llms.txt` file with "when to use" and "when NOT to use" sections
* AI crawlers allowed in `robots.txt`
* Documentation indexed in Context7
* Clear, scannable API reference pages that agents can parse quickly
* Comparison pages that help agents understand your positioning
## The six metrics collected per run
In addition to the two component scores, Devtool Arena records four efficiency metrics on every run:
| Metric | What it measures | Why it matters |
| ------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| **Eval Score** | Task completion rate | Primary signal — did the agent succeed? |
| **Discovery Score** | Agent-unaided API discoverability | Did the agent find you without being told? |
| **Cost** | Claude API spend per run (USD) | Proxy for complexity. High cost often means the agent struggled and iterated excessively. |
| **Tool Calls** | Number of tool calls to complete the task | Fewer calls = simpler, better-documented API. Top performers average 3–12 calls; struggling APIs exceed 50. |
| **Errors** | Number of errors during the run | Each error is a recovery event. High error counts raise cost and time even if the agent eventually succeeds. |
| **Time** | Wall-clock time to complete the task | Directly correlated with cost and tool calls. Top performers finish in under 2 minutes; poor performers can exceed 10 minutes. |
Multiple runs are averaged together to produce a stable score. A single flaky run does not permanently determine your ranking — the score reflects the central tendency across all completed evaluations for your tool.
## The grade system
The overall score maps to a letter grade shown in the **Grade** column:
| Grade | Score range | What it means |
| ----- | ------------ | -------------------------------------------------------------------------------------------------------------------------- |
| **A** | 90 and above | The agent consistently completes the task and discovers the API reliably. Reserved for top-tier performers. |
| **B** | 75–89 | The agent usually succeeds with few errors. Most strong performers sit in this band. |
| **C** | 60–74 | The agent completes the task inconsistently or incurs significant errors. Room for improvement on both Eval and Discovery. |
| **D** | Below 60 | The agent frequently fails or cannot discover the API without significant struggle. Fundamental issues to address. |
As of May 2026, the majority of ranked tools score in the B–C range. A grades are rare — achieving one requires both near-perfect task completion and high discoverability.
### Real leaderboard examples
The following table shows how score components correlate with real outcomes (API leaderboard, May 2026):
| Tool | Score | Grade | Eval | Discovery | Cost | Calls | Errors |
| ---------- | ----- | ----- | ---- | --------- | ------ | ----- | ------ |
| Firecrawl | 86 | B | 81 | 100 | \$0.20 | 10 | 2 |
| OpenRouter | 84 | B | 84 | 86 | \$0.08 | 3 | 0 |
| Jina AI | 84 | B | 90 | 71 | \$0.10 | 4 | 0 |
| Datadog | 83 | B | 76 | 100 | \$0.31 | 11 | 2 |
| Stripe | 82 | B | 75 | 100 | \$0.07 | 5 | 1 |
| WorkOS | 74 | C | 64 | 100 | \$0.76 | 28 | 5 |
| Deepgram | 47 | D | 25 | 100 | \$0.20 | 16 | 6 |
Notice that a Discovery Score of 100 does not guarantee a high overall score — Deepgram's perfect discoverability is undermined by a low Eval Score of 25. Conversely, Jina AI achieves a strong Eval Score of 90 despite a lower Discovery Score of 71. Both components matter.
## The checklist columns
The seven checklist columns (C7, llms, MCP, SDK, API, Skills, CLI) are binary presence checks, not scored metrics. They record whether each asset exists in the tool's publicly available resources at evaluation time.
| Column | What is checked |
| ---------- | --------------------------------------------------------------------------------------------------- |
| **C7** | The tool is indexed in Context7, which agents query for up-to-date library documentation. |
| **llms** | The tool publishes an `llms.txt` file at its root domain. |
| **MCP** | The tool provides a Model Context Protocol server that agents can install and use. |
| **SDK** | The tool offers at least one official SDK (any language). |
| **API** | The tool exposes a REST or similar programmatic API. |
| **Skills** | The tool has published Agent Skills, enabling agents to invoke it through skill-calling interfaces. |
| **CLI** | The tool ships an official command-line interface. |
Checklist completeness correlates strongly with higher scores. In our analysis, tools with `llms.txt` and CLI support score an average of 18 points higher than those without. Tools with working MCP servers dominate their categories in the MCP leaderboard. The relationship is not causal by default — a broken MCP server helps no one — but high-quality presence across multiple channels consistently separates top performers from the rest.
If your Discovery Score is low but your Eval Score is reasonable, start with the checklist: add an `llms.txt` file, ensure AI crawlers are allowed in `robots.txt`, and submit your documentation to Context7. These changes tend to move Discovery Score the fastest.
## Skipped entries
Tools listed as **Skipped** (shown with `—` in score columns) were not evaluated because Lightsage could not obtain a working API key during the evaluation window. Common reasons:
* The API requires a paid plan before issuing credentials
* Signup requires a credit card
* Access requires official company certification or business verification
* The tool is open-source and self-hosted with no hosted API available
* The MCP server requires browser-based OAuth that cannot be automated
Skipped tools are still listed on the leaderboard so the community has a complete picture of the tool landscape. If your tool was skipped and you want to be evaluated, [submit your API](/docs/leaderboard/submit-api) with a note about how to obtain credentials.
***
Ready to get evaluated? Learn what to prepare to maximize your score before you submit.
See how scores have shifted over time and which tools have improved most.