> ## Documentation Index
> Fetch the complete documentation index at: https://lightsage.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Optimize your API for AI coding agent discovery

> Work through the 4-layer coding agent decision stack — training data, web search, context retrieval, and tool execution — to boost your recommendation rate.

Coding agents don't pick APIs the way developers do. A developer reads your docs, evaluates a few options, and makes a decision. A coding agent runs a multi-step process that pulls from training data, performs web searches, retrieves structured context, and ultimately executes code — often in under a minute. If your API isn't optimized for each step in that process, you lose the recommendation before any human is involved.

This guide walks through the 4-layer decision stack and what you can do at each layer to improve your position.

## The 4-layer decision stack

Every recommendation from a coding agent — whether Claude Code, Cursor, or GitHub Copilot — passes through four layers:

| Layer                    | What happens                                        | What influences it                                   |
| ------------------------ | --------------------------------------------------- | ---------------------------------------------------- |
| **1. Training data**     | The agent's base knowledge from pre-training        | Your docs, GitHub repos, Stack Overflow, blog posts  |
| **2. Web search**        | Real-time retrieval (\~20% of prompts trigger this) | robots.txt, SSR, comparison pages, content freshness |
| **3. Context retrieval** | Structured access to your API description           | llms.txt, MCP servers, Claude skills                 |
| **4. Tool execution**    | The agent writes and runs code using your API       | Error messages, response typing, quick start quality |

Most teams optimize only for Layer 1 — hoping their docs were crawled before the model's training cutoff — and ignore the three layers where you have direct, immediate control. That is why smaller, newer libraries sometimes beat dominant players: they win on Layers 3 and 4 while incumbents coast on training data.

## Layer 1: Training data

The base model underlying each coding agent learned about your API from documentation sites, GitHub repositories, Stack Overflow, blog posts, and package registries crawled before its knowledge cutoff. If your library existed and was well-documented before the cutoff, you have a baseline advantage.

**What you can do:**

* **Maintain a strong GitHub README.** This is often the first result for your library name and is heavily represented in training data. Keep it current, include working code examples, and state clearly what your API does and when to use it.
* **Build Stack Overflow presence.** Answer questions about your API. Questions and accepted answers are disproportionately represented in training sets.
* **Publish technical blog posts.** Posts that walk through real use cases — especially on developer-focused platforms — are indexed and included in training data.
* **Keep npm/PyPI metadata accurate.** Package descriptions are included in training data for many models.

### How to verify Layer 1

Ask Claude or ChatGPT (with web search disabled) about your API:

* "When should I use \[Your API] vs \[main competitor]?"
* "Show me a quick start for \[Your API]"

If the responses are inaccurate, outdated, or favor competitors, your Layer 1 presence is weak. You can't fix the past training data, but improvements you make today will be included in future model versions — and Layers 2-4 can compensate in the meantime.

## Layer 2: Web search

According to Vercel's AEO tracking research, coding agents perform web searches in roughly 20% of prompts. When the agent searches, it looks for official documentation, comparison content, recent tutorials, and GitHub READMEs. If your docs don't appear in those results — or appear but contain content the agent can't read — you're invisible for those prompts.

### The robots.txt problem

The most common and most damaging Layer 2 failure is blocking AI crawlers in your robots.txt. Many documentation sites added crawler blocks during a wave of concern about AI scraping, and those blocks are still in place.

**What not to do:**

```text theme={null}
# robots.txt — blocks AI discovery entirely
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /
```

**What to do instead:**

```text theme={null}
# robots.txt — allows AI crawlers, blocks only internal pages
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

# Block admin and internal-only paths
User-agent: *
Disallow: /admin/
Disallow: /internal/
```

This fix takes five minutes and has high impact. It is the first thing to check.

### Other Layer 2 fixes

**Ensure server-side rendering.** Crawlers don't execute JavaScript. If your documentation is built with a client-side-only framework, the crawler sees an empty page. Make sure your docs render meaningful content server-side.

**Create comparison pages.** When an agent searches "X vs Y", it looks for pages that directly answer that query. If you don't have a page titled "\[Your API] vs \[Competitor]", your competitor's page defines the narrative. In Lightsage's testing of 70+ APIs, APIs with dedicated comparison pages consistently outperformed those without.

**Use specific page titles.** "Acme API Quick Start" is indexed and found. "Getting Started" is not distinguishable from thousands of other pages.

### How to verify Layer 2

* Check your robots.txt directly: `curl https://yourdomain.com/robots.txt`
* Search your API name plus common use cases in a web browser and note your ranking
* Disable JavaScript in your browser and visit your docs — if content disappears, crawlers see the same empty page

## Layer 3: Context retrieval

This is where most API teams have zero presence — and where the biggest opportunities exist. Beyond web search, coding agents can access structured context about your API through llms.txt files, MCP servers, and Claude skills.

### llms.txt

An llms.txt file is a machine-readable Markdown file at your domain root that gives AI systems a curated overview of your API. When an agent encounters a relevant prompt and finds your llms.txt, it has immediate context about what your API does, when to use it, and how to make its first call.

See [Add llms.txt to improve coding agent discoverability](/docs/guides/llms-txt) for a full implementation guide.

**Time to implement:** 30 minutes\
**Impact:** High

### MCP servers

The Model Context Protocol (MCP) is an open standard that lets Claude Code connect directly to your API. When your API has an MCP server, the agent can discover it, call your endpoints directly, and verify that responses work — before recommending anything to the developer. That changes the recommendation from "I think this might work" to "I connected to this API and it worked."

See [Build an MCP server for direct coding agent access](/docs/guides/mcp-servers) for a full implementation guide.

**Time to implement:** 1-2 weeks\
**Impact:** High

### Claude skills

Skills are packaged workflows that users install into Claude Code. A skill for your API pre-loads preferred configuration patterns, common workflows, and error handling guidance. When a developer has your skill installed, Claude Code will prefer your API for relevant tasks because the context is already available.

**Time to implement:** 2-4 weeks\
**Impact:** Medium

### How to verify Layer 3

* Visit `yourdomain.com/llms.txt` — a 404 means you don't have one
* Search the [MCP server registry](https://github.com/modelcontextprotocol/servers) for your API name — 85,000+ stars and growing
* Search Claude Code skills marketplaces for your API name

## Layer 4: Tool execution

This is where recommendations become reality. When a coding agent recommends your SDK, it typically writes `npm install your-package`, generates integration code, and sometimes executes that code to verify it works. If any step in that chain fails — install, import, API call, or error handling — the agent may switch to an alternative.

### What goes wrong at Layer 4

| Failure mode                       | What happens                                      | Agent response           |
| ---------------------------------- | ------------------------------------------------- | ------------------------ |
| Install fails                      | Package not found or dependency conflict          | Recommends alternative   |
| Import fails                       | Wrong module path or missing export               | Suggests competitor      |
| API call fails with generic error  | "Error 401" or "Internal server error"            | Switches recommendation  |
| API call fails with specific error | "Invalid API key. Expected format: sk\_live\_xxx" | Recovers and retries     |
| Unclear response shape             | Agent generates code against wrong field names    | Fails silently or errors |

### Error message quality matters more than you think

In Lightsage's testing of 70+ APIs, error message specificity had the largest impact on agent error recovery rate:

| Error message style                                                | Agent recovery rate |
| ------------------------------------------------------------------ | ------------------- |
| "Invalid API key format. Expected: sk\_live\_xxx or sk\_test\_xxx" | 89%                 |
| "Authentication failed. Check your API key."                       | 67%                 |
| "Error 401"                                                        | 34%                 |
| "Internal server error"                                            | 12%                 |

Specific errors let agents self-correct. Generic errors send them into debugging loops that end with a competitor recommendation.

### Layer 4 improvements

**Write a working quick start.** The exact code the agent will generate should be in your docs as a copy-pasteable block that works without modification. Test it against Claude Code and Codex directly.

**Use typed responses.** TypeScript definitions, JSON Schema, and OpenAPI specs help agents understand your API's response shape and generate correct code. In Lightsage's benchmarks, APIs with typed responses had 23% higher tool call success rates.

**Consistent naming.** Don't make the agent guess import paths or method names. If your npm package is `@yourco/api` but your import is `require('yourco')`, agents will get it wrong.

**Single-step auth.** Complex multi-step authentication confuses agents. If your API requires multiple setup steps before making the first call, document each step explicitly in your quick start and in your llms.txt.

### How to verify Layer 4

Ask Claude Code to implement your API's primary use case:

```text theme={null}
Add [Your API] to my Next.js app. My API key is in .env as YOURAPI_KEY.
```

Watch what happens: does the code compile, does the API call succeed, and if it fails, can the agent recover? Run the same test with Codex and Cursor — behavior can differ significantly across agents.

## Optimization priority table

If you're starting from scratch, work through these in order. Each later layer builds on the ones below it:

| Priority | Action                           | Effort    | Impact      |
| -------- | -------------------------------- | --------- | ----------- |
| 1        | Fix robots.txt for AI crawlers   | 5 min     | High        |
| 2        | Add llms.txt with positioning    | 30 min    | High        |
| 3        | Create "X vs Y" comparison pages | 2-4 hours | Medium-High |
| 4        | Improve error messages           | 1-2 days  | Medium      |
| 5        | Build MCP server                 | 1-2 weeks | High        |
| 6        | Publish Claude skill             | 2-4 weeks | Medium      |

<Tip>
  Start with the robots.txt check before anything else. It is the most common fixable problem and the one with the fastest time-to-impact. A blocked crawler makes every other optimization invisible.
</Tip>

## How smaller libraries can beat dominant players

The 4-layer model explains a counterintuitive pattern: smaller, newer libraries sometimes beat dominant players in coding agent recommendations.

| Layer             | Large player advantage                   | Small player advantage                |
| ----------------- | ---------------------------------------- | ------------------------------------- |
| Training data     | More historical content, higher coverage | None                                  |
| Web search        | Better SEO, more backlinks               | Can target specific long-tail queries |
| Context retrieval | Often missing llms.txt and MCP           | Can ship these quickly                |
| Tool execution    | More edge cases, legacy patterns         | Clean modern API, better DX           |

A smaller library with a well-crafted llms.txt, a working MCP server, and descriptive error messages can outrank a dominant player that is relying entirely on its training data presence. This is where your leverage is.

## Related guides

* [Add llms.txt to improve coding agent discoverability](/docs/guides/llms-txt)
* [Build an MCP server for direct coding agent access](/docs/guides/mcp-servers)
* [Detect AI agent visits with agent tracking middleware](/docs/guides/agent-tracking-middleware)
* [How to track coding agent recommendations for your API](/docs/guides/track-coding-agents)