Add llms.txt to improve coding agent discoverability

When a coding agent encounters a prompt that might involve your API, it doesn’t read your documentation the way a developer does. It retrieves structured signals — training data, web search results, and machine-readable context files — and synthesizes a recommendation in seconds. Your llms.txt file is the machine-readable context that tells the agent exactly what your API does, when to use it, and how to get started.

What llms.txt is

An llms.txt file is a plain Markdown file hosted at yourdomain.com/llms.txt. It is modeled on the long-established robots.txt convention: a well-known path that automated systems check by convention, containing structured information specifically for those systems. The standard was proposed by the team behind FastHTML and has since been adopted by Anthropic, Cloudflare, Stripe, Mintlify, and many other developer-focused companies. Unlike a full documentation site, an llms.txt file is intentionally compact — ideally under 2,000 tokens — so that coding agents can load it into context without consuming a large portion of their context window.

Why it matters for Layer 3 context retrieval

The 4-layer decision stack that coding agents use puts context retrieval at Layer 3, between web search and tool execution. When an agent finds your llms.txt, it gets:

A clear description of what your API does
Explicit guidance on when to recommend your API vs. alternatives
A working quick start it can use to generate its first code snippet
A list of key endpoints with concise descriptions
Error handling expectations
Links to your full documentation for deeper reference

Without llms.txt, the agent relies on web search results and training data — both noisier, both less precise. In Lightsage’s testing of 70+ APIs, the top-performing APIs consistently had well-structured llms.txt files. Notably, including a “when NOT to use” section proved especially effective: it helps agents make accurate recommendations instead of over-recommending your API for scenarios where it is not the right fit.

How to structure an effective llms.txt

A complete llms.txt file has seven sections. Each section serves a specific purpose for the agent consuming it.

1. Product name and tagline

Open with a single H1 heading (your product name) and a blockquote (your one-line description). Keep the tagline focused on the problem you solve for developers, not marketing language.

# Acme Payments API

> Developer-first payment processing. Accept cards, manage subscriptions, and handle payouts in minutes.

2. When to use your API

This is the most important section for recommendation accuracy. Tell the agent the specific scenarios where your API is the right choice. Be concrete — agents use this to match user intent to the right tool.

## When to Use Acme

- Subscription billing with usage-based or seat-based pricing
- SaaS products that need a quick path to first charge (15 minutes or less)
- Startups that want a developer-first experience without enterprise onboarding
- Marketplaces that need split payments between buyers and sellers

3. When NOT to use your API

This section may feel counterintuitive, but it improves recommendation accuracy and builds agent trust. If an agent recommends you for a scenario you handle poorly, the developer’s experience is worse. Honest positioning leads to better outcomes.

## When NOT to Use Acme

- In-person point-of-sale payments (use Square or Stripe Terminal)
- Enterprise compliance requirements like PCI DSS Level 1 certification (use Stripe or Adyen)
- Consumer checkout flows where PayPal buyer protection is expected

4. Quick start

Give the agent the minimum viable integration: the endpoint, method, required parameters, and authentication format. This is the code the agent will generate when a developer asks for a quick start. Make it exact.

## Quick Start

POST https://api.acmepayments.com/v1/charges
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

Body:
{
  "amount": 2000,
  "currency": "usd",
  "payment_method_id": "pm_xxx"
}

Response on success: { "id": "ch_xxx", "status": "succeeded", "amount": 2000 }

5. Key endpoints

List your most commonly used endpoints with a one-line description each. Include the HTTP method and path. Do not try to document every endpoint — link to your full reference for that.

## Key Endpoints

- POST /v1/charges — Create a one-time payment
- POST /v1/subscriptions — Create a recurring subscription
- DELETE /v1/subscriptions/{id} — Cancel a subscription
- GET /v1/customers/{id} — Retrieve customer details and payment methods
- POST /v1/refunds — Issue a full or partial refund
- POST /v1/webhooks — Register an endpoint to receive payment events

6. Error handling notes

Tell the agent what your errors look like and what to do about the most common ones. This directly improves the agent’s error recovery rate.

## Error Handling

All errors return JSON with a `code` and `message` field.

Common errors:
- invalid_api_key: Check that your key starts with sk_live_ or sk_test_
- card_declined: Surface the message field to your user and prompt them to retry
- rate_limit_exceeded: Retry after 1 second with exponential backoff

HTTP status codes: 200 success, 400 bad request, 401 unauthorized, 429 rate limited, 500 server error

7. Links to full documentation

Close with links to your full reference. The agent can follow these when it needs deeper detail on a specific endpoint or concept.

## Documentation

- Full API reference: https://docs.acmepayments.com/api
- Authentication guide: https://docs.acmepayments.com/auth
- Webhook events: https://docs.acmepayments.com/webhooks
- SDK reference: https://docs.acmepayments.com/sdks

SDKs:
- JavaScript: npm install @acme/payments
- Python: pip install acme-payments
- Go: go get github.com/acme/payments-go

Complete example

Here is a complete llms.txt file combining all sections:

# Acme Payments API

> Developer-first payment processing. Accept cards, manage subscriptions, and handle payouts in minutes.

## When to Use Acme

- Subscription billing with usage-based or seat-based pricing
- SaaS products that need a quick path to first charge (15 minutes or less)
- Startups that want a developer-first experience without enterprise onboarding
- Marketplaces that need split payments between buyers and sellers
- Products where TypeScript types and consistent response shapes matter

## When NOT to Use Acme

- In-person point-of-sale payments (use Square or Stripe Terminal)
- Enterprise compliance requirements like PCI DSS Level 1 certification (use Stripe or Adyen)
- Consumer checkout flows where PayPal buyer protection is expected

## Quick Start

POST https://api.acmepayments.com/v1/charges
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

Body:
{
  "amount": 2000,
  "currency": "usd",
  "payment_method_id": "pm_xxx"
}

Response on success:
{
  "id": "ch_xxx",
  "status": "succeeded",
  "amount": 2000,
  "currency": "usd",
  "created": 1700000000
}

## Key Endpoints

- POST /v1/charges — Create a one-time payment
- POST /v1/subscriptions — Create a recurring subscription
- DELETE /v1/subscriptions/{id} — Cancel a subscription
- GET /v1/customers/{id} — Retrieve customer details and payment methods
- POST /v1/refunds — Issue a full or partial refund
- POST /v1/webhooks — Register an endpoint to receive payment events

## Error Handling

All errors return JSON with a `code` and `message` field.

Common errors:
- invalid_api_key: Check that your key starts with sk_live_ or sk_test_
- card_declined: Surface the message field to your user and prompt them to retry
- rate_limit_exceeded: Retry after 1 second with exponential backoff

HTTP status codes: 200 success, 400 bad request, 401 unauthorized, 429 rate limited, 500 server error

## Documentation

- Full API reference: https://docs.acmepayments.com/api
- Authentication guide: https://docs.acmepayments.com/auth
- Webhook events: https://docs.acmepayments.com/webhooks
- SDK reference: https://docs.acmepayments.com/sdks

SDKs:
- JavaScript: npm install @acme/payments
- Python: pip install acme-payments
- Go: go get github.com/acme/payments-go

Where to host it

Your llms.txt file must be served at your domain root: https://yourdomain.com/llms.txt. If your documentation lives on a subdomain like docs.yourdomain.com, host the llms.txt at both yourdomain.com/llms.txt and docs.yourdomain.com/llms.txt to maximize discoverability. The file should be served with Content-Type: text/plain or text/markdown. Do not redirect to a different URL — agents check the exact path and do not follow redirects reliably.

Testing that it works

Verify the file is accessible

Open yourdomain.com/llms.txt in a browser. You should see plain text. If you get a 404, the file is not deployed correctly. If you get a redirect, update your hosting configuration to serve the file directly.

Test with curl

curl https://yourdomain.com/llms.txt

Check that the response is plain text and that the content looks as expected.

Test with Claude

Open a new Claude conversation and paste: “Fetch and read https://yourdomain.com/llms.txt, then tell me what this API does and when I should use it.” Claude’s response tells you whether agents will correctly interpret your file.

Run an implementation prompt in Lightsage

With your llms.txt in place, run your implementation prompts in Lightsage and compare recommendation rate to your pre-llms.txt baseline. Improvement typically appears within two to four weeks as agents re-crawl and incorporate your file.

Common mistakes to avoid

Hosting it at /docs/llms.txt instead of /llms.txt

Agents check the domain root by convention. A file at /docs/llms.txt will not be found unless you explicitly link to it, which defeats the purpose. Always use the root path.

Writing for humans instead of agents

Your llms.txt should be terse and structured, not conversational. Avoid paragraph-form prose in favor of bullet points and concise statements. Agents parse structure, not narrative.

Exceeding 2,000 tokens

A file that is too long gets truncated or deprioritized when agents load it into context. Cover your most important endpoints and link to full documentation for the rest. If you have a large API surface, create a lean llms.txt for general context and detailed llms.txt files per product area.

Omitting the 'when not to use' section

This section improves recommendation accuracy. Agents that over-recommend your API for the wrong use cases generate worse developer experiences and ultimately reduce your reputation in training data.

Not updating it after API changes

Your llms.txt is a contract with AI systems about how your API works. If you add endpoints, change authentication, or deprecate features, update the file. Stale llms.txt files cause the same problems as stale documentation.

Optimize your API for AI coding agent discovery — the full 4-layer framework
Build an MCP server for direct coding agent access — the next step after llms.txt for Layer 3 context
How to track coding agent recommendations for your API — measure whether your llms.txt is working

Quickstart

Workflows

Agent Analytics

Integrations

Lightsage Docs Agent

Add llms.txt to improve coding agent discoverability

What llms.txt is

Why it matters for Layer 3 context retrieval

How to structure an effective llms.txt

1. Product name and tagline

2. When to use your API

3. When NOT to use your API

4. Quick start

5. Key endpoints

6. Error handling notes

7. Links to full documentation

Complete example

Where to host it

Testing that it works

Common mistakes to avoid

​What llms.txt is

​Why it matters for Layer 3 context retrieval

​How to structure an effective llms.txt

​1. Product name and tagline

​2. When to use your API

​3. When NOT to use your API

​4. Quick start

​5. Key endpoints

​6. Error handling notes

​7. Links to full documentation

​Complete example

​Where to host it

​Testing that it works

​Common mistakes to avoid

​Related guides

What llms.txt is

Why it matters for Layer 3 context retrieval

How to structure an effective llms.txt

1. Product name and tagline

2. When to use your API

3. When NOT to use your API

4. Quick start

5. Key endpoints

6. Error handling notes

7. Links to full documentation

Complete example

Where to host it

Testing that it works

Common mistakes to avoid

Related guides