Skip to main content
Lightsage Docs Agent checks whether AI coding agents can use your documentation changes before they ship. It runs API Performance evals against a staging docs URL from a GitHub pull request, then records the results in Lightsage. Use it when you want to catch agent-facing documentation regressions during review. A run can test whether agents can find the right page, understand the workflow, write the expected code, and complete the task with your API.

​
How it works

1

Open a docs pull request

Create a pull request for the documentation change you want to test.
2

Deploy preview docs

Make sure the pull request has a staging or preview docs URL. Lightsage can use URLs from PR comments, the PR body, and deployments.
3

Ask Lightsage to run evals

Comment /lightsage run evals on the pull request.
4

Review the results

Open Custom evals in Lightsage and filter history by PR bot to review the runs from GitHub.

​
What Lightsage evaluates

Lightsage runs the custom evals configured for the connected docs source. Each eval uses the staging docs URL as the documentation source for that run. Common PR checks include:
  • Whether a coding agent can follow a getting started guide.
  • Whether a changed SDK page still leads to working code.
  • Whether a new endpoint guide has enough context for an agent to make a correct request.
  • Whether a docs change introduces ambiguity that causes agents to call the wrong endpoint.

​
Requirements

To run docs evals from GitHub:
  • Install the Lightsage GitHub App on the repository.
  • Connect the repository to a Lightsage docs source.
  • Configure custom evals for that docs source.
  • Make a staging docs URL available in the pull request.
  • Comment from a GitHub account with write, maintain, or admin access to the repository.
Lightsage does not queue a run when it cannot find a docs URL. It posts a pull request comment with the reason instead.

​
Next step

See Run docs evals from GitHub for the pull request commands and URL discovery behavior.