Skip to main content
Use GitHub PR evals to test documentation changes against a staging docs URL before merging. Lightsage runs the configured custom evals for the connected docs source and stores the results in your eval history.

​
Commands

Put the command on the first non-empty line of a pull request comment.

​
Run evals

1

Confirm the pull request has a staging docs URL

The URL can appear in the PR body, a PR comment, or a deployment attached to the pull request.
2

Comment on the pull request

Add the run command:
3

Wait for Lightsage to reply

Lightsage posts a comment when the evals are queued or when it needs more information.
4

Review the eval history

In Lightsage, open Custom evals and filter the history source to PR bot.

​
Include the docs URL in the command comment

If the pull request has more than one preview URL, include the docs URL in the same comment as the command:
Lightsage uses that URL for the eval run.

​
How Lightsage finds the docs URL

Lightsage searches the pull request context for a docs URL. It checks:
  • The /lightsage run evals comment.
  • The pull request body.
  • Other pull request comments.
  • Deployments attached to the pull request.
If Lightsage cannot find a docs URL, it does not queue evals. It posts a comment explaining that a staging docs URL is required.

​
Permissions

The comment author must have write, maintain, or admin access to the repository. This prevents drive-by comments from running evals and using credits. The repository also needs the Lightsage GitHub App installed and connected to a Lightsage docs source.

​
Check status

Use /lightsage status to see the latest Lightsage eval activity for the pull request:
Lightsage replies with the latest queued or running eval status it can find for that PR.

​
Troubleshooting

Add the staging docs URL to the /lightsage run evals comment, the PR body, or a PR comment. Then run the command again.
Ask someone with write, maintain, or admin access to run the command, or update the comment author’s repository permissions.
Configure custom evals for the connected docs source in Lightsage, then run the command again.
Run the command again and include the intended URL directly in the command comment.