Evals

An eval stores one or more prompts, success criteria, optional agents and runtime tools, a persona, and the number of times to repeat each prompt-agent pair. Use Eval runs to execute an eval and Running evals for an end-to-end workflow.

Eval commands

Command	Description
`lightsage evals list`	List evals.
`lightsage evals create`	Create an eval.
`lightsage evals retrieve --eval-id <eval_id>`	Retrieve one eval.
`lightsage evals update --eval-id <eval_id>`	Update one eval.
`lightsage evals delete --eval-id <eval_id>`	Delete one eval.

Create an eval

lightsage evals create \
  --name "Lightsage interface simulation" \
  --prompt "Use the Lightsage CLI to create an eval." \
  --prompt "Use the Lightsage MCP server to retrieve an eval." \
  --judge "The requested workflow completes successfully." \
  --persona "Platform engineer" \
  --agent codex:gpt-5-4 \
  --agent claude-code:claude-opus-4-8 \
  --cli "brew install lightsagehq/tools/lightsage" \
  --mcp @lightsage-mcp.json \
  --setup-command "npm install" \
  --cleanup-command "npm run db:cleanup" \
  --runs 2 \
  --tag smoke-test

Repeat --prompt, --judge, --agent, --cli, --mcp, --setup-command, --cleanup-command, and --tag once per value. --persona is one shared value. The total attempt count is prompts × agents × runs and cannot exceed 100. A --cli value can be a saved CLI UUID or an installation command. An --mcp value can be a saved MCP UUID, an inline JSON object, or @path/to/file.json. Creating an eval does not accept or store secret environment values. Setup commands run sequentially after the repository and declared tools are prepared but before the agent starts. Cleanup commands run best-effort after result capture and before sandbox deletion.

List and retrieve evals

lightsage evals list --output-format json
lightsage evals retrieve --eval-id eval_123 --output-format json

Responses use the same public fields: prompts, judges, persona, agent, clis, mcps, setup_commands, cleanup_commands, runs, and tags.

Update an eval

lightsage evals update \
  --eval-id eval_123 \
  --body '{
    "prompts": ["Run the narrower scenario."],
    "agent": ["codex:gpt-5-4"],
    "setup_commands": ["npm install"],
    "cleanup_commands": ["npm run db:cleanup"],
    "runs": 3
  }'

Omitted fields remain unchanged. Empty agent, clis, mcps, setup_commands, cleanup_commands, or tags arrays clear those saved values. Send persona: null in the request body to clear the saved persona.

Delete an eval

lightsage evals delete --eval-id eval_123

Deleting an eval also deletes its saved configurations.

Saved configurations

Configurations remain available for reusable repositories, environment profiles, skills, and legacy tool selections:

lightsage evals configurations create \
  --eval-id eval_123 \
  --name "Configured runtime" \
  --repository-id repo_123 \
  --environment-id env_123 \
  --skill-ids skill_123 \
  --skills resend/resend-skills

The configuration is optional when the saved eval already contains everything the run needs.

Getting Started

Visibility

Eval resources

Evals

Eval commands

Create an eval

List and retrieve evals

Update an eval

Delete an eval

Saved configurations

​Eval commands

​Create an eval

​List and retrieve evals

​Update an eval

​Delete an eval

​Saved configurations

Eval commands

Create an eval

List and retrieve evals

Update an eval

Delete an eval

Saved configurations