> ## Documentation Index
> Fetch the complete documentation index at: https://lightsage.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Create and manage reusable coding-agent simulations from the Lightsage CLI.

An eval stores one or more prompts, success criteria, optional agents and
runtime tools, a persona, and the number of times to repeat each prompt-agent
pair.

Use [Eval runs](/docs/cli-reference/eval-runs) to execute an eval and [Running
evals](/docs/guides/running-evals) for an end-to-end workflow.

## Eval commands

| Command | Description |
| - | - |
| `lightsage evals list` | List evals. |
| `lightsage evals create` | Create an eval. |
| `lightsage evals retrieve --eval-id <eval_id>` | Retrieve one eval. |
| `lightsage evals update --eval-id <eval_id>` | Update one eval. |
| `lightsage evals delete --eval-id <eval_id>` | Delete one eval. |

## Create an eval

```bash theme={null}
lightsage evals create \
  --name "Lightsage interface simulation" \
  --prompt "Use the Lightsage CLI to create an eval." \
  --prompt "Use the Lightsage MCP server to retrieve an eval." \
  --judge "The requested workflow completes successfully." \
  --persona "Platform engineer" \
  --agent codex:gpt-5-4 \
  --agent claude-code:claude-opus-4-8 \
  --cli "brew install lightsagehq/tools/lightsage" \
  --mcp @lightsage-mcp.json \
  --setup-command "npm install" \
  --cleanup-command "npm run db:cleanup" \
  --runs 2 \
  --tag smoke-test
```

Repeat `--prompt`, `--judge`, `--agent`, `--cli`, `--mcp`, `--setup-command`,
`--cleanup-command`, and `--tag` once per value. `--persona` is one shared
value. The total attempt count is `prompts × agents × runs` and cannot exceed
100\.

A `--cli` value can be a saved CLI UUID or an installation command. An `--mcp`
value can be a saved MCP UUID, an inline JSON object, or `@path/to/file.json`.
Creating an eval does not accept or store secret environment values.

Setup commands run sequentially after the repository and declared tools are
prepared but before the agent starts. Cleanup commands run best-effort after
result capture and before sandbox deletion.

## List and retrieve evals

```bash theme={null}
lightsage evals list --output-format json
lightsage evals retrieve --eval-id eval_123 --output-format json
```

Responses use the same public fields: `prompts`, `judges`, `persona`, `agent`,
`clis`, `mcps`, `setup_commands`, `cleanup_commands`, `runs`, and `tags`.

## Update an eval

```bash theme={null}
lightsage evals update \
  --eval-id eval_123 \
  --body '{
    "prompts": ["Run the narrower scenario."],
    "agent": ["codex:gpt-5-4"],
    "setup_commands": ["npm install"],
    "cleanup_commands": ["npm run db:cleanup"],
    "runs": 3
  }'
```

Omitted fields remain unchanged. Empty `agent`, `clis`, `mcps`,
`setup_commands`, `cleanup_commands`, or `tags` arrays clear those saved
values. Send `persona: null` in the request body to clear the saved persona.

## Delete an eval

```bash theme={null}
lightsage evals delete --eval-id eval_123
```

Deleting an eval also deletes its saved configurations.

## Saved configurations

Configurations remain available for reusable repositories, environment
profiles, skills, and legacy tool selections:

```bash theme={null}
lightsage evals configurations create \
  --eval-id eval_123 \
  --name "Configured runtime" \
  --repository-id repo_123 \
  --environment-id env_123 \
  --skill-ids skill_123 \
  --skills resend/resend-skills
```

The configuration is optional when the saved eval already contains everything
the run needs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.