> ## Documentation Index
> Fetch the complete documentation index at: https://lightsage.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Running evals

> Create, fan out, monitor, and inspect coding-agent evals with the Lightsage CLI.

Use `lightsage evals run` to create and start an eval in one command. One
request can test multiple prompts across multiple agents and repeat every pair.

> [!NOTE]
>
> The number of attempts is `prompts × agents × runs`. One request can create
> at most 100 attempts.

## Set up the CLI

Authenticate once, then list the reusable resources available to your
organization:

```bash theme={null}
lightsage auth login
lightsage agents list
lightsage repositories list
lightsage clis list
lightsage mcp-servers list
```

## Create and run directly

Use the direct form for a one-off eval:

```bash theme={null}
lightsage evals run \
  --name "Lightsage interface simulation" \
  --prompt "Use the Lightsage CLI to create an eval." \
  --prompt "Use the Lightsage MCP server to retrieve an eval." \
  --judge "The requested workflow completes successfully." \
  --persona "Platform engineer" \
  --agent codex:gpt-5-4 \
  --agent claude-code:claude-opus-4-8 \
  --cli "brew install lightsagehq/tools/lightsage" \
  --mcp '{
    "name": "lightsage",
    "type": "http",
    "url": "https://mcp.lightsage.com/mcp",
    "headers": {
      "X-Lightsage-Api-Key": "${LIGHTSAGE_API_KEY}"
    }
  }' \
  --setup-command "npm install" \
  --setup-command "npm run db:migrate" \
  --cleanup-command "npm run db:cleanup" \
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \
  --runs 2 \
  --output-format json
```

This command creates eight attempts: two prompts × two agents × two runs. The
persona applies to every attempt without adding another fan-out dimension.

The response contains the persisted `eval_id` and the parent run `id`. Keep the
run ID to monitor the full matrix.

### Attach multiple CLIs and MCP servers

Repeat `--cli` and `--mcp` in request order. Each value can reference a saved
resource or define one inline:

```bash theme={null}
lightsage evals run \
  --prompt "Exercise every available Lightsage interface." \
  --judge "The workflow completes successfully." \
  --agent codex:gpt-5-4 \
  --cli 230f0c71-ed91-4a4a-9932-9cef267411bc \
  --cli "npm install --global @example/tools" \
  --mcp 550e8400-e29b-41d4-a716-446655440000 \
  --mcp @lightsage-mcp.json \
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \
  --runs 3
```

For `--cli`, a UUID references a saved CLI. Any other value is an installation
command.

For `--mcp`, pass a saved UUID, one inline JSON object, or `@path/to/file.json`.
An MCP file can contain either the canonical object with a `name` field or a
one-server MCP config map:

```json theme={null}
{
  "lightsage": {
    "type": "http",
    "url": "https://mcp.lightsage.com/mcp",
    "headers": {
      "X-Lightsage-Api-Key": "${LIGHTSAGE_API_KEY}"
    }
  }
}
```

The `${LIGHTSAGE_API_KEY}` reference is resolved from repeated `--env KEY=VALUE`
fields when the sandbox starts. Lightsage does not return or add the resolved
value to public run metadata.

Saved MCP IDs and inline MCP definitions use the same runtime path after
resolution. Lightsage translates the portable MCP definition into the native
configuration for Claude Code, Codex, Cursor, GitHub Copilot, Antigravity,
Gemini CLI, OpenClaw, OpenCode, Hermes, Pi, Kilo, and Grok.

Use repeatable `--skill owner/repository` values to install public agent skills
before the selected harness starts. Skill installation and both saved and
inline CLI installation commands run during the same pre-execution setup.

### Run setup and cleanup commands

Repeat `--setup-command` to run repository-specific preparation after CLI and
skill installation but before the agent starts. Setup commands run in order in
the prepared repository. A failed setup command stops the attempt before agent
execution.

Repeat `--cleanup-command` for best-effort teardown after the agent result is
captured and before the sandbox is deleted. Cleanup runs for successful and
failed agent attempts. If one cleanup command fails, Lightsage records the
failure and continues with the remaining cleanup commands without replacing
the attempt's primary result.

### Prompt an agent to run the eval

Use this prompt when you delegate the workflow to a coding agent:

```text Lightsage CLI theme={null}
Use the Lightsage CLI to directly create and run a one-off eval with
`lightsage evals run`.

Create a simple eval about crawling a public webpage about fish. Use
`codex:gpt-5-4`, include a clear success criterion, and run it once. Return the
created eval ID and run ID. Monitor the run until it reaches a terminal state,
then retrieve its analysis.
```

## Create a reusable eval

Save the simulation matrix when you expect to run it again:

```bash theme={null}
lightsage evals create \
  --name "SDK installation" \
  --prompt "Install the SDK and send a successful request." \
  --judge "The SDK is installed and the request succeeds." \
  --persona "Application developer" \
  --agent codex:gpt-5-4 \
  --cli "brew install lightsagehq/tools/lightsage" \
  --mcp @lightsage-mcp.json \
  --setup-command "npm install" \
  --cleanup-command "npm run db:cleanup" \
  --runs 2 \
  --tag onboarding
```

Saved evals can contain prompts, judges, agents, persona, CLIs, MCP servers,
setup commands, cleanup commands, repeat count, and tags. They do not store
inline secret values.

Create a saved configuration only when the eval also needs a reusable
repository, environment profile, skill set, or legacy resource selection:

```bash theme={null}
lightsage evals configurations create \
  --eval-id eval_123 \
  --name "Configured runtime" \
  --repository-id repo_123 \
  --environment-id env_123 \
  --skill-ids skill_123 \
  --skills resend/resend-skills
```

Run the saved defaults:

```bash theme={null}
lightsage evals run \
  --eval-id eval_123 \
  --configuration-id config_123 \
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY"
```

The configuration is optional. You can also override the saved prompts,
persona, agents, CLIs, MCP servers, setup commands, cleanup commands, or repeat
count for one launch:

```bash theme={null}
lightsage evals run \
  --eval-id eval_123 \
  --prompt "Run this narrower scenario." \
  --agent codex:gpt-5-4 \
  --setup-command "npm install" \
  --cleanup-command "npm run db:cleanup" \
  --runs 5
```

In the public API, send an explicit empty `setup_commands`, `cleanup_commands`,
`clis`, or `mcps` array to disable the saved collection for one launch.

## Monitor progress

Monitor all active eval runs:

```bash theme={null}
lightsage evals monitor
```

Monitor one run or retrieve a finite status response:

```bash theme={null}
lightsage evals monitor --run-id run_123
lightsage eval-runs retrieve --run-id run_123 --output-format json
```

## Inspect analysis

After the run reaches a terminal state, retrieve its persisted judge analysis:

```bash theme={null}
lightsage evals analysis --run-id run_123 --output-format json
```

Each attempt includes its prompt, prompt index, agent, and repeat index so you
can compare every matrix cell.

## Related reference

> **[Evals](/docs/cli-reference/evals)**
>
> Create reusable simulation definitions and configurations.

> **[Eval runs](/docs/cli-reference/eval-runs)**
>
> Start runs and monitor their progress.

> **[CLIs](/docs/cli-reference/clis)**
>
> Find saved CLI IDs that can be attached to an eval.

> **[MCP servers](/docs/cli-reference/mcp-servers)**
>
> Find saved MCP server IDs that can be attached to an eval.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.