Eval runs

lightsage evals run either runs a saved eval or creates and starts one directly. A run fans out across every prompt and agent, then repeats each pair according to --runs.

Commands

Command	Description
`lightsage evals run`	Start a saved or direct eval run.
`lightsage evals monitor`	Stream live progress for active runs.
`lightsage eval-runs list`	List eval runs.
`lightsage eval-runs retrieve --run-id <id>`	Retrieve or poll one run.
`lightsage evals analysis --run-id <id>`	Retrieve persisted judge analysis.

Create and run directly

lightsage evals run \
  --prompt "Use the CLI to create an eval." \
  --prompt "Use the MCP server to retrieve an eval." \
  --judge "The requested workflow succeeds." \
  --agent codex:gpt-5-4 \
  --agent claude-code:claude-opus-4-8 \
  --cli "brew install lightsagehq/tools/lightsage" \
  --mcp @lightsage-mcp.json \
  --setup-command "npm install" \
  --cleanup-command "npm run db:cleanup" \
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \
  --runs 2

Repeat --prompt, --judge, --agent, --cli, --mcp, --setup-command, --cleanup-command, --env, --skill, and --tag once per value. The response includes the created eval_id and the parent run id. Setup commands run sequentially before agent execution and fail the attempt fast if one exits unsuccessfully. Cleanup commands run best-effort after result capture and before sandbox deletion, including when agent execution fails.

Run a saved eval

Run the saved defaults:

lightsage evals run --eval-id eval_123

Add a configuration when the run needs a saved repository, environment profile, skill set, or legacy tool selection:

lightsage evals run \
  --eval-id eval_123 \
  --configuration-id config_123

You can replace saved prompts, persona, agents, CLIs, MCP servers, setup commands, cleanup commands, or repeat count for one launch. An explicitly empty clis, mcps, setup_commands, or cleanup_commands array in a JSON body disables the saved collection for that launch.

Monitor and inspect a run

lightsage evals monitor --run-id run_123
lightsage eval-runs retrieve --run-id run_123 --output-format json
lightsage evals analysis --run-id run_123 --output-format json

The parent run reports its agents, persona, prompt count, runs per matrix cell, attempt count, status, and progress. Attempt-level analysis and trace records include the exact prompt, prompt index, agent, and repeat index. Inline environment values are write-only. Lightsage keeps them in private job configuration and does not return them in run, trace, result, or analysis responses.

Getting Started

Visibility

Evals

Eval resources

Commands

Create and run directly

Run a saved eval

Monitor and inspect a run

​Commands

​Create and run directly

​Run a saved eval

​Monitor and inspect a run

Commands

Create and run directly

Run a saved eval

Monitor and inspect a run