Skip to main content
Custom eval runs are first-class Lightsage CLI workflows. Start configured custom evals, list active and completed executions, poll one run, and retrieve its trace with the same run_id.

​
Commands

​
Run custom evals

Run selected custom eval definitions:
If you omit --eval-ids, Lightsage uses the custom eval set configured for your organization. You can override configured coding-agent targets for a one-off run:
The response includes a parent batch_id, a dispatch_id, and the individual executions in runs. Use each child run_id with lightsage get run and lightsage get trace.

​
List eval executions

Filter active executions or one custom eval definition:
Each row includes lifecycle status, stage, elapsed time, trace availability, and a full dashboard_url containing your organization ID and the selected run ID.

​
Get a run

While the execution is active, poll this command to read the latest status, stage, and elapsed_seconds. When the result is persisted, result_id is populated and trace_available becomes true. Statuses include pending, running, completed, failed, cancelled, and interrupted.

​
Get a trace

For an active execution, the response returns trace_available: false with the current status, stage, and elapsed time. After completion, the response includes the prompt, response, tool calls, conversation turns, execution output, generated files, and exit code when available.