curl --request POST \
--url https://api.lightsage.com/v2/eval-runs \
--header 'Content-Type: application/json' \
--header 'X-Lightsage-Api-Key: <api-key>' \
--data '
{
"eval_id": "eval_123",
"env": {
"LIGHTSAGE_API_KEY": "secret-value"
}
}
'{
"id": "<string>",
"job_id": "<string>",
"status": "<string>",
"eval_id": "<string>",
"eval_name": "<string>",
"configuration_id": "<string>",
"persona": "<string>",
"agent": [
"<string>"
],
"prompt_count": 1,
"runs": 1,
"attempt_count": 1,
"stage": "<string>",
"dispatch_id": "<string>",
"progress": {
"total": 0,
"queued": 0,
"running": 0,
"completed": 0,
"failed": 0,
"cancelled": 0,
"retrying": 0,
"interrupted": 0,
"percent": 0
},
"estimated_cost": 123,
"error": "<string>",
"created_at": "<string>",
"started_at": "<string>",
"completed_at": "<string>"
}{
"detail": "Environment profile 'Production' contains empty or placeholder values for: ACME_API_KEY. Update those values before running the evaluation."
}{
"detail": {
"error": "Insufficient credits",
"required": 123,
"balance": 123
}
}{
"detail": "Your current plan does not include: cursor"
}{
"detail": [
{
"loc": [
"<string>"
],
"msg": "<string>",
"type": "<string>",
"input": "<unknown>",
"ctx": {}
}
]
}Start an eval run
Queues a durable eval job and returns immediately with a polling ID. Poll GET /v2/eval-runs/{run_id} until its status reaches a terminal value; once finished it also returns the results and analysis. Follow a run live with the monitor operation.
Send one of these flat request forms:
- Saved eval: provide
eval_id; optionally replace its prompts, persona, agents, CLIs, MCPs, setup commands, cleanup commands, or runs for this launch. A saved configuration can still provide repository, environment, skills, and legacy tool selections. - Direct create-and-run: provide
prompts,judges, andagent. You may also provide a persona, tags, repository, write-only environment variables, saved or inline CLIs and MCP servers, skills, setup commands, and cleanup commands.
The run fans out across prompts × agents × runs. Persona is shared metadata and does not multiply attempts. No separate mode field is required. If the organization lacks the estimated credits, the request fails with 402 before dispatching work.
A saved or inline environment containing an empty or placeholder credential is rejected with 400 before any evaluation is dispatched.
curl --request POST \
--url https://api.lightsage.com/v2/eval-runs \
--header 'Content-Type: application/json' \
--header 'X-Lightsage-Api-Key: <api-key>' \
--data '
{
"eval_id": "eval_123",
"env": {
"LIGHTSAGE_API_KEY": "secret-value"
}
}
'{
"id": "<string>",
"job_id": "<string>",
"status": "<string>",
"eval_id": "<string>",
"eval_name": "<string>",
"configuration_id": "<string>",
"persona": "<string>",
"agent": [
"<string>"
],
"prompt_count": 1,
"runs": 1,
"attempt_count": 1,
"stage": "<string>",
"dispatch_id": "<string>",
"progress": {
"total": 0,
"queued": 0,
"running": 0,
"completed": 0,
"failed": 0,
"cancelled": 0,
"retrying": 0,
"interrupted": 0,
"percent": 0
},
"estimated_cost": 123,
"error": "<string>",
"created_at": "<string>",
"started_at": "<string>",
"completed_at": "<string>"
}{
"detail": "Environment profile 'Production' contains empty or placeholder values for: ACME_API_KEY. Update those values before running the evaluation."
}{
"detail": {
"error": "Insufficient credits",
"required": 123,
"balance": 123
}
}{
"detail": "Your current plan does not include: cursor"
}{
"detail": [
{
"loc": [
"<string>"
],
"msg": "<string>",
"type": "<string>",
"input": "<unknown>",
"ctx": {}
}
]
}Authorizations
Body
Start an eval run using one of two flat request forms. Saved runs require eval_id and use saved simulation defaults unless the request replaces them. Direct create-and-run requests require prompts, judges, and agent. No separate mode field is required.
Saved-run form only. Existing eval definition to run. Omit this field to use the direct create-and-run form.
1Saved-run form only. Optional saved repository, environment, skills, and legacy tool configuration.
1Direct create-and-run form only. Display name for the durable eval. Defaults to the first prompt.
1 - 160Task prompts. Required in direct mode; in saved mode, replaces saved prompts for this launch. CLI clients expose repeatable --prompt.
1 - 20 elementsSuccess criteria. Required in direct mode; in saved mode, replaces saved judges for this launch. CLI clients expose repeatable --judge.
1 - 20 elementsRunnable agents from GET /v2/agents. Required unless the saved eval already has agents. CLI clients expose repeatable --agent.
1 - 10 elementsOptional perspective for every attempt in this launch. It does not modify prompts or add a fan-out dimension.
160Saved CLI IDs or shell installation commands. UUID values resolve saved CLIs from GET /v2/clis; every other value is run as an installation command. In saved mode, an explicit empty array disables saved CLIs for this launch. CLI clients expose repeatable --cli.
20Saved MCP references or inline portable definitions. In saved mode, an explicit empty array disables saved MCPs for this launch.
20- V2SavedMcpReference
- V2InlineMcpDefinition
Show child attributes
Show child attributes
Shell commands run sequentially in the prepared repository before the agent starts. In saved mode, an explicit empty array disables saved setup commands for this launch. CLI clients expose repeatable --setup-command.
20Best-effort shell commands run sequentially after agent execution and result capture, before sandbox deletion. In saved mode, an explicit empty array disables saved cleanup commands for this launch. CLI clients expose repeatable --cleanup-command.
20Independent attempts for every prompt-agent pair. Defaults to the saved eval value or 1 for a direct run.
1 <= x <= 20Direct create-and-run form only. Optional labels saved on the created eval. CLI clients may expose this string array as a repeatable --tag flag.
20Direct create-and-run form only. A saved repository ID, built-in alias such as nextjs-starter, or a public GitHub HTTPS URL.
1Write-only environment variables for this launch. Values are stored only in the queued job's private configuration.
Show child attributes
Show child attributes
Additional public skill packages to install before the agent starts in either request form, using GitHub owner/repository references such as resend/resend-skills. CLI clients may expose this string array as a repeatable --skill flag.
20Direct create-and-run form only. Reusable skill IDs from GET /v2/skills to install before the agent starts.
20Web-retrieval policy for this launch. Overrides the selected configuration or saved eval when provided; direct runs default to default.
disabled, enabled, default Response
Successful Response
Run ID to use when polling eval-run status.
Alias of id for the durable parent job.
Terminal values: completed, failed, and cancelled. Recoverable interruptions are reported as running with stage retrying.
Saved eval definition ID, including one created by a direct run.
Display name captured for the eval when the run was created.
Perspective applied to every attempt in this simulation.
Runnable agents selected for this simulation.
Number of prompts included in the simulation.
x >= 1Independent attempts requested for every prompt-agent pair.
x >= 1Total concrete attempts after all fan-out dimensions.
x >= 1Show child attributes
Show child attributes