> ## Documentation Index > Fetch the complete documentation index at: https://lightsage.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Start an eval run > Queues a durable eval job and returns immediately with a polling ID. Poll `GET /v2/eval-runs/{run_id}` until its status reaches a terminal value; once finished it also returns the results and analysis. Follow a run live with the monitor operation. Send one of these flat request forms: - **Saved eval:** provide `eval_id`; optionally replace its prompts, persona, agents, CLIs, MCPs, setup commands, cleanup commands, or runs for this launch. A saved configuration can still provide repository, environment, skills, and legacy tool selections. - **Direct create-and-run:** provide `prompts`, `judges`, and `agent`. You may also provide a persona, tags, repository, write-only environment variables, saved or inline CLIs and MCP servers, skills, setup commands, and cleanup commands. The run fans out across prompts × agents × runs. Persona is shared metadata and does not multiply attempts. No separate `mode` field is required. If the organization lacks the estimated credits, the request fails with `402` before dispatching work. A saved or inline environment containing an empty or placeholder credential is rejected with `400` before any evaluation is dispatched. ## OpenAPI ````yaml /openapi.json post /v2/eval-runs openapi: 3.1.0 info: title: Lightsage Public API description: Public API for Lightsage CLI and external integrations. version: 1.0.0 servers: - url: https://api.lightsage.com description: Production security: [] tags: - name: Status description: Check API availability and validate the calling API key. - name: Visibility description: >- Measure visibility, citations, competitors, and prompt performance across answer platforms. - name: Prompts description: Manage tracked prompts and topics, and inspect their stored runs. - name: Opportunities description: Review ranked opportunities computed from prompt visibility results. - name: Tasks description: Manage the persisted action queue and task lifecycle. - name: Site Audits description: Review website audit results and page-level findings. - name: Evals description: Define eval tasks, judges, subjects, and saved configurations. - name: Eval Templates description: >- Select reusable judges, targets, schedules, metadata, and runtime settings with `POST /v2/eval-templates`, then copy only those settings into prompt-specific evals with `POST /v2/evals/batch`. Templates never contain prompts and remain independent from evals created from them. - name: Eval Runs description: Start eval runs and inspect their lifecycle state. - name: Eval Results description: Review completed eval outcomes and diagnostic details. - name: Agent Preference description: >- Compare your company head-to-head with a tracked competitor. Create a comparison with `POST /v2/agent-preference/evals`, run it with `POST /v2/eval-runs`, then read which company agents recommended. - name: Agents description: List runnable coding-agent harness and model combinations. - name: Repositories description: Manage reusable Git repository workspaces. - name: Environments description: List reusable runtime environments without exposing secret values. - name: MCP Servers description: List reusable MCP server integrations without exposing credentials. - name: CLIs description: List reusable command-line interfaces without exposing setup commands. - name: Skills description: List reusable skills available to workflows and evals. paths: /v2/eval-runs: post: tags: - Eval Runs summary: Start an eval run description: >- Queues a durable eval job and returns immediately with a polling ID. Poll `GET /v2/eval-runs/{run_id}` until its status reaches a terminal value; once finished it also returns the results and analysis. Follow a run live with the monitor operation. Send one of these flat request forms: - **Saved eval:** provide `eval_id`; optionally replace its prompts, persona, agents, CLIs, MCPs, setup commands, cleanup commands, or runs for this launch. A saved configuration can still provide repository, environment, skills, and legacy tool selections. - **Direct create-and-run:** provide `prompts`, `judges`, and `agent`. You may also provide a persona, tags, repository, write-only environment variables, saved or inline CLIs and MCP servers, skills, setup commands, and cleanup commands. The run fans out across prompts × agents × runs. Persona is shared metadata and does not multiply attempts. No separate `mode` field is required. If the organization lacks the estimated credits, the request fails with `402` before dispatching work. A saved or inline environment containing an empty or placeholder credential is rejected with `400` before any evaluation is dispatched. operationId: v2_eval_runs_create requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/CreateV2EvalRunRequest' examples: saved_eval: summary: Run a self-contained saved eval description: >- Provide `eval_id` and any environment values required by its saved MCP definitions. A configuration remains optional. value: eval_id: eval_123 env: LIGHTSAGE_API_KEY: secret-value direct_eval: summary: Create and run an eval directly description: >- Omit `eval_id` and provide `prompts`, `judges`, and one or more values in `agent`. The created eval ID is returned with the run. value: agent: - codex:gpt-5-4 - claude-code:claude-opus-4-8 prompts: - Complete the task with the CLI. - Complete the task with the MCP server. judges: - The requested workflow succeeds. - The generated project passes its tests. persona: Platform engineer runs: 2 tags: - sdk - smoke-test repository: nextjs-starter clis: - brew install lightsagehq/tools/lightsage - npx -y firecrawl-cli@latest init --all setup_commands: - npm install - npm run db:migrate cleanup_commands: - npm run db:cleanup mcps: - name: lightsage type: http url: https://mcp.lightsage.com/mcp headers: X-Lightsage-Api-Key: ${LIGHTSAGE_API_KEY} env: LIGHTSAGE_API_KEY: secret-value skills: - resend/resend-skills responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/V2EvalRun' '400': description: >- The saved/direct mode fields, referenced resource, or environment credentials are invalid. content: application/json: schema: type: object properties: detail: type: string required: - detail examples: invalid_environment_credentials: summary: Environment credentials are missing value: detail: >- Environment profile 'Production' contains empty or placeholder values for: ACME_API_KEY. Update those values before running the evaluation. '402': description: >- The organization does not currently have enough credits for the requested run. content: application/json: schema: type: object properties: detail: type: object properties: error: type: string example: Insufficient credits required: type: number description: Usage-based credits the requested run needs. balance: type: number description: The organization's current credit balance. required: - error - required - balance required: - detail '403': description: >- The requested coding agent is unavailable on the organization's current plan. content: application/json: schema: type: object properties: detail: type: string required: - detail example: detail: 'Your current plan does not include: cursor' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' security: - LightsageApiKeyAuth: [] components: schemas: CreateV2EvalRunRequest: properties: eval_id: type: string minLength: 1 title: Eval Id description: >- Saved-run form only. Existing eval definition to run. Omit this field to use the direct create-and-run form. configuration_id: type: string minLength: 1 title: Configuration Id description: >- Saved-run form only. Optional saved repository, environment, skills, and legacy tool configuration. name: type: string maxLength: 160 minLength: 1 title: Name description: >- Direct create-and-run form only. Display name for the durable eval. Defaults to the first prompt. prompts: anyOf: - items: type: string type: array maxItems: 20 minItems: 1 - type: 'null' title: Prompts description: >- Task prompts. Required in direct mode; in saved mode, replaces saved prompts for this launch. CLI clients expose repeatable `--prompt`. judges: anyOf: - items: type: string type: array maxItems: 20 minItems: 1 - type: 'null' title: Judges description: >- Success criteria. Required in direct mode; in saved mode, replaces saved judges for this launch. CLI clients expose repeatable `--judge`. agent: anyOf: - items: type: string type: array maxItems: 10 minItems: 1 - type: 'null' title: Agent description: >- Runnable agents from `GET /v2/agents`. Required unless the saved eval already has agents. CLI clients expose repeatable `--agent`. persona: type: string maxLength: 160 title: Persona description: >- Optional perspective for every attempt in this launch. It does not modify prompts or add a fan-out dimension. clis: anyOf: - items: type: string type: array maxItems: 20 - type: 'null' title: Clis description: >- Saved CLI IDs or shell installation commands. UUID values resolve saved CLIs from `GET /v2/clis`; every other value is run as an installation command. In saved mode, an explicit empty array disables saved CLIs for this launch. CLI clients expose repeatable `--cli`. mcps: anyOf: - items: anyOf: - $ref: '#/components/schemas/V2SavedMcpReference' - $ref: '#/components/schemas/V2InlineMcpDefinition' type: array maxItems: 20 - type: 'null' title: Mcps description: >- Saved MCP references or inline portable definitions. In saved mode, an explicit empty array disables saved MCPs for this launch. setup_commands: anyOf: - items: type: string type: array maxItems: 20 - type: 'null' title: Setup Commands description: >- Shell commands run sequentially in the prepared repository before the agent starts. In saved mode, an explicit empty array disables saved setup commands for this launch. CLI clients expose repeatable `--setup-command`. cleanup_commands: anyOf: - items: type: string type: array maxItems: 20 - type: 'null' title: Cleanup Commands description: >- Best-effort shell commands run sequentially after agent execution and result capture, before sandbox deletion. In saved mode, an explicit empty array disables saved cleanup commands for this launch. CLI clients expose repeatable `--cleanup-command`. runs: type: integer maximum: 20 minimum: 1 title: Runs description: >- Independent attempts for every prompt-agent pair. Defaults to the saved eval value or 1 for a direct run. tags: items: type: string type: array maxItems: 20 title: Tags description: >- Direct create-and-run form only. Optional labels saved on the created eval. CLI clients may expose this string array as a repeatable `--tag` flag. repository: type: string minLength: 1 title: Repository description: >- Direct create-and-run form only. A saved repository ID, built-in alias such as `nextjs-starter`, or a public GitHub HTTPS URL. env: additionalProperties: type: string type: object maxProperties: 100 title: Env description: >- Write-only environment variables for this launch. Values are stored only in the queued job's private configuration. writeOnly: true skills: items: type: string type: array maxItems: 20 title: Skills description: >- Additional public skill packages to install before the agent starts in either request form, using GitHub `owner/repository` references such as `resend/resend-skills`. CLI clients may expose this string array as a repeatable `--skill` flag. skill_ids: items: type: string type: array maxItems: 20 title: Skill Ids description: >- Direct create-and-run form only. Reusable skill IDs from `GET /v2/skills` to install before the agent starts. search_grounding: anyOf: - $ref: '#/components/schemas/SearchGroundingMode' - type: 'null' description: >- Web-retrieval policy for this launch. Overrides the selected configuration or saved eval when provided; direct runs default to `default`. additionalProperties: false type: object title: CreateV2EvalRunRequest description: >- Start an eval run using one of two flat request forms. Saved runs require `eval_id` and use saved simulation defaults unless the request replaces them. Direct create-and-run requests require `prompts`, `judges`, and `agent`. No separate mode field is required. examples: - env: LIGHTSAGE_API_KEY: secret-value eval_id: eval_123 - agent: - codex:gpt-5-4 - claude-code:claude-opus-4-8 judges: - The requested workflow succeeds. - The generated project passes its tests. name: SDK simulation persona: Platform engineer prompts: - Complete the task with the CLI. - Complete the task with the MCP server. repository: nextjs-starter runs: 2 skills: - resend/resend-skills tags: - sdk - smoke-test V2EvalRun: properties: id: type: string title: Id description: Run ID to use when polling eval-run status. job_id: type: string title: Job Id description: Alias of `id` for the durable parent job. eval_id: anyOf: - type: string - type: 'null' title: Eval Id description: Saved eval definition ID, including one created by a direct run. eval_name: anyOf: - type: string - type: 'null' title: Eval Name description: Display name captured for the eval when the run was created. configuration_id: anyOf: - type: string - type: 'null' title: Configuration Id persona: anyOf: - type: string - type: 'null' title: Persona description: Perspective applied to every attempt in this simulation. agent: items: type: string type: array title: Agent description: Runnable agents selected for this simulation. prompt_count: type: integer minimum: 1 title: Prompt Count description: Number of prompts included in the simulation. default: 1 runs: type: integer minimum: 1 title: Runs description: Independent attempts requested for every prompt-agent pair. default: 1 attempt_count: type: integer minimum: 1 title: Attempt Count description: Total concrete attempts after all fan-out dimensions. default: 1 status: type: string title: Status description: >- Terminal values: `completed`, `failed`, and `cancelled`. Recoverable interruptions are reported as `running` with stage `retrying`. stage: anyOf: - type: string - type: 'null' title: Stage dispatch_id: anyOf: - type: string - type: 'null' title: Dispatch Id progress: $ref: '#/components/schemas/PublicEvalRunProgress' estimated_cost: anyOf: - type: number - type: 'null' title: Estimated Cost error: anyOf: - type: string - type: 'null' title: Error created_at: anyOf: - type: string - type: 'null' title: Created At started_at: anyOf: - type: string - type: 'null' title: Started At completed_at: anyOf: - type: string - type: 'null' title: Completed At type: object required: - id - job_id - status title: V2EvalRun HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError V2SavedMcpReference: properties: id: type: string minLength: 1 title: Id description: Saved MCP server ID from `GET /v2/mcp-servers`. additionalProperties: false type: object required: - id title: V2SavedMcpReference V2InlineMcpDefinition: properties: name: type: string maxLength: 100 minLength: 1 title: Name description: Stable server name used in the selected agent's MCP config. type: type: string enum: - http - sse - stdio title: Type url: anyOf: - type: string - type: 'null' title: Url description: HTTPS endpoint for an `http` or `sse` MCP server. headers: additionalProperties: type: string type: object maxProperties: 50 title: Headers description: >- HTTP headers. Reference run secrets with `${VARIABLE_NAME}` instead of embedding credentials. command: anyOf: - type: string - type: 'null' title: Command description: Executable command for a `stdio` MCP server. args: items: type: string type: array maxItems: 100 title: Args env: additionalProperties: type: string type: object maxProperties: 100 title: Env description: >- Environment mapping for a `stdio` MCP server. Values may reference run secrets with `${VARIABLE_NAME}`. additionalProperties: false type: object required: - name - type title: V2InlineMcpDefinition SearchGroundingMode: type: string enum: - disabled - enabled - default title: SearchGroundingMode description: Requested web-retrieval behavior for an eval run. PublicEvalRunProgress: properties: total: type: integer title: Total default: 0 queued: type: integer title: Queued default: 0 running: type: integer title: Running default: 0 completed: type: integer title: Completed default: 0 failed: type: integer title: Failed default: 0 cancelled: type: integer title: Cancelled default: 0 retrying: type: integer title: Retrying description: >- Recoverable child jobs waiting to be retried. These remain active and are excluded from `percent`. default: 0 interrupted: type: integer title: Interrupted description: Deprecated compatibility alias for `retrying`; use `retrying`. default: 0 deprecated: true percent: type: number title: Percent description: >- The percentage of child jobs in a final state (`completed`, `failed`, or `cancelled`). It can reach `100` while the parent job is still summarizing, so use `status` to detect completion. default: 0 type: object title: PublicEvalRunProgress ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError securitySchemes: LightsageApiKeyAuth: type: apiKey in: header name: X-Lightsage-Api-Key ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.