Skip to main content
POST
Create Eval

Authorizations

X-Lightsage-Api-Key
string
header
required

Body

application/json
name
string
required
Required string length: 1 - 160
prompts
string[]
required

Task prompts. CLI clients expose this as repeatable --prompt.

Required array length: 1 - 20 elements
judges
string[]
required

Plain-text success criteria. CLI clients expose this as repeatable --judge.

Required array length: 1 - 20 elements
agent
string[]

Runnable agents from GET /v2/agents. An empty array saves the eval without default agents. CLI clients expose repeatable --agent.

Maximum array length: 10
persona
string

Optional perspective applied to the whole eval. It is stored as metadata and does not change prompts or multiply attempts.

Maximum string length: 160
clis
string[]

Saved CLI IDs or shell installation commands. UUID values resolve saved CLIs from GET /v2/clis; every other value is run as an installation command. CLI clients expose repeatable --cli.

Maximum array length: 20
mcps
(V2SavedMcpReference · object | V2InlineMcpDefinition · object)[]

Saved MCP references or inline portable MCP definitions.

Maximum array length: 20
setup_commands
string[]

Shell commands run sequentially in the prepared repository before the agent starts. CLI clients expose repeatable --setup-command.

Maximum array length: 20
cleanup_commands
string[]

Best-effort shell commands run sequentially after agent execution and result capture, before sandbox deletion. CLI clients expose repeatable --cleanup-command.

Maximum array length: 20
search_grounding
enum<string>
default:default

Web-retrieval policy for every run. disabled hides native search and fetch tools and requests network isolation; enabled exposes retrieval tools, requires live retrieval in the runtime prompt, and qualifies the run only when retrieval is observed; default exposes the platform's normal tools without requiring retrieval.

Available options:
disabled,
enabled,
default
runs
integer
default:1

Independent attempts for every prompt-agent pair.

Required range: 1 <= x <= 20
tags
string[]

Optional labels for organizing the eval. CLI clients may expose this string array as a repeatable --tag flag.

Maximum array length: 20

Response

Successful Response

id
string
required
name
string
required
prompts
string[]
judges
string[]
agent
string[]
persona
string | null
clis
(V2SavedCliReference · object | V2InlineCliReference · object)[]
mcps
(V2SavedMcpReference · object | V2InlineMcpDefinition · object)[]
setup_commands
string[]
cleanup_commands
string[]
search_grounding
enum<string>
default:default

Saved web-retrieval policy: disabled, enabled, or default.

Available options:
disabled,
enabled,
default
runs
integer
default:1
Required range: 1 <= x <= 20
tags
string[]
created_at
string | null
updated_at
string | null