Custom eval definitions

Custom eval definitions are higher-level tasks Lightsage should test with coding agents. Use them for workflows that span multiple operations, such as installing an SDK, authenticating, and sending a first request. Use Custom eval runs to execute these definitions and inspect active or completed runs.

Commands

Command	Description
`lightsage api-performance custom-eval-definitions list`	List custom eval definitions.
`lightsage api-performance custom-eval-definitions create`	Create a custom eval definition.
`lightsage api-performance custom-eval-definitions retrieve --custom-eval-id <custom_eval_id>`	Retrieve one custom eval definition.
`lightsage api-performance custom-eval-definitions update --custom-eval-id <custom_eval_id>`	Update one custom eval definition.
`lightsage api-performance custom-eval-definitions delete --custom-eval-id <custom_eval_id>`	Delete one custom eval definition.

List custom eval definitions

lightsage api-performance custom-eval-definitions list --output-format json

Example response:

{
  "data": [
    {
      "id": "custom_eval_123",
      "prompt": "Install the SDK and crawl https://example.com",
      "prompt_template": null,
      "description": "Measures whether an agent can complete the first SDK request.",
      "competitor": null,
      "prompt_competitor": null,
      "prompt_competitor_group": null,
      "category": {
        "name": "Getting started",
        "slug": "getting-started"
      },
      "eval": {
        "id": "eval_123",
        "eval_type": "integration",
        "enabled": true
      }
    }
  ],
  "meta": {
    "count": 1
  }
}

Create a custom eval definition

lightsage api-performance custom-eval-definitions create \
  --prompt "Install the SDK and crawl https://example.com" \
  --category-name "Getting started" \
  --description "Measures whether an agent can complete the first SDK request."

Configure the generated eval at creation time:

lightsage api-performance custom-eval-definitions create \
  --prompt "Install the SDK and crawl https://example.com" \
  --description "The agent should complete the first working request." \
  --eval '{"docs_mode":"include","include_env_vars":true,"mcp_enabled":false}'

Use a prompt template when Lightsage should render the prompt for a specific company. Templates support {{company_name}} and {{company_domain}}.

lightsage api-performance custom-eval-definitions create \
  --body '{"prompt":"Build an integration for Example API.","prompt_template":"Build an integration for {{company_name}} using docs from {{company_domain}}.","category_name":"Getting started"}'

Retrieve a custom eval definition

lightsage api-performance custom-eval-definitions retrieve \
  --custom-eval-id custom_eval_123

Update a custom eval definition

lightsage api-performance custom-eval-definitions update \
  --custom-eval-id custom_eval_123 \
  --description '"Covers SDK install, authentication, and first crawl request."'

Update the attached eval definition:

lightsage api-performance custom-eval-definitions update \
  --custom-eval-id custom_eval_123 \
  --eval '{"expected_behavior":"The agent installs the SDK, authenticates, and prints the crawl result."}'

Updateable fields:

Flag	Description
`--prompt`	Task prompt.
`--prompt-template`	Optional prompt template. Supports `{{company_name}}` and `{{company_domain}}`.
`--category-name`	Category label for organizing custom evals.
`--description`	Expected task outcome.
`--competitor`	Competitor ID for competitor-specific eval variants.
`--prompt-competitor-group`	Base custom eval ID for competitor variants.
`--eval`	Eval definition update object as JSON.

Delete a custom eval definition

lightsage api-performance custom-eval-definitions delete \
  --custom-eval-id custom_eval_123

Deleting a custom eval definition removes it from future custom eval runs.

CLI Reference

Prompts

API Performance

Custom eval definitions

Commands

List custom eval definitions

Create a custom eval definition

Retrieve a custom eval definition

Update a custom eval definition

Delete a custom eval definition

​Commands

​List custom eval definitions

​Create a custom eval definition

​Retrieve a custom eval definition

​Update a custom eval definition

​Delete a custom eval definition

Commands

List custom eval definitions

Create a custom eval definition

Retrieve a custom eval definition

Update a custom eval definition

Delete a custom eval definition