> ## Documentation Index
> Fetch the complete documentation index at: https://lightsage.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom eval definitions

> Create and manage custom API Performance eval definitions from the Lightsage CLI.

Custom eval definitions are higher-level tasks Lightsage should test with coding agents. Use them for workflows that span multiple operations, such as installing an SDK, authenticating, and sending a first request.

Use [Custom eval runs](/docs/cli-reference/eval-runs) to execute these definitions and inspect active or completed runs.

## Commands

| Command                                                                                        | Description                          |
| ---------------------------------------------------------------------------------------------- | ------------------------------------ |
| `lightsage api-performance custom-eval-definitions list`                                       | List custom eval definitions.        |
| `lightsage api-performance custom-eval-definitions create`                                     | Create a custom eval definition.     |
| `lightsage api-performance custom-eval-definitions retrieve --custom-eval-id <custom_eval_id>` | Retrieve one custom eval definition. |
| `lightsage api-performance custom-eval-definitions update --custom-eval-id <custom_eval_id>`   | Update one custom eval definition.   |
| `lightsage api-performance custom-eval-definitions delete --custom-eval-id <custom_eval_id>`   | Delete one custom eval definition.   |

## List custom eval definitions

```bash theme={null}
lightsage api-performance custom-eval-definitions list --output-format json
```

Example response:

```json theme={null}
{
  "data": [
    {
      "id": "custom_eval_123",
      "prompt": "Install the SDK and crawl https://example.com",
      "prompt_template": null,
      "description": "Measures whether an agent can complete the first SDK request.",
      "competitor": null,
      "prompt_competitor": null,
      "prompt_competitor_group": null,
      "category": {
        "name": "Getting started",
        "slug": "getting-started"
      },
      "eval": {
        "id": "eval_123",
        "eval_type": "integration",
        "enabled": true
      }
    }
  ],
  "meta": {
    "count": 1
  }
}
```

## Create a custom eval definition

```bash theme={null}
lightsage api-performance custom-eval-definitions create \
  --prompt "Install the SDK and crawl https://example.com" \
  --category-name "Getting started" \
  --description "Measures whether an agent can complete the first SDK request."
```

Configure the generated eval at creation time:

```bash theme={null}
lightsage api-performance custom-eval-definitions create \
  --prompt "Install the SDK and crawl https://example.com" \
  --description "The agent should complete the first working request." \
  --eval '{"docs_mode":"include","include_env_vars":true,"mcp_enabled":false}'
```

Use a prompt template when Lightsage should render the prompt for a specific company. Templates support `{{company_name}}` and `{{company_domain}}`.

```bash theme={null}
lightsage api-performance custom-eval-definitions create \
  --body '{"prompt":"Build an integration for Example API.","prompt_template":"Build an integration for {{company_name}} using docs from {{company_domain}}.","category_name":"Getting started"}'
```

## Retrieve a custom eval definition

```bash theme={null}
lightsage api-performance custom-eval-definitions retrieve \
  --custom-eval-id custom_eval_123
```

## Update a custom eval definition

```bash theme={null}
lightsage api-performance custom-eval-definitions update \
  --custom-eval-id custom_eval_123 \
  --description '"Covers SDK install, authentication, and first crawl request."'
```

Update the attached eval definition:

```bash theme={null}
lightsage api-performance custom-eval-definitions update \
  --custom-eval-id custom_eval_123 \
  --eval '{"expected_behavior":"The agent installs the SDK, authenticates, and prints the crawl result."}'
```

Updateable fields:

| Flag                        | Description                                                                     |
| --------------------------- | ------------------------------------------------------------------------------- |
| `--prompt`                  | Task prompt.                                                                    |
| `--prompt-template`         | Optional prompt template. Supports `{{company_name}}` and `{{company_domain}}`. |
| `--category-name`           | Category label for organizing custom evals.                                     |
| `--description`             | Expected task outcome.                                                          |
| `--competitor`              | Competitor ID for competitor-specific eval variants.                            |
| `--prompt-competitor-group` | Base custom eval ID for competitor variants.                                    |
| `--eval`                    | Eval definition update object as JSON.                                          |

## Delete a custom eval definition

```bash theme={null}
lightsage api-performance custom-eval-definitions delete \
  --custom-eval-id custom_eval_123
```

Deleting a custom eval definition removes it from future custom eval runs.