> ## Documentation Index
> Fetch the complete documentation index at: https://lightsage.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval definitions

> List, retrieve, and update API Performance eval definitions from the Lightsage CLI.

Eval definitions contain the prompt, expected behavior, starter project, and runtime options Lightsage uses for API Performance runs.

## Commands

| Command                                                        | Description                   |
| -------------------------------------------------------------- | ----------------------------- |
| `lightsage api-performance evals list`                         | List eval definitions.        |
| `lightsage api-performance evals retrieve --eval-id <eval_id>` | Retrieve one eval definition. |
| `lightsage api-performance evals update --eval-id <eval_id>`   | Update one eval definition.   |

## List eval definitions

```bash theme={null}
lightsage api-performance evals list --output-format json
```

Filter to enabled evals for one source:

```bash theme={null}
lightsage api-performance evals list \
  --source-id source_123 \
  --enabled true
```

Example response:

```json theme={null}
{
  "data": [
    {
      "id": "eval_123",
      "operation_id": "operation_123",
      "source_id": "source_123",
      "source": {
        "id": "source_123",
        "type": "rest_api",
        "label": "REST API"
      },
      "eval_type": "integration",
      "prompt": "Use the API to crawl a page.",
      "custom_prompt": null,
      "expected_behavior": "The agent completes the task successfully.",
      "starting_project_id": "starter_123",
      "docs_mode": "include",
      "include_env_vars": true,
      "env_profile_ids": ["prod"],
      "model_ids": ["claude-sonnet-4.5"],
      "compare_skills": false,
      "skills_mode": "selected",
      "skills_enabled": true,
      "skill_ids": ["skill_123"],
      "mcp_enabled": false,
      "run_context": {},
      "enabled": true
    }
  ],
  "meta": {
    "count": 1
  }
}
```

## Retrieve an eval definition

```bash theme={null}
lightsage api-performance evals retrieve --eval-id eval_123
```

## Update an eval definition

Change the custom prompt and expected behavior:

```bash theme={null}
lightsage api-performance evals update --eval-id eval_123 \
  --custom-prompt '"Use the TypeScript SDK to crawl https://example.com."' \
  --expected-behavior '"The agent installs the SDK, sends a valid request, and prints the crawl result."'
```

Attach a starter project:

```bash theme={null}
lightsage api-performance evals update --eval-id eval_123 \
  --starting-project-id '"starter_123"'
```

Control runtime context:

```bash theme={null}
lightsage api-performance evals update --eval-id eval_123 \
  --docs-mode '"include"' \
  --include-env-vars true \
  --env-profile-ids '["prod"]' \
  --model-ids '["claude-sonnet-4.5"]' \
  --mcp-enabled false \
  --run-context '{"framework":"nextjs","package_manager":"pnpm"}'
```

Control skill usage:

```bash theme={null}
lightsage api-performance evals update --eval-id eval_123 \
  --skills-mode '"selected"' \
  --skill-ids '["skill_123"]' \
  --compare-skills true
```

Set `--skills-mode '"all"'` to use all available skills for the organization, or `--skills-mode '"none"'` to disable skill context. Use `--compare-skills true` when you want Lightsage to compare runs with skill context enabled.

Disable an eval without deleting it:

```bash theme={null}
lightsage api-performance evals update --eval-id eval_123 --enabled false
```

Updateable fields:

| Flag                    | Description                                                                                     |
| ----------------------- | ----------------------------------------------------------------------------------------------- |
| `--custom-prompt`       | Replacement prompt text for the eval.                                                           |
| `--expected-behavior`   | Expected behavior used by graders and diagnosis.                                                |
| `--starting-project-id` | Starter project cloned before the eval. Pass an empty string to clear it.                       |
| `--docs-mode`           | Docs behavior for the eval. Use `default`, `include`, or `exclude`.                             |
| `--include-env-vars`    | Include configured environment variable names in the eval prompt.                               |
| `--env-profile-ids`     | Environment profile IDs to use for this eval as a JSON array. Empty array clears the selection. |
| `--model-ids`           | Model IDs to use for this eval as a JSON array. Empty array uses config-level models.           |
| `--skills-mode`         | Skill behavior for the eval. Use `none`, `all`, or `selected`.                                  |
| `--skill-ids`           | Skill IDs to use when `--skills-mode '"selected"'`, as a JSON array.                            |
| `--compare-skills`      | Compare skill-enabled runs against the default eval path.                                       |
| `--mcp-enabled`         | Enable MCP context for the eval.                                                                |
| `--run-context`         | Additional runtime context as JSON.                                                             |
| `--enabled`             | Enable or disable the eval.                                                                     |