Eval definitions

Eval definitions contain the prompt, expected behavior, starter project, and runtime options Lightsage uses for API Performance runs.

Commands

Command	Description
`lightsage api-performance evals list`	List eval definitions.
`lightsage api-performance evals retrieve --eval-id <eval_id>`	Retrieve one eval definition.
`lightsage api-performance evals update --eval-id <eval_id>`	Update one eval definition.

List eval definitions

lightsage api-performance evals list --output-format json

Filter to enabled evals for one source:

lightsage api-performance evals list \
  --source-id source_123 \
  --enabled true

Example response:

{
  "data": [
    {
      "id": "eval_123",
      "operation_id": "operation_123",
      "source_id": "source_123",
      "source": {
        "id": "source_123",
        "type": "rest_api",
        "label": "REST API"
      },
      "eval_type": "integration",
      "prompt": "Use the API to crawl a page.",
      "custom_prompt": null,
      "expected_behavior": "The agent completes the task successfully.",
      "starting_project_id": "starter_123",
      "docs_mode": "include",
      "include_env_vars": true,
      "env_profile_ids": ["prod"],
      "model_ids": ["claude-sonnet-4.5"],
      "compare_skills": false,
      "skills_mode": "selected",
      "skills_enabled": true,
      "skill_ids": ["skill_123"],
      "mcp_enabled": false,
      "run_context": {},
      "enabled": true
    }
  ],
  "meta": {
    "count": 1
  }
}

Retrieve an eval definition

lightsage api-performance evals retrieve --eval-id eval_123

Update an eval definition

Change the custom prompt and expected behavior:

lightsage api-performance evals update --eval-id eval_123 \
  --custom-prompt '"Use the TypeScript SDK to crawl https://example.com."' \
  --expected-behavior '"The agent installs the SDK, sends a valid request, and prints the crawl result."'

Attach a starter project:

lightsage api-performance evals update --eval-id eval_123 \
  --starting-project-id '"starter_123"'

Control runtime context:

lightsage api-performance evals update --eval-id eval_123 \
  --docs-mode '"include"' \
  --include-env-vars true \
  --env-profile-ids '["prod"]' \
  --model-ids '["claude-sonnet-4.5"]' \
  --mcp-enabled false \
  --run-context '{"framework":"nextjs","package_manager":"pnpm"}'

Control skill usage:

lightsage api-performance evals update --eval-id eval_123 \
  --skills-mode '"selected"' \
  --skill-ids '["skill_123"]' \
  --compare-skills true

Set --skills-mode '"all"' to use all available skills for the organization, or --skills-mode '"none"' to disable skill context. Use --compare-skills true when you want Lightsage to compare runs with skill context enabled. Disable an eval without deleting it:

lightsage api-performance evals update --eval-id eval_123 --enabled false

Updateable fields:

Flag	Description
`--custom-prompt`	Replacement prompt text for the eval.
`--expected-behavior`	Expected behavior used by graders and diagnosis.
`--starting-project-id`	Starter project cloned before the eval. Pass an empty string to clear it.
`--docs-mode`	Docs behavior for the eval. Use `default`, `include`, or `exclude`.
`--include-env-vars`	Include configured environment variable names in the eval prompt.
`--env-profile-ids`	Environment profile IDs to use for this eval as a JSON array. Empty array clears the selection.
`--model-ids`	Model IDs to use for this eval as a JSON array. Empty array uses config-level models.
`--skills-mode`	Skill behavior for the eval. Use `none`, `all`, or `selected`.
`--skill-ids`	Skill IDs to use when `--skills-mode '"selected"'`, as a JSON array.
`--compare-skills`	Compare skill-enabled runs against the default eval path.
`--mcp-enabled`	Enable MCP context for the eval.
`--run-context`	Additional runtime context as JSON.
`--enabled`	Enable or disable the eval.

CLI Reference

Prompts

API Performance

Commands

List eval definitions

Retrieve an eval definition

Update an eval definition

​Commands

​List eval definitions

​Retrieve an eval definition

​Update an eval definition

Commands

List eval definitions

Retrieve an eval definition

Update an eval definition