How Coding Agents Actually Decide Which CLI to Use

When a developer asks Claude Code to "deploy my app to Vercel," the agent doesn't just recommend an SDK. It often reaches for a CLI tool, and the decision process is different from how it picks libraries.

CLIs are the power tools of the coding agent world. They let agents execute real operations, not just generate code. But only 31 out of 73 APIs in our Devtool Arena benchmarks have CLI support. That's a massive gap and an opportunity.

Here's how Claude Code, Codex, and Cursor actually decide which CLI to use.

Table of Contents

• TL;DR: The 4-Layer Decision Stack
• Why CLIs Matter More Than You Think
• Layer 1: Training Data
• Layer 2: Web Search
• Layer 3: Context Retrieval
• Layer 4: Shell Execution
• The Full Picture: How Recommendations Happen
• Why Newer CLIs Can Win
• How to Measure Your CLI's Position
• The Optimization Priority
• FAQ

TL;DR: The 4-Layer Decision Stack for CLIs

The short version: Agents prefer CLIs they can run non-interactively, parse reliably, and recover from when things go wrong. The --json flag alone is worth +18 points in our benchmarks.

CLI Leaders by Category

The Decision Stack

Most devtool teams build an SDK and stop there. The teams winning in coding agents ship a CLI that agents can actually run: structured output, clear errors, and simple auth.

Why CLIs Matter More Than You Think

Coding agents have a superpower SDKs can't match: they can execute shell commands directly.

When Claude Code uses an SDK, it generates code that the developer must run. When it uses a CLI, it can run the command itself in its own sandbox, computer use environment, or the developer's terminal.

This changes the calculus:

CLIs are the difference between Claude Code saying "try this" and "I did it."

Layer 1: Training Data (The Foundation)

Every coding agent's base model learned about CLI tools from:

• Man pages and --help output crawled from documentation sites
• GitHub repositories containing shell scripts and CI/CD configs
• Stack Overflow answers showing CLI usage patterns
• Blog posts and tutorials with command examples
• Package manager metadata (Homebrew formulae, npm package.json)

What This Means for CLIs

If your CLI existed before the model's training cutoff and appeared in enough shell scripts and tutorials, the agent has baseline familiarity. It knows the command name, common flags, and typical usage patterns.

But training data has CLI-specific limits:

• Flag names and behaviors may have changed since the cutoff
• New subcommands won't exist in the agent's knowledge
• Authentication patterns may be outdated
• The agent may generate deprecated flag combinations

This is why Layer 1 alone isn't enough. A CLI that shipped last quarter won't exist in training data, but it can still get recommended through the other layers.

Installation Methods in Training Data

Agents learn installation patterns from training data. The more installation methods that appear in tutorials and READMEs, the more reliably agents can install your CLI:

# Agents have seen these patterns thousands of times
brew install stripe/stripe-cli/stripe
npm install -g vercel
pip install awscli
cargo install ripgrep

CLIs available through multiple package managers (Homebrew, npm, pip, apt) have higher installation success rates because agents can fall back to alternatives when one method fails.

Layer 2: Web Search (Real-Time Discovery)

According to Vercel's AEO tracking research, coding agents perform web searches in roughly 20% of prompts.

CLI-related searches often look different from SDK searches:

• "how to deploy with vercel cli" (task-oriented)
• "aws cli s3 sync example" (command-specific)
• "stripe cli vs dashboard" (comparison)
• "gh cli authentication" (setup-focused)

What the Agent Searches For

When Claude Code searches for CLI guidance, it prioritizes:

1. Official CLI documentation, especially quick starts and command references
2. "How to X" tutorials, which rank well for task-oriented queries
3. GitHub README files, often the first result for [tool] cli
4. Stack Overflow answers showing real-world usage patterns and edge cases

What Blocks Discovery

Your CLI won't appear in agent searches if:

# robots.txt blocking AI crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Or if your CLI docs are buried inside a larger product site with no dedicated CLI section.

How to Win at Layer 2

Layer 3: Context Retrieval (Machine-Readable CLIs)

This layer is where CLIs have unique advantages, and where most teams miss opportunities.

llms.txt for CLIs

A good llms.txt for a CLI tool includes command structure, not just API endpoints. Here's a comprehensive example:

# Acme CLI

> Command-line tool for managing Acme cloud deployments and infrastructure.

## Installation
brew install acme-io/tap/acme
npm install -g @acme/cli
curl -fsSL https://acme.io/install.sh | bash

## Authentication
# Environment variable (preferred for automation)
export ACME_API_KEY=ak_live_xxxxx
acme whoami  # Verify authentication

# Interactive login (for humans)
acme auth login

# Token flag (for CI/CD)
acme --api-key $ACME_API_KEY deploy

## Common Commands
acme deploy                    # Deploy current directory
acme deploy --prod             # Deploy to production
acme deploy --preview          # Create preview deployment
acme logs --follow             # Stream logs in real-time
acme logs --json --since 1h    # JSON logs from last hour
acme env pull                  # Pull env vars to .env.local
acme env push                  # Push local env vars

## JSON Output
All commands support --json for machine-readable output:
  acme deploy --json           # Returns {"url": "...", "id": "..."}
  acme list --json             # Returns array of deployments
  acme logs --json             # Returns newline-delimited JSON

## Exit Codes
0 = Success
1 = General error (see stderr for details)
2 = Authentication error
3 = Resource not found
4 = Rate limited (retry after delay)

## When to Use CLI vs SDK
- CLI: Deployments, quick operations, CI/CD, scripting, one-off tasks
- SDK: Application code, webhooks, complex workflows, type safety

## When NOT to Use Acme CLI
- Real-time event streaming (use SDK with WebSocket support)
- High-frequency operations (SDK has better rate limit handling)
- Complex conditional workflows (SDK provides better control flow)

The CLI-specific sections (installation methods, auth patterns, exit codes, and --json documentation) are critical for agent success. The "When NOT to Use" section prevents over-recommendation.

MCP Servers

MCP servers can expose CLI commands as callable tools. When your CLI has MCP presence:

1. The agent discovers available commands
2. The agent calls them with proper arguments
3. The agent parses structured output

APIs with both CLI and MCP coverage dominate their categories because agents have multiple ways to accomplish tasks.

Shell Completions

Shell completions aren't just for humans. When an agent is uncertain about a flag or subcommand, it may reference completion scripts to understand available options:

# Completions provide structured command metadata
_acme_completions() {
  local commands="deploy logs env auth"
  local deploy_flags="--prod --preview --json --help"
  ...
}

Well-structured completions are machine-readable documentation.

Layer 4: Shell Execution (The Proof)

This is where CLI recommendations succeed or fail, and where the gap between good and bad CLIs becomes massive.

When a coding agent runs your CLI, it:

1. Constructs the command with flags, arguments, and environment
2. Executes it in a shell, sandbox, or container
3. Captures output including stdout, stderr, and exit code
4. Parses the result to verify success and extract data
5. Handles errors or gives up and tries something else

The Factors That Determine Success

Our Devtool Arena CLI benchmarks found massive variance in execution success. Here's what separated winners from losers:

The --json Flag Is Non-Negotiable

CLIs with --json output scored 18 points higher on average in our benchmarks.

Why? Agents need to parse output programmatically. Human-readable tables are ambiguous:

# Human-readable: hard for agents to parse
$ acme list
NAME        STATUS    CREATED
my-app      running   2 days ago
test-app    stopped   1 week ago

# JSON: unambiguous for agents
$ acme list --json
[{"name":"my-app","status":"running","created":"2026-08-28T..."},
 {"name":"test-app","status":"stopped","created":"2026-08-23T..."}]

The agent can reliably extract my-app from JSON. Parsing the table requires guessing column boundaries.

Authentication Patterns Matter

In our benchmarks, 83% of successful CLI authentications used one of two patterns:

1. Environment variable: ACME_API_KEY=xxx acme deploy
2. Flag: acme deploy --api-key xxx

Interactive browser flows work for humans but break agent workflows. The best CLIs support both:

# Interactive (for humans)
$ acme auth login
Opening browser to authenticate...

# Non-interactive (for agents and CI)
$ acme auth login --token $ACME_API_TOKEN
Authenticated as [email protected]

Only 1 out of 35 CLI tools in our benchmarks supported keyless authentication. That's an opportunity for differentiation.

Error Messages That Help Agents Recover

We tracked error recovery rates across CLIs:

Specific errors with actionable guidance = high recovery. Generic errors = agent gives up.

Exit Codes Signal Success

Agents rely on exit codes to determine if a command succeeded:

# Good: Clear exit code contract
$ acme deploy --prod
Deployed to https://my-app.acme.app
$ echo $?
0

$ acme deploy --prod --invalid-flag
Error: Unknown flag --invalid-flag
$ echo $?
1

CLIs that always exit 0 (even on failure) break agent decision-making.

Composability: The Unix Philosophy Pays Off

Agents love CLIs that compose well with other tools. The Unix philosophy (small tools that do one thing well and work together via pipes) maps perfectly to how agents chain operations:

# Agent chains commands to accomplish complex tasks
vercel list --json | jq '.[0].url' | xargs curl -s | head -20

# Filter logs, extract errors, count by type
acme logs --json --since 1h | jq 'select(.level=="error")' | jq -s 'group_by(.code) | map({code: .[0].code, count: length})'

CLIs that output clean JSON, accept stdin, and work with jq, grep, and xargs give agents superpowers. CLIs that only work interactively or produce unparseable output force agents to work around them or give up.

Timeouts and Long-Running Commands

Some CLI operations take time: deployments, builds, large uploads. Agents need to know:

1. Is it still running? Progress indicators help humans but confuse agents.
2. How long should I wait? Reasonable timeouts prevent agents from hanging.
3. Did it actually finish? Exit codes and final status lines matter.

The best CLIs provide --wait flags that block until completion, or --no-wait flags that return immediately with a status URL:

# Good: Returns immediately with status URL
$ acme deploy --no-wait
Deployment started. Status: https://acme.app/deployments/abc123
$ echo $?
0

# Good: Blocks until complete with clear final status
$ acme deploy --wait
Deploying... (typically takes 30-60 seconds)
✓ Deployed to https://my-app.acme.app
$ echo $?
0

The Full Picture: How a CLI Recommendation Happens

Let's trace a real example. A developer asks Claude Code:

"Deploy my Next.js app to Vercel"

Layer 1 (Training): Claude knows about the Vercel CLI from training data. It has seen vercel deploy in countless GitHub repos and tutorials.

Layer 2 (Search): Claude may search "vercel cli deploy nextjs" to check for recent changes or best practices. Vercel's CLI docs rank well.

Layer 3 (Context): Claude checks for llms.txt and MCP availability. The Vercel CLI has strong context signals.

Layer 4 (Execution): Claude constructs and runs the command:

$ vercel deploy --yes --json
{
  "id": "dpl_abc123",
  "url": "https://my-app-abc123.vercel.app",
  "readyState": "READY",
  "alias": ["my-app.vercel.app"]
}

The --yes flag skips interactive prompts. The --json flag gives parseable output. Claude extracts the URL, verifies the deployment is READY, and reports: "Deployed to https://my-app.vercel.app".

Why Vercel wins at every layer:

Contrast with a losing CLI: A deployment tool that requires interactive project selection, outputs progress bars instead of JSON, and takes 3 minutes to complete would fail at Layer 4 even if Layers 1-3 were strong.

Why Newer CLIs Can Win

The 4-layer model explains something counterintuitive: newer CLIs can beat established players in agent recommendations.

A new CLI built with --json output, environment variable auth, clear errors, and an MCP server can outrank an established tool that's coasting on training data alone.

Case Study: How Daytona Won the Sandbox Category

In our benchmarks, Daytona moved from lower rankings to #1 in the sandbox category. Here's what they did:

Before optimization:

• Interactive-first CLI design
• Limited JSON output
• Complex multi-step authentication
• No MCP presence

After optimization:

• Full --json support on all commands
• DAYTONA_API_KEY environment variable auth
• Non-interactive workspace creation: daytona create --repo github.com/user/repo --yes
• MCP server for direct agent integration
• Clear error messages with fix suggestions

The result: Agents could go from "create a dev environment for this repo" to a working workspace URL in under 60 seconds, with zero interactive prompts. Competitors requiring manual setup steps couldn't keep up.

This pattern repeats across categories. The CLI that makes agent execution easiest wins, regardless of market share or brand recognition.

How to Measure Your CLI's Position

For each layer:

Layer 1: Training Data

• Test: Ask Claude/GPT (without web search) about your CLI
• Check: Does it know the correct command name and common flags?
• Metric: Baseline familiarity in chat models

Layer 2: Web Search

• Test: Search "[your CLI] deploy example" and similar queries
• Check: Do your docs appear? What position?
• Metric: Search visibility for task-oriented queries

Layer 3: Context Retrieval

• Test: Does yourdomain.com/llms.txt include CLI documentation?
• Check: Is your CLI available via MCP?
• Metric: Presence in agent context systems

Layer 4: Shell Execution

• Test: Ask Claude Code to run a task with your CLI
• Check: Does it succeed? How many retries?
• Metric: Task completion rate, error recovery rate

The Devtool Arena CLI leaderboard benchmarks CLIs across all four layers so you can see where you stand against competitors.

The Optimization Priority

Based on effort vs. impact from our benchmarks:

Most teams skip straight to marketing and ignore the mechanical factors (1-3) that determine execution success.

Related Reading

• How Coding Agents Actually Decide Which SDK to Use: The parallel framework for library selection
• We Tested 70+ APIs in Claude Code and Codex: The benchmark data behind these findings
• Why Claude Code Isn't Recommending Your Library: The 4 fixable reasons and how to address each
• The Devtool Visibility Stack in 2026: The measurement framework for API teams
• AEO/GEO for Dev Tools: Why consumer GEO tools don't work for CLIs

FAQ

Do coding agents prefer CLIs over SDKs?

It depends on the task. For one-off operations, deployments, and quick commands, agents often prefer CLIs because they can execute and verify immediately. For application code and complex workflows, SDKs provide better type safety and integration. The best APIs offer both, and agents choose based on context:

What's the most important CLI feature for coding agents?

The --json flag. In our benchmarks, CLIs with JSON output scored 18 points higher on average. Agents need to parse output programmatically, and JSON is unambiguous.

Second most important: non-interactive authentication. CLIs that require browser flows block agent execution entirely.

Should my CLI support interactive mode?

Yes, for humans. But also support fully non-interactive operation for agents and CI/CD:

# Interactive (for humans)
$ acme deploy
? Select project: [Use arrows to select]
  my-app
  test-app

# Non-interactive (for agents and CI)
$ acme deploy --project my-app --yes --json
{"url": "https://my-app.acme.app", "status": "deployed"}

Common flags for non-interactive mode: --yes, --force, --no-input, --quiet, --json.

How do I know if agents can use my CLI?

Test it directly. Ask Claude Code to complete a real task using your CLI. Track:

1. Did it succeed on the first try? Good CLI.
2. Did it need retries but eventually succeed? Check error messages.
3. Did it fail and switch to a competitor? Major problem.
4. Did it not even try your CLI? Discovery problem (Layers 1-3).

For systematic testing, the Devtool Arena CLI leaderboard benchmarks execution success, discoverability, and completion rates.

Why did my CLI work in Claude Code but fail in Codex?

Claude Code and Codex have completely different execution environments:

A CLI optimized for one may fail in the other. Test across multiple agents and track results separately.

Is it worth building a CLI if I already have an SDK?

Yes. Only 31 out of 73 APIs in our benchmarks have CLI support, which means a 58% gap. CLIs let agents execute operations directly instead of generating code for developers to run.

The investment calculation:

If your competitors don't have a CLI, shipping one is a fast way to dominate your category in agent recommendations.

What package managers should my CLI support?

Prioritize based on your audience:

More installation methods = higher installation success rate for agents.

How do I handle CLI commands that take a long time?

Long-running commands (deployments, builds, large operations) need special handling:

# Bad: Agent doesn't know if it's still running
$ acme build
Building... [progress bar that updates in place]

# Good: Clear start/end, or async with status URL
$ acme build --json
{"status": "building", "eta_seconds": 45}
... (waits) ...
{"status": "complete", "artifact_url": "https://..."}

# Also good: Return immediately with polling endpoint
$ acme build --async
Build started. Poll status at: https://api.acme.io/builds/abc123

How do I track my CLI's agent visibility?

Lightsage tracks CLI visibility separately from API visibility. The Devtool Arena CLI leaderboard shows rankings across:

• Execution success rate: Does agent-generated CLI usage work?
• Discoverability: Do agents find and recommend your CLI?
• Completion rate: Do agents finish tasks using your CLI?
• Tool calls: How efficient is the agent when using your CLI?

You can see how your CLI compares to competitors and track improvements over time.

Key Takeaways

1. CLIs are execution, not generation. Agents can run CLIs directly and verify success, which is more powerful than generating code developers must run.

2. Only 42% of APIs have CLI support. This is a massive gap and an opportunity to differentiate.

3. The --json flag is worth +18 points. Structured output lets agents parse results reliably.

4. Non-interactive auth is non-negotiable. Browser flows block agent execution. Support TOOL_API_KEY environment variables.

5. Error messages determine recovery. Specific errors with fix suggestions = 87% recovery rate. Generic errors = 15%.

6. Newer CLIs can win. The decision stack rewards execution quality over market share. Daytona went from lower rankings to #1 in sandboxes by optimizing for agents.

7. Composability matters. CLIs that work with jq, pipes, and scripting give agents superpowers.

Check Your CLI's Agent Visibility

Your CLI might be invisible to coding agents, or breaking every time they try to use it. Find out where you stand.

Free: Check the CLI leaderboard on Devtool Arena. See how your CLI ranks on execution success, discoverability, and completion rates.

Full audit: Get a Lightsage visibility report. Understand exactly why agents aren't using your CLI and get a prioritized fix list.

Community: Join the AI DevTool Demo Night. 3,500+ developer community, 50+ DevTool companies, hosted at AWS SF.