How to See If Coding Agents Pick a Competitor
Direct answer
To see if coding agents pick a competitor, ask buyer questions two ways: open prompts that name no brand, and head-to-head "X or Y" prompts. Run each in three agents, two stacks and five runs. Score mention, recommendation and first choice separately and read the reasons. A competitor is winning only if it comes first across agents, stacks and runs.
Why one answer from one agent tells you nothing
A coding agent's pick depends on the harness, the project it runs in, the wording and chance. Coding agent visibility is whether coding agents find, mention and recommend your product when a developer asks for a tool. Agent discoverability is whether an agent names you in an open question, and agent preference is whether it picks you over a named competitor.
The harness alone changes the answer. Claude Code runs on the same models as Claude, yet across 24,135 responses the two mentioned only about 1 in 5 of the same brands for a prompt (Profound, August 24, 2026). One agent also varies between runs: two Claude Code responses to the same prompt shared about 2 in 5 of the brands they mentioned. Each agent agreed with itself more than with the other, so spread runs across agents before adding more runs of one.
The steps below use one illustration: a search company like TinyFish checking whether coding agents pick Tavily. It shows the method and doesn't describe what TinyFish does.
1. Write open discovery prompts for the jobs you compete on
Write five to ten prompts that each describe a job a developer hands an agent, with no brand names. Mix questions with tasks, because developers ask both ways. We think wording decides which market the agent answers for: a prompt about scraping pages calls up different vendors than one about search for an agent.
What should this agent use to search the web and read the results?
Add live web search to the research agent in this repo.
Our agent gets raw HTML back from search. What should we switch to?
To check this step, confirm that a trial run of each prompt returns vendors from your market. Our guides to writing prompts to track and prompts for search companies go deeper.
2. Write head-to-head prompts that name you and the competitor
Write two to four prompts that ask the agent to choose between you and one competitor for a specific task. Include the context a real buyer gives, such as stack or volume. Write each twice with the names swapped so the first-named product gets no edge. Open prompts show whether you're in the conversation. Head-to-head prompts show who wins once a developer has shortlisted both.
I'm choosing between TinyFish and Tavily for web search in this agent. Which should I use?
I'm choosing between Tavily and TinyFish for web search in this agent. Which should I use?
Our FastAPI service runs about 50,000 agent searches a month. Tavily or TinyFish?
A tie is an answer that declines to choose. To check this step, confirm that every prompt names both products and one task, and has a swapped twin.
3. Run every prompt in two stacks, three agents and five runs
Create two starter projects in different stacks, such as a Next.js app and a FastAPI app. Make each one a git repository, since codex exec expects to run inside one (Codex non-interactive mode). From inside each project, run every prompt in at least three agents, five times each, and save every answer.
We think the project moves the winner too. A coding agent answers from inside the repo, and a Next.js app and a FastAPI app hand it different languages, package managers and dependencies to fit its pick to. As an illustration, a competitor with a strong Python SDK and a thin JavaScript one could win every FastAPI run and lose every Next.js run. Two stacks is the smallest test that shows whether a lead depends on one of them.
Here's the manual version, using Claude Code's print mode (CLI reference) and codex exec:
cd nextjs-starter
P="I'm choosing between TinyFish and Tavily for web search in this agent. Which should I use?"
for i in 1 2 3 4 5; do
claude -p "$P" > claude-h2h-a-$i.txt
codex exec "$P" > codex-h2h-a-$i.txt
done
Repeat the loop in fastapi-starter, then in a third agent such as Cursor or OpenCode. Record whether each run searched the web, because most runs never search. Vercel's early sampling found coding agents searched in roughly 20% of prompts (Vercel, February 9, 2026), and Profound saw Claude Code search in 13% of its responses. Five runs is our floor, since at five runs one odd answer moves a share by 20 points. To check this step, confirm you have one file per prompt, agent, stack and run.
Is there a tool that shows if agents pick a competitor?
Yes. Lightsage's Agent Preference runs the head-to-head half of this matrix against one tracked competitor at a time. Each comparison takes 1 to 20 prompts, up to 10 coding agents in sandboxes, 1 to 20 runs per prompt and agent, and an optional persona. A judge records won, lost or tie for each run and rolls them into a win rate. Running with search grounding on and off separates what the model already believes from what it finds on the web.
Agent Preference has no landing page yet. It sits alongside Agent Discoverability, which runs your open prompts daily across 12 coding agents and 7 answer engines.
The manual method is enough for a one-off decision. If you'd rather count installs than recommendations, phrase each prompt as a task and check what lands in package.json or requirements.txt. Devtool Arena's Agent Landscape ranks tools by how often coding agents recommend them, and our comparison of coding agent visibility tools covers other options.
4. Score mention, recommendation, first choice and no single pick separately
Give every answer five fields: mentioned you, recommended you, first choice, no single pick, and failed run. Add won, lost or tie for head-to-head prompts. Report each result as a share of runs with its base, such as "12 of 30 runs".
The signals come apart. In Profound's sample, Claude Code mentioned 6.6 brands per response on average, so a mention puts you among about six names and only one of them comes first. An agent can also mention a product while advising against it.
| Run | Agent | Stack | Mentioned you | Recommended you | First choice | Reason given |
|---|---|---|---|---|---|---|
| 1 | Claude Code | Next.js | Yes | Yes | Competitor | "Returns results ready for the model" |
| 2 | Codex | Next.js | Yes | Yes | None | Listed both, chose neither |
| 3 | Codex | FastAPI | No | No | Failed run | Empty response |
The rows are illustrative. Give "no single pick" its own column. Claude Code put a list in 94% of the responses Profound sampled and a table in 54%, and we think answers laid out as options produce many "listed both, chose neither" runs. Count a clarifying question back to you as no single pick too. Keep failed runs out of the denominator but in the report. Our guide to agent-led growth metrics explains why mention and recommendation rates stay separate.
5. Read the reason behind every competitor pick
For every run where the competitor came first, copy the sentence the agent used to justify it. Tag it as default choice, stack fit, feature, price, docs or SDK quality, or ecosystem.
The reason names the fix, and the score can't. A competitor that wins as "the default choice" needs a different fix from one that wins on a price the agent quoted. When Claude Code does look something up, it reads pages you control: nearly three-quarters of the Claude Code agent visits Profound observed went to documentation, informational and pricing pages. We think that makes a reason quoting a feature, limit or price the easiest to act on, because you can check the page it came from.
| Reason in the transcript | What it points to |
|---|---|
| "The default choice" or "most popular" | Reputation. Win a narrower job first |
| A feature you also have | The agent doesn't connect it to you. Check your docs |
| A plan limit or price | Your pricing page. Compare the quote with what's live |
| An old product name or a retired API | Stale knowledge. Compare runs with web search on and off |
To check this step, confirm you can name the two tags behind most competitor wins.
6. Call a winner only when it holds across agents, stacks and runs
The competitor is winning when it comes first in more decided runs than you, in at least two agents and in both stacks. That rule is ours, and it's strict on purpose. A narrower lead has a cause you can name. A lead in one agent points to a harness lean, and a lead in one stack points to a gap in your quickstart. A lead on one wording points to a use case you may decide to leave alone.
Then check whether agents can use the winner, because recommended and usable are separate checks. In one of our October 2026 meeting-bot runs on Devtool Arena's Claude Code API board, Claude Code passed the vendor's API task, while the same vendor's MCP server couldn't be set up on the Claude Code or Codex MCP boards. A setup block like that says nothing about how well an agent works with the server, but it decides whether an agent can use it at all.
Lightsage Agent Usability runs the same kind of task against your own API, MCP server or CLI. To check this step, fill in this sentence: "The competitor came first in N of M decided runs, in three agents and both stacks, mostly for stack fit." If you can't, keep testing.
How to measure progress
Track two numbers for each agent and stack: the competitor's first-choice share on your open prompts, and your head-to-head win rate. Rerun the same prompts monthly and after every docs, SDK or pricing change, logging the agent and model version. Count a change as real when it beats one run's worth (20 points at five runs) in at least two agents. The reason tags should move with it.
Frequently asked questions
Is there a tool that can tell me when agents pick a competitor?
Yes, if it reruns the same prompts on a schedule. Lightsage Agent Discoverability reruns tracked prompts daily and records whether each brand was the primary recommendation. Agent Preference reports wins, losses and ties against one named competitor. For a one-off check, the manual method in this guide costs only your agent usage.
Is there a platform that measures agent preference across OpenCode and Codex?
Lightsage Agent Preference runs one head-to-head comparison across up to 10 coding agents, and Codex and OpenCode are both among the 12 coding agents Lightsage supports. Codex is on every plan, and OpenCode needs the Growth plan, Enterprise or the 7-day trial.
Is there cheap software that shows if agents pick a competitor for my CLI?
The cheapest option is the manual method: run your prompts with claude -p and codex exec inside a sample repo, which costs only your agent usage. Lightsage starts at $100 a month after a 7-day trial with 1,000 credits, and its Startup plan covers Claude Code and Codex. For a CLI, phrase prompts as tasks in a repo so the agent has a reason to reach for a command-line tool.
Next step
Once the transcripts show why the competitor wins, use our guide to ranking higher in coding agent search to work on the fix.