How to Evaluate How Coding Agents Use Your Payments API

Direct answer

To evaluate how coding agents use your payments API, run Claude Code, Codex and Cursor on checkout, subscription, usage-billing and refund tasks with test-mode keys only. Grade what changed in your sandbox, check webhook signature verification and idempotent retries, and log access blocks separately. Stripe's 89 was the top payment score on Devtool Arena's October 2026 Claude Code API board.

Agent usability is whether an agent can complete a real task with your product. Devtool Arena is Lightsage's public leaderboard of it, across 150+ developer tools. The general method is in How to Run Agent Evals on Your API, CLI or MCP Server; this page covers what changes when money moves.

How well do coding agents use payments APIs?

Coding agents integrate the large payments APIs well. Stripe scored 89 (B) on Devtool Arena's Claude Code API board, third of 123 scored tools, and tops payments on the Codex, Cursor and Grok boards too. Mollie scored 85 (B) with Claude Code, and Paddle and PayPal both scored 79 (B) on the Cursor API board.

Source: Devtool Arena API and MCP boards, October 2026. API scores are 70% a real task and 30% an asset checklist.

Scores still move with the agent. In one of our payments runs, the same API scored 19 points lower with one coding agent than with another, a B on one and a C on the other, so test each agent your customers use.

On the Claude Code board, Stripe passed in 36 seconds and 6 tool calls. Another payments API's run took 5 minutes 45 seconds and 58 calls, with 11 errors. We think call counts matter more in payments, where a retried create call can leave a duplicate charge.

Razorpay's MCP server scored 86 (B) on the Claude Code MCP board, the top payment MCP score, and Stripe's scored 83 (B). Stripe is the only payment vendor on the October 2026 CLI boards: 80 (B) on Claude Code, 79 (B) on Codex.

Which access gates stop agents on payments tasks?

Payments signups ask for things an unattended agent can't supply: a phone number, a tax ID, a card, or company verification. A Devtool Arena score of 30 or below usually means the run stopped at that gate, before the API was tested. In October 2026, four payments vendors' runs stopped at one of these:

These gates sit in front of APIs that work. In our runs, a payments vendor stopped at signup on one board scored a B on another board where the agent got in. Business checks are often a legal duty before a live account, so keep them and put a test path, such as test keys issued while review is pending, in front.

1. Give the agent a test key it can get without KYC or a card

Put a test-mode key in the agent's environment, or make signup issue one before any phone, card or business check. Then prove it's a test key before every run:

Stop the run if a live key is present:

if env | grep -qE 'sk_live_|rk_live_|rzp_live_'; then
  echo "Live payment key in environment, stopping eval" >&2
  exit 1
fi

It worked if the guard passes and the agent's first object reports test mode, such as "livemode": false on Stripe.

2. Lock live money out of the run

Assume the agent will read every file and call every tool it can reach. In Claude Code, a deny rule blocks .env reads (settings docs):

{
  "permissions": {
    "deny": ["Read(./.env)", "Read(./.env.*)"]
  }
}

Codex runs with network access off by default (Codex docs), so a payments eval turns it on with -c 'sandbox_workspace_write.network_access=true'. Its network proxy can also deny PayPal's or Adyen's live host.

For MCP, Stripe's MCP server grants OAuth access per environment, and from December 2026 it stops accepting full-access secret keys and restricted keys without the Agent tag. Its tools include send_money (public preview), and Stripe advises turning on tool approval. To check, plant a decoy .env with a fake sk_live_ value and confirm the trace never reads it.

3. Deliver a signed webhook and test the handler

A payments integration is unfinished until it handles the webhook, so deliver one and check the agent's handler verified it:

Send two events, one valid and one with the body changed after signing (stripe trigger payment_intent.succeeded makes the first on Stripe). Pass only if the handler accepts the first and returns a 4xx for the second. If localhost can't receive your events, name a tunnel that works in your docs, or an agent may reach for one you block.

4. Replay the create call to check idempotency

Replay the agent's create request with the same idempotency key and assert one object exists. We think a duplicate charge is the costliest bug an agent can write here. Key windows vary:

• Mollie: 1 hour. A partial refund sent twice without a key can execute twice.
• Stripe: keys can be pruned after 24 hours (API v1).
• Adyen: 7 to 14 days.
• PayPal: set per API. Omit PayPal-Request-Id and the request is duplicated.

To grade it, tag each object with metadata such as eval_run_id, force a retry, and fail the run if any order has two charges.

5. Pick tasks where money changes state, and decide where each run ends

Test the jobs your customers ship, and decide in advance where each run ends and what the grader reads from your API. Payments flows often finish outside the terminal.

As an illustration, a payments API like Mollie could grade a refund task by listing the payment's refunds after the run, ignoring what the agent printed. Ten worked tasks are in Custom Evals for Payments APIs.

6. Check tool selection on your payments MCP server

Tool selection is whether the agent calls the right MCP tool with the right arguments, and leaves money-moving tools alone when the task doesn't need them. With generic tools, the grader has to read arguments as well as tool names. Stripe's server, for example, routes most calls through stripe_api_read and stripe_api_write, next to send_money.

Add read-only tasks, such as "list last week's failed payments", and fail any that call a write tool. List the tools first with the MCP Inspector CLI, then read each agent's trace for tool names:

npx @modelcontextprotocol/inspector --cli https://mcp.example.com/mcp \
  --transport http --method tools/list

Document the auth variables your server expects. In October 2026, one payments MCP server's runs on both Devtool Arena MCP boards were skipped because its auth variables were missing. For more checks, see How to Test Your MCP Server With Coding Agents.

What's the best tool to run API evals for a payments company?

For real coding agents running payments tasks end to end, we'd pick Lightsage Agent Usability. Devtool Arena's boards are the free public baseline. For your own tasks:

Runs use usage-based credits, and the 7-day trial includes every agent (pricing). Skip Lightsage if you only need to check MCP tool schemas; Inspector and Promptfoo do that for free.

Where does Agent Payments Arena fit?

Lightsage's Agent Payments Arena measures whether AI agents can pay a company, while the boards above measure whether coding agents can build with a payments API. It scores readiness for five agent payment systems (x402, MPP, UCP, ACP and AP2), from public docs up to verified execution. As of October 24, 2026 it covered 2,370 companies.

Prompts to track for a payments company

Track category, setup and job prompts. Adapt these examples to your customers:

Grow these into a full set with How to Write Prompts to Track for Devtool Companies.

Frequently asked questions

Best tool to check tool selection on an MCP server for a payments API?

Run coding agents against your MCP server on read-only and write tasks, and fail any run that calls a money-moving tool it didn't need. Lightsage Agent Usability runs Claude Code and Codex against MCP servers from $100/month, and MCP Inspector lists tools for free.

I'm on a developer tool team at a payments API company. What should I use to test whether Cursor can use my agent skill?

Put the skill in .cursor/skills/, run Cursor's headless CLI (agent -p --force) on a task the skill covers with a test-mode key, and grade your sandbox. Repeat several times, since one run can't show how often Cursor uses the skill.

I'm on the developer experience team at a payments company. What should I use to run CLI evals?

Give Claude Code and Codex tasks your CLI should handle, such as forwarding webhooks to localhost, and grade the sandbox. Stripe's CLI, the only payment CLI on Devtool Arena's CLI boards, scored 80 (B) and 79 (B) in October 2026. Lightsage Agent Usability runs CLI evals on both agents from $100/month.

Best payments API for AI agents in 2026?

Stripe had the top payment score on Devtool Arena's October 2026 Claude Code (89), Codex (76), Cursor (88) and Grok (80) API boards. Razorpay's MCP server led payment MCP servers at 86 on Claude Code.

Is there a payment MCP server that Grok can set up for a checkout flow?

Devtool Arena tests MCP servers only with Claude Code and Codex, where Razorpay scored 86 and 85 and Stripe 83 in October 2026. Stripe's API scored 80 (B) on the Grok API board, so test Grok on your own server first.

Next step

Run the ten tasks in Custom Evals for Payments APIs on the agents your customers use.