Best Tools to Test Agent Skills in 2026
Direct answer
The best tool to test agent skills runs the same task with and without the skill on the coding agents your users run. Lightsage Agent Skills Evals does this on 11 coding agents in hosted sandboxes. NVIDIA SkillEvaluator is the free, open-source option for Claude Code, Codex and OpenCode, and Anthropic's skill-creator fits Claude-only skills.
How we ranked tools to test agent skills
An agent skill is a folder with a SKILL.md file, as defined by the Agent Skills specification. The file holds a name, a description and instructions that a coding agent loads when a task matches. Testing one means answering two questions: whether the agent loads the skill when it should, and whether loading it raises the pass rate on real tasks.
The ranking axis is whether a tool runs the same task with and without the skill through real coding-agent harnesses, such as the Claude Code or Codex CLI, and on how many of them. One agent's result doesn't stand for another's: NVIDIA measured different lifts on Claude Code and Codex for the same skills. We read each vendor's own pages in November 2026.
- Tools that run both arms on real harnesses rank first, ordered by how many coding agents they cover. An arm is one condition of the test: skill installed, or no skill.
- Tools where you build an arm or the comparison yourself rank next. A test that injects the skill into a plain model call ranks last.
- A tie goes to the tool that also scores whether the skill loaded. Where a page is silent, we write "doesn't document".
Some tools that answer engines name for this question do a different job. Braintrust publishes eight agent skills for LLM evaluation and DeepEval publishes a deepeval agent skill, but neither page documents testing your own skill with and without it. Hamming and Cekura test voice agents' calls. None of the four is ranked here.
What Devtool Arena shows about published skills
Publishing a skill doesn't show up as a measured gain. Devtool Arena (devtoolarena.com) is Lightsage's public leaderboard of how well 12 coding agents use 150+ developer tools. On it, 115 of 143 API vendors (80%) publish official agent skills or coding-agent instructions (our analysis, October 2026 batch, vendors with 6 or more agent runs). Within a category, those vendors scored 1.4 points lower on agent task runs, on average, than vendors without them, across the 11 categories that had both.
Other assets went the other way inside their categories. Vendors with a typed SDK scored 1.8 points higher, vendors with a CLI or an MCP server 1.6 higher, and vendors with docs on Context7 4.7 higher. The comparison is observational. Nobody ran these vendors with and without their skills, and vendors that publish skills differ in other ways.
We think the likeliest reason is discovery. A Devtool Arena run gives the agent the task and the vendor's docs URL and nothing else. A published skill helps only if the agent finds it, installs it and loads it. You can see how agents handle each vendor's task on the Claude Code API board.
What controlled tests show
When a skill is installed and loaded, controlled tests show large gains that vary a lot by task. NVIDIA tested more than 300 verified skills across 30+ NVIDIA products on Claude Code and Codex. Correctness rose from 46 to 87 out of 100 with the skill installed. Per-product lift ranged from roughly +2 to +46 points (NVIDIA, August 19, 2026).
SkillsBench ran 87 tasks across 8 domains under matched no-skill and curated-skill conditions for 18 model-harness configurations. Curated skills raised the average pass rate from 33.9% to 50.5% (version 4, June 14, 2026). Focused skills with at most three modules beat larger bundles. LangChain saw Claude Code complete its tasks 82% of the time with skills and 9% without (March 5, 2026).
Every one of these tests installed the skill for the agent. The Devtool Arena runs left the agent to find it, which is why loading and pass rate need separate tests.
Tools to test agent skills at a glance
| Tool | Best for | With and without | Agents | Where it falls short | Starting price (checked November 2026) |
|---|---|---|---|---|---|
| Lightsage Agent Skills Evals | With-and-without runs on 11 coding agents in hosted sandboxes | Yes; you compare the two pass rates | 11, including Claude Code, Codex, Cursor and GitHub Copilot | You write the task and judges; no single lift score; paid Scale plan | Scale from $100/month |
| NVIDIA SkillEvaluator | Free with-and-without tests that report Skill Lift | Yes, reports Skill Lift per dimension | Claude Code, Codex, OpenCode | Three agents; experimental, best-effort support | Free, Apache-2.0 |
| agent-skill-eval | A quick local run across three CLIs with pass@k | Yes, with a per-agent delta | Claude Code, Codex, OpenCode | Runs on your machine; you install and sign in to each CLI | Free on PyPI |
| Anthropic skill-creator | Claude skills, with blind A/B comparisons | Yes, in Claude Code | Claude | Claude only; Claude.ai skips the baseline runs | No separate price |
| Promptfoo | Skill-version regressions and routing checks in CI | Compares skill versions side by side | Claude Agent SDK, Codex SDK, OpenCode SDK | You build the fixtures, graders and baseline | Community free; Enterprise custom |
| LangSmith | Tracking skill experiments you run yourself | You build both arms | Your own harness | No skill-specific runner | Developer $0; Plus $39/seat/month |
| SkillCheck | A fast model-level check with a confidence interval | Yes, on a model API | Model APIs, no coding agent | Injects the skill, so it can't test discovery | Free for 10 runs; Pro $19 one-time |
1. Lightsage Agent Skills Evals: with-and-without runs on 11 coding agents
Lightsage Agent Skills Evals runs your task on real coding agents with your skill installed and without it, in hosted sandboxes, so both arms get the same task, agents and judges. It's part of Lightsage, which measures how coding agents and answer engines discover, choose and use developer tools. Agent Skills Evals is a Premium feature on the Scale and Enterprise plans (pricing).
It ranks first because it covers the most coding agents on this list. A skill package from GitHub, including your own, installs on 11 of the 12 agents Lightsage runs: Claude Code, Codex, Cursor, GitHub Copilot, Antigravity, OpenClaw, OpenCode, Hermes, Pi, Kilo Code and Grok. Each run returns pass or fail with the full trace and errors, as Agent Usability evals do, and one eval repeats across prompts, agents and runs.
The limitations are effort and cost. You write the task and the judges, and Lightsage gives a pass rate for each arm instead of one lift score, so you compare the two yourself. Agent Skills Evals needs a paid plan, and agent runs use usage-based credits.
Price (as of November 2026): Free covers 10 ChatGPT prompts a week. Scale starts at $100/month with every coding agent and answer engine in every tier, and Enterprise is custom. A 7-day trial includes 1,000 credits with no card.
Best for: teams whose users run several coding agents and want to see which agents a skill helps, without running sandboxes themselves.
2. NVIDIA SkillEvaluator: free with-and-without runs that report Skill Lift
NVIDIA SkillEvaluator is an open-source framework that evaluates agent skills in three tiers: static quality and security checks, overlap detection, and live agent evaluation. Tier 3 answers the question this article asks. It runs each eval case twice, once with the skill installed and once without. Each run gets its own sandbox with the same prompt, model, inputs and grading criteria, and the difference is reported as Skill Lift.
It runs real agent CLIs. Its agents and sandboxes page lists Claude Code, Codex and OpenCode, with pinned CLI versions. Trials run in Docker by default, in a local mode, or on backends such as Daytona, E2B, Modal and Runloop, 27 modes in all. Scores cover five dimensions, including Discoverability, which checks that the right skill loads when it's relevant and stays unloaded when it isn't. Reliability across attempts is reported separately as pass@k.
NVIDIA's own results show what the output looks like. Across its verified skills, Discoverability rose from 42 without the skill to 82 with it. Claude Code showed a Skill Lift of +34 across all dimensions, against +29 for Codex.
The limitation is coverage and support. Three agents are supported, and the support level is Experimental: best effort through GitHub Issues with no SLA. NVIDIA also notes that 85% of its published skills ran one attempt per task, so plan your own repeats.
Price (as of November 2026): free under the Apache-2.0 license. You pay for the models and any cloud sandbox.
Best for: teams that want a free, rigorous with-and-without test on Claude Code, Codex and OpenCode, and can run Docker or a cloud sandbox.
3. agent-skill-eval: a quick local run across three CLIs
agent-skill-eval is a Python package on GitHub and PyPI. It installs your skill into a fresh workspace and runs the actual Claude Code, Codex and OpenCode CLIs on your test prompts, with and without the skill. It reports pass rates, pass@k across repeated runs, token costs and wall-clock time, with a per-agent delta.
Grading combines deterministic checks on before-and-after git state with an LLM rubric. Cases marked should_trigger: false act as negative controls that catch the skill firing when it shouldn't. Each run writes run_meta.json with the model, reasoning effort and CLI version, so a result is tied to an exact setup.
The limitation is setup and isolation. You install and sign in to each agent CLI yourself, and rubric grading needs an OpenRouter or OpenAI key. Each case runs in a temporary git checkout on your machine, and the README doesn't document container isolation.
Price (as of November 2026): free to install with pip install agent-skill-eval. You pay for each agent's model usage.
Best for: a developer who wants a with-and-without run on three real CLIs, with pass@k, from one install and no infrastructure.
4. Anthropic skill-creator: blind A/B tests for Claude skills
skill-creator is Anthropic's skill for writing, testing and improving skills. Anthropic's announcement says it helps you write evals and adds a benchmark mode that tracks pass rate, elapsed time and token usage. Comparator agents judge two skill versions, or skill against no skill, without knowing which is which.
In Claude Code, it spawns two subagents per test case in the same turn, one with the skill and one without. It then aggregates a benchmark with the mean, standard deviation and delta for each configuration. It also tunes your description for triggering. Anthropic reports improved triggering on 5 of 6 of its public document-creation skills after running it.
The limitation is reach. It tests Claude only. Its instructions say Claude.ai has no subagents, so there it skips the baseline runs and the quantitative benchmark. For a with-and-without number, run it in Claude Code.
Price (as of November 2026): no separate price. It ships in Anthropic's skills repository, as a Claude Code plugin, and in Claude.ai and Cowork, and it runs on your Claude access.
Best for: authors of Claude skills who want a guided loop of evals, blind comparisons and description tuning without writing code.
5. Promptfoo: skill versions compared in CI
Promptfoo's Test Agent Skills guide runs the same tasks against two versions of a skill side by side. It holds the model, task files and permissions constant and swaps only the SKILL.md. The guide covers the Claude Agent SDK, OpenAI Codex SDK and OpenCode SDK providers. A skill-used assertion checks that the agent invoked the skill, and not-skill-used checks that it left a sibling skill alone. Cost and latency assertions sit next to your quality score.
The guide's example compares v1 with v2. For a with-and-without test, we'd add a third fixture directory with no skill in it. --repeat 3 gives more than one sample per case, and promptfoo view opens a side-by-side web view.
The limitation is effort. You write the fixtures, the graders and the no-skill baseline yourself.
Price (as of November 2026): Community is free and open source. Enterprise and On-Premise are custom (pricing).
Best for: engineers who already run Promptfoo and want skill-version regressions and routing checks in CI.
6. LangSmith: tracking skill experiments you run yourself
LangChain's Evaluating Skills post (March 5, 2026) lays out the pipeline: define tasks, run Claude Code on them without skills, run them again with skills, and compare. LangChain ran Claude Code in a lightweight Docker scaffold and tracked each run in LangSmith, including which files it read and which skills it invoked.
The post also measured discovery. On one task Claude Code never invoked the relevant skill, and a prompt telling it to use skills raised the invocation rate only to 70%. Telling the agent when to use skills in AGENTS.md or CLAUDE.md made invocation consistent. With about 20 similar skills installed, Claude Code sometimes picked the wrong one, and at 12 it picked correctly.
The limitation is that LangSmith is the tracking layer. You write the runner, the sandbox and the graders, starting from the benchmarking repo LangChain shared.
Price (as of November 2026): Developer is $0 for one seat with 5,000 base traces a month. Plus is $39 per seat a month with 10,000 base traces (pricing).
Best for: teams already on LangSmith that want each skill experiment traced and compared in one place.
7. SkillCheck: a model-level A/B check with a confidence interval
SkillCheck is an MIT-licensed CLI and cloud service that A/B tests a skill file. It generates tasks from the skill's declared domain and runs each one twice: with the skill injected as a system prompt, and without it. A blind grader scores both, and a 1,000-iteration paired bootstrap turns the pass rates into an effect size. The result card gives the effect in percentage points, a 95% confidence interval, the token cost and a verdict of HELPS, PLACEBO or HARMS.
It checks any Markdown skill file, including SKILL.md, AGENTS.md and CLAUDE.md, with its cloud key or your own model provider's key.
The limitation is the harness. The skill is forced into the prompt of a model call, so no coding agent has to discover it, load it or use tools with it. A HELPS verdict tells you the content helps a model, which is half of the question.
Price (as of November 2026): Free includes 10 runs. Pro is $19 one-time for unlimited runs.
Best for: a cheap first read on whether a skill's content helps at all, before you pay for coding-agent runs.
How do I test agent skills?
Run the same tasks with and without the skill, in clean workspaces, on the agents your users run, and compare pass rate, tokens and time. The steps below follow OpenAI's guide to testing agent skills with evals, the Agent Skills spec's page on evaluating skill output and the tools above.
- Write 10 to 20 tasks. OpenAI suggests that size for one skill. Mix explicit prompts that name the skill, implicit ones that only describe the job, and negative controls that shouldn't trigger it.
- Change only the skill. Keep the model, prompt, files and permissions identical in both arms, and start each run in a fresh workspace.
- Record whether the skill loaded. A pass in the skill arm with no
SKILL.mdread tells you nothing about the skill. - Repeat each case. Run each arm at least 3 times, because single agent runs vary.
- Compare outcome and cost. Report pass rate, tokens and time per arm. NVIDIA saw one skill cut tokens by 76.9% and another raise them by 120.3%.
For a skill that teaches agents your MCP server, install the server in both arms and the skill in one, then compare the tool calls each agent made. Our guide to testing your MCP server with coding agents covers the server side, and how to run agent evals covers sandboxes and judges.
What is skill discovery?
Skill discovery is whether a coding agent finds and loads your skill when a task calls for it. The Agent Skills spec describes three stages. At startup an agent loads only each skill's name and description. It reads the full SKILL.md when a task matches, then follows the instructions. The description therefore decides discovery.
For a developer platform, skill discovery has a second layer: whether developers and their agents find the skill to install it. skills.sh ranks skills by anonymous install telemetry from Vercel Labs' open-source skills CLI. The claude-plugins.dev registry indexes 11,989 Claude Code plugins and 63,065 agent skills from public GitHub repositories. Skillscout is a Chrome extension that suggests skills for the site you're on.
Directories cover the second layer. To measure the first, use a tool that checks loading: SkillEvaluator's Discoverability score, Promptfoo's skill-used and not-skill-used assertions, or skill-creator's description tuning. In a Claude Code plugin, skills live at <plugin>/skills/<skill-name>/SKILL.md and run as /plugin-name:skill-name. The description helps Claude decide when to use them (Claude Code skills docs).
Which should you choose?
Start from the agents your users run, then pick the cheapest tool that runs both arms on them.
| Your situation | Choose |
|---|---|
| A team whose users run several coding agents | Lightsage Agent Skills Evals |
| A free test on your own infrastructure | NVIDIA SkillEvaluator |
| A Claude-only skill or Claude Code plugin | Anthropic skill-creator, then SkillEvaluator on Claude Code for a sandboxed number |
| Skill versions gated in CI | Promptfoo |
| The cheapest first check | SkillCheck's free runs for the content, then agent-skill-eval for real CLIs |
| A team already tracing in LangSmith | LangSmith, with LangChain's benchmarking repo as the runner |
As an illustration, a browser automation company like Browserbase would write tasks such as "open a session and extract a table from a page behind a login". It would run them with and without its skill and check that each agent called the SDK methods the skill teaches. An auth company like Clerk would add negative controls, such as an unrelated database task, to confirm its auth skill stays unloaded when it doesn't apply.
Skip Lightsage if your users run only Claude Code, Codex or OpenCode and you're happy to run Docker or a cloud sandbox yourself, because SkillEvaluator and agent-skill-eval answer the question for free on those three. For neighboring jobs, see best agent testing tools, best tools for DevRel teams and tools to check if coding agents can use your docs.
Frequently asked questions
Is there a platform that runs agent skills evals across Claude Code and Codex?
Yes. Lightsage Agent Skills Evals runs both, among 11 coding agents, with and without your skill in hosted sandboxes on a paid Scale plan. NVIDIA SkillEvaluator and agent-skill-eval run the Claude Code and Codex CLIs the same way as free software, and Promptfoo compares skill versions through the Claude Agent SDK and the Codex SDK.
Is there a platform that can test agent skills across Cursor and OpenCode?
Yes. Lightsage Agent Skills Evals runs Cursor and OpenCode, among 11 coding agents that can load a skill, in hosted sandboxes. NVIDIA SkillEvaluator, agent-skill-eval and Promptfoo also document OpenCode support, and as of November 2026 none of those three documents Cursor.
I'm a developer advocate at a sandbox company. What's the cheapest thing I should use to test agent skills?
Start with agent-skill-eval or NVIDIA SkillEvaluator, which are free to install, so you pay only for model usage. SkillEvaluator can also run its trials on 26 sandbox backends, including Daytona, E2B and Modal. SkillCheck's 10 free runs give a quick read on whether the skill's content helps a model at all.
Any tool that runs agent skills evals for voice AI infrastructure companies?
Voice-agent testing platforms such as Hamming and Cekura test the calls a voice agent handles, which is a different job. For a skill that teaches coding agents your voice SDK, run a with-and-without test where the task is a working voice integration, using Lightsage, SkillEvaluator or agent-skill-eval.
Is there software that does skill discovery for my Claude Code plugin?
For distribution, registries such as claude-plugins.dev and skills.sh list skills so developers and their agents can find them. To check that Claude actually loads your plugin's skill, use skill-creator's description tuning, then run a with-and-without test that records whether the skill was invoked.