Evaluation tooling for AI agents

An evaluation harness for AI agents is software that runs an agent on a task, records what it did and scores the result. For coding agents such as Claude Code, Codex CLI or Gemini CLI, the agent itself runs inside a “harness” (the command-line program that supplies the system prompt, tools and file access around a language model), so evaluating it means driving that program, usually in its non-interactive mode, and reading its output. A newer class of tools does this for instruction packages that agents load on demand (see Agent skills), comparing runs with and without the package, repeating them, checking files and transcripts automatically, simulating users over several turns and testing whether the instructions are triggered at all. The field is young, mostly documented by the tools’ own pages, and contested on one central point: whether scores from different harnesses can be compared, since the harness, the model and the way instructions are loaded vary together. For the design of such evaluations see Evaluating agent instructions; for model-based graders see LLM-as-a-judge.

History

Anthropic announced Agent Skills on 2025-10-16, with support in Claude Code (Anthropic, 2025a). The format was developed by Anthropic and later released as an open standard (Agent Skills project, n.d.a); the open specification repository’s first commit is dated 2025-12-16 (agentskills/agentskills contributors, 2025). Other harnesses followed within weeks, by the earliest release notes found: Codex gained experimental support in a commit dated 2025-12-02 (openai/codex contributors, 2025), GitHub announced Copilot support on 2025-12-18 (GitHub, 2025), OpenCode listed it in v1.0.186 on 2025-12-22 (OpenCode team, 2025), and Gemini CLI’s v0.24.0-preview.0 on 2026-01-07 lists its first infrastructure work (Google Gemini CLI team, 2026). Dates are the earliest notes found, not proven first, and Claude Code’s changelog entry for skills (version 2.0.20) carries no date (Anthropic, n.d.e).

Evaluation tools came after. Before skill-specific tools, instruction files such as AGENTS.md were evaluated by paired runs with and without the file on the same tasks, or observationally from merged pull requests (Gloaguen et al., 2026; Lulla et al., 2026; Arabat & Sayagh, 2026). Anthropic added evals and a benchmark mode to its skill-creator on 2026-03-03 (Anthropic, 2026a); the SkillTester, SkillAudit and SkillsBench papers each describe a gap in benchmark measurement of whether skills help (Wang et al., 2026b; Yu et al., 2026; Li et al., 2026d). The open-source tools compared below date from 2026: Caliper was created on 2026-05-13 (GitHub, 2026b), Alibaba’s skill-up on 2026-05-09 (GitHub, 2026a), and NVIDIA announced SkillEvaluator on 2026-08-19 (Abramovitch et al., 2026).

Core concepts

The landscape: skill-specific tools and general frameworks

Four open-source tools say they drive real agent command-line programs with a skill. Caliper runs Claude Code, Codex, Pi and Hermes from one specification, choosing the engine at run time (Donadei, n.d.). agent-skill-eval runs Claude Code, Codex and OpenCode with and without a skill (tardigrde, 2026a). Skillgrade lists Gemini CLI, Claude Code, Codex CLI, OpenCode, any ACP-compatible agent or any command-line tool, supports deterministic, rubric and human graders, and runs tasks in Docker by default (Gechev, n.d.). skill-ab-eval runs the claude, codex, gemini and Antigravity (agy) CLIs and reports lift as the pass-rate difference (cskwork, n.d.). Only Skillgrade and skill-ab-eval list Gemini CLI, and agent-skill-eval records each run’s CLI version and effective model configuration (tardigrde, 2026a). Smaller tools include agent-exam, which normalises transcripts from Claude Code, Codex CLI, Copilot CLI and OpenCode (zytedata, n.d.a).

Vendors ship narrower or newer tooling. Anthropic’s skill-creator covers Claude products only (Anthropic, 2026a). OpenAI published a guide to evaluating skills with codex exec --json traces, a how-to pattern rather than a packaged tool, for Codex only (OpenAI, n.d.c). Alibaba’s skill-up lists claude_code, codex, qodercli and qwen_code as engines plus custom engines (Alibaba, n.d.), and NVIDIA’s SkillEvaluator runs live evaluations against codex, claude-code or opencode and marks support Experimental with no SLA (NVIDIA, n.d.).

General evaluation frameworks can also run an agent as the system under test. Promptfoo has a Codex SDK provider with a set working directory and sandbox, where skill use is only a heuristic (Promptfoo, n.d.b), and a Claude Agent SDK provider that starts without settings or skills and runs in a temporary directory, so skills must be enabled explicitly (Promptfoo, n.d.a). Inspect SWE runs Claude Code, Codex CLI and Gemini CLI, among others, as standard Inspect agents in a sandbox with model calls proxied back to Inspect (Meridian Labs, n.d.).

Adoption differs sharply by age. Promptfoo has 26,950 GitHub stars (created 2023-04), DeepEval 19,363 and Inspect 3,204, while Inspect’s agent add-on inspect_swe has 41 (GitHub, 2026c). Among skill tools, Skillgrade has 735 stars, agent-skill-eval 4, skill-ab-eval 0, agent-exam 2 and agent-eval-harness 0 (GitHub, 2026d); Caliper has 230 (GitHub, 2026b) and skill-up 1,439 (GitHub, 2026a). Stars measure attention, not use or maintenance.

Running with and without the instructions

The basic design is a paired comparison: the same task run once with the instructions installed and once without. SkillTester defines a skill’s utility relative to a matched no-skill run (Wang et al., 2026b). Tools differ in how they produce the two conditions. Caliper installs the skill where each agent looks for it, in a fresh home with an empty working directory, supports an ablation run without it (Donadei, 2026), and tests activation (whether the skill was loaded) separately from task success (Donadei, 2026). agent-skill-eval creates a fresh workspace per case, installs the skill into each harness’s native directory (.claude/skills, .codex/skills, .opencode/skills) and runs with and without it (tardigrde, 2026b). skill-ab-eval builds a harness-by-condition matrix in which each cell is a fresh context, on the argument that reuse would leak the skill into the baseline (cskwork, 2026a), but its command-line mode does not install the skill: the with-skill side pastes the SKILL body into the prompt, and the without side sends only the task (cskwork, 2026b).

Isolation from the experimenter’s own configuration is uneven. agent-skill-eval records the effective configuration rather than blocking it, and Codex inherits ~/.codex/config.toml unless flags are set (tardigrde, 2026b). skill-ab-eval’s own committed example shows the user’s global CLAUDE.md leaking into claude -p runs, so that baseline was not clean (cskwork, 2026a). The harnesses offer controls of different kinds. Claude Code’s --bare flag skips auto-discovery of skills, hooks, CLAUDE.md and memory, but it also withholds the list of available skills, which matters for skill evaluations (Anthropic, n.d.b); without it, claude -p loads the host’s skills, hooks and CLAUDE.md (Anthropic, n.d.b). Codex has flags to ignore the user’s config and rules (OpenAI, n.d.a). Directories also differ: Gemini CLI reads .gemini/skills and an .agents/skills alias, with the alias ranked higher (Google (gemini-cli project), n.d.a); OpenCode reads .opencode/skills plus .claude/skills and .agents/skills (OpenCode, n.d.); Copilot reads .github/skills, .claude/skills or .agents/skills (GitHub, n.d.); and Claude Code’s documentation lists .claude/skills paths and no .agents/skills (Anthropic, n.d.c). The Agent Skills guide calls .agents/skills/ a widely adopted convention that the specification does not mandate (Agent Skills project, n.d.b). A Google codelab says Antigravity CLI reads .agents/skills/ in a project, gives inconsistent global paths, and says skills in ~/.agents/skills are not picked up (Google, n.d.).

Repeated runs and variance

Agent runs are nondeterministic, so one run says little. The Agent Skills guide handles this with repeated runs and a trigger rate with a default threshold of 0.5 (Agent Skills project, n.d.b); Skillgrade offers trial presets of 5, 15 and 30 runs (Gechev, n.d.). Schmid names nondeterminism as a reason skill evaluations are hard and recommends repeated trials, as opinion without measured data (Schmid, 2026b). A 2026 preprint on continuous evaluation reports paired lift (the with-skill score minus the matched baseline) because absolute scores confound skill quality with agent strength; it found 87 of 947 paired cases with negative lift (Kevin et al., 2026).

Automatic checks of files and transcripts

Deterministic checks avoid judge noise. Caliper runs Python checks on the attempt’s working directory and a no-judge check of which skills loaded, while transcript-based expectations go to a language-model judge (Donadei, 2026). agent-exam has assertions on files and tool use (file_exists, tool_called, tool_not_called) and a skill_invoked assertion (zytedata, n.d.b). Promptfoo’s guide lists skill-used, not-skill-used and trajectory:step-count assertions plus custom JavaScript over tool calls, but no dedicated file-content assertion (Promptfoo, n.d.d). In contrast, skill-ab-eval’s page documents only judge-graded assertions (cskwork, n.d.), and agent-skill-eval combines state-diff checks with a rubric graded by a model (tardigrde, 2026a).

Transcripts differ by harness, so tools normalise them. Caliper stores each attempt as an ordered array of turns with a common schema (role, content, tool name, input and output) whatever the backend (Donadei, n.d.); for Hermes it recovers the tool-call trajectory with a separate session-export command after a one-shot run (Donadei, n.d.). agent-exam normalises transcripts from four harnesses so assertions are harness-neutral, and adding a harness needs an adapter (zytedata, n.d.a). Promptfoo exposes tool calls to JavaScript assertions as metadata.toolCalls and infers Codex skill use from reads of SKILL.md (Promptfoo, n.d.d).

The harnesses supply machine-readable output for this. Claude Code’s -p mode offers json and stream-json, the JSON carrying total_cost_usd and per-model cost (Anthropic, n.d.b), and the stream includes tool_use blocks and system init metadata (Anthropic, n.d.b). Gemini CLI’s headless mode gives JSON with usage statistics and a JSONL stream of tool_use, tool_result and result events, with defined exit codes (Google (gemini-cli project), n.d.b). codex exec --json emits a JSONL stream with turn.completed and item.* events (OpenAI, n.d.a).

Testing whether instructions are triggered

Skills load on demand: in most clients the model reads SKILL.md with a file tool or calls a dedicated activation tool, and the harness does not match triggers itself (Agent Skills project, n.d.b). Gemini CLI loads a skill when the model calls activate_skill, after user approval (Google (gemini-cli project), n.d.a). The method described in the Agent Skills guide is to label queries that should and should not trigger the skill, run each several times, detect loading from the client’s logs or tool-call history, and compute a trigger rate; it tells users to swap in detection for their own client (Agent Skills project, n.d.b). Its worked example is Claude Code specific, looking for a Skill tool call in JSON output (Agent Skills project, n.d.b), as does Anthropic’s skill-creator trigger test, which counts a Skill or Read tool call naming the skill (Anthropic, n.d.f).

Detection therefore needs per-harness rules. Caliper takes the union of a dedicated skill tool call naming the skill (Claude Code’s Skill, Hermes’s skill_view) and any tool call whose input contains a <name>/SKILL.md path, which is how Codex and Pi load a skill, since they have no dedicated tool (edonadei (Caliper project), n.d.). It matches on the path suffix because Codex gave the same activation as an absolute path in one run and a relative one in another (edonadei (Caliper project), n.d.). agent-exam reads Codex activations from a command that reads a skills/<name>/SKILL.md path (zytedata, n.d.c), OpenCode activations from a tool_use event whose tool is skill (zytedata, n.d.e), and Copilot CLI activations from toolRequests named skill (zytedata, n.d.d). agent-skill-eval offers natural-trigger cases and negative controls and can force invocation by prefixing the prompt (tardigrde, 2026b), but infers triggering from side effects in git state (branches, commits, pushes, pull requests) rather than a transcript record of loading (tardigrde, 2026c).

Simulated multi-turn users

Some tasks need several turns, with a language model playing the user. General frameworks supply this; skill-specific tools mostly do not. Promptfoo has a built-in simulated-user provider that alternates an LLM user, driven by an instructions template, with the target until maxTurns or a ###STOP### reply (Promptfoo, n.d.c). By default it resends the full transcript each turn; a stateful option sends only the latest user message and relies on the target to keep history (Promptfoo, n.d.c), reading a sessionId from the target’s response and reusing it (Promptfoo, n.d.e). Promptfoo’s Claude Agent SDK provider returns session IDs and can resume, continue or fork a session, which a stateful simulated user needs (Promptfoo, n.d.a). DeepEval’s ConversationSimulator has an LLM play the user from a scenario and persona but calls a model callback that the user writes for each assistant reply, so wrapping an agent command-line program is left to the user (Confident AI, n.d.). The extension inspect_multiturn adds a conversational solver to Inspect that alternates a simulated user with any Inspect agent, but its README says it is in early development (aschriner, n.d.). Strands Evals’ ActorSimulator generates a persona profile from a test case, tracks the goal and stops on goal completion or a turn limit; the authors report that ad-hoc “act like a user” prompting drifts between runs, a reason to structure persona and goal (Singh et al., 2026).

On the skill-specific side, agent-skill-eval defines a case as a single prompt with no documented multi-turn mode (tardigrde, 2026b), and the Agent Eval Harness README describes dataset-driven cases with check and judge scoring and does not mention multi-turn use (EmilienM, 2026). A practitioner runner in the dev-skills project alternates the agent under test with a simulated user configured by a free-text briefing and ends at a maximum number of turns or a [DONE] reply, but it is built around the claude CLI, so it is not cross-harness as documented (chriscalo, n.d.).

Judges, briefly

Most of these tools let a language model grade open-ended outcomes. Promptfoo lets the user choose the judge model separately from the system under test, globally, per test or per assertion (Promptfoo, 2026); Caliper lets the judge engine differ from the engine under test and warns when compared runs used different judges (Donadei, n.d.); Inspect supports a panel of grader models combined by majority vote, and advises temperature 0 and multiple epochs to reduce judge nondeterminism (UK AI Security Institute (Inspect), n.d.). Anthropic’s skill-creator includes a blind comparator that judges with-skill and without-skill outputs without knowing which is which (Anthropic, n.d.a). Whether such judges agree with people, and what biases they carry, is the subject of LLM-as-a-judge.

Cost and access for scripted runs

Many-run evaluations meet usage limits. Claude Pro and Max plans have a five-hour session limit and a weekly limit shared across Claude and Claude Code (Anthropic, n.d.d). Anthropic states that advertised limits assume ordinary individual use of Claude Code and the Agent SDK, which is not defined for batch evaluation runs, and that developers building on the Agent SDK should use API keys (Anthropic, n.d.a). It paused a planned split of Agent SDK and claude -p usage from subscription limits; as of the update note, plan sign-in runs still draw from subscription limits (Anthropic, 2026b). Claude Code’s bare mode, recommended for scripted calls, does not use the subscription login and needs an API key or cloud credentials (Anthropic, n.d.c). OpenAI documents the Codex API-key option as the one for CI automation, billed at API rates, while ChatGPT sign-in draws on plan limits (OpenAI, n.d.b). Gemini CLI allows 1,000 requests per user per day with Google login and 250 with an unpaid API key, on Flash only, and the unpaid tier and Google One users moved to Antigravity CLI on 2026-06-18 (Gemini CLI documentation, n.d.). The tools add their own requirements: agent-skill-eval needs the agent CLIs installed and authenticated plus a separate OpenRouter or OpenAI key for model grading, and warns that the claude CLI prices runs at Anthropic list prices regardless of endpoint (tardigrde, 2026b).

Questions, controversies, and new developments

Can scores be compared across harnesses?

A position paper argues that the harness can matter more than the model, so cross-harness comparisons are misleading unless the harness is disclosed, and reports harness-induced variance that can exceed model-induced variance, including ranking reversals (Zhang et al., 2026b). Sources list several confounds. Model and harness vary together: LH-Bench compares harness-by-model configurations and notes that Claude models are trained with awareness of the Agent Skills specification, which may advantage them (Chandwani & Gupta, 2026). Loading differs: in LH-Bench, Claude Code and Gemini CLI scan skill directories while Codex CLI has skill content inlined into its system prompt, so the with-skill condition is not the same mechanism (Chandwani & Gupta, 2026). Its skill ablation found gains that differed by harness (Codex +0.87, Claude Code +0.15, Gemini CLI +0.05), from only 2 to 3 runs per harness, which its authors call underpowered and directional (Chandwani & Gupta, 2026).

Accounting differs too. Codex’s input tokens include cache reads while Claude Code and OpenCode report them separately, so raw token counts are not comparable, and Codex CLI reports tokens but no cost, so cost needs a user-supplied price table (tardigrde, 2026b). A single-author blog post adds that comparison needs one canonical accounting method, that Claude Code’s parallel tool calls share one timestamp, and that error formats differ and produce different recovery patterns (Bob, 2026). Another blog estimates that built-in system prompts differ several-fold (Claude Code core about 3,000 to 5,000 tokens, Gemini CLI about 5,000 to 6,000, Codex CLI about 6,000), so a skill is read amid different backgrounds (Vaughan, 2026). The two blogs are low-trust sources; the position paper and LH-Bench are preprints.

Does running the instructions beat inspecting them?

Existing tooling that inspects skills falls, according to a 2026 preprint, into four classes that all scan the skill document and none runs it; the same preprint measured that static scores do not predict live lift (62 skills, Spearman correlation about zero), which it takes to mean that scanning cannot stand in for running (Kevin et al., 2026). It also reports that absolute judge values depend on the judge, and that judge agreement and human calibration on live runs were not measured (Kevin et al., 2026). It is a single preprint.

Do instruction files help at all?

The paired design predates skills. A 2026 study of repository context files found no general gain in task success and a rise in inference cost of over 20 percent on average (Gloaguen et al., 2026). A paired study on 124 pull requests measured runtime and tokens and found comparable task completion (Lulla et al., 2026), an observational study found instruction files did not reliably improve merge rate (Arabat & Sayagh, 2026), and one skill for testing such rules reports an A/B test where nearly all assertions passed with and without the file (vltansky, 2026). Skills add a problem that always-loaded files lack: in Vercel’s evaluation of documentation delivery, skills were often never invoked (Gao, 2026a). The pytest plugin pytest-codingagents tests real coding agents through their SDKs and targets both skills and Copilot instructions, but is Copilot-focused with one release series in 2026-02 and no stars (ecosyste.ms, 2026).

Why are the tools few and little used?

Practitioners say skill evaluation is rare. A Google DeepMind engineer writes, without data, that almost nobody tests skills and that they are often judged by feel (Schmid, 2026b). The Expo team reports informal checks and testing only by invoking skills by name before building a harness, and now runs scenarios in CI before merging (Shukla et al., 2026b). A conference talk listing says that of over 50,000 indexed GitHub skills almost none carry evaluations, but gives no method (Schmid, 2026a), and the SkillsBench paper reports 47,150 unique skills with no count of how many have evaluations in the part read (Li et al., 2026d). A qualitative study of 238 skills found that 21.4 percent of SKILL.md bodies contain an Evaluation component and 9.2 percent Running Tests instructions, which are instructions inside the skill rather than evaluations of it (Hong et al., 2026). The authors of the evaluation papers state a measurement gap (Wang et al., 2026b; Yu et al., 2026; Li et al., 2026d); SkillAudit adds that fixed task suites conflate a skill’s contribution with model strength and that developers adopt skills on README claims, stars and downloads (Yu et al., 2026). Vendors are less explicit: NVIDIA says it built SkillEvaluator to measure whether skills improve trajectories and outputs but names no gap in earlier tools (Abramovitch et al., 2026), and skill-up presents itself as replacing ad hoc run folders (Alibaba, n.d.). The tools are months old, and NVIDIA states no service commitment (NVIDIA, n.d.).

Broader connections

This tooling applies general evaluation practice to a new target. The questions of which baseline to use, how many runs are enough and who grades the output are treated in Evaluating agent instructions; the instruction packages themselves, and how they call and chain one another, in Agent skills; and how language-model graders are validated in LLM-as-a-judge.

Further reading

• Agent Skills project (n.d.b): the open guide to trigger testing with labelled queries, repeated runs and a trigger-rate threshold.
• Donadei (2026): the clearest single description of a cross-harness skill harness (deterministic checks, activation tests, ablation).
• Zhang et al. (2026b): the position paper on why harness and model cannot be separated in agent comparisons.
• Chandwani and Gupta (2026): an ablation of the same skill across three harnesses, with the authors’ own caveats.
• Kevin et al. (2026): a preprint that measures paired lift and compares static scores with live results.
• Meridian Labs (n.d.): how a general evaluation framework runs several agent command-line programs as agents under test.