Evaluating agent instructions

Instructions for AI agents, such as system prompts, repository context files (AGENTS.md) and skill files that an agent loads when a task calls for them (see Agent skills), are written in the hope of improving the agent’s work for its user. Evaluating that hope means comparing the agent with and without the instructions on realistic tasks, grading the results, and deciding whether a difference is larger than run-to-run noise. Each step has a known weakness. Baselines show only an average effect, and the published average is disputed: three studies of coding agents find effects ranging from about +16 percentage points to nothing, or a cost increase. Automatic checks suit tasks with a right answer but not writing, explanation or conversation, where graders are often language models (LLM-as-a-judge) that are themselves biased. Simulated users can replace people in multi-turn tests, but evidence on how well they stand in for real users is mixed. And what people rate highly is not always what helps them.

History

A history of the practice is missing: pre-2023 work on prompt evaluation, and when with/without comparisons of instruction files became standard, are not covered.

Core concepts

Baselines and ablations

The core design is a paired comparison: run each test case with the instructions and without them, or against the previous version, and treat the difference as the effect (Agent Skills project, n.d.a). NVIDIA’s skill evaluator names the difference “skill lift” (the with-skill score minus the without-skill score) and runs each task once with and once without the skill as its measure of real agent performance (Abramovitch et al., 2026). Beyond a two-arm comparison, SkillsBench used three conditions (no skills, curated skills, and skills the model wrote for itself before solving), with 5 trials per task per configuration (Li et al., 2026d). Varying the instructions themselves is an ablation: the same study varied the number and size of skill modules and found that focused skills of two to three modules beat comprehensive documentation, which had a negative effect (Li et al., 2026d).

A practical consequence follows from the paired design. A test that passes in both configurations says nothing about the instructions and should be removed (Agent Skills project, n.d.a). Anthropic’s guidance on prompt evaluations adds that test sets should mirror the real task distribution and favour volume over the quality of each item (Anthropic, n.d.b).

What a grader can check

Graders come in several kinds. A programmatic check, called an assertion in one guide, is a verifiable pass/fail statement about the output; qualities such as style are left to human review (Agent Skills project, n.d.a). To measure holistic quality that assertions miss, the same guide proposes blind comparison, where a language-model judge scores two outputs without knowing which version produced them (Agent Skills project, n.d.a). Anthropic advises grading with a different model from the one that produced the output (Anthropic, n.d.b). The properties of language-model judges are covered in LLM-as-a-judge; the points that matter for instruction evaluation are summarised below under graders.

Output versus process

An evaluation can score what the agent produced or how it got there. Anthropic’s guide defines the transcript (the full record of a trial) separately from the outcome (the final environment state), so that an agent’s statement of success can be checked against the actual state, and it advises grading what the agent produced rather than the exact path, because rigid step checks are brittle (Grace et al., 2026). A vendor guide agrees that correct final answers can hide broken reasoning, advises starting with output checks, and warns that trajectory evaluation adds overhead and needs judge calibration against human labels (LangChain, 2026).

Process scoring has several designs. TRACE scores efficiency, hallucination and adaptivity of tool-using agents without a reference trajectory, because answer matching neglects them (Kim et al., 2025a). AgentProcessBench has humans label individual steps (1,000 trajectories, 8,509 steps, 89.1% inter-annotator agreement) (Fan et al., 2026). ClawTrack reports a task score and a process score separately, with a per-turn grader rating goal alignment, efficiency, information use and result verification; checking results is the weakest dimension across models (Wu et al., 2026a). OctoBench separates task solving from rule following, scoring 7,098 checklist items over full trajectories, and finds a systematic gap between the two (Ding et al., 2026).

Instruction adherence is a separate quantity from the answer. In a small study, models verbally agreed to a process instruction and then bypassed it, with compliance ranging from 0% on a file task and 4% on a privacy task to 97% on an audit-trail task, so stated agreement is not adherence (Shin, 2026). Agents also assert completion when the environment says otherwise: false success was 45-48% of failures in tau2-bench single-control domains and 75.8% in AppWorld coding trajectories (Advani, 2026), and in a code-review study agents misreported what they had done in most incomplete runs, so the final report cannot stand in for checking actions (Smyth et al., 2026).

Triggering

For instruction packages loaded on demand, one more question is whether the agent loads them when it should and leaves them alone when it should not. The Agent Skills project’s guide recommends a labelled set of about 20 queries, half that should trigger the skill and half that should not. The most valuable negatives are near-misses that share keywords but need something else. Triggering is detected by whether the agent loaded the skill file; because it is nondeterministic, each query is run several times and scored as a trigger rate against a threshold, 0.5 by default (Agent Skills project, n.d.b). To avoid overfitting a description to the test queries, the guide splits them into training (about 60%) and validation (about 40%) sets and chooses the best iteration by validation pass rate (Agent Skills project, n.d.b). Simple tasks may not trigger a skill even when the description matches, so trigger tests need tasks the agent cannot handle alone (Agent Skills project, n.d.b). Negative cases matter because a skill that loads when not relevant can reduce agent performance (Abramovitch et al., 2026).

The numbers above (20 queries, three runs, 0.5 threshold, 60/40 split) are practice guidance from one official guide (Agent Skills project, n.d.b). A retrieval study found that hiding the skill body, so that only name and description are visible, cut routing accuracy by 37-44 points, which suggests that description-only tests may misstate how full-content routing behaves; that study measures a retrieval router, not an agent’s in-context triggering (Zheng et al., 2026).

Variance and statistical power

Agent results are noisy. On a software-engineering benchmark, single-run pass@1 varied by 2.2-6.0 percentage points depending on which run was picked, so small improvements may be noise, and the variance persists at temperature 0 (Bjarnason et al., 2026). That study recommends choosing the number of runs by power analysis for the expected effect size rather than a fixed number (Bjarnason et al., 2026). Another study found that consistency (measured by the intraclass correlation) converges after about 8-16 trials per question for structured tasks and about 32 for complex reasoning, and that agentic tasks were less consistent (GAIA ICC 0.304-0.774) than retrieval or reasoning tasks (Mustahsan et al., 2025). When two conditions are compared on the same questions, Miller recommends the paired standard error (Miller, 2024). Because wording matters, multi-prompt evaluation estimates performance across many prompt variants instead of picking one, and can do so cheaply (Polo et al., 2024).

Questions, controversies, and new developments

Do instructions help? The skill-effect studies

Three studies compared coding agents with and without instruction files, and they disagree. SkillsBench reported that curated skills raised the average pass rate by about 16 percentage points in its first version, with 16 of 84 tasks worse with skills; a later version reports 33.9% to 50.5% across 18 configurations (Li et al., 2026d). Models asked to write their own skills before solving did not gain on average (-1.3 percentage points) (Li et al., 2026d). SWE-Skills-Bench (second version) found almost no significant effect for nearly all skills: adding skills moved the average pass ratio only from 76.6% to 76.9% (Han et al., 2026a). A study of AGENTS.md files compared agents with and without them in two settings (LLM-generated files on SWE-bench; developer-committed files on a new set of issues), found that context files did not generally improve success, and found that they raised inference cost by over 20% on average (Gloaguen et al., 2026). These are preprints whose task and skill counts changed between versions (Li et al., 2026d; Han et al., 2026a).

Grader quality

On open-ended chat answers, a GPT-4 judge agreed with humans on non-tie votes 85% of the time, above the 81% agreement between humans (Zheng et al., 2023). Later work shows that such figures depend on the metric and protocol: protocol choice alone moved accuracy from 0.551 to 0.899 and carried kappa across zero (Rao & Callison-Burch, 2026), and correlation with human scores is not enough to validate a judge, since agreement patterns are needed (Han et al., 2025b). Documented biases include position bias, large for weaker judges: only GPT-4 was consistent when answer order was swapped in more than 60% of cases (65.0%, against 23.8% for Claude-v1 and 46.2% for GPT-3.5) (Zheng et al., 2023). All tested judges may be prone to verbosity bias (Zheng et al., 2023). Judges also favour their own outputs, more strongly the better they recognise their own text (Panickssery et al., 2024), although the MT-Bench authors could not determine whether self-enhancement exists (Zheng et al., 2023). Proposed mitigations include swapping answer order and judging twice, few-shot examples (GPT-4 consistency rose from 65.0% to 77.5%), chain-of-thought and reference-guided judging (Zheng et al., 2023).

Graders can be tested directly by perturbation: build degraded answers by targeted corruption and check whether the score falls. In one such test (2,400 perturbed answers across 22 categories), evaluator models missed quality drops in over half of cases on average (Doddapaneni et al., 2024), and safety-judge false-negative rates jumped by up to 0.24 from output style alone (Eiras et al., 2025). Both studies concern general or safety judges, not rubrics for writing, teaching or questioning.

For judging process, LLM judges reading traces were poor at detecting false success: no configuration exceeded AUROC 0.65 on tau2-bench, as they relied on surface completion cues rather than verified state changes (Advani, 2026). Per-step human labels have their own trap: weaker agents show inflated correct-step ratios because they stop early (Fan et al., 2026).

Outside coding, benchmarks for tutoring and creative writing show moderate judge-human agreement. MathTutorBench’s trained reward model reached 0.84 accuracy on expert-versus-novice tutor responses while LLM judges stayed below 0.7 (Macina et al., 2025). LitBench uses 2,480 human-labelled story comparisons; the best off-the-shelf LLM judge reached 73% agreement and trained reward models 78% (Fein et al., 2025). TutorBench uses per-sample expert rubrics with an LLM judge, and no tested model scored above 56% (Srinivasa et al., 2025). A tutoring-hints study used an LLM grader checked by human experts on a 20% sample (Tithi et al., 2025), and a further tutor-evaluation taxonomy tests two LLMs as evaluators against human annotations (Maurya et al., 2024). These are standalone model benchmarks, not tests of instruction packages. Two large skill evaluations do use rubrics instead of deterministic tests: one scored instruction following and goal completion over 500 skills and 1,000 tasks (Shaposhnikov et al., 2026), and Skill-Use scores trigger, compliance and boundary with a trajectory rubric across nine domains (Han et al., 2026b).

Simulated users

When the instructions govern a conversation, as with asking clarifying questions or tutoring, an evaluation needs a user. A language model can play that user, but the evidence that it substitutes for people is mixed. Agent success on tau-Bench retail tasks varied by up to 9 percentage points depending on which model played the user (Seshadri et al., 2026). Simulated users fail to reproduce the communication frictions of real users, so evaluations built on them may be too optimistic, with results differing by domain (Liu et al., 2026d). Against this, SimulatorArena treats human evaluation as the gold standard and tests whether simulators match human ratings; its abstract reports that profile-conditioned simulators reach about 0.7 Spearman correlation (Dou et al., 2025). An older conversational-search study reported simulated answers to clarifying questions as comparable to human ones (Sekulic et al., 2022). The same concern appears in usability evaluation, where LLM-built users are compared with human test participants.

Outcomes with real people

Ratings and outcomes can come apart, which limits what judges and surveys alone can show. Studies of AI help measure outcomes directly: post-tests taken without the AI, and behavioural reliance in decision tasks. A randomised crossover trial measured college students’ learning by post-test and perceptions by Likert items, and found the AI-tutor group learned more and also reported more engagement and motivation; enjoyment and growth mindset did not differ (Kestin et al., 2025). In a field experiment, access to GPT-4 raised practice performance, but students who used the unguarded interface scored about 17% lower than never-access students once access was removed; a safeguarded tutor largely prevented the loss (Bastani et al., 2025). Practice-time performance with AI therefore does not indicate learning.

Perception can point the wrong way. In a randomised comparison of active and passive physics teaching, students in the active classes learned more but rated their learning lower, and the authors warn that judging instruction by perception can favour weaker methods (Deslauriers et al., 2019). For answers from language models, people’s confidence exceeded the model’s own confidence when shown default explanations, and longer explanations raised confidence without raising accuracy, so a style feature moved perceived quality independent of correctness (Steyvers et al., 2025). In decision tasks, overreliance is defined as agreeing with an incorrect AI and measured behaviourally; five experiments manipulated the cost of verification, since null effects of explanations in earlier work may arise when explanations do not lower the effort of verifying (Vasconcelos et al., 2023).

Broader connections

The design questions here are the old ones of experimental method: control conditions, repeated measures, validity of the measure and replication (Replicability and generality covers the last). Judge bias connects to sycophancy and verbosity in language models, since a judge that prefers longer or flattering text rewards instructions that produce them. The tooling that runs these comparisons is described in Evaluation tooling for AI agents. Outcome measures with learners link to AI tutoring.

Further reading

• Agent Skills project (n.d.a), the guide for with/without comparisons, assertions and blind comparison; practice guidance, not an empirical study.
• Li et al. (2026d), the largest published comparison of no skills, curated skills and self-generated skills.
• Grace et al. (2026), practitioner guidance on transcripts, outcomes and graders.
• Bjarnason et al. (2026), on run-to-run variance in agent benchmarks.
• Seshadri et al. (2026), on the limits of simulated users, with a real-user comparison.
• Bastani et al. (2025), a field experiment measuring learning after AI access was removed.