LLM-as-a-judge

An LLM judge is a language model asked to grade the output of another model or agent, either by scoring one output against a rubric or assertions or by choosing the better of two. It replaces or supplements a human rater, and it inherits the questions that apply to any measuring instrument: whether it agrees with people, whether its verdict depends on irrelevant features such as answer order, and whether anyone has checked. The main disputes are how a judge should be validated, and which agreement statistic to trust when it is.

History

The origin of the technique and early judge studies are still to be written.

Core concepts

Judges as scorers inside evaluation tools

Evaluation tools treat the judge as one kind of grader beside deterministic checks. Caliper, for example, runs Python checks on the attempt’s working directory and a no-judge check of which skills loaded, and sends transcript-based expectations to an LLM judge (Donadei, 2026). Skillgrade supports deterministic, LLM-rubric and human graders (Gechev, n.d.). The tools themselves are the subject of Evaluation tooling for AI agents; here the concern is the judge.

Choosing and repeating the judge

Several tools let the user choose the judge model separately from the system under test. Promptfoo allows this globally, per test or per assertion (Promptfoo, 2026). Caliper lets the judge engine differ from the engine under test and warns when compared runs used different judges (Donadei, n.d.). A preprint on skill evaluation reports that absolute judge values depend on the judge, and that judge agreement and human calibration on live runs were not measured (Kevin et al., 2026).

To reduce a single judge’s idiosyncrasies, Inspect supports a panel of grader models combined by majority vote, with the individual votes recorded (UK AI Security Institute (Inspect), n.d.). Inspect advises reducing judge nondeterminism with temperature 0 and multiple epochs (UK AI Security Institute (Inspect), n.d.).

Pairwise comparison and blind judging

A judge can be given two outputs rather than one. Anthropic’s skill-creator includes a blind comparator that judges skill versus no-skill outputs, or two skill versions, without knowing which produced which, which reduces bias toward a variant (Anthropic, n.d.a). This is a prompt for Claude agents, not a validated cross-harness judge (Anthropic, n.d.a). DeepEval’s arena comparison masks contestant names and randomizes their positions, which mitigates order bias by randomizing it rather than measuring it (DeepEval (Confident AI), n.d.).

Position bias and swap checks

Position bias is the tendency of a judge’s verdict to change when the same two answers are presented in the opposite order. A 2026 preprint defines position flip rate as the fraction of items for which the verdict changes when the response order is swapped (Norman et al., 2026). Arize Phoenix’s pairwise evaluator randomizes answer order per example with a reproducible seed and offers a swap-and-confirm variant that returns a winner only when both orders agree, at twice the API cost (Arize Phoenix, n.d.). Potato, an annotation platform rather than an agent-evaluation tool, reports position-swap consistency as a judge bias check (Potato project, n.d.). An MLflow article describes a manual check of running each pair in both orders and counting flips, and does not present it as a built-in feature (Kevin, 2026).

Validating a judge against human labels

The common validation is to have people label a sample and compare the judge’s verdicts with theirs. The tools differ in what they compute.

• The kappa tool scores a judge against human labels in a JSONL file with Cohen’s kappa and a confidence interval, and reports the result as inconclusive when the interval spans the ship bar. It makes no LLM calls, works on verdicts already logged, and only lints the judge prompt for bias rather than testing order swaps (yishaik, n.d.).
• Potato measures agreement with human gold labels using Cohen’s kappa and a confusion matrix (Potato project, n.d.).
• Ragas has a judge-alignment metric that checks per-item match between judge and human label, with no chance correction (Ragas, n.d.). Its how-to reports raw match rate, with an example baseline of 121 of 160 (Ragas, n.d.).
• MLflow aligns a judge to human feedback on traces; its example validation uses simple accuracy, not kappa (MLflow, n.d.). The align() method builds a new judge from human feedback; at least 10 traces work and 50 to 100 are better (Databricks, 2026). An MLflow article calls the 10-trace minimum a floor, not a trustworthy calibration set (Kevin, 2026).
• The validate-evaluator procedure splits about 100 human-labelled traces into train, dev and test sets and measures true-positive and true-negative rates. It sets dev-set targets above 90 percent for each (minimum 80 percent) and requires a single run on a held-out test set (hamelsmu (Hamel Husain), n.d.).

Questions, controversies, and new developments

Which agreement statistic?

The tools disagree on the metric. The kappa tool and Potato use Cohen’s kappa, a chance-corrected agreement statistic, while Ragas and MLflow’s example use raw match or accuracy, with no chance correction (yishaik, n.d.) (Ragas, n.d.). The 2026 preprint that defines position flip rate also finds that exact-match agreement overstates chance-corrected agreement (Norman et al., 2026). The validate-evaluator procedure instead recommends against kappa for judge-versus-human checks, reserving it for measuring agreement between two human annotators, and prefers true-positive and true-negative rates (hamelsmu (Hamel Husain), n.d.). No source cited here settles which is right; the question is contested.

How much validation do tools provide?

Documentation for the major tools offers little. Promptfoo’s model-graded docs contain no guidance on validating the judge with human labels or order swapping (Promptfoo, 2026), and Inspect’s page offers no human-label validation (UK AI Security Institute (Inspect), n.d.). Position-swap measurement is defined in a preprint and listed in Potato (Norman et al., 2026).

Judges and measurement validity

The preprint on skill evaluation adds a caution about what judged scores mean. Static scores of a skill did not predict its live lift across 62 skills (Spearman correlation about zero) (Kevin et al., 2026), and absolute scores confound skill quality with agent strength, which is why it reports paired lift; it found 87 of 947 paired cases with negative lift (Kevin et al., 2026). The paper itself treats the judge as a measurement dependence (Kevin et al., 2026).

Broader connections

Judging belongs to the wider practice of evaluating agent instructions, where a judge is one grader among deterministic checks and human outcome measures.

Further reading

• hamelsmu (Hamel Husain) (n.d.), validate-evaluator: a step-by-step procedure for validating a judge with human-labelled splits and TPR/TNR.
• Norman et al. (2026): a large 2026 preprint on judge agreement, consistency and bias, and the source of position flip rate.
• Kevin (2026): an MLflow article on measuring, correcting and reporting judge calibration, including a manual position-bias check.
• Arize Phoenix (n.d.): pairwise evaluator documentation, showing randomized order and swap-and-confirm.
• yishaik (n.d.): the README of a small tool that scores a judge against human labels with kappa and a confidence interval.