Usability evaluation

Usability evaluation is the set of methods for finding out whether people can use an interface to do what they came to do, and what stops them. Two families dominate. In a usability test, representative users attempt tasks while observers record where they fail or hesitate. In an inspection, a person who is not the user, such as an expert or a developer, examines the design against a list of principles (heuristic evaluation) or steps through the actions a user would take (cognitive walkthrough). Each method is disputed on the same points: how many problems it finds, how many of the problems it reports are real, and how far two evaluators agree. Newer work asks whether language models can run an inspection or play the user. Evaluation is one step in the larger sequence described in the interface design process; the visual and layout rules that evaluation checks are the subject of interface design.

History

The earliest comparisons on the record are from the early 1990s. A study at Hewlett-Packard found that heuristic evaluation found the most problems (more than half of the total) and the most serious ones at the lowest cost, while also producing many low-priority and one-time problems that user testing avoided (Jeffries et al., 1991). A 1992 poster abstract reported the reverse: empirical laboratory testing identified four to five times as many problems as heuristic evaluation (Desurvire et al., 1992). A practitioner article from 2003 cites a third study of the period for the claim that a heuristic review alone yields a long list of false alarms without highlighting the real problems (Straub, 2003). These early studies set the terms of the debate that the sections below follow.

The origins of the think-aloud test, the heuristic lists and the cognitive walkthrough are not covered by the research; add them before this section is complete.

Core concepts

Formative and summative evaluation

Nielsen Norman Group defines formative evaluation as iterative improvement before production, and summative evaluation as the comparison of a shipped product with a benchmark (Kendrick, 2019). The standard ISO 9241-210 makes evaluation one of four design activities, alongside understanding the context of use, specifying user requirements and producing design solutions, and states that the process is iterative and driven by user-centred evaluation (International Organization for Standardization, 2010). Most of the evidence below concerns the formative kind.

What the comparison studies measure

Studies that compare evaluation methods score thoroughness (how many real problems a method finds, so few are missed), validity (how few false alarms it raises) and effectiveness as the product of the two; the author of one dissertation notes that measures and criteria were inconsistent across studies (Andre, 2000). A figure such as “finds 60% of problems” therefore depends on what counts as a real problem.

Usability tests

A usability test has users attempt tasks with a prototype or product. In a think-aloud test, users voice their reasoning as they work. Remote testing follows two patterns. In one review, remote synchronous tests matched laboratory tests, while asynchronous remote methods identified fewer problems and cost participants more time (Bastien, 2010). In that review, Tullis et al. found high correlations between laboratory and remote tests on completion and time data, with both finding the most critical issues (Bastien, 2010). Andreasen et al. compared three remote methods with a laboratory think-aloud test (Andreasen et al., 2007).

A test is good at identifying problems and poor at suggesting solutions. In a 48-participant test, usability testing even with several designs was not an effective source of constructive suggestions from users; it identifies problems, not solutions (Tohidi et al., 2006). The same study found that each design scored higher on average when seen alone than when seen among three alternatives, with a significant difference for two of three designs (Tohidi et al., 2006).

Heuristic evaluation

In heuristic evaluation, several analysts inspect an interface against a set of usability heuristics and list the violations. Nielsen reports that some problems in telephone-company case studies were so domain-specific that heuristic evaluation could not realistically find them without user testing, and that the method may miss problems that testing finds when evaluators lack domain expertise (Nielsen, 1995). A newsletter summary of Fu, Salvendy and Turley (2002) reports that heuristic evaluation found slightly more problems (87%) than usability testing (54%), but that testing found significantly more problems at the knowledge-based level of behaviour (Straub, 2003).

Reporting format may matter more than the heuristics themselves. Cockton, Woolrych and Hindmarch report validity of 31% in an initial study and 48% to 63% in later studies in which analysts used an extended structured report that asked for a rationale for each problem; that is a comparison across studies, not within one experiment (Cockton et al., 2004). The authors saw no bogus reports in the pilot or replication, which they call a side effect of self-reporting (Cockton et al., 2004). By contrast, changing the reporting medium (web tool versus paper) and the heuristic set made no significant difference to the effectiveness, efficiency or reliability of evaluators (Hvannberg et al., 2006). Giving evaluators personas changed little: evaluators with personas reported more navigation findings and those without reported more design findings, and those who built their own personas included more complex issues (Friess, 2015).

Cognitive walkthrough

In a cognitive walkthrough, the evaluator steps through the actions of a task and asks at each step whether a user would know what to do. Beer et al. report that cognitive walkthroughs find the most severe problems while think-aloud tests find all types, so the two are complementary (Beer et al., 1997).

Questions, controversies, and new developments

How many test users are enough

Nielsen states that a first five-participant study finds 85% of usability problems, based on a typical single-user detection rate of 31% across the projects his group studied, and recommends several small tests rather than one large one (Nielsen, 2000). In 2012 he still gave five users as the default for a qualitative study, but added that each distinct user group needs its own participants, so five applies per group, and that quantitative studies need at least 20 users (Nielsen, 2012).

Critics dispute the figure. Faulkner (2003), as reported by Straub, sampled five-user groups from 60 participants and found coverage varied from nearly everything to 55%; Spool and Schroeder (2001), as reported by Straub, found the first five users revealed 35% of the problems the larger set identified (Straub, 2004). The abstract of Faulkner’s study gives the same spread and adds that with 10 users the worst set found 80% and with 20 users 95% (Faulkner, 2003). Schmettow argues that problems differ in how visible they are and a study is incomplete, so the classic formula understates the sample needed, and he doubts that 80% of problems can be found with 10 users; in one full dataset the 90% confidence interval for the users needed to reach 80% discovery spans 37 to 165 (Schmettow, 2012). Lewis and Sauro (2024) say discovery rises with sample size with diminishing returns, that rare problems are hard to find with small samples, and that estimating a mean problem probability from a sample consistently overestimates it (Lewis & Sauro, 2024).

The two sides use different standards. Nielsen’s claim is about average discovery and about iterating; the critics report the spread between samples and the problems that small samples miss.

Do evaluators agree

Evidence that evaluators differ is strong and consistent. A review of eleven studies found that different evaluators using the same method detect markedly different sets of problems, with two-evaluator overlap of about 5 to 65 percent (Hertzum & Jacobsen, 2003). When four evaluators analysed the same usability-test sessions, only 20% of 93 problems were found by all four, and none of their selected severe problems appeared on all four top-10 lists (Jacobsen et al., 1998). A 2014 replication with experienced professionals found that evaluators reported on average about a third of the problems reported by all, and disagreed in severity ratings (Hertzum et al., 2014). Across 33 studies the median any-two agreement between evaluators was 27%, against 4% in the original CUE study (Sauro, 2018). This variation between evaluators is known as the evaluator effect.

How much of an inspection’s output is real

Studies disagree on whether inspection finds more or fewer problems than testing, and the reported rates are not comparable. An inspection technique found more problems per unit effort than a user test in one 2020 comparison, but only 19.8% of its problems were confirmed in the test, and it found about half of the problems that occurred in the test (Nakamura et al., 2020). In a study of 99 analysts applying heuristic evaluation, 65% of all predicted problems were false when checked against user-test problems, and the heuristics did not support discovery of possible usability problems (Cockton & Woolrych, 2001). The Hewlett-Packard result above and the 87% against 54% summary point the other way on counts, while the 1992 abstract points back (Jeffries et al., 1991; Straub, 2003; Desurvire et al., 1992). What the studies share is that inspection reports include many unconfirmed items; whether the net result is cheaper or more thorough depends on the study.

Review by people who did not build the design

The authors of a heuristic evaluation of a web prototype report relatively few differences between the developers who built it and user-interface experts, although the experts found 102 heuristic violations and the four developers found 64 (Levi & Conrad, 1996). A practitioner column says confirmatory bias makes designers less likely to discard faulty designs, while recording practitioners who think the advantages of testing one’s own design probably outweigh the bias; it cites no measured result (Sherman, 2009). The evaluator-effect results above bear on the question indirectly: independent reviewers disagree with each other, so a second reviewer is not a fixed standard.

Scenario-based reading

Experienced engineers found up to three times as many exceptions by stepping through scenario steps as by examining user requirements (Alexander, 2000). The controlled evidence on inspection techniques is from software requirements rather than interfaces, and its scenarios are class-specific search procedures (each reviewer looks for one kind of fault), not walks through the steps of a user’s task. In a randomized experiment with 48 graduate students inspecting requirements documents, reviewers using such scenarios detected more faults than reviewers using ad hoc or checklist methods, and checklist reviewers did no better than ad hoc ones (Porter et al., 1995). The Maryland group summarises about 35 percent more defects for defect-based readers than for ad hoc and checklist readers (Experimental Software Engineering Group, 1998).

Replications were mixed. Basili, Shull and Lanubile report five independent replications of defect-based reading, with students and practitioners obtaining different results; they hypothesize that the technique is too sophisticated for undergraduates and that practitioners may revert to familiar techniques (Basili et al., 1999). Two replications with undergraduates comparing perspective-based reading with a checklist gave conflicting results on one of two documents, and each technique uncovered defects the other did not (Fabbri et al., 2004). A meta-analysis of perspective-based reading experiments found no clear positive effect, with the method beating ad hoc reading but doing worse than checklists (Ciolkowski, 2009). For interfaces specifically, perspective-based usability inspectors found more problems overall than heuristic evaluators, about 30 percent more when pooling three inspectors (24 professionals) (Zhang et al., 2000).

Prototype fidelity

Tests are often run on sketches or paper prototypes. In two experiments with 10 subjects per condition, low- and high-fidelity prototypes found largely the same usability problems; some problems appeared in only one condition, which the authors attribute mainly to small-sample variability without direct evidence (Virzi et al., 1996). In a 48-user study, reduced-fidelity prototypes (paper, 3D mock-up) generally predicted the usability of the real appliance, with implications depending on fidelity (Sauer et al., 2010). Experts reported more usability problems than novices but of lower severity (Sauer et al., 2010). A mobile-phone study found significant differences between low- and high-fidelity prototypes on each factor, so results are not uniform (Zhang et al., 2012). A practitioner summary reports that Walker et al. (2002) found no difference in the number of issues between paper and computer prototypes but that fidelity changed the types of issues found, and that Sefelin et al. (2003) found equal critiques on both while 92% of participants preferred the computer prototype (Sauro, 2017).

Retesting after changes

Testing again after fixes does not guarantee improvement. In the Census Bureau’s American FactFinder redesign, task accuracy in iteration 3 (52%) dropped from iterations 2 and 2.5; the problems were so large that earlier fixes could not be evaluated, and the team said further testing was needed to see whether fixes raise performance or cause new decrements (Olmsted-Hawala et al., 2011). A two-round library website study (8 students) found that the second round caught a problem the changes had created unintentionally (DeStasio & Jeitner, 2024). A newsletter summary of Tan et al. (2001) reports that fixing problems on a website created 9 new problems and cut the problem count by only 37% (Bailey, 2002). These are small, uncontrolled reports.

Language models as inspectors and as simulated users

Recent work asks whether language models can replace human evaluators. As heuristic evaluators of web screenshots, GPT-4o found only about a fifth of the issues human experts found, reported 27 issues the experts had not, and generated several false positives from hallucination and from trying to predict issues (Guerino et al., 2025). Two other models found many new defects but with more false positives and redundant reports than four human inspectors; human inspectors had the highest precision and coverage, and the authors conclude AI cannot replace inspectors (Campos et al., 2025). One synthetic heuristic evaluation reached the opposite figures, identifying 73% and 77% of usability issues on two interfaces against 57% and 63% for five human evaluators, though it struggled to recognise some interface components and design conventions (Zhong et al., 2025b).

For cognitive walkthroughs, language-model agents completed tasks more often than humans but identified fewer potential failure points; with additional prompting they could predict failure points humans identified (Zhong et al., 2025a). The UICrit authors state that language-model interface evaluation does not yet match human evaluators, though few-shot and visual prompting with expert critiques raised feedback quality by 55 percent (Duan et al., 2024b). The authors of PerceptUI say earlier multimodal evaluators gave surface-level critiques or judgments reflecting the model’s own biases rather than a particular user’s response (Bougie et al., 2026).

Agents that act as test users are less studied. A vendor comparison on two prototype sites reports AI browser agents had task success of 0-25% on most tasks in one project while humans reached 62-95%, and that agents could not flag problems beyond basic navigation struggles; the vendor concludes AI testing cannot yet replace human testers (Biddle, 2025). A study of language-model-simulated users in agent evaluations (benchmarks, not interface usability) found they underestimate agent performance on challenging tasks and surface different failure patterns than human users (Seshadri et al., 2026). The broader question of using models as graders is treated in LLM-as-a-judge.

Broader connections

Usability evaluation checks the output of the steps in the interface design process and bears on AI-generated interfaces, where the model-based methods above are proposed as a substitute for human evaluators. Its yardsticks, such as problem counts and agreement between raters, are those of research methods generally; see methods for how inferences from behaviour are validated in cognitive science.

Further reading

• Nielsen (2012): the current form of the five-user guideline, with its limits (per group, qualitative only).
• Schmettow (2012): the main argument that the classic discovery formula understates the sample needed.
• Hertzum et al. (2014): a replication of the evaluator effect with experienced professionals.
• Cockton et al. (2004): how structured reporting changed the validity of heuristic evaluation.
• Zhong et al. (2025b) and Guerino et al. (2025): two opposite results on language-model heuristic evaluation.