Cognitive task analysis

Cognitive task analysis (CTA) is, in one applied study’s definition, a family of methods for gaining access to the cognitive processes that underlie performance of tasks (Fackler et al., 2009). It is used in cognitive systems engineering and human factors to find out what people doing complex work notice, judge and decide, so that the findings can shape training, simulations and the design of displays and tools. Its methods rely mostly on interviews with experts, and so inherit the limits of what people can recall and put into words. The main disputes concern how reliable and valid the methods are, whether any method can be recommended over another, and how large the benefit of CTA-based training is.

History

The critical decision method (CDM), the best known CTA interview, was described in 1989 by Klein, Calderwood and MacGregor as a retrospective interview strategy that applies a set of cognitive probes to actual nonroutine incidents that required expert judgment or decision making (Klein et al., 1989). Its first applications were to firefighters (Klein et al., 1989). A later paper by Hoffman, Crandall and Shadbolt used the method as a case study in the methodology of CTA, addressing questions of data quality, method reliability, efficiency and utility (Hoffman et al., 1998).

The field around CTA is cognitive systems engineering, which a university laboratory says was founded by David Woods and Erik Hollnagel in the aftermath of the Three Mile Island accident (Cognitive Systems Engineering Laboratory, n.d.); see Cognitive systems engineering. CTA also draws on the naturalistic decision making (NDM) community. Klein, Hoffman and Militello say that NDM names a community of researchers rather than a cognitive process or model, and that it studies sensemaking, problem detection and planning as well as decisions (Klein et al., 2016).

The origins of CTA before 1989, and of its use in instructional design, still need to be written.

Core concepts

What the methods try to capture

The target of CTA is what experts know and do that a list of task steps would miss. Klein, Hoffman and Militello use the term macrocognition for sensemaking, coordinating, managing uncertainty and other cognitive functions that let people cope with complexity (Klein et al., 2016). Klein defines tacit knowledge as a set of beliefs and capabilities acquired through experience and expressed through pattern matching, rather than as knowledge that cannot be put into words. He places perceptual knowledge, pattern recognition, recognition of familiarity or anomaly, mental models and mindsets within it, and says mental models are used to form expectancies that flag missing data and events (Klein, 2023).

Two objects recur in the interview methods: cues, the features of a situation an expert attends to, and strategies, what the expert does about them. In the CDM interview much of the time is spent uncovering cues (Klein et al., 1989). A cue’s meaning depends on its context (Fackler et al., 2009).

The critical decision method

In the original procedure, the interviewee picks a nonroutine incident that was challenging and on which an expert might have decided differently from someone with less experience (Klein et al., 1989). If no case comes to mind, several can be screened briefly, and the authors learned to avoid memorable cases, such as a death, in which the interviewee did not play a key decision role (Klein et al., 1989). The interview then has these steps (Klein et al., 1989):

1. An uninterrupted account of the incident from start to end. It gives the interviewer context, activates memory and casts the interviewer as listener.
2. A timeline of the sequence and duration of events, including the person’s own thoughts and perceptions. Building it exposes inconsistencies and missing facts.
3. Identification of decision points. A point is probed if the interviewee agrees that other reasonable courses of action were possible, or that another person, with less or greater expertise, might have chosen differently, even if the person did not see it as a decision.
4. Probing of each decision point with a set of probes.

The probe set covers cues (“What were you seeing, hearing, smelling?”), knowledge, analogues (whether the person was reminded of a previous experience), goals, options, basis, experience, aiding, time pressure, situation assessment and hypotheticals, such as what difference it would have made if a key feature of the situation had been different (Klein et al., 1989). The authors advise eliciting specific rather than general goals, and report that reasons for an action are most often found by asking about choices that were not made or were rejected (Klein et al., 1989). Sessions are planned to last about two hours, are recorded for verbatim transcripts, and the interviewee is usually allowed to draw a diagram of the incident (Klein et al., 1989). The authors use two interviewers where possible, because one person can struggle to listen, take notes and begin the timeline at once, and because a pair lets a less experienced interviewer learn at low risk (Klein et al., 1989).

Later applied studies describe the interview in four sweeps: incident identification, timeline verification, deepening, and “what-if” queries. In the deepening sweep the interviewer probes decision points, judgments and the decision process; the what-if sweep asks hypothetical questions to identify implicit cues (Islam et al., 2014). A health-care toolkit describes the whole method as a sequence from defining the scenario and choosing probes in advance, through timeline, phases and probing, to transcription and tables, with the probes defined before analysis to prevent irrelevant information (Agency for Healthcare Research & Quality, n.d.).

Other methods

Applied Cognitive Task Analysis (ACTA) is a set of techniques, a task diagram, a knowledge audit and a simulation interview, developed to complement each other, each tapping a different aspect of cognition (Gore & McAndrew, 2009). Its output is a cognitive demands table (Gore & McAndrew, 2009). A literature coding of CTA studies identified over 100 distinct methods (Yates & Feldon, 2011). In a systematic review of CTA in clinical and health services research, the critical decision method and CTA interviews were used most often, and hierarchical task analysis, task knowledge structures and think-aloud less often (Swaby et al., 2022). A different design applied to medical training has experts perform the target task several times, each pass examining a different aspect, to produce training requirements, performance metrics, scenario requirements and simulator requirements together; it was pilot-tested with two combat casualty tasks (Cannon-Bowers et al., 2013).

Interviews suit exploring tasks, events and scenarios but do not directly capture the representations people use during performance. Verbal protocols, in which people speak their thinking during or immediately after the task, better probe the ongoing processes of representing problems and accomplishing tasks (van de Wiel, 2017).

Concept mapping, goal-directed task analysis and the think-aloud method itself are not yet covered.

Representing the data

Interview data are usually condensed into a table. In ACTA the cognitive demands table is the deliverable intended for practitioners: each row covers a difficult cognitive element, why it is difficult, common errors, and the cues and strategies used, merging the data of earlier steps (Gore & McAndrew, 2009).

Other tables have different columns. In a Navy study of landing signal officers, the decision requirements table listed the critical decisions, critical cues, potential errors, why each decision is difficult, degrees of uncertainty, tools in use and conflicting goals; the study then used it to design displays (Thordsen, 1998). In a training project for air operations commanders, the decision requirements table (DRT) had four columns: Decisions/Assessments, Why Difficult?, Cues/Factors and Strategies (Baxter & Lunsford, 2005). One row, for determining which information sources to use, listed difficulties such as red tape and not knowing all available resources, cues such as information going out of date rapidly, and strategies such as talking to people at all levels and synthesising rather than only gathering data (Baxter & Lunsford, 2005). Each key decision was identified from interview notes with what made it difficult, the factors affecting it and the strategies the subject matter experts suggested (Baxter & Lunsford, 2005).

In an ACTA study for emergency medical care, the knowledge audit table had one row per interviewee statement, with the problem, cues, strategies and the expert’s identifier, and the cognitive demand table kept only demands that at least two interviewees mentioned. One entry noted that measurement methods for monitoring a patient’s condition are unreliable in cold environments, a point nine participants raised, which could confuse inexperienced practitioners (Stanik et al., 2021). The authors found the table a compact summary of all findings of earlier stages, useful to downstream engineers, though not always required (Stanik et al., 2021).

From tables to training and design

In the air operations project, decisions and strategies went into a decision and strategy table for the training simulation, and the cues and factors gave the simulation its cognitive fidelity (Baxter & Lunsford, 2005). The authors report that the tables identified contingencies up front, which cut development time by removing repeated modifications and testing, but CTA covered only nine of the sixty decisions identified (Baxter & Lunsford, 2005). The detailed rules for turning a table into design or training recommendations still need to be written.

Questions, controversies, and new developments

How reliable and valid are the interviews?

The 1989 authors say reliability of a semistructured retrospective interview is hard to assess, because the exact circumstances can never be recreated and the interviewee’s memory of the event alters to some unknown degree after the interview (Klein et al., 1989). They acknowledge that the method relies on verbal reports, which are not a direct window into cognition (Klein et al., 1989). They report three checks from firefighter studies (Klein et al., 1989):

• Two coders agreed 81 to 100 percent on locating decision points in four fireground transcripts; the newer coder over-called standard responses as decision points.
• Classifying decision points into five strategy categories gave only 66.5 percent exact agreement between coders (87.8 percent counting adjacent categories), so the authors dropped the finer-grained analysis.
• In a wildland fire study, protocols taken on site and five months later corresponded from 56 to 100 percent per decision maker, averaging 82.5 percent.

They also say the interview methods are sensitive to the domain experience of the interviewer (Klein et al., 1989). Their own mitigations are to ask for an uninterrupted initial account, as a bulwark against the interviewers’ biases, and to refrain from asking directly why actions were taken (Klein et al., 1989). Because interviewers have heard the whole story before probing, they can adapt the timing and wording of questions to the case (Klein et al., 1989). The 1998 methodology paper addresses data quality and method reliability, efficiency and utility (Hoffman et al., 1998).

What experts cannot or do not report

Yates, Feldon and Clark state that experts omit about 70 percent of the decisions they make when they describe how to perform a complex task, which weakens instruction built on their accounts (Yates et al., 2011). A systematic review lists experts’ inability to articulate what they do, and incomplete verbal accounts, as limitations of CTA, and notes that people often overestimate their expertise or vary in self-assessment with context; it suggests narrowing the expert pool to qualified subgroups (Swaby et al., 2022). Klein’s definition of tacit knowledge as pattern matching, not as something unsayable, offers a different framing of the same difficulty (Klein, 2023). For interviews with people about their own reasoning more generally, see Eliciting self-knowledge.

Is there a basis for choosing among methods?

With over 100 methods identified, Yates and Feldon conclude that methods are matched to applications inconsistently, that CTA is currently more craft than technology, and that comparing the reliability and validity of methods is an open task (Yates & Feldon, 2011). The health research review found no explicit guidance on choosing a method (Swaby et al., 2022). Critics hold that ACTA gathers less comprehensive information than the critical decision method; one applied study found this true for the simulation stage but chose ACTA because it helped structure emergency scenarios (Stanik et al., 2021). Its authors also note that learning the terms and processes of a specialised domain is hard for the analyst and affects the quality of questions (Stanik et al., 2021). ACTA’s authors themselves note that CTA methods have been criticised for problematic data analysis (Gore & McAndrew, 2009).

Does CTA improve training?

A 2013 meta-analysis of CTA-based instruction reported a large overall effect (Hedges g = .871), with effect sizes varying substantially by CTA method and training context; the authors add that there is relatively little controlled research on CTA as a foundation for training (Tofel-Grehl & Feldon, 2013). A PRISMA meta-analysis of 12 studies of surgical training found that trainees given CTA-based training did better than groups given conventional training on procedural knowledge (standardized mean difference 1.36, 95 percent confidence interval 0.67 to 2.05) and technical performance (SMD 2.06, 95 percent confidence interval 1.17 to 2.96) (Edwards et al., 2021). One applied report of a training simulation project reports shorter development time (Baxter & Lunsford, 2005).

Broader connections

CTA is a toolkit of cognitive systems engineering, whose basic unit of analysis is the joint cognitive system, a person and technology together, defined by the ability to modify behaviour from experience (Baxter, 2011). Its outputs, tables of difficult decisions, cues and strategies, feed the design of decision support and of displays, as in the landing signal officer study (Thordsen, 1998). Its interview methods share the problem of retrospective report with other ways of asking people about their own thinking, discussed in Eliciting self-knowledge, and with the think-aloud methods of usability testing. The expertise it studies is the subject of work on problem solving and judgment and decision-making.

Further reading

• Klein, Calderwood and MacGregor (1989) (Klein et al., 1989): the paper that introduced the critical decision method, with its probes, reliability checks and stated limits.
• Gore and McAndrew (2009) (Gore & McAndrew, 2009): a short overview of ACTA and its cognitive demands table.
• Swaby and colleagues (2022) (Swaby et al., 2022): a systematic review of how CTA methods are used in clinical research, with the limitations reported.
• Yates and Feldon (2011) (Yates & Feldon, 2011): a review calling for taxonomic research to compare CTA methods.
• Baxter and Lunsford (2005) (Baxter & Lunsford, 2005): an applied report showing a decision requirements table and its use in building a training simulation.
• van de Wiel (2017) (van de Wiel, 2017): compares interviews and verbal protocols for examining expertise.