Eliciting self-knowledge

Eliciting self-knowledge means asking a person how they work, read, decide or judge, in order to produce a usable account of their preferences, criteria or expertise. The main methods are open self-report, the critical incident technique and its relatives in knowledge elicitation, comparison of concrete samples, and, more recently, interviews run by language-model agents. The central dispute is how far people can report on their own processes: people state preferences reliably, but several lines of evidence show that what they state can diverge from what they do or from what works for them. Methods differ mainly in how they try to get around that gap. Related methods for expert work are covered in Cognitive task analysis, and the wider problem of inferring hidden processes in Methods of cognitive psychology.

History

History is missing: the origins of introspection, the critical incident technique and survey methodology, beyond the dated primary papers cited in this article (for example the 1989 Critical Decision Method paper).

Core concepts

Self-report and its limits

A secondary course summary of Nisbett and Wilson’s 1977 paper “Telling more than we can know” states that people explaining their responses may base their reports on implicit, a priori theories about the causal connection, rather than on memory of the process (Course outline, n.d.). On that account a correct report is incidental to the theory the person holds.

Evidence from learning research shows the same kind of gap in practice. In a randomized comparison, students in the more effective active classes scored about 0.46 SD higher on the exam but reported about 0.56 SD lower feeling of learning, and they preferred and rated more highly the passive lecture that produced lower measured learning (Deslauriers et al., 2019). The authors attribute the mismatch partly to the extra cognitive effort of the better condition, which learners read as poor learning (Deslauriers et al., 2019). A review found virtually no evidence that teaching in a person’s stated preferred style improves learning, even though people reliably state such preferences (Pashler et al., 2008).

Self-reports about behaviour are also imperfect. Both daily end-of-day and single-time reports of smartphone use differed considerably from digital trace data, and respondents usually under-reported (Tkaczyk et al., 2024). In 251 people, a retrospective condom-use questionnaire matched a daily diary for 73 percent, with 15 percent over-reporting and 12 percent under-reporting, and agreement was best for people who always or never did the behaviour (Hoppe et al., 2007). For day-by-day smoking, recall by the Time-Line Follow-Back method was associated only 0.29 with a real-time electronic diary within each person, though the between-person correlation was 0.77 in 232 heavy smokers (Shiffman, 2009). Recall also showed heaping at multiples of 10 and 20 that real-time records did not (Wang et al., 2012). These are health and technology-use studies, not studies of reading or reviewing, so they show the general pattern rather than its size for any particular skill.

Critical incident technique and knowledge elicitation

The critical incident technique asks for a specific incident tied to an outcome, unlike an open example question that gives the participant no direction (Rosala, 2020). Its limits are that details are lost, some incidents are forgotten entirely, and recalling incidents can be demanding for participants (Rosala, 2020).

Knowledge elicitation developed the same idea for expert work. The Critical Decision Method interviews people about an actual, challenging nonroutine incident rather than about their general practice (Klein et al., 1989). The interviewee first gives an uninterrupted account, with the interviewer interrupting only for minor clarification (Klein et al., 1989). The incident is then rebuilt as a timeline, which exposed inconsistencies in the account and missing facts (Klein et al., 1989). The reasons for an action are most often revealed by the options that were not taken or were rejected (Klein et al., 1989), and probes include hypotheticals about a changed situation, such as what difference a different key feature would have made (Klein et al., 1989). The motivation is that experts omit a large share of the decisions they make when describing a complex task, according to one workshop abstract that gives no evidence for its figure (Yates et al., 2011).

Comparing concrete samples

A second way around weak introspection is to have people judge actual objects and then explain. In a study of teachers’ assessment of student work, the tacit sense of work quality was hard to put into words, and comparing pieces and writing a justifying comment made it partly explicit (Persson & Hartell, 2021). Policy-capturing research approaches the same question statistically: in a study of 49 employees judging the fairness of 56 appraisals, self-reported judgment policies matched regression-derived actual policies only to a limited degree (German et al., 2016).

In language-model preference elicitation, one study found that Bayesian-selected pairwise questions beat language-model-selected pairwise questions and overall baselines in a news-recommendation user study of about 40 users per setting (Handa et al., 2024). Open-ended questions, however, far outperformed pairwise ones when answers were evaluated with a language model, and the edge of the Bayesian method came from its own predictor (Handa et al., 2024). A choice-based (revealed-preference) technique predicted real sales better than the best of three stated-preference techniques in one vendor study (Dectech, 2019).

Interviews run by AI agents

Language models can run the interview. In a randomized comparison with university students, an AI interviewer did not produce significantly lower response quality than human interviewers, though respondents found it less interesting and were less likely to repeat it, and both settings left them satisfied (Wuttke et al., 2024). Neither humans nor the AI always followed the guidelines, with 72 violations per AI interview against 64 per human interview (Wuttke et al., 2024). The AI accounted for 88 percent of the violations of the rule to follow up on surprising or unclear answers and occasionally judged respondents’ answers despite instruction, while human interviewers led respondents with suggestions and failed to use active listening (Wuttke et al., 2024) (Wuttke et al., 2024).

In a 1,800-person experiment, chatbots that probed for elaboration got more detailed answers at a slight cost to respondent experience (Barari et al., 2025). In a 399-person study of three interview chatbots, responses scored high on established quality metrics but rarely captured participants’ specific motives or personal examples (Cuevas et al., 2023). Follow-up questions generated by GPT-4o and guided by a list of common interviewer mistake types outperformed human-authored questions (Shen et al., 2025). A related study found that language-model-led open-ended elicitation often yields more informative preference specifications than user-written prompts or labelled examples, with users reporting it takes less effort (Li et al., 2023).

Question design

Conventional survey advice includes avoiding leading or loaded questions and asking about one thing at a time; the source presents this as textbook wisdom, not as a tested effect (Krosnick & Presser, 2009). Agree/disagree rating questions produced much lower response quality than item-specific questions, with acquiescence bias in agree/disagree items documented over 50 years (Saris et al., 2010). Respondents may lack motivation from the start or become fatigued as a questionnaire progresses, and acquiescence is more common when a question is difficult and after many prior questions (Krosnick & Presser, 2009). Two experiments found lower data quality when the same question sits at the end of a web questionnaire than at the beginning (Neuert, 2024). Interviewer presence changes answers mainly on questions where social desirability could play a role, though differences are typically not large and many items show none (Keeter, 2015).

Questions, controversies, and new developments

Are hypothetical questions worse than incident questions?

A common assumption is that questions about a specific past incident are more accurate than questions about general practice or hypothetical situations. In hiring interviews, situational (“what would you do”) questions have corrected validities of 0.41 to 0.47 across meta-analyses, so hypothetical questions are not simply inaccurate there (Oostrom et al., 2016). What a hypothetical answer predicts is the person’s skill at working out what the situation demands, not honest self-description of intentions (Oostrom et al., 2016).

Do stated criteria match actual judgments?

The limited match between stated and inferred policies in the employee-appraisal study is read by some as poor self-insight (German et al., 2016). A commentary argues that a mismatch does not by itself show poor self-insight, because either the statistical or the elicitation exercise may fail to capture the original process (Rakow, 2014).

Can the gap be corrected during the interview?

In a semester-long course, an instructor gave a 20-minute talk on fluency, feeling of learning and actual learning, and over 65 percent of students later reported on a survey that their feelings about active learning had significantly improved; this was self-report only, with no controlled comparison, and the authors recommend presenting the value of effort early though the main experiment did not test this (Deslauriers et al., 2019). In a 150-participant experiment, a metacognitive prompt designed to help people compare their experiences with each strategy’s effectiveness did not improve their strategy decisions (Onan et al., 2022).

Member checking and validating the account

Member checking shares data or interpretation back with participants for feedback (Schafer & Phillippi, 2025). Participants’ views at that time can differ from what they said at data collection, so feedback does not simply confirm the earlier account, and critics say member checking may threaten validity and that agreement between researcher and participant is not necessarily the goal (Schafer & Phillippi, 2025).

Profiles in language-model personalization

Once an elicited profile is given to a language model, its effects are mixed. Profiles help because of personalization signals, mainly past responses the user produced or approved, not because of their semantic content, and position in the prompt matters (Wu et al., 2024). Across 13 language models, user profiles and retrieved memories generally worsened irrelevant personalization, preference narrowing and sycophancy, with average drops of 45.9, 41.7 and 61.7 percent (Wang et al., 2026a). In a two-week study with 38 participants, a condensed user profile in memory had the largest effect on agreeableness (Zewe, 2026). See Sycophancy in language models.

Limits of the AI-interviewer evidence

The comparison of AI and human interviewers rests on a pilot of 5 AI and 6 human interviews with students interested in survey methodology and a closed model, which limits generalisability and replicability, and minor prompt changes can cause unintended side effects (Wuttke et al., 2024) (Wuttke et al., 2024). In a user study of language-model preference elicitation, users found open-ended questions repetitive and not picking up on their input, and pairwise questions made without a model of the user’s preferences failed to probe how features are weighted (Handa et al., 2024). The advantage of generative open questions was shown in content recommendation, not across all domains, and that paper did not test pairwise comparison against open questions (Li et al., 2023).

Broader connections

The methods here are the front end of usability evaluation, and of cognitive task analysis, which applies incident-based elicitation to expert work. One application is a per-user guide to how a person wants text written and work reviewed; the finding that stated preferences about what helps learning do not track measured learning is a caution for any guide built from interview answers alone, and bears on how writing for comprehension is adapted to individual readers and on inclusive design, which adapts systems to individual users.

Further reading

• Klein, Calderwood and MacGregor, “Critical Decision Method for Eliciting Knowledge” (Klein et al., 1989): the original description of incident-based interviewing with timelines and probes.
• Deslauriers and colleagues, on feeling versus measured learning (Deslauriers et al., 2019): a randomized study of how misleading felt learning can be.
• Rosala, “The Critical Incident Technique in UX” (Rosala, 2020): a practitioner overview with strengths and limits.
• Shiffman on smoking recall versus real-time diaries (Shiffman, 2009): a direct comparison of recall and real-time records.
• Wuttke and colleagues on AI conversational interviewing (Wuttke et al., 2024): a randomized comparison of AI and human interviewers.