Replicability and generality

A finding replicates when a new study, following the same method with new participants, obtains the same result. Since 2014, large coordinated projects have repeated dozens of published psychology findings, often across many laboratories at once. Their headline results are sobering: in the Open Science Collaboration’s 2015 project, replication effects were about half the size of the originals across 100 studies (Open Science Collaboration, 2015), and in Many Labs 2, 54 percent of 28 findings replicated while the median effect size fell from d = 0.60 to 0.15 (Klein et al., 2018). Cognitive findings replicate about twice as often as social-psychological ones, 50 against 25 percent, although the comparison is confounded with study design (Open Science Collaboration, 2015). How to read these rates is disputed: Gilbert and colleagues argued that the data are consistent with high reproducibility, and the original authors replied that this rests on statistical misconceptions (Gilbert et al., 2016b; Anderson et al., 2016). A separate question is generality: most research uses Western samples, and critics argue that these are unusual among humans (Henrich et al., 2010; Rad et al., 2018).

History

The Open Science Collaboration published its attempt to replicate 100 psychology studies in 2015. Replication effects were about half the magnitude of the originals (Open Science Collaboration, 2015), and all journals and disciplines showed substantial and similar declines in effect size (Open Science Collaboration, 2015). Earlier, Many Labs 1 had replicated 10 of 13 effects across many samples (the paper’s Discussion gives 11 of 13) (Klein et al., 2014). Many Labs 2 then ran preregistered replications of 28 findings in about 125 samples. Fifteen (54 percent) replicated in the same direction with statistical significance, 21 (75 percent) had smaller effects than the original, and 32 percent went in the opposite direction to the original (Klein et al., 2018).

Core concepts

Measuring replication

There is no single criterion for replication. The Open Science Collaboration counted, among other measures, whether a replication was statistically significant (p < 0.05) in the same direction and whether the original effect fell within the replication’s confidence interval (Open Science Collaboration, 2015; Anderson et al., 2016). Effect sizes are compared with standardised measures such as Cohen’s d, the difference between conditions in standard deviations.

Cognitive versus social effects

In the Open Science Collaboration project, 50 percent of cognitive effects and 25 percent of social effects replicated by the p < 0.05 criterion. The authors attribute the gap partly to weaker original effects in social psychology and to the more powerful within-subject designs common in cognitive work, where each participant is tested in every condition (Open Science Collaboration, 2015). The comparison is confounded with design and journal: the sample had 43 cognitive and 57 social-personality studies, and interaction effects replicated less often (22 percent) than main effects (47 percent) (Open Science Collaboration, 2015).

Multi-lab replications of cognitive effects

Large replications of specific cognitive effects give a mixed picture:

• A multi-lab registered replication of verbal overshadowing, where describing a face impairs later recognition of it, found a robust effect that depended on timing: small when the description immediately followed the event, larger after a 20-minute delay (Simons et al., 2014).
• A registered replication of the congruency sequence effect in cognitive control extended rather than contradicted the original: the effect appeared in reaction times across four tasks, but in accuracy only in the Simon task (Gyurkovics et al., 2020).
• A 23-lab preregistered replication of the sequential-task ego-depletion effect, the idea that exerting self-control impairs performance on a following task, with 2,141 participants, found an effect indistinguishable from zero (d = 0.04) (Hagger et al., 2016).

Records for individual fields are kept in each article: Memory, Attention, Perception, Language, Judgment and decision-making and Embodied cognition.

WEIRD samples

Henrich, Heine and Norenzayan argue that samples from Western, educated, industrialised, rich and democratic (WEIRD) societies are frequent outliers among human populations, so claims about human nature drawn from them are unsafe (Henrich et al., 2010). They base this on variation across populations in domains including visual perception and spatial reasoning, and call for less confident claims (Henrich et al., 2010). A 2018 analysis states that almost all research published in Psychological Science relies on Western samples while drawing inferences about humans in general (Rad et al., 2018).

Questions, controversies, and new developments

How bad is the replication rate?

Gilbert, King, Pettigrew and Wilson argued that the Open Science Collaboration paper contains three statistical errors, concerning error, power and bias, and that its data are in fact consistent with high reproducibility (Gilbert et al., 2016b). They made three points. Many replications differed from the originals in population and procedure. Using Many Labs data, about 34 of the 100 replications would fail by chance alone even if every original finding were true. And the design of single replications had low power: a Many Labs-style set of single replications would succeed 34 percent of the time rather than the 85 percent obtained with many labs (Gilbert et al., 2016b).

The Open Science Collaboration authors replied that this optimistic assessment rests on statistical misconceptions and on selective interpretation of correlational data, and that neither optimistic nor pessimistic conclusions are yet warranted (Anderson et al., 2016). They calculated that sampling error alone would give an expected replication rate of 78.5 percent on the confidence-interval measure, against the 47.4 percent actually observed. They noted that the 85 percent rate of Many Labs 1 rests on a small, ad hoc sample of effects, and that the similar Many Labs 3 gave 30 percent (Anderson et al., 2016). They conceded that differences between laboratories and populations reduce reproducibility on the confidence-interval measure, and that original effect sizes are probably inflated by publication bias, since 83 percent of replications gave smaller effects (Anderson et al., 2016).

In their rejoinder, Gilbert and colleagues reported that replications endorsed by the original authors were four times more likely to succeed, while acknowledging that endorsement is an imperfect measure of fidelity that could reflect the original authors’ confidence in their own findings (Gilbert et al., 2016a). The overall estimate remains disputed. The critique was aimed at the overall estimate; it did not address the cognitive-versus-social figure (Gilbert et al., 2016b).

Do findings generalise across cultures?

The WEIRD critique predicts that effects should vary between Western and other samples. Many Labs 2, with 15,305 participants in 36 countries, found little heterogeneity between more and less WEIRD cultures in its 28 effects, a counterpoint to that worry (Klein et al., 2018). The question is open.

Replication record

The projects described here are the replication record of the field as a whole. The ego-depletion effect failed its multi-lab test (Hagger et al., 2016); verbal overshadowing and the congruency sequence effect survived theirs (Simons et al., 2014; Gyurkovics et al., 2020).

Broader connections

Model mimicry and the limits of inference discussed in Methods of cognitive psychology are a second, independent reason for caution about what cognitive findings show. Human cognition gives an overview of the processes these findings concern.

Further reading

• Open Science Collaboration (2015): the 100-study replication project.
• Gilbert et al. (2016b) and Anderson et al. (2016): the critique and the reply.
• Klein et al. (2018): Many Labs 2, 28 findings across 36 countries.
• Henrich et al. (2010): the WEIRD critique.