How to make research reproducible: psychology protocol gives 86% success rate
Four groups in the field of experimental psychology successfully replicate each other’s work by following best practices
Research teams achieved highly replicable results after steps such as preregistration of their study protocols.Credit: sshepard/Getty
Nowhere has the replication crisis in science struck harder over the past decade than in experimental psychology. A series of high-profile failures to reproduce findings has seen critics line up to dismiss work in the field as unreliable and riddled with methodological flaws.
In a bid to restore its reputation, experimental psychology has now brought its A game to the laboratory. A group of heavy-hitters in the field spent five years working on new research projects under the most rigorous and careful experimental conditions possible and getting each other’s labs to try to reproduce the findings.
Published today in Nature Human Behaviour1, the results show that the original findings could be replicated 86% of the time — significantly better than the 50% success rate reported by some systematic replication efforts.
The study, the authors say, shows that research in the field can indeed be top quality if all of the right steps are taken.
“People are worried that there’s all these problems that undermine the credibility or replicability of findings,” says Brian Nosek, a psychologist at the University of Virginia in Charlottesville and a co-author of the study. “What if we ran a study where we tried to eliminate all those problems and do it as rigorously as possible?” The idea, he says, was to use best practices to identify a benchmark of replicability.
Team effort
Rather than trying to replicate existing published studies, the organizers of the work asked four prominent research groups based at US universities to devise and plan their own separate projects to address new questions in social psychology.
Each lab carried out its chosen projects using practices that are known to increase experimental rigour and the likelihood of replication. When research at the pilot stage suggested an interesting effect, the original lab ran a full-scale confirmatory study with a sample size of at least 1,500 participants. Both the pilot-phase and the full-scale studies were preregistered, which means that the authors specified and submitted a research plan in advance to a database.
Four candidate new discoveries were selected by each lab for the self-confirmatory testing phase. After this, the other three labs each ran a full-scale repeat of these four chosen studies, again with a sample size of at least 1,500. For the replication efforts, the labs relied on the preregistered research plans and other relevant experimental materials (such as instructional videos for participants) shared by the founding lab.
As well as checking the overall findings, these replication efforts also looked at the effect size, to see whether there was any evidence of the ‘decline effect’, in which the strength of a finding reduces with subsequent experiments. No such decline was observed: the effect sizes in the replication trials were the same as those measured in the original labs’ self-confirmatory experiments.
In principle, the results could apply to psychology more broadly, and across other fields of social science, Nosek says.
Serious science
Nosek stresses that the research topics chosen for replication were not trivial questions with obvious answers, which would have been relatively simple to replicate. Instead, the projects assessed serious research questions in marketing, advertising, political science, communication, and judgement and decision-making. Several of the labs involved have already produced published papers about their findings.
One paper2, published last year in Scientific Reports, by a group led by Jonathan Schooler at the University of California, Santa Barbara, showed that people can misattribute the ‘a-ha!’ feeling they get when they solve an anagram to the truth of the statement in which the anagram is embedded.
The new replication effort is also a “political communication to demonstrate that not all of social sciences is trash”, says Malte Elson, a metascientist at the University of Bern. “But that’s a good thing. I think it’s very useful to show both to the community, but also to the general public, that social sciences are not inherently flawed.”
doi: https://doi.org/10.1038/d41586-023-03486-5
Additional reporting by Anil Oza.
This story originally appeared on: Nature - Author:David Adam
