So you got a null result. Will anyone publish it? Nature asks whether they are working
Researchers have tried a bunch of strategies to get more negative results into the literature
Evolutionary biologist Natalie Pilakouta thought it would be an easy theory to test: fish living in Iceland’s geothermal hot springs prefer warmer water than do members of the same species that live in cooler lakes nearby. Yet, when she came to the end of her two-year study, what she found was inconclusive — given the choice, both populations of fish preferred the same, cooler waters. Her postdoctoral supervisor urged her to set aside the findings and move on to other studies: “It’s a failed experiment,” she was told. “You must have done something wrong.”
The words stung because publications are a crucial piece of academic currency, particularly for an early-career researcher, and she knew she’d face an uphill struggle to find a home for her results. Moreover, she felt a sense of urgency to share the counter-intuitive findings, which undermine the assumption that aquatic life might evolve a preference for higher temperatures in response to global warming.
Pilakouta, who is based at the University of St Andrews, UK, was one of the lucky ones. After submitting her findings to seven journals over six years, her study was finally published1 in January 2023. But her experience illustrates academia’s oft-bemoaned ‘file-drawer problem’, in which findings with null or negative results — those that fail to find a relationship between variables or groups, or that go against the preconceived hypothesis — gather dust in favour of studies with positive or significant findings. A 2022 survey of scientists in France, for instance, found that 75% were willing to publish null results they had produced, but only 12.5% were able to do so2. Over time, this bias in publications distorts the scientific record, and a focus on significant results can encourage researchers to selectively report their data or exaggerate the statistical importance of their findings. It also wastes time and money, because researchers might duplicate studies that had already been conducted but not published. Some evidence suggests that the problem is getting worse, with fewer negative results seeing the light of day3 over time.
Funders, publishers and researchers are not sitting idle. Many journals now encourage teams to submit plans and protocols for experiments before conducting them, so that the journals can review the proposals and commit to publishing the results, whatever the outcome. Hundreds of journals now offer such ‘registered reports’, and the number of journals adopting this approach has doubled since 2018.
A laser focus on positive results is not the only way to do science, says Brian Nosek, executive director of the Center for Open Science in Charlottesville, Virginia. Nosek and a constellation of researchers across the world have been pushing to rewrite how research is conducted, challenging the very definition of success. This includes a crackdown on the nefarious side of science — misconduct such as plagiarism — but also a call to curb some of the ‘softer’ transgressions such as selective reporting, with a view to publishing more negative findings. These changes have started to materialize across the publishing industry, as preprint servers proliferate and publishers adopt new manuscript formats, launch journals dedicated to null results and call for special issues.
“We can surely do better,” Nosek says.
Hidden results
Researchers have noted the file-drawer problem for decades. But the bigger problems it was causing did not become clear until the early 2010s, when they set out to reproduce the findings of several foundational experiments in psychology and medical science, and found that they could not. Scientists began to study the extent of this ‘replication crisis’ and the problem of publication bias.
Their research laid bare just how often negative results were being buried. In an analysis of more than 300,000 scientific conference presentations, informal posters or talks that scientists often endeavour to turn into papers, fewer than 40% were published in peer-reviewed journals, and negative or null findings were far less likely to be published than positive results4.
The extent of publication bias varies by discipline and by country, but the problem seems to have worsened over time. An analysis of 4,600 papers from 1990 to 2007 found that publication bias had increased by 22% over that period3.
There could be real-world implications to such a skew in publications. Among 74 registered clinical trials evaluating antidepressants, for example, nearly one-third remained unpublished; these trials were much more likely to show negative than positive results5. Judging by the publications alone, 94% of the trials looked as if they returned positive results, whereas a drug-approval panel judged that 51% did.
This selective reporting creates an inflated perception of drug efficacies, one compounded by meta-analyses — surveys of the published literature — that contain mainly studies with positive results.
The bias is present despite the fact that investigators conducting clinical trials in the United States are mandated by law to report their results, regardless of outcome; there can be billions of dollars at stake and trial participants who have given their time and expect the results to be published. These results illustrate “how high a hill there is to climb”, Nosek says.
Adding to the likelihood of bias, studies with negative or null findings are often given stricter scrutiny than those with positive findings, especially if the positive findings “confirm something we think is true”, says Steven Goodman, founder of the Stanford Program on Research Rigor and Reproducibility at the Stanford School of Medicine in California.
Jessica Payne, a cognitive neuroscientist at the University of Notre Dame in South Bend, Indiana, says that there’s still a perception that scientists must have had some flaw in their research design if a study returns negative or null results.
Indeed, according to a survey of 480 economists6, studies with null results are perceived to be less publishable, of lower quality and less important than studies with large and significant results, even when features such as sample size are held constant — a phenomenon known as the null result penalty. If anything, Goodman says, a study with a large effect size should be scrutinized much more than one with a null finding.
Cultural bias
The replication crisis made one fact crystal clear: incentive structures in academia are not always in line with research integrity and reproducibility. That has a large role in why so few negative studies are published, says Anne Scheel, a metascientist at Utrecht University in the Netherlands.
At the crux of both academic misconduct and publication bias is the same ‘publish or perish’ culture, perpetuated by academic institutions, research funders, scholarly journals and scientists themselves, that rewards researchers when they publish findings in prestigious venues, Scheel says.
But these academic gatekeepers have biases, say some critics, who argue that funders and top-tier journals often crave novelty and attention-grabbing findings. Journal editors worry that pages full of null results will attract fewer readers, says Simine Vazire, a psychologist at the University of Melbourne in Australia and editor of the journal Psychological Science.
This creates a tight feedback loop between researchers and journals. To attract journals with findings that seem new and noteworthy, some scientists might be tempted to change their hypothesis after seeing the results, or to release only a portion of the data, or to perform statistical tricks, Nosek says.
Positive solutions
To encourage more researchers to report null results, journals and funders are trying several schemes. One of the most significant changes to come out of the replication crisis is the expansion of preregistration (see ‘Registrations on the rise’), in which researchers must state their hypothesis and the outcomes they intend to measure in a public database at the outset of their study (this is already the norm in clinical trials).
Source: OSF.IO
The preregistration model nudges researchers to be faithful to the original intent of their study, but it doesn’t address biases that might affect whether they submit their findings to a journal, nor the biases of journal editors and reviewers in deciding what to publish, Nosek says.
Instead, he and his colleagues have been focusing on promoting and evaluating the registered report model — similar to a preregistered report, but with the initial plan published by a journal, along with a commitment to peer-review and publish the results.
Preliminary data look promising: when Scheel and her colleagues compared the results of 71 registered reports with a random sample of 152 standard psychology manuscripts, they found that 44% of the registered reports had positive results, compared with 96% of the standard publications7 (see ‘Intent to publish’). And Nosek and his colleagues found that reviewers scored psychology and neuroscience registered reports higher on metrics of research rigour and quality compared with papers published under the standard model8.
Source: Ref. 7
When the format launched in 2012, only a handful of journals published registered reports; now more than 300 offer the format, including PLoS ONE and Nature, which is published by Springer Nature (Nature’s news team is editorially independent from its journal team). Since starting to offer the format in February 2023, Nature has yet to publish any registered reports, but its sibling journal Nature Human Behaviour has.
Although the format has gained in popularity, there are still some kinks to be ironed out, researchers say. Earlier this year, Christine Blume, a sleep researcher at the University of Basel in Switzerland, published her first registered report9 on how light affects human circadian rhythms in Nature Human Behaviour. Although she liked receiving feedback on her study design before data collection — “it made me feel that I had the best study design to answer the question I set out to address,” she says — she found it frustrating that the feedback process can span months, even though researchers have a limited time in which to spend grant money.
These practical concerns are important to address, Nosek says. He admits that his own paper about the quality of registered reports8 was not itself a registered report, because the grant money was expiring and the team didn’t have time to go through a lengthy approval process and complete its analysis. “We cannot dismiss pragmatics, but what we can do is think about how we lower the barrier so that more of these circumstances can be dealt with,” he says.
Journals that offer registered reports are not spread equally across disciplines; most are in psychology and, more recently, neuroscience. Few physical-science journals offer the format — even though null results, such as the failure of the Large Hadron Collider near Geneva in Switzerland to find new subatomic particles since the Higgs boson, have been an important part of progress. Emily Sena, a translational-medicine researcher and metascientist at the University of Edinburgh, UK, says that few academics in preclinical fields have been keen to try the format, especially when there is already so much red tape before researchers can begin their experiments.
The format has been slow to catch on among researchers, says Vazire. “We’re not receiving many registered-report submissions.”
Sena and her colleagues have been spreading the word about registered reports and helping journal editors to feel equipped to review submissions, she says. Some funders are providing cash incentives: in 2022, the Center for Open Science offered up to US$50,000 to consciousness researchers willing to publish a registered report for their work.
It will be important to track how these interventions affect marginalized groups in academia, Sena says. Academics of colour are more likely to be on a fixed contract, so they have less wiggle room to embrace formats that might be better for science overall but less helpful for individual scientists, she says.
The Center for Open Science is planning to run trials in which researchers are randomly assigned to use either the standard publication model or a registered report, to evaluate the rigour, acceptance rate and timelines of the resulting publications. Results are expected by 2027.
Not every effort to reduce publication bias has borne fruit. One that has seldom worked, Nosek says, is to set up journals with the express purpose of publishing null results. These efforts are well intentioned, he says, but often do not work because a journal can become identified with studies that weren’t able to be published elsewhere. “It can’t provide the reward that researchers need,” he says.
Payne was a co-editor at one of these journals, Experimental Results, published by Cambridge University Press. After only three years, the journal ceased publication in 2023 despite carrying the “imprimatur of Cambridge”, she says.
There’s an increasingly popular do-it-yourself route to publishing negative results: posting a manuscript on a preprint server. Publishing a preprint can offer an opportunity to showcase research without the pressure of journal submission. This option can be especially helpful for early-career researchers, Pilakouta says. Still, it takes time to write up a result, regardless of where it appears, and publishing on preprint servers is unlikely to offer researchers enough of an incentive to justify the time, Goodman says.
Null nuance
Advocates acknowledge that not every study that returns a null result is worthy of publishing. Goodman says he encourages researchers to publish null and negative findings that are “informative”, meaning they come from studies and analyses that are designed rigorously, question previous results and open up fresh areas for exploration.
For example, there is a long-held idea that the womb is sterile — that the uterus and fetus are free of microorganisms. But, beginning in 2010, a series of papers found microbial contamination in the placenta, calling the hypothesis into question and suggesting that some complications of pregnancy could be linked to bacteria. It wasn’t until 2019 that a study of placental samples from 537 women — by far the largest number in an analysis of this kind — rigorously showed the absence of any bacterial signal. That study set a benchmark for investigating the microbiome of tissues that carry few microorganisms and that can therefore give rise to false-positive results, and suggested that bacterial infection is not a common cause of problems in pregnancy10.
Blume says it’s important to extract something insightful from the data, even if they are inconclusive. For example, in 2022 she found that although artificial light suppresses the hormone melatonin, that didn’t equate to a change in sleep quality11. The message that melatonin isn’t necessarily a proxy for sleep quality might have helped that study to get published, she says.
As long as researchers continue to seek publication in prestigious outlets, publication bias won’t go away, Goodman predicts. Still, he is surprised at how much progress has been made in the past decade: top-tier journals pledging to accept rigorous studies, regardless of outcome, would have been “unheard of” even five or ten years ago, he says.
Pilakouta now leads a laboratory and can set an example for her undergraduate and graduate students. But she’s also seen first-hand how deeply engrained the thirst for positive findings is. “It concerns me how early it starts,” she says. Next time she gets a null result, she says, she’s hopeful that it won’t take seven years to publish it.
Nature 631, 728-730 (2024)
doi: https://doi.org/10.1038/d41586-024-02383-9
This story originally appeared on: Nature - Author:Max Kozlov
