Data from giant project show how withdrawn research propagates through the literature

Exclusive: the papers that most heavily cite retracted studies

Data from giant project show how withdrawn research propagates through the literature

In January, a review paper¹ about ways to detect human illnesses by examining the eye appeared in a conference proceedings published by the Institute of Electrical and Electronics Engineers (IEEE) in New York City. But neither its authors nor its editors noticed that 60% of the papers it cited had already been retracted.

The case is one of the most extreme spotted by a giant project to find papers whose results might be in question because they cite retracted or problematic research. The project’s creator, computer scientist Guillaume Cabanac at the University of Toulouse in France, shared his data with Nature’s news team, which analysed it to find the papers that most heavily cite retracted work yet haven’t themselves been withdrawn (see ‘Retracted references’).

“We are not accusing anybody of doing something wrong. We are just observing that in some bibliographies, the references have been retracted or withdrawn, meaning that the paper may be unreliable,” Cabanac says. He calls his tool a Feet of Clay Detector, referring to an analogy, originally from the Bible, about statues or edifices that collapse because of their weak clay foundations.

The IEEE paper is the second-highest on the list assembled by Nature, with 18 of the 30 studies it cites withdrawn. Its authors didn’t respond to requests for comment, but IEEE integrity director Luigi Longobardi says that the publisher didn’t know about the issue until Nature asked, and that it is investigating.

Cabanac, a research-integrity sleuth, has already created software to flag thousands of problematic papers in the literature for issues such as computer-written text or disguised plagiarism. He hopes that his latest detector, which he has been developing over the past two years and describes this week in a Comment article in Nature, will provide another way to stop bad research propagating through the scientific literature — some of it fake work created by ‘papermill’ firms.

Further scrutiny

Cabanac lists the detector’s findings on his website, but elsewhere online — on the paper-review site PubPeer and on social media — he has explicitly flagged more than 1,700 papers that caught his eye because of their reliance on retracted work. Some authors have thanked Cabanac for alerting them to problems in their references. Others argue that it’s unfair to effectively cast aspersions on their work because of retractions made after publication that, they say, don’t affect their paper.

Retracted references don’t definitively show that a paper is problematic, notes Tamara Welschot, part of the research-integrity team at Springer Nature in Dordrecht, the Netherlands, but they are a useful sign that a paper might benefit from further scrutiny. (Nature’s news team is independent of its publisher, Springer Nature.)

Some researchers argue that retraction of references in a narrative review — which describes the state of research in a field — doesn’t necessarily invalidate the original paper. But when studies assessed by a systematic review or meta-analysis are withdrawn, the results of that review should always be recalculated to keep the literature up to date, says epidemiologist Isabelle Boutron at Paris City University.

Retracted references

These studies have the highest proportion of retracted papers in their reference lists, according to Nature’s analysis of articles flagged by the Feet of Clay Detector.

Source: Nature analysis of data from the Feet of Clay Detector. Figures for references and retractions were hand-checked and altered where necessary; detector data sources do not always give accurate counts.

Picking up fraudsters

Some of the papers that cite high proportions of retracted work are authored by known academic fraudsters who have had many of their own papers retracted.

These include engineering researcher Ali Nazari, who was dismissed from Swinburne University of Technology in Melbourne, Australia, in 2019, after a university misconduct investigation into his activities. He previously worked at Islamic Azad University in Saveh, Iran, and his current whereabouts are unclear. After Nature told publishers about his extant papers^2,3 topping Cabanac’s lists — including Elsevier and Fap-Unifesp, a non-profit foundation that supports the Federal University of São Paulo in Brazil — they said that they would look into the articles. One of the relevant journals was discontinued in 2013, Elsevier noted.

Cabanac’s detector also flags papers⁴ by Chen-Yuan Chen, a computer scientist who worked at the National Pingtung University of Education in Taiwan until 2014. He was behind a syndicate that faked peer review and boosted citations, which came to light in 2014 after an investigation by the publisher SAGE. Some of Chen’s papers that are still in the literature were published by Springer Nature, which says it hadn’t been aware of the issue but is now investigating. Neither Chen nor Nazari responded to Nature’s requests for comment.

Another flagged study⁵ is by Ahmad Salar Elahi, a physicist affiliated with the Islamic Azad University in Tehran who has already had dozens of his papers retracted, in many cases because of excessive self-citation and instances of faked peer review. In 2018, the website Retraction Watch (which also wrote about the Nazari and Chen cases) reported that according to Mahmoud Ghoranneviss, then-director of the Plasma Physics Research Centre where Elahi worked, Elahi was likely to be dismissed from the university. Now, Ghoranneviss — who has retired — says that Elahi was barred only from that centre and not the rest of the university. Elahi continues to publish papers, sometimes listing co-authors including Ghoranneviss, who says he wasn’t aware of this. Neither Elahi nor the university responded to Nature’s queries. The IEEE and Springer Nature, publishers of the journals that ran the Elahi papers, say they’re investigating.

Unhappy authors

Some authors are unhappy about Cabanac’s work. In May 2024, editors of the journal Clinical and Translational Oncology placed an expression of concern on a 2019 review paper⁶ about RNA and childhood cancers, warning that it might not be reliable because it cited “a number of articles that have been retracted”. The journal’s publishing editor, Ying Jia at Springer Nature in Washington DC, says the team was alerted by one of Cabanac’s posts on social media last year.

Cabanac’s analysis finds that just under 10% of the article’s 637 references have been retracted — almost all after the review was published. However, the paper’s corresponding author, María Sol Brassesco, a biologist at the University of São Paulo, says that removing these references doesn’t change the conclusions of the review, and that she has sent the journal an updated version, which it hasn’t published. Because the cited works were retracted after publication, the expression of concern “felt like we were being punished for something that we could not see ahead”, she says. Jia says that editors felt that adding the notice was the most appropriate action.

In other cases, authors disagree about what to do. Nature examined three papers^7,8,9 in which between 5 and 16% of the references have now been retracted, all co-authored by Mohammad Taheri, a genetics PhD student at Friedrich Schiller University of Jena in Germany. He says that criticisms of his work on PubPeer “lack solid scientific basis”. Yet, in May, a co-author of two of those works, Marcel Dinger, dean of science at the University of Sydney in Australia, told Retraction Watch that he was reassessing review papers that cited retracted articles. He now says that his team has submitted corrections for the works, but Frontiers, which published one paper, says it hasn’t received the correspondence and will investigate. Elsevier — which published the other two papers — also says that it is examining the issue.

Catching problems early

Examples in which papers cite already-retracted work suggest that publishers could do a better job of screening manuscripts. For instance, 20 studies cited by a 2023 review paper¹⁰ about RNA and gynaecological cancers in Frontiers in Oncology had been retracted before the article was submitted. Review co-author Maryam Mahjoubin-Tehran, a pharmacist at Mashhad University of Medical Sciences in Iran, told Nature that her team didn’t know about the retractions, and does not plan to update or withdraw the paper. The publisher, Frontiers, says it is investigating.

Until recently, publishers have not flagged citations to retracted papers in submitted manuscripts. However, many publishers say they are aware of Cabanac’s tool and monitor issues he raises, and some are bringing in similar screening tools.

Last year, Wiley announced it was checking Retraction Watch’s database of retracted articles to flag issues in reference lists, and Elsevier says it is also rolling out a tool that assesses manuscripts for red flags such as self-citations and references to retracted work. Springer Nature is piloting an in-house tool to look for retracted papers in manuscript citations and Longobardi says the IEEE is considering including Feet of Clay or similar solutions in its workflow. A working group for the STM Integrity Hub — a collaboration between publishers — has also tested the Feet of Clay Detector and “found it useful”, says Welschot.

Medical trend

Medical reviews that cite studies in areas later shown to be affected by fraud are a recurring theme in Cabanac’s findings.

In theory, meta-analyses or systematic reviews should be withdrawn or corrected if work they have cited goes on to be retracted, according to a policy issued in 2021 by the Cochrane Collaboration, an international group known for its gold-standard reviews of medical treatments.

Boutron, who directs Cochrane France in Paris, is using Cabanac’s tool to identify systematic reviews that cite retracted work, and to assess the impact the retracted studies had on the overall results.

However, a 2022 study¹¹ suggests that authors are often reluctant to update reviews, even when they are told the papers cite retracted work. Researchers e-mailed the authors of 88 systematic reviews that cited now-retracted studies in bone health by a Japanese fraudster, Yoshihiro Sato. Only 11 of the reviews were updated, the authors told Nature last year.

Retraction alerts

Authors aren’t routinely alerted if work cited in their past papers is withdrawn — although in recent years, paper-management tools for researchers such as Zotero and EndNote have incorporated Retraction Watch’s open database of retracted papers and have begun to flag papers that have been taken down. Cabanac thinks publishers might use tools like his to create similar alerts.

In 2016, researchers at the University of Oxford, UK, began developing a tool called RetractoBot, which automatically notifies authors by e-mail when a study that they have previously cited has been retracted. The software currently monitors 20,000 retracted papers and about 400,000 papers, published after 2000, that cite them. The team behind it is running a randomized trial to see whether papers flagged by RetractoBot are subsequently cited less than those not flagged by the tool, and will publish its results next year, says project lead Nicholas DeVito, a integrity researcher at Oxford.

The team has alerted more than 100,000 researchers so far. DeVito says that a minority of authors are annoyed about being contacted, but that others are grateful. “We are merely trying to provide a service to the community to reduce this practice from happening,” he says.

doi: https://doi.org/10.1038/d41586-024-02719-5

This story originally appeared on: Nature - Author:Richard Van Noorden