‘Substandard and unworthy’: why it’s time to banish bad-mannered reviews
The publisher of Physics World and other journals is marking Peer Review Week 2024 by calling for more courtesy in the process
Even Cathy Foley, Australia’s chief scientist, has encountered unhelpful peer-review comments on her work.Credit: Fairfax Media Archives/Getty
Electromagnetics researcher Akhlesh Lakhtakia is head of a leading US department of engineering science at Pennsylvania State University in University Park, the author of more than 840 journal articles and a fellow of 9 learned societies. But in 1988, when he was an assistant professor, a peer reviewer said of a paper he had submitted for publication: “This is rubbish. Obviously the author or authors had no EM [electromagnetics] training nor physical intuition.”
Peer review, which has for centuries been the standard tool to determine an academic paper’s suitability for publication, is known to be flawed1. Now, one of its major weaknesses, sheer bad manners on the part of the reviewer, has been highlighted in a YouTube video from IOP Publishing (IOPP), headquartered in Bristol, UK — a society-owned publisher of more than 90 journals.
Released to mark Peer Review Week, which this year runs from 23 to 27 September, the one-minute film features four scientists who hold placards showing the rude, inappropriate or irrelevant reviews that they received when they were early-career researchers. At the same time, an overlay lists their stellar achievements since. Activities taking place during this year’s Peer Review Week, involving more than 35 organizations around the world, will focus on innovation and technology, including artificial intelligence (AI) and how it can be used to automate administrative tasks in the peer-review process.
A study published in 2019 revealed that six in ten researchers in the international science, technology, engineering and mathematics (STEM) community have received at least one unprofessional review — and of those, seven in ten have received several2.
Such reviews don’t always cause long-term damage. Lakhtakia tells Nature that he felt “outraged for a few weeks” after he was refused the chance to submit a revised manuscript that would refute the reviewer’s criticisms. “Then,” he adds, “I wrote a monograph on the broader research topic that led to my elevation to the fellowship of a major learned society in 1992.”
Physicist Cathy Foley, Australia’s chief scientist, who has published 112 refereed papers in international journals, recalls a reviewer’s unhelpful comments on a manuscript that she and a younger colleague had “put our heart and soul into drafting” in 2009, when she was a research-programme leader.
“It was written in a very personal way that suggested our team was substandard and we are not worthy of being researchers,” she says. “It took a lot of discussion and coaching to help us see beyond the nasty comments and look for the research advice. Focusing on that enabled us to revise the paper and move on.”
But not everyone shrugs off insulting remarks. Their impact on self-confidence, productivity and career trajectories can be significant, says Laura Feetham-Walker, IOPP’s reviewer-engagement manager, who led the video project. Here, she explains why mean-spirited peer-review comments should be challenged, and why the science community needs to discuss this commonplace humiliation of its younger members.
When did you realize unprofessional reviewer comments were an issue?
I first heard academics discussing this problem shortly after I joined IOPP in 2020 as its first reviewer-engagement manager, while I was running training workshops for early-career reviewers. Before that, I had worked at BMJ Group and The Lancet. I see my role as engaging with reviewers and supporting them in submitting excellent, constructive reports.
Akhlesh Lakhtakia, who leads a US department of engineering science, holds a piece of paper showing a rude reviewer comment that he received early in his career.Credit: IOP Publishing
It became clear at the workshops how many researchers have been affected by this issue — including the senior reviewers, who spoke up about the rude comments they’d received early in their careers.
Is there a fine line between useful criticism and rudeness?
Not at all. The two are very different. You can have a critical, even very negative, review that is not at all problematic. In the 2019 study, which received feedback from 1,106 STEM professionals who had been first authors on manuscripts submitted to peer-reviewed journals, the definition was clear-cut: unprofessional peer review is that which is unethical, irrelevant, mean-spirited or cruel and lacking constructive criticism2.
Should academics learn to be a bit more thick-skinned?
Maybe. Lakhtakia advises junior researchers to “dispassionately evaluate criticism and then proceed accordingly”. But people from groups that are historically under-represented in STEM — women, non-binary people and those from ethnic minorities — are most likely to report that their confidence as a scientist has been undermined by rude reviews2. They are also the groups most likely to report long-term setbacks in their productivity and career advancement. It’s easy to see how that might happen: if you’ve already got imposter syndrome, and your confidence is low, a mean comment might really get to you. This matters. The STEM community can’t afford to allow unprofessional peer review to disempower effective researchers or lead to important work going unpublished.
What can be done?
It’s an editor’s job to sift out unprofessional comments in reviews. IOPP policy, in line with guidance from the Committee on Publication Ethics, a non-profit organization in Eastleigh, UK, is for the editor to ‘rescind’ problematic reviews and ask the reviewer to revise and resubmit them. If the reviewer declines, then the editor might make minor amendments to remove any problematic comments. But editors are under time pressure and deal with many peer-review reports every day, so some inappropriate comments slip through the net. We need to define unprofessional reviewing, to make it easier to track and to filter such comments out.
We also need to acknowledge that most reviewers do a brilliant job in difficult circumstances and will welcome support to improve their skill and confidence as a reviewer. IOPP now offers a free peer-review training course, available to everyone. Those who complete it earn a certificate, and early-career reviewers can include it on their CV.
But most of all, scientists need to talk more about rude reviews. That’s why we made the video.
Do anonymous reviews encourage rudeness?
Perhaps, yes. There is some evidence of an ‘online disinhibition effect’ — or a lack of restraint that people feel with they communicate over the Internet — so rudeness might have increased as peer review has moved online, although more research is needed in this area.
What about open peer review? Is that helping to end rude reviews?
It has had a major impact, undoubtedly. At IOPP, in February 2022, we introduced a version of open peer review (OPR), called transparent peer review (TPR), throughout our open-access journals. TPR shows the complete peer-review process, from initial review to final decision, with the reviewer reports published alongside accepted articles. It requires both authors and reviewers to opt in. Anecdotally, senior staff on TPR journals say they have never seen rude or unprofessional comments in TPR reviewer reports. But TPR would have to be mandatory for reviewers to completely eliminate rudeness.
Is there a downside to TPR?
No. But uptake has been modest. Only about half of our authors choose to make their reviewer reports visible. The number of STEM journals that use TPR or OPR is relatively low, so often it isn’t an option for authors and reviewers.
In 2021, we introduced double-anonymous (DA) peer review. We were the first physics publisher to adopt this approach across our entire portfolio. Under DA, both authors and reviewers are anonymous. The main aim is to reduce bias in science publishing with respect to gender, race, country of origin and affiliation, the latter reflecting the little-acknowledged risk of ‘prestige bias’, or the favouring of work by scientists associated with elite institutions.
We were also the first society publisher to combine DA and TPR throughout our open-access journals. The reviewer and author are anonymous throughout the review process, but the reviewers’ names and full reports are published with the final research.
We know DA peer review works. Women and non-binary people who choose to submit through DA are 11% more likely to have their papers accepted. In particular, sub. mitting authors in Australasia (9.8%) and Africa (7.7%) are more likely to have their manuscript accepted if it is anonymized, and only authors in western Europe benefit from their papers not being anonymized. We also know, again anecdotally, that DA reduces the frequency of rude comments.
Why do you think that is?
Remember that reviewers are always ‘seen’ by their editors. Our internal editors rate every review on a scale of 1 to 5. Those who get a 5 achieve IOP trusted reviewer status. A review containing unprofessional comments will almost always get a score of 1, or occasionally 2 if the comments are borderline unprofessional.
Last November, we introduced reviewer feedback, which allows reviewers to request their review’s score direct from the editorial team. More than 23,000 reviewers have already opted to receive this feedback. We also offer an overview of how to critique a scientific manuscript, which includes examples of reports, for papers from eight fields of physics, that were rated 1, 3 or 5.
Reviewers’ comments can also be seen by other reviewers during co-reviewing, whereby a reviewer formally invites a colleague to collaborate with them. More than eight in ten reviewers who have participated in co-review have told us that they find it useful or very useful.
The signs all indicate that these mechanisms work together to improve reviewers’ understanding of how to do good peer review — and that will benefit science at large.
doi: https://doi.org/10.1038/d41586-024-02943-z
A spokeperson for Springer Nature says that Nature introduced DA and TPR with DA across all its journals in 2015. BMC Series journals moved from OPR to TPR in 2020. Springer Nature’s peer-review policy includes guidance to protect against unprofessional and or rude reviews, with a guidance statement reminding reviews to keep feedback professional, constructive and courteous in Snapp, its new submission system.
This story originally appeared on: Nature - Author:Jane Feinmann
