³ÉÈË´óƬ

What motivated this study?

David Maslove: Working as an editor for Critical Care Medicine, I saw a huge increase in the volume of research manuscripts submitted to us during the COVID-19 pandemic. Finding reviewers and providing feedback to authors in a timely manner was a challenge. We needed to get important research out to clinicians and policy makers, but the peer review process just wasn’t fast enough. There has been much discussion about whether reviewers should be paid, and we decided to test if this would have any impact on reviewing timelines and the peer review process overall.

Peer review is, traditionally, volunteer work – why did you decide to use monetary incentives?

Christopher Cotton: Some journals in the fields of business and economics have started paying their referees. The main motivation was that review times were usually pretty long, and there is some evidence that the incentives have been helping reduce these times. In biomedicine and health research, as far as we know, this has not yet been tested.

How did you test it?

David Maslove: We partnered with Critical Care Medicine to conduct an experiment in which reviewers were randomly split into two groups: a control group that would receive no incentives, and a test group that would receive a payment of $250 (USD) to review a research paper.

Did you see any impact of the incentives on the peer review process?

Christopher Cotton: Paying reviewers had positive impacts on the review process. However, these impacts tended to be relatively modest in magnitude. For example, review times in the test group were shorter, but not much shorter, compared to the control group (11 days compared to 12 days on average).

David Maslove: We also saw that the cash incentive resulted in an increase in the share of people who accepted to review requests from the journal (49.8 per cent in the test group, versus 42.2 per cent in the control group). But we didn't see that the payments made any difference in the quality of the reviews.

What conclusions can you draw from those results?

Christopher Cotton: Our results suggest that cash incentives aren’t a perfect solution to the delayed editorial process that we see at many journals. Paying reviewers can quickly add up to large financial costs at journals. The impact of paying referees that we found likely are not big enough to justify the additional costs to most journals. But that doesn’t mean that this wouldn’t be a potentially good strategy to use in specific circumstances, such as the height of a pandemic or a crisis where we really do need to speed things up.

What other changes in the peer review process could have a positive impact?

Christopher Cotton: A variety of factors could be causing delays at journals. For example, the time it takes editors to make decisions on papers, or the administrative processes of getting a manuscript ready for publication, or even the sheer number of revisions that papers often have to go through at journals before they're accepted.

What could be the next steps in studying peer review and its processes?

David Maslove: The effects of each intervention will probably be different depending on the journal where the experiment is done, so it would be interesting to see what happens in higher impact journals versus lower impact journals, or specialty journals versus general journals, etc. Peer review is notoriously difficult to study. Although there is a lot of conversation about it, at least in biomedical and clinical literature, there's not a lot of experimentation. So, for me, one of the positive takeaways of this study is that this can actually be done! Instead of musing about what-ifs, the research community should look at injecting some evidence into the debate about different aspects of peer review.