In a correspondence addressed to PLoS Biology, entitled "An Incentive Solution to the Peer review Problem", Marc Hauser and Ernst Fehr propose that editors should punish slow reviewers. The proposed "incentive solution"? Deliberately delaying submissions from slow or bad reviewers! "You were two weeks late with your report? I am going to sit on your manuscript for four..." 
It is rather difficult to articulate what sort of violation to the editorial principles this would actually represent! Even if I am willing to accept that the proposal is half a joke and half a provocation, its purely coercive nature strikes me - in fact it even hurts. It is clear that when scientists accept to review a manuscript–and do so for free, when they are most of the time submerged by traveling, teaching and administrative duties– it is certainly not because editors are pointing a gun to their chest. There must be some positive motivations: could it be a sense of duty, scientific interest and intellectual stimulation to critically analyze a study, desire to stay updated on the latest advances in the field?
Having said that, it is true that it becomes incredibly frustrating when reviewers are awfully late, return a useless two-liner or worst never send any report at all. The simplest action is then to eliminate the person from the circuit and avoid using his/her services in the future. But this very measure makes the shortage of reviewers even more acute. So what can we do?
Beside extreme measures, like suppressing peer-reviewing altogether or in contrary increasing drastically the number of editorial rejections, one has to admit that it is not easy to come up with convincing suggestions for concrete measures, and even more so to find positive incentives.
Should reviewers be paid? How much and by whom? To give some numbers, in the 12 past months, a journal like the EMBO Journal has involved more than 2500 reviewers writing more than 4500 reports! Perhaps an iPod (or a week of vacation on an internet-free island) offered to the best top 10 reviewers would be feasible, but would this represent a strong incentive? Should there be an Academy Award to nominate the Best Reviewers?
Another idea is that acting as a referee on a regular basis for a journal should be highly regarded when hiring. Should journals issue official certificates to join to the CV? Along the same line, should journals more often advertise on their websites lists of regular referees? Again, is this a strong measure?
Altogether, these appear however rather modest incentives (except maybe the holiday on an island). One important problem might be the problem of time management. Even slow reviewers may in fact be motivated to do their job but they are late simply because they are overwhelmed and generally underestimate the time required for writing a decent report.Perhaps reviewers who proved to be slow in the past could be helped by being asked to provide a definite date and time of the day when they will do the job? At the level of education, young scientists would probably benefit from being well trained early on to read fast and render a critical assessment on papers under strong time pressure. See for example the forthcoming EMBO Young Investigator PhD course for an example of such training. Another alternative is to offer young scientists the possibility to spend some time in an editorial office to make them aware of the entire publication process and the crucial importance of good reviewers (see A short trial period in science publishing on nodalpoint).
More creative (and positive) suggestions are clearly required if we want to avoid transforming the scientific arena into a stereotypical confrontation between the Good Author, the Bad Reviewer and the Ugly ... Editor ;-)
Comments (5)
Posted by Pedro Beltrao | April 20, 2007 8:10 PM
One possibility would be have some kind of central repository of aggregated referee information. Ideally this should be common to all journals. This database could show aggregated scores of who referee papers for what journal during a period of 6 months or a year. This should also include information on the average delay.
I would not be surprised if the availability of the information, by itself, would provide a strong incentive for refereeing without delay.
Many people include this information on their CV, so it must be prestigious to referee for top journals.
This simple database would provide with a quick reward system (refereed X times for journals Y,Z,etc) and help journals find referees.
Posted by Ricardo Almeida | April 22, 2007 12:34 PM
To some extent, early access to research data is the main motivation for peer reviewers.
However, prompt referees that evaluate papers and return their reports quickly already are subsequently more frequently sought out by the journals. While this is certainly a welcomed fact by more junior faculty, established researchers can easily get swamped by requests from several journals. Since its not a transparent process, I can see how many get demotivated from performing their refereeing roles efficiently.
I've discussed this matter with a few PIs in the past that mentioned that refereeing papers at their career stage is more of a burden than anything else. Some try to address this by rejecting increasingly more manuscripts. Others prefer not to be cut out of the loop and employ the help of members of their lab. Alternatively, like you mentioned in your post, some delay their reply for a couple of weeks (or more) to stop the flow of requests.
Monetary gratification will not constitute an even motivation for reviewers of distinct stages of their career. Giving publication privileges to well-behaved referees is also not ethical, since quality of science and not reputation should determine access to publication. I tend to like more the idea of journals issuing certificates to their referees mentioning their contribution. Refereeing manuscripts for top journals is indeed prestigious so acknowledging a good referee might go a long way in motivating others.
Posted by Maxine | April 24, 2007 7:58 PM
A common line of argument is that peer-reviewers and authors and editors are all the same people: everyone wins as they are all part of the publication cycle, albeit different parts of it at different times.
I'd also suggest that peer-reviewing is a great learning experience for a young student or postdoc: nature has acquired many good peer-reviewers in this way.
I remember Prof Andrew Huxley once suggested a sliding payment scale for peer-reviewers, in which they get paid a relatively large sum for delivering the report within 24 hours, but the sum progressively decreased with time. He did not say whether the slow peer-reviewer would end up owing the journal or the author money, though.
Posted by Jakob Christensen-Dalsgaard | May 10, 2007 2:43 PM
As an associate editor dealing with referees it is clear to me that what is needed is not any new punishments that will make it even harder to get referees.
Instead, I think an idea could be to have an international and institutionalized merit badge for referees. The merit badge should be a metric reflecting the amount of work put into refereeing an MS, for example depending on the number of pages in the MS and the number of pages in the referee report, and could also reflect any delays on the referees part, but the merit would always be positive, if a report is delivered. The sum of merits would be quotable on a CV and would, I think, be an incentive in itself for most of us (vain as we are)
I am sure that ISI or other bibliometricians could come up with a number. It would then be necessary that journals subscribed to the merit service, but the benefit to these journals -ease of getting referees and maintaining the important peer review -may be a sufficient incentive for them.
Posted by Meister Eckhardt | May 14, 2007 9:10 PM
What's lost here is the fact that the increasing need for reviewers is fueled primarily by the mindless, ever-increasing publish-or-perish mentality in academia.
As numerous studies have shown, most papers aren't cited and most end up in that dark section of the library that you passed by a few times as a grad student.
I'm in favor of rewarding referees, but it won't change the amount of refuse we have to wade through in order to find significant research.