There’s been quite a bit of recent criticism of the peer review process, notably in The Scientist and by Michael Hochberg in his editorial ‘The tragedy of the reviewer commons‘ in Ecology Letters. As someone who sees thousands of decisions every year, I actually think it works quite well, and my opinion of it has certainly improved since I became managing editor. Unfortunately, most of the discussion of peer review is based on personal impressions and this inevitably means that the (rare) bad decisions get more more press than the everyday good ones. Furthermore, most researchers only see 5-10 decisions per year, either as reviewers or authors, and this is clearly too few for drawing informed opinions.
What’s really lacking are some data – and hence I hope to be doing some analysis for the 2011 editorial. I’ve picked out three perceptions of the review process that I’d like to examine:
1) more and more people are refusing to review.
2) that there is a large and growing community of ‘freeloaders’ who only submit papers whilst doing no reviews of others’ papers.
3) That personal biases/anonymous reviewing/negligent editors mean that papers routinely receive decisions disproportionate to their ‘value’ to science.
Testing these will probably be quite labour intensive, but I’m certainly interested to hear other ideas that we could examine.