Is peer review broken?

There’s been quite a bit of recent criticism of the peer review process, notably in The Scientist and by Michael Hochberg in his editorial ‘The tragedy of the reviewer commons‘ in Ecology Letters. As someone who sees thousands of decisions every year, I actually think it works quite well, and my opinion of it has certainly improved since I became managing editor. Unfortunately, most of the discussion of peer review is based on personal impressions and this inevitably means that the (rare) bad decisions get more more press than the everyday good ones. Furthermore, most researchers only see 5-10 decisions per year, either as reviewers or authors, and this is clearly too few for drawing informed opinions.

What’s really lacking are some data – and hence I hope to be doing some analysis for the 2011 editorial. I’ve picked out three perceptions of the review process that I’d like to examine:

1) more and more people are refusing to review.

2) that there is a large and growing community of ‘freeloaders’ who only submit papers whilst doing no reviews of others’ papers.

3) That personal biases/anonymous reviewing/negligent editors mean that papers routinely receive decisions disproportionate to their ‘value’ to science.

Testing these will probably be quite labour intensive, but I’m certainly interested to hear other ideas that we could examine.

RedditDiggMendeleyPocketShare and Enjoy

About Tim Vines

I'm the managing editor of Molecular Ecology and Molecular Ecology Resources.
This entry was posted in peer review and tagged . Bookmark the permalink.
  • Dilara Ally

    I would be interested in knowing whether there is a relationship between academic career stage and those who refuse to review.

    What is the metric used to determine someone is a freeloader? A ratio of number of submissions as author/number of requests for review? Again is there a relationship between career stage and the metric used to assess ‘cheating.’

    And finally, how do you independent of the peer review process decide the “value” of someone’s work in science?

    Just some thoughts!

  • Tim Vines

    With respect to career stage I could group people by title (Mr/Ms vs Dr vs Prof) but there would be quite a lot of noise arising from the different uses of Prof in Europe vs the US, to the extent that I would worry about the validity of the results. We don’t really have any other data on age or career stage that we could use a proxy.

    We’ve honed the questions on whether or not people do their fair share of reviewing:

    1) what is the relationship between number of submissions and reviews invited/agreed/completed? This should be between 2 and 3 for most people, as each paper gets 2 – 3 reviews.

    2) how much variance is there around a plot of number of reviews completed against number of submissions? Do most researchers do c 2.5 reviews per submission or is there a group with <1 and a group with >3? The latter scenario would imply that some researchers aren’t pulling their reviewer weight.

    One obvious criticism is that this is just ‘submissions to Mol Ecol’ and ‘reviews for Mol Ecol’, but I would counter that a) the field of molecular ecology is somewhat discrete from other parts of ecology/evolution, and hence we are drawing from a finite set of researchers and b) this criticism would apply no matter how many journals you consider- each has its ‘core’ authors and reviewers and its ‘periphery’, who will contribute less, such that adding more journals won’t help.

    The crude measure of ‘value’ I was planning to use is citations per month- a paper would be given a score based on the reviewers’ enthusiasm and this would be compared to how useful the community as a whole found the paper. Papers would be divided by whether they were accepted for Mol Ecol or rejected outright, which means we will need to find papers rejected by us that subsequently appeared elsewhere.

    We can’t really use our Manuscript Central data for this, as only the papers accepted in 2008 have been around long enough to accumulate many citations, so we’ll have to go back to the records from when Harry was the ME (e.g. 2005-2007). Still, I’m looking forward to seeing the results.