Crowd-sourced Peer Review

At Ars Technica, Chris Lee's Journal tries crowdsourcing peer reviews, sees excellent results takes off from a column at Nature by a journal editor, Benjamin List, entitled Crowd-based peer review can be good and fast. List and his assistant Denis Höfler have come up with a pre-publication peer-review process that, while retaining what they see as its advantages, has some of the attributes of post-publication review as practiced, for example, by Faculty of 1000. See also here. Below the fold, some commentary.

List's column describes an experiment in which he compared conventional pre-publication reviewing with what he and Höfler call "selected crowd-sourced peer review" of the same papers:

I am not proposing what is sometimes referred to as crowdsourced reviewing, in which anyone can comment on an openly posted manuscript. I believe that anonymous feedback is more candid, and that confidential submissions give authors space to decide how to revise and publish their work. I envisioned instead a protected platform whereby many expert reviewers could read and comment on submissions, as well as on fellow reviewers’ comments. This, I reasoned, would lead to faster, more-informed editorial decisions.

The experiment worked like this. They:

recruited just over 100 highly qualified referees, mostly suggested by our editorial board. We worked with an IT start-up company to create a closed online forum and sought authors’ permission to have their submissions assessed in this way. Conventional peer reviewers evaluated the same manuscripts in parallel. After an editorial decision was made, authors received reports both from the crowd discussion and from the conventional reviewers. ... we put up two manuscripts simultaneously and gave the crowd 72 hours to respond.

The results were encouraging:

Each paper received dozens of comments that our editors considered informative. Taken together, responses from the crowd showed at least as much attention to fine details, including supporting information outside the main article, as did those from conventional reviewers. ... So far, we have tried crowd reviewing with ten manuscripts. In all cases, the response was more than enough to enable a fair and rapid editorial decision. Compared with our control experiments, we found that the crowd was much faster (days versus months), and collectively provided more-comprehensive feedback.

The authors liked the new process. They plan to switch their journal to it, tweaking it as they gain experience.

As I've been saying since the first post to this blog more than a decade ago, conventional pre-publication review is long overdue for a revolution. Chris Lee's Ars Technica piece is well worth reading. He describes List and Höfler's experiment in the context of a broader discussion of the problems of conventional per-publication peer review:

The utter randomness of peer review is frustrating for everyone. Papers get delayed, editors get frustrated, the responsible reviewers get overloaded. Even when everyone is trying their best, any set of three reviewers can disagree so wholeheartedly about your work that the editor has to actually think about a decision—something no editor ever wants to be faced with.

But, more interestingly, Lee looks at the peer-review process from a signal-processing viewpoint:

I'd suggest that there is a physical analog to traditional peer review, called noise. Noise is not just a constant background that must be overcome. Noise is also generated by the very process that creates a signal. The difference is how the amplitude of noise grows compared to the amplitude of signal. For very low-amplitude signals, all you measure is noise, while for very high-intensity signals, the noise is vanishingly small compared to the signal, even though it's huge compared to the noise of the low-amplitude signal. Our esteemed peers, I would argue, are somewhat random in their response, but weighted toward objectivity. Using this inappropriate physics model, a review conducted by four reviewers can be expected (on average) to contain two responses that are, basically, noise. By contrast, a review by 100 reviewers may only have 30 responses that are noise.

It might seem that this simply multiplies the work demanded from already-overloaded reviewers. But Lee ingeniously and credibly argues that this isn't the case. This argument is the best part of the piece, and I urge you to stop reading me and start reading Lee.

Spoiler:

For those of you too lazy to take my advice, here is the Cliff Notes version. A larger number of reviewers brings a broader range of expertise to the review. Since they can all see each other's contributions, each reviewer can focus on the part that matches their specific expertise, and thus avoid the need to cover areas with which they are less familiar, and which thus involve more work in validating references, using less familiar analysis tools, etc.