Below the fold is an edited version of a talk I prepared for some discussion of the future of research communication at Stanford, in which I build on my posts on What's Wrong With Research Communication? and What Problems Does Open Access Solve? to argue that the top priority for reform should be improving peer review.
What distinguishes academic publishing from other kinds of publishing on the Web? Peer review. But despite the changes in other aspects of academic publishing there has been little change in peer review. This is probably because to most academics having their work successfully peer reviewed is a life-and-death matter, so changes to the system appear very threatening.
I'm an engineer. Peer reviewed publishing is important to engineers, but it is less important than in other fields. It is nice that the LOCKSS Program is the only digital preservation program to have spawned prize-winning peer reviewed papers, but it is far more important that the LOCKSS system is in production use at libraries world-wide, and that we are the only financially self-sustaining digital preservation program, having been cash flow positive for 5 years with no soft money.
Thus I'm in a better position than most to ask the fundamental question, what is peer review for? It is estimated to consume about $3B a year in unpaid work by academics. This is double the profits of Elsevier, Springer and Wiley combined, so it is likely that, were reviewers to be paid for their work, the entire industry would be economically unsustainable. But what do we get for this enormous expenditure? Here I should acknowledge a debt to the fine work of the UK House of Commons Science and Technology Committee, on which some of the following is based.
The purpose of peer review used to be clear, to prevent bad science being published. That is no longer possible; everything will be published, the only question now is where. For example, PLoS ONE publishes every submission that meets its criteria of technical soundness. Nevertheless, over 40% of the papers it rejects as unsound are published in other peer reviewed journals.
So, is the purpose of peer review to ensure that journals maintain their quality standards, so that where a paper is published indicates its quality? If so, peer review is failing miserably. The graphs in this paper show that the dispersion in article quality, as measured by citation count, among the articles in a journal is so great that the journal conveys no information as to the article quality.
Is the purpose to improve the article's presentation of its science? Why would one imagine that peer scientists would be any better at English or graphic design than the authors?
Or is it to improve the scientific content of the article? If so, once again peer review is failing miserably. It has repeatedly proven to be incapable of detecting not merely rampant fraud, but even major errors.
Why is the peer review system such poor value for $3B/yr? Clearly, because the entire publishing system is designed to exploit the reviewers. They are not paid, they are not credited, they cannot develop a reputation for excellence, they are not given tools to assist their work, they are not given access to the data, software, materials, experimental setups and so on to be able to check the work. And in many cases they have severe, undeclared conflicts of interest hidden behind reviewer anonymity. Which, in any case, is mostly illusory. Most fields are now so specialized that the pool of reviewers is small enough that reviewer anonymity is ineffective. Most fields now have such good informal communication that blinding papers for review is also ineffective.
There have been experiments in moving from pre-publication to post-publication review. PLoS ONE's restriction of pre-publication review to establishing technical soundness, leaving quality to be determined by post-publication review, has been very successful in one sense, but it has not really transformed the system. Other experiments have mostly been failures. This is likely because of their tentative nature, and lack of a clear vision as to what reviewing is for.
Any attempt to reform the system of research communication should start by answering the question of what reviewing is for. The primary goal of review should be to improve the quality of the research, rather than to improve the quality of the publication. Given that goal, we are faced with the following questions:
- Who should review?
- How should they review?
- What should the reward for reviewing be?
How? Just as reviewers' opinions as to the quality of an article vary, so will opinions as to the constructive nature of reviews. It won't be possible to prevent non-constructive comments being published, so the nature of comments will also have to be determined post-publication. This is a problem that many existing web-based communities, such as Reddit, Stack Overflow and Slashdot, have already solved using reputation systems linked to real or persistently pseudonymous identities. Possible reviewing system architectures are:
- Per-publishing platform systems (c.f. Slashdot comments) each with its own reputation system. Diverse reputation systems are not likely to be effective in enhancing careers.
- Publisher-independent commenting and reputation systems (c.f. Disqus comments). In practice there would be several competing systems, which would hamper their effectiveness.
- A general Web annotation system with inter-operable reputation infrastructure allowing multiple reputation systems to exchange data (c.f. Hypothes.is). A prototype of the annotation part of such a system is available here.
The content to be reviewed will be much more mutable and executable than currently (c.f. myexperiment). The advent of HTML5 means the Web is transitioning from a document model to a programming environment reliant on Web services. For post-publication review to be effective in improving the science:
- Reviewers must have access to the content, the code and the data.
- The content, the code and the data must evolve in response to the reviews.
- Reviews that caused evolution must redound to the credit of the reviewers.
The developers of successful on-line communities such as Reddit all say that the route to success is to give the community the tools they need to create the system that matches their needs, rather than design and build the system you think they need. Supply the infrastructure, let the community define the experience.