Mass-market scholarly communication

I attended the Workshop on Repositories sponsored by the NSF (US) and the JISC (UK). I apologize in advance for the length of this post, which is a follow-up. As I wrote it new aspects kept emerging and more memories of the discussion came back.

In his perceptive position paper for the workshop, Don Waters cites a fascinating paper by Harley et al. entitled "The Influence of Academic Values on Scholarly Publication and Communication Practices". I'd like to focus on two aspects of the Harley et al paper:

• They describe a split between "in-process" communication which is rapid, flexible, innovative and informal, and "archival" communication. The former is more important in establishing standing in a field, where the latter is more important in establishing standing in an institution.
• They suggest that "the quality of peer review may be declining" with "a growing tendency to rely on secondary measures", "difficult[y] for reviewers in standard fields to judge submissions from compound disciplines", "difficulty in finding reviewers who are qualified, neutral and objective in a fairly closed acacdemic community", "increasing reliance ... placed on the prestige of publication rather than ... actual content", and that "the proliferation of journals has resulted in the possibility of getting almost anything published somewhere" thus diluting "peer-reviewed" as a brand.
  

In retrospect, I believe Malcolm Read made the most important observation of the workshop when he warned about the coming generational change in the scholarly community, to a generation which has never known a world without Web-based research and collaboration tools. These warnings are particularly important because of the inevitable time lags in developing and deploying any results from the policy changes that the workshop's report might advocate.

Late in the workshop I channeled my step-daughter, who is now a Ph.D. student. Although I was trying to use her attitudes to illuminate the coming changes, in fact she is already too old to be greatly impacted by any results from the workshop. She was in high school as the Web was exploding. The target generation is now in high school, and their equivalent experience includes blogs and MySpace.

I'd like to try to connect these aspects to Malcolm's warnings and to the points I was trying to communicate by channeling my step-daughter. In my presentation I used as an example of "Web 2.0 scholarship" a post by Stuart Staniford, a computer scientist, to The Oil Drum blog, a forum for discussion of "peak oil" among a diverse group of industry professionals and interested outsiders, like Stuart. See comments and a follow-on post for involvement of industry insiders.

I now realize that I missed my own basic point, which is:

Blogs are bringing the tools of scholarly communication to the mass market, and with the leverage the mass market gives the technology, may well overwhelm the traditional forms.

Why is it that Stuart feels 2-3 times as productive doing "blog-science"? Based on my blog experience of reading (a lot) and writing (a little) I conjecture as follows:

• The process is much faster. A few hours to a few days to create a post, then a few hours of intensive review, then a day or two in which the importance of the reviewed work becomes evident as other blogs link to it. Stuart's comment came 9 hours into a process that accumulated 217 comments in 30 hours. Contrast this with the ponderous pace of traditional academic communication.
• The process is much more transparent. The entire history of the review is visible to everyone, in a citable and searchable form. Contrast this with the confidentiality-laden process of traditional scholarship.
• Priority is obvious. All contributions are time-stamped, so disputes can be resolved objectively and quickly. They're less likely to fester and give rise to suspicions that confidentiality has been violated.
• The process is meritocratic. Participation is open to all, not restricted to those chosen by mysterious processes that hide agendas. Participants may or may not be pseudonymous but their credibility is based on the visible record. Participants put their reputation on the line every time they post. The credibility of the whole blog depends on the credibility and frequency of other blogs linking to it - in other words the same measures applied to traditional journals, but in real time with transparency.
• Equally, the process is error-tolerant. Staniford says "recognition on all our parts that this kind of work will have more errors in any given piece of writing, and its the collaborative debate process that converges towards the truth." This tolerance is possible because the investment in each step is small, and corrections can be made quickly. Because the penalty for error is lower, participants can afford to take more creative risk.
• The process is both cooperative and competitive. Everyone is striving to improve their reputation by contributing. Of course, some contributions are negative, but the blog platforms and norms are evolving to cope with this inevitable downside of openness.
• Review can be both broad and deep. Staniford says "The ability for anyone in the world, with who knows what skill set and knowledge base, to suddenly show up ... is just an amazing thing". And the review is about the written text, not about the formal credentials of the reviewers.
  
• Good reviewing is visibly rewarded. Participants make their reputations not just by posting, but by commenting on posts. Its as easy to assess the quality of a participant reviews as to assess their authorship; both are visible in the public record.

Returning to the Harley et al. paper's observations, it is a commonplace that loyalty to employers is decreasing, with people expecting to move jobs frequently and often involuntarily. Investing in your own skills and success makes more sense than investing in the success of your (temporary) employer. Why would we be surprised that junior faculty and researchers are reluctant to put effort into institutional repositories for no visible benefit except to the institution? More generally, it is likely that as the mechanisms for establishing standing in the field diverge from those for establishing standing in the institution, investment will focus on standing in the field as being more portable, and more likely to be convertible into standing in their next host institution.

It is also very striking how many of the problems of scholarly communication are addressed by Staniford's blog-science:

• "the proliferation of journals has resulted in the possibility of getting almost anything published somewhere" - If scholarship is effectively self-published then attention focusses on tools for rating the quality of scholarship, which can be done transparently, rather than tools for preventing low-rated scholarship being published under the "peer-reviewed" brand. As the dam holding back the flood of junk leaks, the brand looses value, so investing in protecting it becomes less rewarding. Tools for rating scholarship, on the other hand, reward investment. They will be applied to both branded and non-branded material (cf. Google), and will thus expose the decreased value of the brand, leading to a virtuous circle.
• "increasing reliance ... placed on the prestige of publication rather than ... actual content" - Blog-style self-publishing redirects prestige from the channel to the author. Clearly, a post to a high-traffic blog such as Daily Kos (500,000 visits/day) can attract more attention, but this effect is lessened by the fact that it will compete with all the other posts to the site. In the end the citation index effect works, and quickly.
  
• "a growing tendency to rely on secondary measures" - If the primary measures of quality were credible, this wouldn't happen. The lack of transparency in the traditional process makes it difficult to regain credibility. The quality rating system for blogs is far from perfect, but it is transparent, it is amenable to automation, and there is an effective incentive system driving innovation and improvement for the mass market.
  
• "difficult[y] for reviewers in standard fields to judge submissions from compound disciplines" - This is only a problem because the average number of reviewers per item is small, so each needs to span most of the fields. If, as with blogs, there are many reviewers with transparent reputations, the need for an individual reviewer to span fields is much reduced.
  
• "difficulty in finding reviewers who are qualified, neutral and objective in a fairly closed acacdemic community" - This is only a problem because the process is opaque. Outsiders have to trust the reviewers; they cannot monitor their reviews. With a completely transparent, blog-like process it is taken for granted that many reviewers will have axes to grind, the process exists to mediate these conflicting interests in public.
  

Of the advantages I list above, I believe the most important is sheer speed. John Boyd, the influential military strategist, stressed the importance of accelerating the OODA (Observation, Orientation, Decision, Action) loop. Taking small, measurable steps quickly is vastly more productive than taking large steps slowly, especially when the value of the large step takes even longer to become evident.

Why did arXiv arise? It was a reaction to a process so slow as to make work inefficient. Successive young generations lack patience with slow processes; they will work around processes they see as too slow just as the arXiv pioneers did. Note that once arXiv became institutionalized, it ceased to evolve and is now in danger of loosing relevance as newer techologies with the leverage of the mass market overtake it. Scientists no longer really need arXiv; they can post on their personal web sites and Google does everything else (see Peter Suber), which reinforces my case that mass-market tools will predominate. The only mass-market tool missing is preservation of personal websites, which blog platforms increasingly provide. Almost nothing in the workshop was about speeding up the scholarly process, so almost everything we propose will probably get worked around and become irrelevant.

The second most important factor is error tolerance. The key to Silicon Valley's success is the willingness to fail fast, often and in public; the idea that learning from failure is more important than avoiding failure. Comments in the workshop about the need for every report to a funding agency to present a success illustrate the problem. If the funding agencies are incapable of hearing about failures they can't learn much.

What does all this mean for the workshop's influence on the future?

• Unless the institutions' and agencies' efforts are focussed on accelerating the OODA loop in scholarship, they will be ignored and worked-around by a coming generation notorious for its short attention span. No-one would claim that institutional repositories are a tool for accelerating scholarship; thus those workshop participants describing their success as at best "mixed" are on the right track. Clearly, making content at all scales more accessible to scholars and their automated tools is a way to accelerate the process. In this respect Peter Murray-Rust's difficulties in working around restrictions on automated access to content that is nominally on-line are worthy of particular attention.
• Academic institutions and funding agencies lack the resources, expertise and mission to compete head-on with mass market tools. Once the market niche has been captured, academics will use the mass market tools unless the productivity gains from specialized tools are substantial. Until recently, there were no mass-market tools for scholarly communication, but that's no longer true. In this case the mass-market tools are more productive that the specialized ones, not less. Institutions and agencies need to focus on ways to leverage these tools, not to deprecate their use and arm-twist scholars into specialized tools under institutional control.
• Insititutions and agencies need to learn from John Boyd and Silicon Valley themselves. Big changes which will deliver huge value but only in the long term are unlikely to be effective. Small steps that may deliver a small increment in value but will either succeed or fail quickly are the way to go.
• Key to effective change are the incentive and reward systems, since they close the OODA loop. The problem for institutions and agencies in this area is that the mass-market tools have very effective incentive and reward systems, based on measuring and monetizing usage. Pay attention to the way Google runs vast numbers of experiments every day, tweaking their systems slightly and observing the results on user's behavior. Their infrastructure for conducting these experiments is very sophisticated, because the rewards for success flows straight to the bottom line. The most important change institutions and agencies can make is to find ways to leverage the Web's existing reward systems by measuring and rewarding use of scholarly assets. Why does the academic structure regard the vast majority of accesses to the Sloan Digital Sky Survey as being an unintended, uninteresting by-product? Why don't we even know what's motivating these accesses? Why aren't we investing in increasing these accesses?

I tend to be right about the direction things are heading and very wrong about how fast they will get there. With that in mind, here's my prediction for the way future scholars will communicate. The entire process, from lab notebook to final publication, will use the same mass-market blog-like tools that everyone uses for everyday cooperation. Everything will be public, citable, searchable, accessible by automated scholarly tools, time-stamped and immutable. The big problem will not be preservation, because the mass-market blog-like platforms will treat the scholarly information as among the most valuable of their business assets. It will be more credible, and thus more used, and thus generate more income, than less refined content. The big problem will be a more advanced version of the problems currently plaguing blogs, such as spam, abusive behavior, and deliberate subversion. But, again, since the mass-market systems have these problems too, scholars will simply use the mass-market solutions.