Saturday, April 21, 2007

Mass-market scholarly communication

I attended the Workshop on Repositories sponsored by the NSF (US) and the JISC (UK). I apologize in advance for the length of this post, which is a follow-up. As I wrote it new aspects kept emerging and more memories of the discussion came back.

In his perceptive position paper for the workshop, Don Waters cites a fascinating paper by Harley et al. entitled "The Influence of Academic Values on Scholarly Publication and Communication Practices". I'd like to focus on two aspects of the Harley et al paper:
  • They describe a split between "in-process" communication which is rapid, flexible, innovative and informal, and "archival" communication. The former is more important in establishing standing in a field, where the latter is more important in establishing standing in an institution.
  • They suggest that "the quality of peer review may be declining" with "a growing tendency to rely on secondary measures", "difficult[y] for reviewers in standard fields to judge submissions from compound disciplines", "difficulty in finding reviewers who are qualified, neutral and objective in a fairly closed acacdemic community", "increasing reliance ... placed on the prestige of publication rather than ... actual content", and that "the proliferation of journals has resulted in the possibility of getting almost anything published somewhere" thus diluting "peer-reviewed" as a brand.


In retrospect, I believe Malcolm Read made the most important observation of the workshop when he warned about the coming generational change in the scholarly community, to a generation which has never known a world without Web-based research and collaboration tools. These warnings are particularly important because of the inevitable time lags in developing and deploying any results from the policy changes that the workshop's report might advocate.

Late in the workshop I channeled my step-daughter, who is now a Ph.D. student. Although I was trying to use her attitudes to illuminate the coming changes, in fact she is already too old to be greatly impacted by any results from the workshop. She was in high school as the Web was exploding. The target generation is now in high school, and their equivalent experience includes blogs and MySpace.

I'd like to try to connect these aspects to Malcolm's warnings and to the points I was trying to communicate by channeling my step-daughter. In my presentation I used as an example of "Web 2.0 scholarship" a post by Stuart Staniford, a computer scientist, to The Oil Drum blog, a forum for discussion of "peak oil" among a diverse group of industry professionals and interested outsiders, like Stuart. See comments and a follow-on post for involvement of industry insiders.

I now realize that I missed my own basic point, which is:

Blogs are bringing the tools of scholarly communication to the mass market, and with the leverage the mass market gives the technology, may well overwhelm the traditional forms.

Why is it that Stuart feels 2-3 times as productive doing "blog-science"? Based on my blog experience of reading (a lot) and writing (a little) I conjecture as follows:
  • The process is much faster. A few hours to a few days to create a post, then a few hours of intensive review, then a day or two in which the importance of the reviewed work becomes evident as other blogs link to it. Stuart's comment came 9 hours into a process that accumulated 217 comments in 30 hours. Contrast this with the ponderous pace of traditional academic communication.
  • The process is much more transparent. The entire history of the review is visible to everyone, in a citable and searchable form. Contrast this with the confidentiality-laden process of traditional scholarship.
  • Priority is obvious. All contributions are time-stamped, so disputes can be resolved objectively and quickly. They're less likely to fester and give rise to suspicions that confidentiality has been violated.
  • The process is meritocratic. Participation is open to all, not restricted to those chosen by mysterious processes that hide agendas. Participants may or may not be pseudonymous but their credibility is based on the visible record. Participants put their reputation on the line every time they post. The credibility of the whole blog depends on the credibility and frequency of other blogs linking to it - in other words the same measures applied to traditional journals, but in real time with transparency.
  • Equally, the process is error-tolerant. Staniford says "recognition on all our parts that this kind of work will have more errors in any given piece of writing, and its the collaborative debate process that converges towards the truth." This tolerance is possible because the investment in each step is small, and corrections can be made quickly. Because the penalty for error is lower, participants can afford to take more creative risk.
  • The process is both cooperative and competitive. Everyone is striving to improve their reputation by contributing. Of course, some contributions are negative, but the blog platforms and norms are evolving to cope with this inevitable downside of openness.
  • Review can be both broad and deep. Staniford says "The ability for anyone in the world, with who knows what skill set and knowledge base, to suddenly show up ... is just an amazing thing". And the review is about the written text, not about the formal credentials of the reviewers.
  • Good reviewing is visibly rewarded. Participants make their reputations not just by posting, but by commenting on posts. Its as easy to assess the quality of a participant reviews as to assess their authorship; both are visible in the public record.


Returning to the Harley et al. paper's observations, it is a commonplace that loyalty to employers is decreasing, with people expecting to move jobs frequently and often involuntarily. Investing in your own skills and success makes more sense than investing in the success of your (temporary) employer. Why would we be surprised that junior faculty and researchers are reluctant to put effort into institutional repositories for no visible benefit except to the institution? More generally, it is likely that as the mechanisms for establishing standing in the field diverge from those for establishing standing in the institution, investment will focus on standing in the field as being more portable, and more likely to be convertible into standing in their next host institution.

It is also very striking how many of the problems of scholarly communication are addressed by Staniford's blog-science:

  • "the proliferation of journals has resulted in the possibility of getting almost anything published somewhere" - If scholarship is effectively self-published then attention focusses on tools for rating the quality of scholarship, which can be done transparently, rather than tools for preventing low-rated scholarship being published under the "peer-reviewed" brand. As the dam holding back the flood of junk leaks, the brand looses value, so investing in protecting it becomes less rewarding. Tools for rating scholarship, on the other hand, reward investment. They will be applied to both branded and non-branded material (cf. Google), and will thus expose the decreased value of the brand, leading to a virtuous circle.
  • "increasing reliance ... placed on the prestige of publication rather than ... actual content" - Blog-style self-publishing redirects prestige from the channel to the author. Clearly, a post to a high-traffic blog such as Daily Kos (500,000 visits/day) can attract more attention, but this effect is lessened by the fact that it will compete with all the other posts to the site. In the end the citation index effect works, and quickly.
  • "a growing tendency to rely on secondary measures" - If the primary measures of quality were credible, this wouldn't happen. The lack of transparency in the traditional process makes it difficult to regain credibility. The quality rating system for blogs is far from perfect, but it is transparent, it is amenable to automation, and there is an effective incentive system driving innovation and improvement for the mass market.
  • "difficult[y] for reviewers in standard fields to judge submissions from compound disciplines" - This is only a problem because the average number of reviewers per item is small, so each needs to span most of the fields. If, as with blogs, there are many reviewers with transparent reputations, the need for an individual reviewer to span fields is much reduced.
  • "difficulty in finding reviewers who are qualified, neutral and objective in a fairly closed acacdemic community" - This is only a problem because the process is opaque. Outsiders have to trust the reviewers; they cannot monitor their reviews. With a completely transparent, blog-like process it is taken for granted that many reviewers will have axes to grind, the process exists to mediate these conflicting interests in public.

Of the advantages I list above, I believe the most important is sheer speed. John Boyd, the influential military strategist, stressed the importance of accelerating the OODA (Observation, Orientation, Decision, Action) loop. Taking small, measurable steps quickly is vastly more productive than taking large steps slowly, especially when the value of the large step takes even longer to become evident.

Why did arXiv arise? It was a reaction to a process so slow as to make work inefficient. Successive young generations lack patience with slow processes; they will work around processes they see as too slow just as the arXiv pioneers did. Note that once arXiv became institutionalized, it ceased to evolve and is now in danger of loosing relevance as newer techologies with the leverage of the mass market overtake it. Scientists no longer really need arXiv; they can post on their personal web sites and Google does everything else (see Peter Suber), which reinforces my case that mass-market tools will predominate. The only mass-market tool missing is preservation of personal websites, which blog platforms increasingly provide. Almost nothing in the workshop was about speeding up the scholarly process, so almost everything we propose will probably get worked around and become irrelevant.

The second most important factor is error tolerance. The key to Silicon Valley's success is the willingness to fail fast, often and in public; the idea that learning from failure is more important than avoiding failure. Comments in the workshop about the need for every report to a funding agency to present a success illustrate the problem. If the funding agencies are incapable of hearing about failures they can't learn much.

What does all this mean for the workshop's influence on the future?

  • Unless the institutions' and agencies' efforts are focussed on accelerating the OODA loop in scholarship, they will be ignored and worked-around by a coming generation notorious for its short attention span. No-one would claim that institutional repositories are a tool for accelerating scholarship; thus those workshop participants describing their success as at best "mixed" are on the right track. Clearly, making content at all scales more accessible to scholars and their automated tools is a way to accelerate the process. In this respect Peter Murray-Rust's difficulties in working around restrictions on automated access to content that is nominally on-line are worthy of particular attention.
  • Academic institutions and funding agencies lack the resources, expertise and mission to compete head-on with mass market tools. Once the market niche has been captured, academics will use the mass market tools unless the productivity gains from specialized tools are substantial. Until recently, there were no mass-market tools for scholarly communication, but that's no longer true. In this case the mass-market tools are more productive that the specialized ones, not less. Institutions and agencies need to focus on ways to leverage these tools, not to deprecate their use and arm-twist scholars into specialized tools under institutional control.
  • Insititutions and agencies need to learn from John Boyd and Silicon Valley themselves. Big changes which will deliver huge value but only in the long term are unlikely to be effective. Small steps that may deliver a small increment in value but will either succeed or fail quickly are the way to go.
  • Key to effective change are the incentive and reward systems, since they close the OODA loop. The problem for institutions and agencies in this area is that the mass-market tools have very effective incentive and reward systems, based on measuring and monetizing usage. Pay attention to the way Google runs vast numbers of experiments every day, tweaking their systems slightly and observing the results on user's behavior. Their infrastructure for conducting these experiments is very sophisticated, because the rewards for success flows straight to the bottom line. The most important change institutions and agencies can make is to find ways to leverage the Web's existing reward systems by measuring and rewarding use of scholarly assets. Why does the academic structure regard the vast majority of accesses to the Sloan Digital Sky Survey as being an unintended, uninteresting by-product? Why don't we even know what's motivating these accesses? Why aren't we investing in increasing these accesses?

I tend to be right about the direction things are heading and very wrong about how fast they will get there. With that in mind, here's my prediction for the way future scholars will communicate. The entire process, from lab notebook to final publication, will use the same mass-market blog-like tools that everyone uses for everyday cooperation. Everything will be public, citable, searchable, accessible by automated scholarly tools, time-stamped and immutable. The big problem will not be preservation, because the mass-market blog-like platforms will treat the scholarly information as among the most valuable of their business assets. It will be more credible, and thus more used, and thus generate more income, than less refined content. The big problem will be a more advanced version of the problems currently plaguing blogs, such as spam, abusive behavior, and deliberate subversion. But, again, since the mass-market systems have these problems too, scholars will simply use the mass-market solutions.

8 comments:

Prashanth said...

As someone who has often wondered how appropriate the mysterious process of peer-review is, I completely agree that "blog-science" can make a big difference. I agree with most of the advantages David has listed. Here's one more advantage I think would matter a lot:
Most conferences that I have known do not want to run the risk of any perception of unethical behavior. They require that colleagues of the author(s) from the same research group / department / institution or colleagues who have been co-authors in the past (depending on the degree to which they apply this perception of unethical behavior tenet) must not be reviewers and/or must not even contribute to a discussion of pros and cons of the paper during a program committee meeting. They are even asked to step out of the room during a discussion of the concerned paper. This may sound reasonable. But, we are actually artificially limiting the "peer-review" process through these "politically correct" rules. Why shouldn't a colleague contribute to reviewing or to a discussion during the PC meeting? Especially if the colleague happens to be one of only a handful of researchers working in a niche area?

Because of the transparency offered by "blog-science", it should be perfectly fine for anybody to review and comment on a piece of work. It doesn't matter "who" says something; what matters is "what" is being said! So, colleagues and former co-authors are also more than welcome to say anything they wish. What they say or how they vote is not kept a secret through a mysterious process, because of which they could have been suspected of playing nepotism or being biased. Rather, what they say is out there for the whole world to see!

Corinne said...

I absolutely love the idea of enabling research to receive feedback online in real-time. And, the possibility of tracing the research process throughout its development, within a blog, allows those interested in taking the research a step further to do so more effectively.

As a PhD student just finishing my third year, I have already observed and been frustrated by the inefficiencies of academic research. I have three specific "pet peeves", which I expect "blog science" would solve:
1) Least Publishable Unit: numbers count when it comes to papers published, and students in my lab and others play into this rule by turning what could be a single stellar paper into as many papers as possible with incredible amounts of overlap. In the worst cases, the same research is published time and again in different packages. Blog science would allow researchers to keep a constant log of their research as discoveries are made. The quality of an academic would be based not on the number of papers listed on their CV, but on a peer voting system, and perhaps number of links to the work (i.e. usefulness of research).
2) Academic Rivalry - as mentioned above, the small community that is able to review a given paper topic is affected by competition. A good friend of mine began submiting a paper for review last year, but was rejected multiple times by reviewers working on the same problem. I suspect that these researchers wanted to supress his work until theirs was ready to publish first. These researchers have since adopted aspects of his approach. Blog science would have allowed him to post his ideas without this competitive censure.
3) Time: this is a generational and efficiency problem, as mentioned above by David. I have very little patience for the long wait time of peer reviewed journals or even conferences to receive input on research. Getting feedback on research I did over a year ago is simply not that useful.

David's ideas are excellent, and I would like to start using this method of sharing scientific discoveries and ideas immediately. In my opinion, three things are necessary to enable "blog-science"
1) Latex enabled blogging
2) An effective voting system, as David discusses above
3) Acceptance by the academic community of "blog science". I believe this third point would quickly follow from effective implementation and use of the first two.

Dennis McDonald said...

This is great stuff. I've tried to address these issues from the perspective of someone who started out in academia but switched to consulting. What I see happening is people who blog a lot rediscovering a lot of what is already known by people who study the processes (formal and informal) by which research is conducted and disseminated. I've published some of my thinking about blogging and peer reviewed journals on my blog; here is one post: "More Comparisons of Journal Peer Review and Blogging" (http://www.ddmcd.com/more_comparisons.html).

mckusick said...

Blogs certainly do provide a very effective means for developing and disseminating information. However, I do not think that they will completely displace other methods. Really fast moving ideas tend to be developed on IRC or other more immediate channels. I also think that publications will continue to have value for describing established research ideas once they have come to some conclusions on the blogs. Blogs seldom provide the coherent structure of a well written paper: background, idea, realization of idea, results of putting the idea into practice, future directions.

David. said...

Thank you all for your comments. I'm hopeful that this discussion will influence the final report of the workshop, so more comments are welcome. Some quick notes:

* Peter Suber's influential Open Access News blog notices this post!

* There is already a plugin for the WordPress blog platform that supports math symbols. It uses a public server that interprets LaTex tagged as "tex" into an image that gets embedded in your HTML. A quick search couldn't find a equivalent for Blogger or other platforms, but it would clearly make a big difference to blog-science.

* The problem of the least publishable unit is real. It is exacerbated by the proliferation of low-quality journals I referred to in this post. In computer science it is handled to some extent by workshops, which are treated as a less formal mode of publishing partial or work-in-progress results without disqualifying them from later, more formal publication. In a sense blog-science provides a continuous workshop of this kind.

* Its interesting that both the comments from grad students stress the problems caused by the lack of transparency of the peer-review process. As research fields become more and more specialized, and thus the community of peers smaller, these problems become worse and worse.

* Clearly, I should have searched better before writing! Dennis McDonald's first and second posts comparing blogging and peer-review from last November make some important points.

* As I said, I'm not good at time-scales. So no-one should expect blog-science to obsolete traditional publishing quickly (if at all). Note that it was 11 years after arXiv started before a major discovery (Perlman's proof of the Poincare conjecture) was published there without being published in traditional journals.

* Far-sighted traditional journals are gradually becoming more blog-like. The British Medical Journal has long been a leader in this area with its "Rapid Response" feature (see a typical paper and its responses. Disclosure: BMJ is hosted on Stanford's HighWire Press). Subscribers can, and frequently do, submit comments on papers, editorials and other elements of the BMJ which get moderated and attached to the relevant "post".

David. said...

Clearly, some staff at scientific publishers are scared enough of the encroaching blogs to over-react. From Slashdot:

"Recently, the well-read science blog Retrospectacle posted an article on a scientific paper that concluded that alcohol augments the antioxidant properties of fruit. The blog post reproduced a chart and a table from the original article and everything was fully attributed. When the publisher John Wiley & Sons found out, they threatened legal action unless the chart and table were removed."

Shelly Batts, who posted the offending material, complied with the takedown request but reconstructed the table and chart by hand in Excel. Science proceeded majestically on its way. After a storm of blog and e-mail protest, Wiley graciously apologized and acknowledged that she had fair use rights under the University of Michigan's subscription.

David. said...

Stuart Staniford has posted an impressive follow-up to the post that originally inspired this discussion.

David. said...

Way down in the over 200 comments on Staurt Staniford's follow-up is this sub-thread discussing whether to set up a Transactions part of The Oil Drum where posts voted by the community to be of archival quality would be migrated after re-formatting and editing to take account of the comments.

Also, James Hamilton posted this more readable summary of Stuart's follow-up for those finding the monster to hard to follow.