A Brief History of E-Journal Preservation

The workshop on the Future of Research Communication opened with a set of talks about the past, providing background for the workshop discussions. My talk covered the history of e-journal preservation, and the lessons that can be drawn from it. An edited text of the talk, with links to the sources, is below the fold.

Rebuttals

I'll save those who might point to evidence against some of my positions the trouble by pointing to the evidence myself:

• I've used the comparison between libraries and Starbucks several times. According to Jim Romenesko via Gawker and Yves Smith we learn that Starbucks is succeeding so well that the students and other "laptop hobos" are driving out people who want to drink coffee. So maybe I was wrong and libraries are showing how hard it is to compete with free.
• On the other hand, I have been very skeptical of the New York Times' paywall. In the Columbia Journalism Review Felix Salmon points to Seth Mnookin reporting on the first 4 months of the paywall.

  its digital-subscription plan has thus far been an enormous success. The ... the goal was to amass 300,000 online subscribers within a year of launch. On Thursday, the company announced that after just four months, 224,000 users were paying for access to the paper’s website. Combined with the 57,000 Kindle and Nook readers who were paying for subscriptions and the roughly 100,000 users whose digital access was sponsored by Ford’s Lincoln division, that meant the paper had monetized close to 400,000 online users. (Another 756,000 print subscribers have registered their accounts on the Times’ website.)

  Clearly that is encouraging, and Salmon goes on to point out that:

  Those paying digital subscribers, however, are much more valuable than their subscription streams alone would suggest. They’re hugely loyal, they read loads of stories, they’re well-heeled, and advertisers will pay a premium to reach them. ... it seems as though total digital ad revenues are going up, not down, as subscriptions get introduced: the holy grail of paywalls.

  However, this is only one digital subscriber for every two paper subscribers, and the NYT is making much less from each digital subscriber even after advertising revenue, so I'm sticking to my position.
• I have been skeptical of the long-term prospects of large publishers such as Elsevier, although I was careful to note my track record of being right way too soon. In the short term, Elsevier's parent Reed Elsevier recently turned in modestly better resultsfor the first half of 2011. Its stock has been performing roughly in line with the S&P500. I still believe that the scope for Elsevier to acquire large numbers of lucrative new customers is limited, and their ability to squeeze more income from the existing customers is obviously limited by research and teaching budgets.

Fujitsu agrees with me

I've been saying for some time that flash memory is unlikely to be the long-term solid state memory technology. In an interview with Chris Mellor at The Register, Fujitsu's CTO agrees with me:

[Joseph Reger] reckons Phase Change Memory (PCM) is the closest, in terms of time to become a usable technology, than other post-flash contenders such as HP's Memristor.

In the more interesting part of the story, he agrees with me that the real potential of these post-flash technologies is that they can be packaged as persistent RAM rather than block storage:

[Reger] asks if everything will be rewritten and re-orchestrated to work with data memory management. Is there effectively only going to be one tier, memory in one form or another?

"Currently, having data in storage means it's not in memory. Is it going to stay like that?" After all, storage was invented to deal with memory-size limitations. If those limitations go away then who needs storage?

Reger said: "I truly believe we are going to have a data orientation rather than memory and storage orientations." But this is really far out in the future.

I'm old enough to remember when computer memory persisted across power cycles because it was magnetic cores. I'd love to see this feature return. The major software change that would be needed is far more than simply using in-memory databases. The RAM data structures would need to be enhanced with metadata and backups, especially for long-term integrity, if we were to get rid of block storage entirely.

Flash more reliable than hard disk?

Andrew Ku has an interesting article at Tom's Hardware showing that, despite claims by the vendors, the evidence so far available as to the reliability of flash-based SSDs in the field shows that they fail at rates comparable with hard drives. These failures have nothing to do with the limited write lifetime of flash, these are enterprise SSDs which have been in service a year or two and are thus nowhere close to their write life.

Although it is easy to believe that SSDs' lack of moving parts should make them more reliable, these results are not surprising:

• Experience shows that vendors tend to exaggerate the reliability of their products.
• Root-cause analysis of failures in hard-drive systems shows that 45-75% of failures are not due to the disks themselves, so that even if SSDs were a lot more reliable that hard drives but were used in similar systems, the effect on overall system reliability would be small. In fact, most of the SSDs surveyed were used in hybrid systems alongside disks.
• The flash SSDs have even more software embedded in them than disk drives. Thus even if their raw storage of bits was more reliable than the disk platters, they would be more vulnerable to software errors.

These are early days for the kinds of mass deployments of SSDs that can generate useful data on field reliability, so there is no suggestion that they should be avoided. The article concludes:

The only definitive conclusion we can reach right now is that you should take any claim of reliability from an SSD vendor with a grain of salt.

"Paying for Long-Term Storage" Revisited

In this comment to my post onPaying for Long-Term Storage I linked to this speech by Andrew Haldane and Richard Davies of the Bank of England. They used historical data from 624 UK and US companies to estimate the extent to which short-term thinking is reflected in the prices of their stock. In theory, the price of each company's stock should reflect the net present value of the stream of future dividends. They find that:

First, there is statistically significant evidence of short-termism in the pricing of companies’ equities. This is true across all industrial sectors. Moreover, there is evidence of short-termism having increased over the recent past. Myopia is mounting. Second, estimates of short-termism are economically as well as statistically significant. Empirical evidence points to excess discounting of between 5% and 10% per year.

In other words, the interest rate being charged when using standard discounted cash flow computations to justify current investments is systematically 5-10% too high, resulting in investments that would be profitable not being made.

This is obviously a serious problem for all efforts to preserve content for future readers; the value the content eventually delivers to the future readers has to be enormous to justify even minimal investment now in preserving it. Below the fold I describe how even this analysis fails to capture the scale of the problem of short-termism.

More on de-duplicating flash controllers

My ACM Queue piece describing the problems caused by storage devices invisibly doing de-duplication attracted the attention of Robin Harris who actually asked SandForce and other manufacturers to comment. The details of SandForce's response are in Robin's StorageMojo article but the key claims are:

There is no more likelihood of DuraWrite loosing data than if it was not present.

and:

That is why SandForce created RAISE (Redundant Array of Independent Silicon Elements) and includes it on every SSD that uses a SandForce SSD Processor. ... if the ECC engine is unable to correct the bit error RAISE will step in to correct a complete failure of an entire sector, page, or block. ...This combination of ECC and RAISE protection provides a resulting UBER of 10^-29 virtually eliminates the probabilities of data corruption.

I would regard both claims with considerable skepticism:

• SandForce has not disclosed the details of their technology, but the research performed by Michael Wei and his co-authors at UCSD revealed that it definitely includes de-duplication. Thus the first claim is not credible; there is only a single copy of the supposedly redundant data in the flash array instead of multiple copies. The metadata must therefore be at higher risk.
• As regard the second claim, in my earlier ACM Queue article I show that manufacturers claims of error rates such as 10^-29 are not credible because they are not the result of experiments, but of models which are unrealistic and unverified.

Library of Congress interview

An interview with me inaugurates an interview series called Insights on the Library of Congress' digital preservation blog The Signal.