Yet more bad news about disks

Chris Mellor at The Register reviews the prospects for the 4TB disk generation and reports that manufacturers are finding the transition to the technologies it needs more difficult and expensive than expected.

This reinforces the argument of my earlier post, based on Dave Anderson's presentation (PDF) to the 2009 Library of Congress Storage workshop, that the exponential drop in cost per byte we expect from disks is about to flatten out.

JCDL 2010 Keynote

On June 23 I gave a keynote address entitled to the joint JCDL/IACDL 2010 conference at Surfer's Paradise in Queensland, Australia. Below the fold is an edited text of the talk, with links to the resources.

Even more bad news about disks

I attended the 25th (plus one) anniversary celebrations for the Andrew project at Carnegie-Mellon University. As part of these James Gosling gave a talk. He stressed the importance of parallelism in programming, reinforcing the point with a graph of CPU clock rate against time. For many years, clock rate increased. Some years ago, it stopped increasing. Did Moore's Law stop working? Not at all, there were no strong technological barriers to increasing the clock rate. What happened was that increasing the clock rate stopped being a way to make money. Mass-market customers wanted lower power, lower price CPUs, not faster ones. So that's what the manufacturers made.

Follow me below the fold to see the analogous phenomenon happening to disks, and why this is bad news for digital preservation.

Spring CNI Plenary: The Video

CNI has now posted the video of Cliff Lynch's introduction, my plenary presentation, and the questions.

How Are We Ensuring the Longevity of Digital Documents? from CNI Video Editor on Vimeo.

I gave a significantly shortened version of this talk at the Sun PASIG meeting in Malta June 26.

Hard Disk Drives: The Good, the Bad and the Ugly

Jon Elerath just published a wonderful paper in the June 2009 Communications of the ACM entitled "Hard Disk Drives: The Good, the Bad and the Ugly". Everyone, especially anyone who believes bit preservation is a solved problem, should read it. He clearly communicates the incredible complexity of the technology inside the familiar 3.5" drive form factor.

Elerath reviews the range of hard disk failure modes, and shows how difficult it will be for disk manufacturers to maintain the drive reliability constant as disks get bigger. And even if they succeed in keeping drive reliability constant while the disk gets bigger, the bit reliability they deliver goes down. He says:

Multi-terabyte capacity drives using perpendicular recording will be available soon, increasing the probability of both correctable and uncorrectable errors by virtue of the narrowed track widths, lower flying heads, and susceptibility to scratching by softer particle contaminants.

Thus, as I have been saying for a while, just as we are trying to preserve larger and larger numbers of bits, the technologies we use to make those bits reliable are not keeping pace. Elerath concludes:

Only when these high-probability [failure] events are included in the optimization of the RAID operation will reliability improve. Failure to address them is a recipe for disaster.

I agree that RAID technology needs to adapt to the decreasing bit reliability and longer time to repair of newer disk drives. But, as I argued in my iPRES2008 paper (pdf), even if we do a good job of adapting RAID to cope with these problems we will still be many orders of magnitude below the reliability levels digital preservation needs.

Sheila Morrissey's comment

Portico's Sheila Morrissey posted a valuable comment on the post that provided the background and sources for my CNI plenary. It set out the conventional wisdom against which I was arguing, but at such length that I felt it was inhibiting discussion. It was also difficult to respond to by adding a comment, among other reasons because there was no easy way to connect my responses to their targets in the comment. I therefore saved the text of Sheila's comment, deleted it from the original post, and reproduced it below the fold, together with my responses. Portico has posted a version of her comment here.

Spring CNI Plenary: The Remix

This post provides the text of the slides, sources and commentary for the opening plenary that I just gave at the CNI Spring Task Force meeting. The actual slides are available here (PDF). Follow me below the fold for the full details.