Even more bad news about disks

I attended the 25th (plus one) anniversary celebrations for the Andrew project at Carnegie-Mellon University. As part of these James Gosling gave a talk. He stressed the importance of parallelism in programming, reinforcing the point with a graph of CPU clock rate against time. For many years, clock rate increased. Some years ago, it stopped increasing. Did Moore's Law stop working? Not at all, there were no strong technological barriers to increasing the clock rate. What happened was that increasing the clock rate stopped being a way to make money. Mass-market customers wanted lower power, lower price CPUs, not faster ones. So that's what the manufacturers made.

Follow me below the fold to see the analogous phenomenon happening to disks, and why this is bad news for digital preservation.


At the 2009 Library of Congress workshop on Architectures for Digital Preservation, Dave Anderson of Seagate gave two fascinating talks about the disk drive business.

He described Seagate's investigation of the idea of a disk drive specifically for archival use. Technologically, it was easy. They could build a very reliable, long-lived drive. But there was no way to make money building it. One reason was that customers wouldn't buy it; the economics for them of replacing older drives with newer ones that were identical in all respects except for greater capacity are irresistible. They don't want old, small but highly reliable drive cluttering up their machine rooms. The other reason was that even if customers did want to buy these drives, they would be a niche product sold in small volumes. So they would cost a lot more per byte than the consumer drives. Customers are, with good reason, skeptical of manufacturer's claims for reliability. Thus even if the special archival disks actually did repay the additional cost in greater reliability, it would likely not be possible to persuade customers of this.

He also described Seagate's look forward for consumer 3.5" drives. Current drives are 2TB. The next generation is 4TB. The generation after that would be 8TB. There are no technological barriers to building an 8TB 3.5" drive in the 2013 timeframe, The technology, called HAMR, is in hand. But there's no way to make money building 8TB 3.5" drives and selling them for $100. Why is this?

• Up until a few years ago, disk drives were pretty much always full. But nowadays, they rarely fill up. The 2TB 3.5" drive generation is more than almost all consumers need. So what value does an 8TB drive add? Let alone the problem of backing it up.
• More importantly, laptops and now netbooks and tablets are destroying the market for the desktop boxes that 3.5" drives go in to. Even customers who want a box sitting on their desk now prefer small form factor (SFF) PCs such as the Mac Mini.

What this means is that the $100 3.5" drive will go away. There will still be a market for 3.5" drives, but because they won't be manufactured in consumer quantities, they will cost more than $100. There will still be a market for $100 drives, but they will be 2.5" and say 1/3 or 1/4 the capacity of the 3.5" drive that people were expecting.

And what this means is that one of the articles of faith underlying the economics of disk drives, the exponential decrease in cost per byte described by Clayton Christensen in The Innovator's Dilemma, won't hold. As we work through the transition to 2.5" drives, and possibly through the subsequent transition to solid state storage such as flash, the cost per byte curve will at best flatten or more likely go up for some years.

Although it is true, as shown by the work of the San Diego Supercomputing Center, that storage media cost is only about 1/3 of the total cost of long-term storage, the other costs do not fall as fast. Thus the effect of the disk cost curve is more than the 1/3 would indicate. But, worse, the large storage farms at the heart of "the cloud" are built from 3.5" consumer drives. Cloud providers, which are seen as key to the future of long-term storage, have bet heavily on storage cost's exponential decline. Failure to fulfill these expectations could be disruptive.