Follow me below the fold to see the analogous phenomenon happening to disks, and why this is bad news for digital preservation.
At the 2009 Library of Congress workshop on Architectures for Digital Preservation, Dave Anderson of Seagate gave two fascinating talks about the disk drive business.
He described Seagate's investigation of the idea of a disk drive specifically for archival use. Technologically, it was easy. They could build a very reliable, long-lived drive. But there was no way to make money building it. One reason was that customers wouldn't buy it; the economics for them of replacing older drives with newer ones that were identical in all respects except for greater capacity are irresistible. They don't want old, small but highly reliable drive cluttering up their machine rooms. The other reason was that even if customers did want to buy these drives, they would be a niche product sold in small volumes. So they would cost a lot more per byte than the consumer drives. Customers are, with good reason, skeptical of manufacturer's claims for reliability. Thus even if the special archival disks actually did repay the additional cost in greater reliability, it would likely not be possible to persuade customers of this.
He also described Seagate's look forward for consumer 3.5" drives. Current drives are 2TB. The next generation is 4TB. The generation after that would be 8TB. There are no technological barriers to building an 8TB 3.5" drive in the 2013 timeframe, The technology, called HAMR, is in hand. But there's no way to make money building 8TB 3.5" drives and selling them for $100. Why is this?
- Up until a few years ago, disk drives were pretty much always full. But nowadays, they rarely fill up. The 2TB 3.5" drive generation is more than almost all consumers need. So what value does an 8TB drive add? Let alone the problem of backing it up.
- More importantly, laptops and now netbooks and tablets are destroying the market for the desktop boxes that 3.5" drives go in to. Even customers who want a box sitting on their desk now prefer small form factor (SFF) PCs such as the Mac Mini.
And what this means is that one of the articles of faith underlying the economics of disk drives, the exponential decrease in cost per byte described by Clayton Christensen in The Innovator's Dilemma, won't hold. As we work through the transition to 2.5" drives, and possibly through the subsequent transition to solid state storage such as flash, the cost per byte curve will at best flatten or more likely go up for some years.
Although it is true, as shown by the work of the San Diego Supercomputing Center, that storage media cost is only about 1/3 of the total cost of long-term storage, the other costs do not fall as fast. Thus the effect of the disk cost curve is more than the 1/3 would indicate. But, worse, the large storage farms at the heart of "the cloud" are built from 3.5" consumer drives. Cloud providers, which are seen as key to the future of long-term storage, have bet heavily on storage cost's exponential decline. Failure to fulfill these expectations could be disruptive.