I'm David Rosenthal from the LOCKSS (Lots Of Copies Keep Stuff Safe) Program at the Stanford University Libraries. I started working on the problem of long-term digital preservation 15 years ago. I thought it would be semi-retirement; it has turned into a job for life. There are three parts to the problem. Stuff must be ingested, preserved and disseminated. The essential task of the preservation part is storing the bits for the long term. The problems of storing bits for the long term come in two flavors, technical and economic.
I discussed the technical problems in a 2010 article for ACM Queue entitled Keeping Bits Safe: How Hard Can It Be? I used the example of a black box keeping a Petabyte for a century with a 50% chance that every bit survives unchanged. Consider each bit like a radioactive atom, subject to a random process that flips its state. The specification implies a half-life for the bits. It is about 60 million times the age of the universe. What this means is that although a storage system vendor may claim their product meets your specification, benchmarking it to validate their claim is infeasible.
Further, the set of threats against which stored data must be preserved includes not merely random media failures, which can be modelled with some realism, but also threats such as insider abuse and external attack which cannot. Edward Snowden's revelations show the capabilities nation-state actors had a few years ago. A few years from now many of them will be available in the exploit market for anyone motivated to corrupt your data to use.
The economic problem of storing data for the long term has historically been considered insignificant. The 30-year history of Kryder's Law, the exponential increase in bit density, led to an exponential drop in dollars per byte. This meant that, if you could afford to store the data for a few years, you could afford to store it forever. And this led to the concept of endowing stored data, depositing it together with a capital sum believed sufficient to pay for its eternal storage. In economic terms, the endowment is the net present value of the stream of future payments for storage.
In 2010 Serge Goldstein of Princeton described their endowed data service, based on his analysis that if they charged double the initial cost they could store data forever. I was skeptical, not least because what Princeton actually charged was $3K/TB. This meant either that they were paying $1.5K/TB for disk at a time when Fry's was selling disks for $50/TB, or that they were skeptical too.
So I built an economic model of long-term storage. The two key parameters are:
- The interest rate that the as yet un-expended part of the endowment obtains.
- The Kryder rate, the rate at which the cost per byte drops.
There is nothing to worry about, right? The Kryder rate has been 30-40% for 30 years, so it is bound to continue just as Moore's Law is.
In late 2011 the floods in Thailand destroyed 40% of the world's disk manufacturing capacity. The price per byte almost doubled, and more than 2 years later is still far above what it would have been absent the floods. Even the perennially optimistic industry road-maps now project not more than 20%/yr for the next 5 years. Industry consolidation, and the fearsome cost of the transition from PMR to HAMR let alone to BPM, mean that the days of 30-40% Kryder rates are over.
Every few months there is another press release announcing that some new, quasi-immortal medium such as stone DVDs has solved the problem of long-term storage. But the problem stays resolutely unsolved. Why is this? Very long-lived media are inherently more expensive, and are a niche market, so they lack economies of scale. Seagate did a study of the market for disks with an archival service life, which they could easily make, and discovered that no-one would pay the extra for them.
The fundamental problem is that long-lived media only make sense at very low Kryder rates. Even if the rate is only 10%/yr, after 10 years you could store the same data in 1/3 the space. Since space in the data center or even at Iron Mountain isn't free, this is a powerful
There is one long-term storage medium that might eventually make sense. DNA is very dense, very stable in a shirtsleeve environment, and best of all it is very easy to make Lots Of Copies to Keep Stuff Safe. DNA sequencing and synthesis are improving at far faster rates than magnetic or solid state storage. Right now the costs are far too high, but if the improvement continues DNA might eventually solve the archive problem. But access will always be slow enough that the data would have to be really cold before being committed to DNA.
The reason that the idea of long-lived media is so attractive is that is suggests that you can design a system ignoring the possibility of media failures. You can't, and even if you could it wouldn't make economic sense. As Brian Wilson, CTO of BackBlaze points out, in their long-term storage environment:
Double the reliability is only worth 1/10th of 1 percent cost increase. I posted this in a different forum: Replacing one drive takes about 15 minutes of work. If we have 30,000 drives and 2 percent fail, it takes 150 hours to replace those. In other words, one employee for one month of 8 hour days. Getting the failure rate down to 1 percent means you save 2 weeks of employee salary - maybe $5,000 total? The 30,000 drives costs you $4m.Let me leave you with another graph. It is based on three industry numbers:
The $5k/$4m means the Hitachis are worth 1/10th of 1 per cent higher cost to us. ACTUALLY we pay even more than that for them, but not more than a few dollars per drive (maybe 2 or 3 percent more).
Moral of the story: design for failure and buy the cheapest components you can. :-)
- According to IDC, the demand for storage each year grows about 60%.
- According to IHS iSuppli, the bit density on the platters of disk drives will grow no more than 20%/year for the next 5 years.
- According to computereconomics.com, IT budgets in recent years have grown between 0%/year and 2%/year.
In the discussion after this introductory talk, someone from IBM reported large customers asking for large-scale, write-once on-line storage at low cost. I am skeptical that this combination of properties can be delivered more cheaply with custom components than, for example, Backblaze is delivering based on commodity components. Even more interesting, Dave Anderson reported that industry projections for the Kryder rate are now down to around 12%.