Paying for Long-Term Storage

I was part of a panel on Economics at the Personal Digital Archiving 2011 conference at the Internet Archive. I talked about the idea of endowing data with a sum sufficient to pay for its indefinite storage. I first blogged about this in 2007. Below the fold is an edited and expanded version of my talk with links to sources.


Paying For Long-Term Storage

The LOCKSS Program at the Stanford Library builds and supports tools libraries can use to collect and preserve material published on the Web, such as academic journals, e-books and government documents, so we face somewhat different problems from personal digital archives. But I'm here to talk about a problem we all share, paying for long-term storage.

Kryder's Law

I'm sure you've all seen this graph, called Kryder's Law: It shows the now 30-year history of exponential decrease in dollars-per-byte of hard disks. There are three main business models for long term digital storage, and two of them depend heavily on Kryder's Law's continuing validity.

Business Models

Amazon's S3 is an example of the simplest of these business models, namely charging rent for the space occupied. The rent per byte can be adjusted over time to match whatever the underlying cost turns out to be; this model will work just fine even if the price of storage goes up.

Google's Gmail doesn't charge for the space. Instead, it makes money by serving ads around accesses to the stored information. How often do you access mail a decade old? So how much money is Google going to make serving ads against your decade-old mail? Google is betting that Kryder's Law will continue, so that the money they make serving ads against the frequently accessed recent mail will be more than enough to pay for storing the accumulated, much larger, body of rarely accessed old mail.

This is one reason Gmail imposes a limit on how much mail you can store at any one time. They can modulate this limit in response to changes in storage costs, although actually shrinking the space in response to an increase in cost would be rather unpopular.

The third model is the one I'll be talking about. The obvious conclusion to draw from Kryder's Law is that it is possible to deposit data in a preservation service together with an endowment, a capital sum sufficient to pay for its preservation indefinitely. Or, as Serge Goldstein of Princeton calls it "Pay Once, Store Endlessly".

Simple Analysis

Princeton is actually operating an endowed data service, so I will use Goldstein's PowerPoint from last fall's CNI meeting as an example of the typical simple analysis. He computes that if you replace the storage every 4 years and the price drops 20%/yr, you can keep the data forever for twice the initial storage cost.

Benefits of Endowment

There are some important benefits of endowing long-term storage in this way. If you are mandated to preserve data, for example under an NSF Data Management Plan, it allows for the cost of doing so to be rolled-up and funded as part of the term-limited funding of the research. The other models require a separate, continuing funding stream. These are difficult to arrange. Most importantly, unlike paper which survives benign neglect well, bits are very vulnerable to interruptions in the money supply. Endowment is the only way to protect against this threat.

Assumptions

Endowment provides a relatively predictable money supply, but how well can we really predict the money demand, and will they match? In practice, we will need some margin of supply over predicted demand in order to be safe.

What Goldstein is actually charging for Princeton's service is $3000/TB for a single on-line copy. A single tape backup is an extra $2000/TB. This would imply that he pays $3000 each for 2TB SATA drives. They are on the shelves at Fry's for $100. So why is the proponent of endowing data so reluctant to follow his own analysis that he charges a 3000% margin?

The simple analysis rests on a set of assumptions. Here are some of them:

• Storage is the only significant cost.
• Kryder's Law will continue.
• The service will do in the future what you paid for in the past.

Lets look at each of them in turn.

Storage is the Major Cost

Figures from Vijay Gill, who runs Google's farms, show that space, power and cooling are 58% of the three-year cost of a server, whereas the cost of the hardware itself is only 26%. Figures from the San Diego Supercomputer Center (abstract only) show that the media is only about 1/3 the total cost of storage.

Some of these other costs are insensitive to the amount of data being stored, so the cost-per-byte drops with the media. But in most cases, such as power, staff, software licenses, and so on, the absolute cost is subject to inflation. The total cost of storage has been dropping, just not as fast as the raw cost of hard disk. As this trend continues, the effect of Kryder's Law on the total cost decreases, since it affects a smaller proportion of the total.

All these factors seem likely to slow the exponential price drop, thus increasing the margin above the current cost of storage that must be charged to endow data.

Moore's Law

There's another exponential curve I'm sure you're familiar with, namely Moore's Law: It says that the number of transistors on a chip doubles every 2 years. This curve is expected to continue for at least another 5 years. For many years, Moore's Law delivered faster and faster CPUs, so people started believing the faster CPUs were a consequence of the law. But a few years ago the increase in CPU speed stopped, because it turned out that there were more profitable things to use the smaller and smaller transistors for, such as multiple cores and lower power.

Kryder's Law Will Continue

There are a number of reasons to believe something similar is about to happen to storage. First, laptops, netbooks and now tablets are destroying the market for desktop PCs, which is where the bulk of the high-capacity low-cost consumer drives go. As the market switches from 3.5" to 2.5" drives, the cost per byte goes up - right now at Fry's a 3.5" drive is 5c/GB where a 2.5" drive is 16c/GB. As 3.5" volumes decrease their cost per byte will drop more slowly. 2.5" volumes are already higher so they are already on the normal exponential drop. Eventually they will cross, but until then the cost curve will drop more slowly.

Second, although the Kryder's Law curve looks like a smooth exponential, it is actually the result of overlaying a series of exponentials, one for each successive disk technology. The current technology, Perpendicular Magnetic Recording (PMR) has been through 5 generations. It was expected already to have been replaced by either Heat-Assisted Magnetic Recording (HAMR) or
Bit-Patterned Media (BPM). The costs of transitioning to these technologies have turned out to be vastly higher than expected. The industry is having to stretch PMR into a sixth generation. Delaying adoption of these higher-density technologies will slow the cost decrease for a while, and their higher costs will also reduce their initial impact on the cost curve.

Third, solid-state storage, such as Flash and soon Phase Change Memory and Memristors, is eroding the market for hard drives. These technologies need much less power, cooling and rack space. In archival use, they can be expected to last much longer. Their initial cost is much, much higher but the savings can nevertheless be significant because space, power and cooling is such a large part of the total.

All these factors seem likely to slow the exponential price drop, thus increasing the margin above the current cost of storage that must be charged to endow data.

You Will Get What You Paid For

Once you have deposited the data you want preserved, and paid the service, you no longer have much leverage if the service doesn't do a good job. That is why insurance, which has similar characteristics, is a heavily regulated industry. One obvious refinement is to make sure that your money goes not to the service directly up front, but to a third party escrow agent. The agent's job is to audit the preservation service at intervals and, if the service is actually preserving the data it is supposed to, hand over the payment for the interval.

If it turns out that the service you chose is not doing a good job, the escrow agent will have to arrange for your data to be transferred to another, successor service. The endowment will have to include a reserve to cover the costs of such transfers.

Of course, the escrow and audit service has to be paid too, and their costs may well not drop exponentially, so this again increases the margin over the current cost of storage that needs to be charged.

An alternative would be to set up a trust with the capital and simply pay for cloud storage. Using S3's "reduced redundancy storage" with a design goal of 4 nines reliability, and assuming the same 20% per year cost decrease with 5% interest you would need a $4.7K endowment. Using Amazon's full redundancy storage for a goal of 11 nines, the trust would need over $7K. It isn't likely that Princeton is buying storage much cheaper than Amazon, so we see that Princeton isn't charging enough to provide even 4 nines reliability, let alone the much higher levels needed for long-term preservation.

Conclusion

Endowing data has some significant advantages over the competing business models when applied to long-term data preservation. But the assumptions behind the simple analysis are optimistic. Real endowed data services, such as Princeton's, need to charge a massive markup over the cost of the raw storage to insulate themselves from this optimism. The perceived mismatch this causes between cost and value may make the endowed data model hard to sell.