Notes from Desiging Storage Architectures workshop

Below the fold are some notes from this year's Library of Congress Designing Storage Architectures meeting.

There was wide agreement among the industry attendees that in the next 5 years or so Kryder rates of 20% for bit density on disk, and 40% for tape and solid state were to be expected, and that some time in the following 5 years or so the rates for both tape and solid state would slow. Although there are many reasons for doubting that future increases in bit density will translate 1-for-1 into reductions in $/GB, I plan to use this consensus projection in the next stage of my economic modeling work. There was also wide agreement that budgets for digital preservation were at best flat.


There was a striking change from previous meetings, in that many industry attendees at least mentioned cost as a significant problem. However, the recognition that cost was significant hasn't affected their thinking much. They had not yet arrived at the concept of price elasticity of demand (PED):

a measure used in economics to show the responsiveness, or elasticity, of the quantity demanded of a good or service to a change in its price. More precisely, it gives the percentage change in quantity demanded in response to a one percent change in price

The assumption behind most of the presentations was that current growth rates in demand for storage would continue irrespective of the effective increase in the price of storage implied by the reduction in Kryder rates from their historic level. In other words, the assumption is that the demand for storage is completely inelastic; organizations will pay whatever it takes to store their data, and are incapable of reducing their demand to match their budget. I've expressed doubt that this is the case in previous blog posts, and in the graph at the start of my presentation to the meeting.

One interesting data point at the meeting came from the San Diego Supercomputer Center's cloud storage service. At first glance they seem to be undercutting S3 by a substantial amount:

Store your data in SDSC's enterprise class Cloud Service for as low as $3.25/Month for 100GB ($32.50/Terabyte/Month) with no transfer costs.

Alas, this isn't the case. "As low as" $0.0325/GB/mo applies only to UC customers. Other academic customers are "as low as" $0.0569/GB/mo and industry partners are "as low as" $0.065/GB/mo. And what you get for this isn't S3, with three geographically separate replicas. It is a single replica, more like S3's Reduced Redundancy Storage, which has tiered pricing ranging from $0.093 for the first TB to $0.037/GB/mo once you get over 5PB. Counting SDSC's lack of tiers and access charges, they are still somewhat cheaper for preservation use, especially for UC customers and fairly small outsiders.

However, the interesting question for preservation use is not so much the initial price, but how well the price matches the price drop of the underlying storage. If SDSC's prices come down much quicker than Amazon's have, this would be an interesting product. If not, it doesn't make a big enough difference to invalidate my earlier conclusion that:

Unless things change, cloud storage is simply too expensive for long-term use.