Amazon's Announcement of Glacier

Today's announcement from Amazon of Glacier vindicates the point I made here, that the pricing model of cloud storage services such as S3 is unsuitable for long-term storage. In order to have a competitive product in the the long-term storage market Amazon had to develop a new one, with a different pricing model. S3 wasn't competitive. Details are below the fold.

Glacier has some excellent features. In return for accepting a significant access latency (Amazon says typically 3.5 to 4.5 hours, but does not commit to a maximum latency), you get one low price of $0.01/GB/mo. You can access 5% of your data each month free, which provides for data integrity checks at 20-month intervals at no cost other than the compute costs in EC2 needed to perform them. Other than that, each access above this costs as much as a month's storage. There is a $0.05 fee for each 1000 requests, making it important that archived objects are large.

Amazon makes a pitch for Glacier in the digital preservation market:

Digital preservationists in organizations such as libraries, historical societies, non-profit organizations and governments are increasing their efforts to preserve valuable but aging digital content such as websites, software source code, video games, user-generated content and other digital artifacts that are no longer readily available. These archive volumes may start small, but can grow to petabytes over time. Amazon Glacier makes highly durable, cost-effective storage accessible for data volumes of any size. This contrasts with traditional data archiving solutions that require large scale and accurate capacity planning in order to be cost-effective.

They are right that Glacier's advantage in this market is that it avoids the need for capacity planning. The announcement does not, however, address the point I made here, which is that the long-term cost-competitiveness of cloud storage services such as Glacier depends not on their initial pricing, but on how closely their pricing tracks the Kryder's Law decrease in storage media costs. It is anyone's guess how quickly Amazon will drop Glacier's prices as the underlying storage media costs drop. But, based on the history of Amazon's pricing, my guess is that it will follow the pattern of S3's pricing. That is:

• The initial price will be set low, even at a loss, so that competitors are deterred and Amazon captures the vast bulk of the market.
• Subsequent media price drops will not be used to reduce prices all round, but will be used to create tiered prices, so that the part of the value of lower media prices not captured by Amazon itself goes to their largest customers.

Thus customers, especially those with short planning horizons, will get themselves locked-in to Glacier and will in the long term pay a lot more than they could have by doing it themselves. The lock-in effect is strengthened in that getting bulk data out of Glacier quickly is expensive, equivalent to between 5 months and a year of storage. But it will be difficult for other vendors to compete with Amazon in this space; they are trading on the short-termism identified in this paper from the Bank of England (PDF).

There is more detail on Werner Vogel's blog, and on the AWS developer's blog. But note that none of these are specific about the technology underlying Glacier. The unspecified but hours-long access latency would allow them to use tape robots, but The Register believes Glacier uses the same underlying disk storage as S3. If it does, one can see why Amazon wouldn't be specific about the technology. It would reveal that suspicions as to the enormous margins Amazon enjoys from S3 are correct.