Tuesday, October 1, 2013

End-of-life in the Cloud

The Register reports on the demise of Nirvanix, an enterprise cloud storage startup. Nirvanix customers were told on Sept. 18:
Customers had to get all their data out by the end of September or, in effect, face losing it.
They had 13 days to do it. Below the fold I ask what would happen if Amazon made a similar announcement about S3 - not because I think that is possible but to show how impossible it is.

Lets assume that Amazon decided to shut S3 down and gave its customers the same 13 days to recover their data that Nirvanix did. As of last April Amazon claimed that S3 held 2*1012 objects. They don't say how big the average object is, but lets assume that it is 106 bytes. That means that the outbound bandwidth needed to extract the data in 13 days is about 1.5*1013bit/s or 15GTb/s, a 20% increase in 2011's global fixed-line Internet traffic. Amazon charges 5c/GB for up to 500TB/month outbound bandwidth. At this price (there are discounts for more), they would charge their customers in aggregate $108 to get their data out.

Even assuming the customers have enough inbound bandwidth to accept their share of the traffic, they need a place to store the data they are extracting. So in 13 days they need to buy 5*105 4TB disks, or 12% of the world's total storage production in that time. At retail they would pay $7.5*107. When the Thai floods destroyed 40% of the world's disk production capacity prices doubled overnight. So S3's customers are going to have to pay over the odds for their drives. Add in the costs of staff time and general disruption, and we are looking at $109. A billion dollars.

What this thought experiment shows is that AWS, and S3 in particular, has become "too big to fail" just like the big banks. A threat by Amazon to shut down would be so disruptive that governments would be forced to step in to prevent it. Note that governments would themselves be among the customers most affected by the threat.


pch said...

Good read, as ever.

I haven't stopped to do the sums, but shouldn't "15Gb/s" be "15Tb/s"?

Max Guernsey said...

You've definitely shown that shutting down Amazon's S3 service in 13 days would be inconvenient for people who chose to depend on it.

If that ever came to pass, I would personally be more concerned with the pandemic, alien invasion, massive asteroid impact, or nuclear war that was causing it. A couple trillion bits that probably aren't even worth as much to me as the splitting maul or shovel rusting in my shed would be pretty much the last thing on my mind.

David. said...

Even if the cloud storage service doesn't vanish into the ether, Dan Tynan's story shows that your account can, and with it all your data, without warning.

Max Guernsey said...

That story is an outrage. I would be so furious. That's why my wife and I NEVER rely on cloud storage alone. It is a backup system, not a primary storage system. The one exception is our source-control service, which is the source of record for certain artifacts but has a backup system that we use to keep copies of our data safe in locations we control.