Dan Geer at RSA

Dan Geer gave a must-read talk at the recent RSA conference. Dan is especially strong on the fragility of the systems upon which society is coming to depend, a theme that also ran through my friend Dewayne Hendricks' recent talk at Stanford's EE380 seminar series. Below the fold, I quibble with one part of Dan's talk to show how persistent projections based on exponential growth are in the face of facts.

Dan says:

Contradicting what I said earlier, there may actually be a difference between the public and the private sector because the private sector will collect data only so long as increased collection can be monetized, whereas government will collect data only so long as increased collection can be stored. With storage prices falling faster than Moore's Law, government's stopping rule may thus never be triggered.

Even experts such as Dan are wrong.

Just as Moore's Law has slowed dramatically, Kryder's Law (the analogous law for magnetic storage) slowed dramatically in 2010. The Thai floods in 2011 nearly doubled $/GB and we have only just got back to 2010 levels. That means that disk is about 7 times as expensive as it would have been had Kryder's Law continued from 2010 unabated. Industry projections for the next 5 years range from 20%/yr to 10%/yr. If they pan out disk in 2020 will be a couple of orders of magnitude more expensive than it would have been had Kryder's Law continued unabated.

For details on why Kryder's Law slowed see our 2012 UNESCO paper.

"so long as increased collection can be monetized" means "monetized enough to pay for the storage". Storage will be a lot less free than it used to be, so the monetary or intelligence value of the data will need to be a lot higher than people used to think to make it worth storing.

The idea that everything will be stored for ever depends on some new magic medium - it won't be stored on flash, or disk, or tape, or even "in the cloud". We know how much of these media will be made in the next few years because the plant to make it already exists - no-one is going to invest in lots of new flash fabs, disk lines or tape lines at current prices because they wouldn't make money.

Of course, if storage prices increased a lot, more storage might be made, but the lead time on building the fabs is considerable, so this isn't a solution in the short term. It isn't a solution in the long either because, after all, it just makes storage even less free.