Bit.ly's Plan Z

One of the big worries the evolution of the Web has been posing for preservation is the spread of URL shortening services. A failure of one of these services would break a vast number of preserved links. The more polite version of the two description of the problem at urlte.am is:

The URLTeam is the ArchiveTeam subcommittee on URL shorteners. We believe that they pose a serious threat to the internet's integrity. If one of them dies, gets hacked or sells out, millions of links will stop working.

In a fascinating keynote at Digital Preservation 2013, Hilary Mason dropped a hint to ask her after the talk about Bit.ly's Plan Z. So we did. It turns out that Plan Z is Bit.ly doing the right thing, for themselves and to some extent for the world.

Every time Bit.ly shortens a URL, they write a static redirect from the short to the long URL into Amazon's S3. Thus, if their service ever goes down, as for example it did when someone unplugged their data center, all they need need to do is to update the DNS record for bit.ly to point to S3, and the short URLs continue to resolve as they used to.

This is fine for Bit.ly itself and it is a big step towards being fine for the world. Of course, if Bit.ly were to go out of business, they would stop paying the S3 charges and for the bit.ly domain name, and their Plan Z wouldn't help the Web survive intact.

Archive Team has a group called Urlteam that works to back up URL shortening services, compiling a map from short to long URL for each and exposing the result as a torrent that can be downloaded and preserved. This is great, as it gets the data out of the custody of the service but, again, it doesn't on its own solve the problem of keeping the links intact after the service implodes.

If we can get it deployed, Memento is the missing piece of the solution. It is designed to ensure that links resolve even after the original target has gone away. As far as Memento is concerned, a shortened URL is no different from any other URL. If, after the URL shortening service dies, one or more of the backups can provide a site like Plan Z's that exports the static redirects and supports Memento then browsers that support Memento will continue to see the shortened URLs resolve as they originally did.

Talk at Digital Preservation 2013

I was on a panel at the Library of Congress' Digital Preservation 2013 meeting entitled Green Bytes: Sustainable Approaches to Digital Stewardship. Below the fold is the text of my brief presentation, with links to the sources.

Immortal media

Announcements of media technologies with very long lives happen regularly. In 2011 it was stone DVDs, in 2012 it was DNA, now it is a team from Southampton University and Eindhoven's 5-dimensional quartz DVDs. Note that in two presentations at the IEEE Mass Storage meeting back in April Hitachi announced that they already have similar but lower-density DVD technology. Below the fold, my reactions.

The Library of Congress' "Preserving.exe" meeting

In May the Library of Congress held a meeting entitled Preserving.exe: Toward a National Strategy for Preserving Software. I couldn't be there and I've only just managed to read the presentations and other materials. Three quick reactions below the fold.

Dr. Pangloss and the Road-Maps

The renowned Dr. Pangloss takes great pleasure in studying the storage industry's road-maps with their rosy view of the future. I've frequently pointed to this 2008 Seagate road-map for disk technology, showing Perpendicular Magnetic Recording (PMR) being supplanted by Heat Assisted Magnetic Recording (HAMR) starting in about 2009. Its 5 years later and no vendor has yet shipped HAMR drives, although HAMR has been demonstrated in the lab at over a trillion bits per square inch, about a 30% improvement over the best current PMR. This illustrates that these vendor road-maps tend to err on the optimistic side. Dr. Pangloss is rubbing his hands with glee at the vendor's latest road-maps; below the fold I look at why he is so happy.