Tuesday, July 16, 2013

Immortal media

Announcements of media technologies with very long lives happen regularly. In 2011 it was stone DVDs, in 2012 it was DNA, now it is a team from Southampton University and Eindhoven's 5-dimensional quartz DVDs. Note that in two presentations at the IEEE Mass Storage meeting back in April Hitachi announced that they already have similar but lower-density DVD technology. Below the fold, my reactions.

So far, announcements of very long-lived media have made no practical difference to large-scale digital preservation because:
  • Media life is only one of many, many threats to which bits are subject. Fixing one sub-problem does not solve the overall problem.
  • Very long-lived media are a niche market, and are thus very expensive. See Dave Anderson's presentation explaining why although it is perfectly possible to build hard disks with archival lives, the economics of doing so are infeasible.
  • The research we have been doing in the economics of long-term preservation demonstrates the enormous barrier to adoption that accounting techniques pose for media that have high purchase but low running costs, such as these long-lived media.
  • The big problem in digital preservation is not keeping bits safe for the long term, it is paying for keeping bits safe for the long term. So an expensive solution to a sub-problem can actually make the overall problem worse, not better.
  • These long-lived media are always off-line media. In most cases, the only way to justify keeping bits for the long haul is to provide access to them (see Blue Ribbon Task Force). The access latency scholars (and general Web users) will tolerate rules out off-line media for at least one copy. As Rob Pike said "if it isn't on-line no-one cares any more".
  • So at best these media can be off-line backups. But the long access latency for off-line backups has led the backup industry to switch to on-line backup with de-duplication and compression. So even in the backup space long-lived media will be a niche product.
  • Off-line media need a reader. Good luck finding a reader for a niche medium a few centuries after it faded from the market - one of the points  Jeff Rothenberg got right two decades ago.
As I've said before, I believe that eventually DNA is likely to have a role in archival storage. It is very dense, very stable and it is very easy to create enormous numbers of copies, which mitigate many of the threats that media such as quartz DVDs don't. The quartz DVD press release indicates that the team don't understand the system-level problems of long-term storage:
“We are developing a very stable and safe form of portable memory using glass, which could be highly useful for organisations with big archives. At the moment companies have to back up their archives every five to ten years because hard-drive memory has a relatively short lifespan,” says Jingyu.
Companies don't "back up" their archives, they migrate them. And the reason for the migration isn't that hard drives have a "relatively short lifespan". The reason is that Kryder's Law has meant that after 4-5 years the old drives no longer justify their slot in the infrastructure against the much denser new drives in terms of $/byte. The drives are designed for a service life of only 4-5 years because the vendors know they will be replaced after that time even if they are still working perfectly. Hard drive media already have archival data retention times of 2-3 decades, but that has nothing to do with their service life.


  1. Simon Sharwood at The Register restrains his skepticism about this month's edition of materials science saves the world.

  2. The BBC has a long piece on Ewan Birney and Nick Goldman's work on DNA as a long-term storage medium. As I have written before, I believe the technical aspects of their work are excellent but their economic analysis is flawed in ways that mean it will be a long time before DNA storage is competitive with conventional media.

  3. Can no-one stop this madness? The University of Twente awarded Jeroen de Vries a Ph. D. for showing that data etched into tungsten then encapsulated in silicon nitride is not degraded after 1 hour in an over at 200C.

    This result hardly surprising, and it has absolutely nothing to do with the practical problem of preserving data for meaningful spans of time.

  4. Microsoft is experimenting with DNA storage. They are spending $0.10 to write a base, about 2 bits. Enough money to store 3 copies of a gigabyte for a year, with a much lower access latency. So, as I have said many times, economic feasibility for DNA storage is a long way away.

  5. A team at TU Delft have stored a kilobyte in chlorine atoms on a copper lattice at -196˚C in a clean vacuum. Press reports describe this as a storage breakthrough despite research leader Sander Otte's comment:

    "I would like to emphasize that in my view the true importance of this work is not necessarily the data storage itself".