Seeds Or Code?

Svalbard Summer '69  

I'd like to congratulate Microsoft on a truly excellent PR stunt, drawing attention to two important topics about which I've been writing for a long time, the cultural significance of open source software, and the need for digital preservation. Ashlee Vance provides the channel to publicize the stunt in Open Source Code Will Survive the Apocalypse in an Arctic Cave. In summary, near Longyearbyen on Spitzbergen is:

the Svalbard Global Seed Vault, where seeds for a wide range of plants, including the crops most valuable to humans, are preserved in case of some famine-inducing pandemic or nuclear apocalypse.

Nearby, in a different worked-out coal mine, is the Arctic World Archive:

The AWA is a joint initiative between Norwegian state-owned mining company Store Norske Spitsbergen Kulkompani (SNSK) and very-long-term digital preservation provider Piql AS. AWA is devoted to archival storage in perpetuity. The film reels will be stored in a steel-walled container inside a sealed chamber within a decommissioned coal mine on the remote archipelago of Svalbard. The AWA already preserves historical and cultural data from Italy, Brazil, Norway, the Vatican, and many others.

Github, the newly-acquired Microsoft subsidiary, will deposit there:

The 02/02/2020 snapshot archived in the GitHub Arctic Code Vault will sweep up every active public GitHub repository, in addition to significant dormant repos as determined by stars, dependencies, and an advisory panel. The snapshot will consist of the HEAD of the default branch of each repository, minus any binaries larger than 100KB in size. Each repository will be packaged as a single TAR file. For greater data density and integrity, most of the data will be stored QR-encoded. A human-readable index and guide will itemize the location of each repository and explain how to recover the data.

Follow me below the fold for an explanation of why I call this admirable effort a PR stunt, albeit a well-justified one.

Nanopore Technology For DNA Storage

DNA assembly for nanopore data storage readout by Randolph Lopez et al from the UW/Microsoft team continues their steady progress in developing technologies for data storage in DNA.

Below the fold, some details and a little discussion.

Cost-Reducing Writing DNA Data

In DNA's Niche in the Storage Market, I addressed a hypothetical DNA storage company's engineers and posed this challenge:

increase the speed of synthesis by a factor of a quarter of a trillion, while reducing the cost by a factor of fifty trillion, in less than 10 years while spending no more than $24M/yr.

Now, a company called Catalog plans to demo a significant step in the right direction:

The goal of the demonstration, says Park, is to store 125 gigabytes, ... in 24 hours, on less than 1 cubic centimeter of DNA. And to do it for $7,000.

That would be 1E11 bits for $7E3. At the theoretical maximum 2 bits/base, it would be $3.5E-8 per base, versus last year's estimate of 1E-4, or around 30,000 times better.

If the demo succeeds, it marks a major achievement. But below the fold I continue to throw cold water on the medium-term prospects for DNA storage.

It Isn't Just Cryptocurrency Mining

Izabella Kaminska's Just because it's digital doesn't mean it's green reports on:

A new report by the carbon emission think-tank The Shift Project out this week highlights that not much has changed since [2014]. ICT still contributes to about 4 per cent of global greenhouse gas emissions, which is still twice that of civil aviation. What is worse, its contribution is growing more quickly than that of civil aviation.

Cryptocurrency mining is definitely a problem, but how big a part of the problem isn't clear. It could be quite big. Follow me below the fold for some surprising details.

Dr. Pangloss and Data in DNA

Last night I gave a 10-minute talk at the Storage Valley Supper Club, an event much beloved of the good Dr. Pangloss. The title was DNA as a Storage Medium; it was a slightly edited section of The Medium-Term Prospects for Long-Term Storage Systems. Below the fold, an edited text with links to the sources.

Archiving Storage Tiers

Tom Coughlin uses Hetzler's touch-rate metric to argue for tiered storage for archives in a two-part series. Although there's good stuff there, I have two problems with Tom's argument. Below the fold, I discuss them.

Facebook's Warm Storage

Last month I was finally able to post about Facebook's cold storage technology. Now, Subramanian Muralidhar and a team from Facebook, USC and Princeton have a paper at OSDI that describes the warm layer between the two cold storage layers and Haystack, the hot storage layer. f4: Facebook's Warm BLOB Storage System is perhaps less directly aimed at long-term preservation, but the paper is full of interesting information. You should read it, but below the fold I relate some details.

More on Facebook's "Cold Storage"

So far this year I've attended two talks that were really revelatory; Krste Asanović's keynote at FAST 13, which I blogged about earlier, and Kestutis Patiejunas' talk about Facebook's cold storage systems. Unfortunately, Kestutis' talk was off-the-record, so I couldn't blog about it at the time. But he just gave a shorter version at the Library of Congress' Designing Storage Architectures workshop, so now I can blog about this fascinating and important system. Below the fold, the details.

A Challenge to the Storage Industry

I gave a brief talk at the Library of Congress Storage Architecture meeting, pulling together themes from a number of recent blog posts. My goal was twofold:

• to outline the way in which current storage architectures fail to meet the needs of long-term archives,
• and to set out what an architecture that would meet those needs would look like.

Below the fold is an edited text with links to the earlier posts here that I was condensing.

Is This The Dawn of DAWN?

More than three years ago, Ian Adams, Ethan Miller and I were inspired by a 2009 paper FAWN: A Fast Array of Wimpy Nodes from David Andersen et al at C-MU. They showed how a fabric of nodes, each with a small amount of flash memory and a very low-power processor, could process key-value queries as fast as a network of beefy servers using two orders of magnitude less power.

We put forward a storage architecture called DAWN: Durable Array of Wimpy Nodes, similar hardware but optimized for long-term storage. Its advantages were small form factor, durability, and very low running costs. We argued that these would outweigh the price premium for flash over disk. Recent developments are starting to make us look prophetic - details below the fold.

Estimating Storage Costs

Ethan Miller points me to a paper on the cost of storage, How Much Does Storage Really Cost? Towards a Full Cost Accounting Model for Data Storage by Amit Kumar Dutta and Ragib Hasan (DH) of the University of Alabama, Birmingham. Unfortunately, the conference at which it was presented, GECON 2013, is one of those whose proceedings are published in Springer's awful Lecture Notes in Computer Science series, so no link. Below the fold, discussion of the relationship between DH and our on-going work on the economics of long-term storage.

Talk at Digital Preservation 2013

I was on a panel at the Library of Congress' Digital Preservation 2013 meeting entitled Green Bytes: Sustainable Approaches to Digital Stewardship. Below the fold is the text of my brief presentation, with links to the sources.

Facebook's "Cold Storage"

Last week Facebook announced they are building a couple of "cold storage" data centers:

Facebook will move older pictures and back-up photos to new-more energy efficient-data centers, called "cold storage" centers.  ... the new "cold storage" centers-which are still in construction-will be five times more energy efficient and will allow users to access old images anytime without noticing any difference.

Facebook's problem is that they are ingesting 315M photos/day, or 7PB/month. Reducing the energy consumed by the backup copies and the older, less frequently accessed pictures is important. Although this is a work-in-progress and Facebook isn't talking about some details, it appears that among the techniques they are using are erasure coding, to operate with a lower replication factor, aggressively spinning down disks, using flash to hold indexes, and perhaps new, low-power drives such as these from Seagate, which claim 27% less power draw. They are working in the context of the Open Vault project, so this technology should eventually be available to others.

DNA as a storage medium

I blogged last October about a paper from Harvard in Science describing using DNA as a digital storage medium. In a fascinating keynote at IDCC2013 Ewan Birney of EMBL discussed a paper in Nature with a much more comprehensive look at this technology. It has been getting a lot of press, much of it as usual somewhat misleading. Below the fold I delve into the details.

Fujitsu agrees with me

I've been saying for some time that flash memory is unlikely to be the long-term solid state memory technology. In an interview with Chris Mellor at The Register, Fujitsu's CTO agrees with me:

[Joseph Reger] reckons Phase Change Memory (PCM) is the closest, in terms of time to become a usable technology, than other post-flash contenders such as HP's Memristor.

In the more interesting part of the story, he agrees with me that the real potential of these post-flash technologies is that they can be packaged as persistent RAM rather than block storage:

[Reger] asks if everything will be rewritten and re-orchestrated to work with data memory management. Is there effectively only going to be one tier, memory in one form or another?

"Currently, having data in storage means it's not in memory. Is it going to stay like that?" After all, storage was invented to deal with memory-size limitations. If those limitations go away then who needs storage?

Reger said: "I truly believe we are going to have a data orientation rather than memory and storage orientations." But this is really far out in the future.

I'm old enough to remember when computer memory persisted across power cycles because it was magnetic cores. I'd love to see this feature return. The major software change that would be needed is far more than simply using in-memory databases. The RAM data structures would need to be enhanced with metadata and backups, especially for long-term integrity, if we were to get rid of block storage entirely.

Solid State Memory for Archival Use

In last year's JCDL keynote I pointed to work at Carnegie-Mellon on FAWN, the Fast Array of Wimpy Nodes and suggested that the cost savings FAWN realizes by distributing computation across a very large number of very low-power nodes might also apply to storage. Now, Ian Adams and Ethan Miller of UC Santa Cruz's Storage Systems Research Center and I have looked at this possibility more closely in a Technical Report entitled Using Storage Class Memory for Archives with DAWN, a Durable Array of Wimpy Nodes. We show that it is indeed plausible that, even at current flash memory prices, the total cost of ownership over the long term of a storage system built from very low-power system-on-chip technology and flash memory would be competitive with disk. More on this below the fold.

How Green Is Digital Preservation?

At iPRES 2010 I was on a panel chaired by Neil Grindley of JISC entitled How Green is Digital Preservation?. Each of the panelists gave a very brief introduction; below the fold is an edited version of mine.