Talks at the Library of Congress Storage Architecture Meeting

Slides from the talks at last September's Library of Congress Storage Architecture meeting are now on-line. Below the fold, links to and commentary on three of them.

Fontana's 2015 analysis

Robert Fontana updated his invaluable survey of storage media trends with 2015 numbers and more detailed discussion. You need to go read the whole thing; extracts cannot do it justice.

Many of the conclusions he drew are similar to those from my post The Future of Storage and earlier posts:

• The Kryder rates for tape, hard disk and NAND flash are similar, and in the 20%/yr range. The days of 40%/yr are gone for good.
• The impact of this change on the costs of storing data for the long haul has yet to sink in. As Fontana says "Storage is more valuable, less replaceable, and must be reliable for longer time periods".
• No medium's $/GB is outpacing the others by a huge margin, although over time flash is gaining ground.

• 

  Fontana EB shipped

  Total Exabytes shipped is increasing linearly, not exponentially, at around 77EB/yr. Storage is not undergoing Mind-Boggling Growth; the IDC numbers for "data generated" have nothing to do with storage demand.
• Total revenue is about flat, now with more of the dollars going to flash and less to hard disk.
• Last year flash shipped 83EB and hard disk shipped 565EB. For flash to displace hard disk immediately would need 32 new state-of-the-art fabs at around $9B each or nearly $300B in total investment. So not going to happen.
• But over the next 4 years Fontana projects NAND flash shipments will grow to 400EB/yr versus hard disk shipments perhaps 800EB/yr. So there will be continued gradual erosion of hard disk market share.

I've long admired the work of Kestutis Patiejunas on Facebook's long-term storage systems. He and Sam Merat presented An Optical Journey: Building the largest optical archival data storage system at Facebook. They described the collaboration between Facebook and Panasonic to get the prototype optical storage system that attracted attention when it was announced in 2014 into production. They deployed 10s of Petabytes of 100GB BluRay disks and found a disk failure rate of 0.015%. In 2017 they expect to deploy 100s of PB of a second-generation system with 300GB disks, and in 2018 to achieve Exabyte scale with a third generation using 500GB disks. Panasonic showed the production hardware at the meeting.

I've consistently been skeptical of the medium-term prospects for DNA storage, as in my post debunking Nature's reporting on a paper from Microsoft. Karin Strauss and Luis Ceze, from the team behind that paper, presented A DNA-Based Archival Storage System. Despite my skepticism, I believe this team is doing very important work. The reason is the same as why Facebook's work on optical storage is interesting; it is the system aspects not the media themselves that are important.

The Microsoft team are researching what a DNA-based storage system would look like, not just trying to demonstrate the feasibility of storing data in DNA. For example, they discuss how data might be encoded in DNA to permit random access. Although this is useful research, the fact remains that DNA data storage requires a reduction in relative synthesis cost of at least 6 orders of magnitude over the next decade to be competitive with conventional media, and that currently the relative write cost is increasing, not decreasing.