Storage Roundup

It is time for another roundup of topics in storage that have caught my eye recently. Below the fold I discuss the possible ending of the HAMR saga and various developments in archival storage technology.

Heat-Assisted Magnetic Recording

Unless you have been tracking storage technology for many years, it is hard to appreciate how long the timescales are. My go-to example for communicating this is Seagate's development of HAMR.

Seagate first demonstrated HAMR in 2002. In 2008 they published this graph, predicting HAMR would supplant Perpendicular Magnetic Recording (PMR) starting in 2009.

I first wrote skeptically about projections of HAMR's deployment twelve years ago. Seagate had just demonstrated HAMR at a terabit per square inch and predicted market entry in 2014.

I wrote again in 2013. In 2015 I wrote more about it. Then in 2016 I wrote about it again.

Seagate's 2018 HAMR roadmap

In 2018 I wrote about Chris Mellor's Seagate HAMRs out a roadmap for future hard drive recording tech::

Seagate has set a course to deliver a 48TB disk drive in 2023 using its HAMR (heat-assisted magnetic recording) technology, doubling areal density every 30 months, meaning 100TB could be possible by 2025/26. ... Seagate will introduce its first HAMR drives in 2020. ... a 20TB+ drive will be rolled out in 2020.

So in a decade the technology had gone from next year to the year after next. The year after next Jim Slater wrote HAMR don’t hurt ’em—laser-assisted hard drives are coming in 2020:

Seagate has been trialing 16TB HAMR drives with select customers for more than a year and claims that the trials have proved that its HAMR drives are "plug and play replacements" for traditional CMR drives, requiring no special care and having no particular poor use cases compared to the drives we're all used to.

But no, it would be another four years before we saw the first signs of HAMR drives in the market. In December 2024 Matthew Connatser reported that Seagate launches 32TB Exos M hard drive based on HAMR technology – Mozaic 3+ drives are the world’s first generally available HAMR HDDs:

Seagate’s biggest-ever hard drive is finally here, coming with 32TB of capacity courtesy of the company’s new HAMR technology (via Expreview).

It has almost been a year since Seagate said it had finally made a hard drive based on heat-assisted magnetic recording (HAMR) technology using its new Mozaic 3+ platform.
...
Exos drives based on Mozaic 3+ were initially released to select customers in small quantities, but now the general release is (nearly) here, thanks to mass production.

Note that the drives that are "(nearly) here" are still not available from Amazon, although they are featured on Seagate's web site. Kevin Purdy writes:

Drives based on Seagate's Mozaic 3+ platform, in standard drive sizes, will soon arrive with wider availability than its initial test batches. The driver maker put in a financial filing earlier this month (PDF) that it had completed qualification testing with several large-volume customers, including "a leading cloud service provider," akin to Amazon Web Services, Google Cloud, or the like. Volume shipments are likely soon to follow.

There is no price yet, nor promise of delivery, but you can do some wishful thinking on the product page for the Exos M, where 30 and 32TB capacities are offered. That's 3TB per platter, and up to three times the efficiency per terabyte compared to "typical drives," according to Seagate.

More indications that volume shipments could happen "next year" comes from Chris Mellor's WD’s HAMR switch could be closer than we think:

Intevac has said there is strong interest in its HAMR disk drive platter and head production machinery from a second customer, which could indicate that Western Digital is now involved in HAMR disk developments following Seagate’s move into volume production.

Intevac supplies its 200 Lean thin-film processing machines to hard disk drive media manufacturers, such as Seagate, Showa Denko and Western Digital. It claims more than 65 percent of the world’s HDD production relies on its machinery. The Lean 200 is used to manufacture recording media, disk drive platters, for current perpendicular magnetic recording (PMR) disks.

Intevac’s main customer for HAMR-capable 200 Lean machines is Seagate, which first embarked on its HAMR development in the early 2000s. It is only this year that a prominent cloud service provider has certified Seagate’s Mozaic 3 HAMR drives for general use, more than 20 years after development first started. The lengthy development period has been ascribed to solving difficulties in producing drives with high reliability from high yield manufacturing processes, and Intevac will have been closely involved in ensuring that its 200 Lean machines played their part in this.

Archival Media

Maybe 2025 will be the year I can finally bring my 12-year-long series about HAMR shipment schedules to a close, 23 years after Seagate started work on the technology. Why have I been HAMR-ing on Seagate all these years, and again now? Not to denigrate Seagate's engineering. Getting a HAMR drive into volume production that meets both the incredibly demanding standards for storage media reliability and performance, and the manufacturing yields needed for profit, is an extraordinarily difficult feat. It is not a surprise that it took a couple of decades.

My criticisms have been aimed at the storage industry's marketing and PR, which hypes developments that are still in the lab as if they are going to solve customers' problems "next year". And at the technology press, which took far too long to start expressing skepticism. Seagate's marketing eventually lost all credibility, with their predictions about HAMR becoming an industry joke.

The situation is far worse when it comes to archival media. The canonical article about some development in the lab starts with the famous IDC graph projecting the amount of data that will be generated in the future. It goes on to describe the density some research team achieved by writing say a gigabyte into their favorite medium in the lab. This conveys four false impressions:

• That there is some possibility they could scale their process up to a meaningful fraction of IDC's projected demand, or even to the microscopic fraction of the projected demand that makes sense to archive. There is no such possibility. Archival media is a much smaller market than regular media.
• That the researcher's favorite medium could make it into the market in the timescale of IDC's projections. As we see with HAMR, where research started twenty-six years ago, it won't. Research on the silica medium started fifteen years ago. Research on the DNA medium started thirty-six years ago. Neither is within five years of market entry.
• That even if the researcher's favorite medium did make it into the market it would be a product that consumers could use. As Kestutis Patiejunas figured out more than a decade ago, the only way to make the economics of archival storage work is to do it at data-center scale and harvest the synergies that come from not needing data-center power, cooling, staffing, etc.
• That anyone either cares or even knows what medium their archived data lives on. Only the hyper-scalers do. Consumers believe their data is safe in the cloud. If anyone really cares about archiving they use a service such as Glacier, when they definitely have no idea what medium is being used.

Consumers already have an affordable, durable archival medium. As I have shown:

Surprisingly, with no special storage precautions, generic low-cost media, and consumer drives, I'm getting good data from CD-Rs more than 20 years old, and from DVD-Rs nearly 18 years old.

The market for DVD-R media and drives is gradually dying because they have been supplanted in the non-archival space by streaming, an illustration that consumers really don't care about archiving their data!

DNA Storage

In 2018's DNA's Niche in the Storage Market I imagined myself as the product marketing guy for an attempt to build a rack-scale DNA storage system, and concluded:

Engineers, your challenge is to increase the speed of synthesis by a factor of a quarter of a trillion, while reducing the cost by a factor of fifty trillion, in less than 10 years while spending no more than $24M/yr.

The only viable market for DNA storage is the data-center, and the two critical parameters are still the write bandwidth and the write cost. As far as I'm aware despite the considerable progress in the last 6 years both parameters are still many orders of magnitude short of what a system would have needed back then to enter the market. Worse, the last six years of data center technology development have increased the need for write bandwidth and reduced the target cost. DNA storage is in a Red Queen's Race and it is a long way behind.

Nevertheless, DNA's long-term potential as an archival storage medium justifies continued research. Among recent publications is Parallel molecular data storage by printing epigenetic bits on DNA by Cheng Zhang et al, which avoids the need to synthesize strands of DNA by attaching the bits to prexisting strands. In principle this can be done in parallel. As is traditional, they start by asserting:

The markedly expanding global data-sphere has posed an imminent challenge on large-scale data storage and an urgent need for better storage materials. Inspired by the way genetic information is preserved in nature, DNA has been recently considered a promising biomaterial for digital data storage owing to its extraordinary storage density and durability.

The paper attracted comment from, among others, The Register, Ars Technica and Nature. In each case the commentary included some skepticism. Here are Carina Imburgia and Jeff Nivala from the University of Washington team in Nature:

However, there are still challenges to overcome. For example, epigenetic marks such as methyl groups are not copied by the standard PCR techniques used to replicate DNA, necessitating a more complex strategy to preserve epi-bit information when copying DNA data. The long-term behaviour of the methyl marks (such as their stability) in various conditions is also an open question that requires further study.

Another challenge is that many applications require random access memory (RAM), which enables subsets of data to be retrieved and read from a database. However, in the epi-bit system, the entire database would need to be sequenced to access any subset of the files, which would be inefficient using nanopore sequencing. Moreover, the overall cost of the new system exceeds that of conventional DNA data storage and of digital storage systems, limiting immediate practical applications;

You have to read a long way into the paper to find that:

we stored 269,337 bits including the image of a tiger rubbing from the Han dynasty in ancient China and the coloured picture of a panda ... An automatic liquid handling platform was used to typeset large-scale data at a speed of approximately 40 bits s⁻¹

This is interesting research but the skepticism in the commentaries doesn't exactly convey the difficulty and the time needed to scale from writing less than 40KB in a bit under 2 hours, to the petabyte/month rates (about 2.8TB every 2 hours) Facebook was writing a decade ago. This would be a speed-up of nearly 11 orders of magnitude to compete with decade-old technology.

Diamonds

Chinese boffins find way to use diamonds as super-dense and durable storage medium by Laura Dobberstein reports that:

The research, published in Nature Photonics, highlights that the breakthrough extends beyond density. It is said to offer significant improvements in write times – as little as 200 femtoseconds – and lives up to the promise that "a diamond is forever" by offering millions of years of allegedly maintenance-free storage. Diamonds are highly stable by nature and the the authors have claimed their medium could protect data for 100 years even if kept at 200°C.

High-speed readout is demonstrated with a fidelity of over 99 percent, according to the boffins.

Scientists have been eyeing diamonds as storage devices for a while. Researchers at City College of New York in 2016 claimed to be the first group to demonstrate the viability of using diamond as a platform for superdense memory storage.

These researchers, like so many others in the field, fail to understand that the key to success in archival storage is reducing total system cost. Long-lived but expensive media like diamonds are thus counter-productive.

Project Silica

I wrote about Microsoft's Project Silica last March, in Microsoft's Archival Storage Research. The more I think about this technology, the more I think it probably has the best chance of impacting the market among all the rival archival storage technologies:

• The media is very cheap, so the effect of Kryder's Law economics driving media replacement and thus its economic rather than technical lifetime is minimal.
• The media is quasi-immortal and survives benign neglect, so opex once written is minimal.
• The media is write-once, and the write and read heads are physically separate, so the data cannot be encrypted or erased by malware. The long read latency makes exfiltrating large amounts of data hard.
• The robotics are simple and highly redundant. Any of the shuttles can reach any of the platters. They should be much less troublesome than tape library robotics because, unlike tape, a robot failure only renders a small fraction of the library inaccessible and is easily repaired.
• All the technologies needed are in the market now, the only breakthroughs needed are economic, not technological.
• The team has worked on improving the write bandwidth which is a critical issue for archival storage at scale. They can currently write hundreds of megabytes a second.
• Like Facebook's archival storage technologies, Project Silica enjoys the synergies of data center scale without needing full data center environmental and power resources.
• Like Facebook's technologies, Project Silica has an in-house customer, Azure's archival storage, with a need for a product like this.

The expensive part of the system is the write head, because it uses costly femtosecond lasers. The eventual system's economics will depend upon the progress made in cost-reducing the lasers.