Solid State Memory for Archival Use

In last year's JCDL keynote I pointed to work at Carnegie-Mellon on FAWN, the Fast Array of Wimpy Nodes and suggested that the cost savings FAWN realizes by distributing computation across a very large number of very low-power nodes might also apply to storage. Now, Ian Adams and Ethan Miller of UC Santa Cruz's Storage Systems Research Center and I have looked at this possibility more closely in a Technical Report entitled Using Storage Class Memory for Archives with DAWN, a Durable Array of Wimpy Nodes. We show that it is indeed plausible that, even at current flash memory prices, the total cost of ownership over the long term of a storage system built from very low-power system-on-chip technology and flash memory would be competitive with disk. More on this below the fold.


We are not alone in thinking that the coming disruption of the storage industry may end up with solid state having a big chunk of the market. A Silicon Valley startup called Pure Storage is just emerging from stealth mode. They, and an impressive list of investors, have bet over $25M on the same prediction.

As I've written, its likely that the solid state storage technology that ends up winning will not be flash. Both the leading contenders to replace it had good news recently. An editorial introducing a recent special issue of Nanotechnology describes much research on phase change memory (PCM), and results from HP's work to understand the details of the mechanism underlying memristors, which has been somewhat mysterious:

Researchers at HP labs in the US have made notable progress in this field, and among the work reported in this special issue they describe means to control the switch resistance and show that limiting the current during electroforming leads to the coexistence of two resistance switching modes in TiO2 memristive devices. They also present spectromicroscopic observations and modelling results for the Joule heating during switching, providing insights into the ON/OFF switching process.
Non-volatile memory based on nanostructures, Kalinin, Yang and Demming

The mergers that seemed likely to leave us with two disk drive suppliers have been put on hold for 90 days by the EU's competition watchdog. My take is that this is unlikely to result in significantly more competition in the disk drive market, so I'm still predicting a violation of Kryder's Law, which will help drive adoption of solid state storage.

The lower total cost of ownership of solid state memory requires paying higher initial capital cost to achieve lower running costs through time. An extreme version of this is the endowment model of paying for storage I discussed at the Personal Digital Archiving conference; one initial payment provides for indefinite storage of the data. Solid state storage is thus a natural fit with the endowment model. However, there is bad news on this front. As Serge Goldstein and I agreed in comments, the endowment model faces a market perception problem; the markup that is needed over the raw cost of storage is perceived as excessive. In a recent talk by Andrew Haldane and Richard Davies of the Bank of England they say:

Our evidence suggests that short-termism is both statistically and economically significant in capital markets. It also appears to be rising. In the UK and the US, cash-flows 5 years ahead are discounted at rates more appropriate 8 or more years hence; 10 year ahead cash-flows are valued as if 16 or more years ahead; and cash-flows more than 30 years ahead are scarcely valued at all.

In other words, decisions are based on unrealistically high discount rates, which makes spending more now to save money later hard to justify. This depressing result has implications not just for the success of solid state storage, but for the entire field of digital preservation.