"National Hosting" of archives

The LOCKSS team are working with some countries to build in-country Private LOCKSS Networks (PLNs) to preserve the content such as e-journals and e-books that they pay for. Other countries are considering outsourcing their national archive of this content to foreign providers. One of the questions that countries ask about these efforts is "where is the data stored?" Recent developments in the US and the UK mean that this is no longer the right question to ask. Follow me below the fold to find out what the right question has become.

TRAC Audit: Lessons

This is the third in a series of posts about CRL's TRAC audit of the CLOCKSS Archive. Previous posts announced the release of the certification report, and recounted the audit process. Below the fold I look at the lessons we and others can learn from our experiences during the audit.

TRAC Audit: Process

This is the second in a series of posts about CRL's audit of the CLOCKSS Archive. In the first, I announced the release of the certification report. In this one I recount the process of being audited and what we did during it. Follow me below the fold for a long story, but not as long as the audit process.

Update: the third post discussing the lessons to be drawn is here.

Post-Flash Solid State Storage Gets Real-er

HGST announced today that they are demonstrating an SSD that is based on Phase-Change Memory (PCM), one of the technologies competing to take over as flash runs out of steam. The selling point of the SSD is that it is extremely fast:

The demonstration shows unprecedented SSD performance levels that are achieved by utilizing a combination of HGST's new, latency-optimized interface protocols with next-generation non-volatile memory components.
The SSD demonstration utilizes a PCIe interface and delivers three million random read IOs per second of 512 bytes each when operating in a queued environment and a random read access latency of 1.5 microseconds (us) in non-queued settings, delivering results that cannot be achieved with existing SSD architectures and NAND Flash memories. This performance is orders of magnitude faster than existing Flash based SSDs, resulting in a new class of block storage devices. 

The SSD is based on 1Gb PCM chips. The new protocols that are needed to squeeze this performance out of PCIe were described by Dejan Vučinić et al in their paper DC Express: Shortest Latency Protocol for Reading Phase Change Memory over PCI Express at this year's FAST conference.

TRAC Certification of the CLOCKSS Archive

The CLOCKSS Archive is a dark archive of e-journal and e-book content, jointly managed by publishers and libraries, implemented using the LOCKSS technology and operated on behalf of the CLOCKSS not-for-profit by the LOCKSS team at the Stanford Library. For well over a year the LOCKSS team and CLOCKSS management have been preparing for and undergoing the Trustworthy Repositories Audit and Certification (TRAC) process for the CLOCKSS Archive with the Center for Research Libraries (CRL).

CRL just released the Certification Report on the CLOCKSS Archive. I'm happy to report that our work was rewarded with an overall score that equals the previous best, and the first ever perfect score in the "Technologies, Technical Infrastructure, Security" category. We are grateful for this wonderful endorsement of the LOCKSS technology.

In the interests of transparency the LOCKSS team have released all the non-confidential documentation submitted during the audit process. As you will see, there is a lot of it. What you see at the link is not exactly what we submitted. It has been edited to correct errors and obscurities we found during the audit, and to add material from the confidential part of the submission that we decided was not really confidential. These documents will continue to be edited as the underlying reality changes, to keep them up-to-date and satisfy one of the on-going requirements of the certification.

This is just a news item. In the near future I will follow up with posts describing the process of being audited, what we did to make the process work, and the lessons we learned that may be useful for future audits.

Update: the post describing the audit process is here and the post discussing the lessons to be drawn is here.

Coronal Mass Ejections

In my talk What Could Possibly Go Wrong last April I referred to a paper on the 2012 Coronal Mass Ejection (CME) that missed Earth by only nine days:

Most of the information needed to recover from such an event exists only in digital form on magnetic media. These days, most of it probably exists only in "the cloud", which is this happy place immune from the electromagnetic effects of coronal mass ejections and very easy to access after the power grid goes down.

NASA has a post discussing recent research into CMEs which is required reading:

Analysts believe that a direct hit by an extreme CME such as the one that missed Earth in July 2012 could cause widespread power blackouts, disabling everything that plugs into a wall socket.  Most people wouldn't even be able to flush their toilet because urban water supplies largely rely on electric pumps.

An extreme CME called the "Carrington Event" actually did hit the Earth in September 1859:

Intense geomagnetic storms ignited Northern Lights as far south as Cuba and caused global telegraph lines to spark, setting fire to some telegraph offices and thus disabling the "Victorian Internet."

And:

A similar storm today could have a catastrophic effect. According to a study by the National Academy of Sciences, the total economic impact could exceed $2 trillion or 20 times greater than the costs of a Hurricane Katrina. 

Not to worry, because:

In February 2014, physicist Pete Riley of Predictive Science Inc. published a paper in Space Weather entitled "On the probability of occurrence of extreme space weather events."  In it, he analyzed records of solar storms going back 50+ years.  By extrapolating the frequency of ordinary storms to the extreme, he calculated the odds that a Carrington-class storm would hit Earth in the next ten years.

The answer: 12%.

Only 12%. I'd say that CMEs need to be part of the threat model of digital preservation systems.

Discounting the far future

In 2011 Andrew Haldane and Richard Davies of the Bank of England (HD) presented research showing that, when making investment decisions, investors applied discount rates much higher than the prevailing interest rates, and that this gap was increasing through time. One way of looking at their results was as an increase in short-termism; investors were increasingly reluctant to make investments with a long-term payoff. This reluctance clearly has many implications, including making dealing with climate change even more difficult. Their work has influenced our efforts to build an economic model of long-term storage, another area where the benefits accrue over a long period of time.

Now, Stefano Giglio of the Booth School and Matteo Maggiori and Johannes Stroebel of the Stern School (GMS) have a post entitled Discounting the very distant future announcing a paper entitled Very Long-Run Discount Rates. Their work, at first glance, seems to contradict HD. Below the fold, I look into this apparent disagreement.