More on Facebook's "Cold Storage"

So far this year I've attended two talks that were really revelatory; Krste Asanović's keynote at FAST 13, which I blogged about earlier, and Kestutis Patiejunas' talk about Facebook's cold storage systems. Unfortunately, Kestutis' talk was off-the-record, so I couldn't blog about it at the time. But he just gave a shorter version at the Library of Congress' Designing Storage Architectures workshop, so now I can blog about this fascinating and important system. Below the fold, the details.

Plenary Talk at 3rd EUDAT Conference

I gave a plenary talk at the 3^rd EUDAT Conference's session on sustainability entitled Economic Sustainability of Digital Preservation. Below the fold is an edited text with links to the sources.

A Challenge to the Storage Industry

I gave a brief talk at the Library of Congress Storage Architecture meeting, pulling together themes from a number of recent blog posts. My goal was twofold:

• to outline the way in which current storage architectures fail to meet the needs of long-term archives,
• and to set out what an architecture that would meet those needs would look like.

Below the fold is an edited text with links to the earlier posts here that I was condensing.

Three Good Reads

Below the fold I'd like to draw your attention to two papers and a post worth reading.

Utah State Archives has a problem

A recent discussion on the NDSA mailing list featured discussion about the Utah State Archives struggling with the costs of being forced to use Utah's state IT infrastructure for preservation. Below the fold, some quick comments.

Two Sidelights on Short-Termism

I've often referred to the empirical work of Haldane & Davies and the theoretical work of Farmer and Geanakoplos, both of which suggest that investors using Discounted Cash Flow (DCF) to decide whether an investment now is justified by returns in the future are likely to undervalue the future. This is a big problem in areas, such as climate change and digital preservation, where the future is some way off.

Now Harvard's Greenwood & Shleifer, in a paper entitled Expectations of Returns and Expected Returns, reinforce this:

We analyze time-series of investor expectations of future stock market returns from six data sources between 1963 and 2011. The six measures of expectations are highly positively correlated with each other, as well as with past stock returns and with the level of the stock market. However, investor expectations are strongly negatively correlated with model-based expected returns.

They compare investors' beliefs about the future of the stock market as reported in various opinion surveys, with the outputs of various models used by economists to predict the future based on current information about stocks. They find that when these models, all enhancements to DCF of one kind or another, predict low performance investors expect high performance, and vice versa. If they have experienced poor recent performance and see a low market, they expect this to continue and are unwilling to invest. If they see good recent performance and a high market they expect this to continue. Their expected return from investment will be systematically too high, or in other words they will suffer from short-termism.

Yves Smith at Naked Capitalism has a post worth reading critiquing a Washington Post article entitled America’s top execs seem ready to give up on U.S. workers. It reports on a Harvard Business School survey of its graduates entitled An Economy Doing Half Its Job. Yves writes:

In the early 2000s, we heard regularly from contacts at McKinsey that their clients had become so short-sighted that it was virtually impossible to get investments of any sort approved, even ones that on paper were no-brainers. Why? Any investment still has an expense component, meaning some costs will be reported as expenses on the income statement, as opposed to capitalized on the balance sheet. Companies were so loath to do anything that might blemish their quarterly earnings that they’d shun even remarkably attractive projects out of an antipathy for even a short-term lowering of quarterly profits.

Note "Companies were so loath". The usually careful Yves falls into the common confusion between companies (institutions) and their managers (individuals). Managers evaluate investments not in terms of their longer-term return to the company, but in terms of their short-term effect on the stock price, and thus on their stock-based compensation. Its the IBGYBG (I'll Be Gone, You'll Be Gone) phenomenon, which amplifies the underlying problems of short-termism.

Two Brief Updates

A couple of brief updates on topics I've been covering, Amazon's margins and the future of flash memory.

Interesting report on Digital Legal Deposit

Last month the International Publishers Association (IPA) put out an interesting report about the state of digital legal deposit for copyright purposes, with extended status reports from the national libraries of Germany, the Netherlands, the UK, France and Italy, and short reports from many other countries. The IPA's conclusions echo some themes I have mentioned before:

• "It is clear that the more voluntary a digital legal deposit scheme is at the outset, the better."
• "The best schemes are those where an emphasis has been put on publishers and librarians collaborating to address key concerns"

My reason for saying these things is based on experience. It shows that, no matter what the law says, if the publishers don't want you to collect their stuff, you will have a very hard time collecting it. On-line publishers need to have robust defenses against theft, which even national libraries would have difficulty overcoming without the publishers' cooperation.

The publishers' reason for saying these things is different. What are the publishers' "key concerns" on which voluntary collaboration is needed?

• "copyright protection, digital security and monitored access"
• "clear, mutually agreed and flexible rules on access which protect publishers' normal exploitation; who is authorized to use deposited material, where they can access it and what they can lawfully do with it."

In other words, they are happy to deposit their content only under conditions that make it almost useless, such as that it only be accessible to one reader at a time physically at the library, just like a paper book.

Given that the finances of many national libraries are in dire straits, the publishers have a helpful suggestion:

"Countries might usefully consider other models, such as larger publishers self-archiving material, agreeing to make it available on request to libraries."

Or, in other words, lets just forget the whole idea of legal deposit.

Note: everything in quotes is from the report, emphasis in the original.