More on de-duplicating flash controllers

My ACM Queue piece describing the problems caused by storage devices invisibly doing de-duplication attracted the attention of Robin Harris who actually asked SandForce and other manufacturers to comment. The details of SandForce's response are in Robin's StorageMojo article but the key claims are:

There is no more likelihood of DuraWrite loosing data than if it was not present.

and:

That is why SandForce created RAISE (Redundant Array of Independent Silicon Elements) and includes it on every SSD that uses a SandForce SSD Processor. ... if the ECC engine is unable to correct the bit error RAISE will step in to correct a complete failure of an entire sector, page, or block. ...This combination of ECC and RAISE protection provides a resulting UBER of 10^-29 virtually eliminates the probabilities of data corruption.

I would regard both claims with considerable skepticism:

• SandForce has not disclosed the details of their technology, but the research performed by Michael Wei and his co-authors at UCSD revealed that it definitely includes de-duplication. Thus the first claim is not credible; there is only a single copy of the supposedly redundant data in the flash array instead of multiple copies. The metadata must therefore be at higher risk.
• As regard the second claim, in my earlier ACM Queue article I show that manufacturers claims of error rates such as 10^-29 are not credible because they are not the result of experiments, but of models which are unrealistic and unverified.

Library of Congress interview

An interview with me inaugurates an interview series called Insights on the Library of Congress' digital preservation blog The Signal.