I agree that "disk, tape and other media are remar...

2008-01-21T07:03:00.000-08:00

I agree that "disk, tape and other media are remarkably reliable". Disk and tape drives contain many layers of astonishing engineering, from the medium though the heads and the signal processing that extracts bits from the noisy analog signal, and the error correcting codes that clean up the bit stream. But these are nowhere near good enough to meet the demands society has for data preservation. I'm not the right person to explain these layers.

So, on top of the actual storage media we have to layer file systems and digital preservation systems to make up for their (measurable) unreliability. They increase the reliability of the bits by another large factor. The problem is not in our ability to continue to add more and more error detection and correction capabilities to the pile. I have worked in these layers off and on for many years.

There are three problems. The first is that, perhaps because they are in awe of the astonishing engineering, very few people are measuring the reliability their pile is delivering. The second is that the scanty evidence we have is that the pile is failing to deliver the reliability we would expect. The third is that, even if the pile was delivering the reliability we need, we could not perform the experiments needed to prove that it was doing so.

So, yes, it is amazing that we can store bits so reliably. But in the context of digital preservation, patting ourselves on the back about this achievement is counter-productive.

Hi David, I think you somewhat misrepresent my com...

2008-01-21T02:46:00.000-08:00

Hi David, I think you somewhat misrepresent my comment on the other post. I was ONLY addressing the "bit half life 100M times the age of the universe" factoid. For the record, I do believe that keeping a petabyte for a century is hard (I know someone who is trying to do it), and I accept many of your other evidence and arguments. However, it's clear that we can keep data which represents bit half lives >> the age of the universe, even after only 60 years of trying (I don't think I quite emphasised how hard a GB for a decade would have seemed in 1968, when I first met a 8 MB disk drive the size of a fridge).

I would like to understand HOW we can achieve such miraculous-seeming results; I guess it's through distributing checks widely across many layers and many hardware devices.

Comments on DSHR's Blog: How Hard Is "A Petabyte for a Century"?

I agree that "disk, tape and other media are remar...

Hi David, I think you somewhat misrepresent my com...