Internet Archive Storage

The Internet Archive is a remarkable institution, which has become increasingly important during the pandemic. It has been for many years in the world's top 300 Web sites and is currently ranked #209, sustaining almost 60Gb/s outbound bandwidth from its collection of almost half a trillion archived Web pages and much other content. It does this on a budget of under $20M/yr, yet maintains 99.98% availability.

Jonah Edwards, who runs the Core Infrastructure team, gave a presentation on the Internet Archive's storage infrastructure to the Archive's staff. Below the fold, some details and commentary.

Among the highlights:

• 750 servers, some up to 9-years old
• 1,300 VMs
• 30K storage devices
• >20K spinning disks (in paired storage), a mix of 4,8,12,16TB drives, about 40% of the bytes are on 16TB drives.
• almost 200PB of raw storage
• growing the size of the archive >25%/yr.
• adding 10-12PB of raw storage per quarter
• with 16TB drives it would need 15 racks to hold a copy
• currently running ~75 racks
• currently serving about 55GB/s, planning for ~80GB/s soon

Edwards reports that the primary outage causes are:

• Fiber cuts
• power quality issues
• power outages

Going forward, Edwards is asking "whether paired storage the right model?" The current constraints are:

• Items in the archive are directories on disk
• basic unit of storage is the disk
• disks are replicated across datacenters
• content is served from all (=both?) copies

The big issue with treating disk as the unit of paired storage is that when a disk fails a new member of the pair has to be created by reading the whole of the good member and writing the whole of the new member. This takes time, during which the good member is under high load and thus likely to suffer a correlated failure. The new member will be at the start of its life so subject to infant mortality, although it is fair to say that drive manufacturers have paid a lot of attention to reducing infant mortality. Edwards reports that the more recent drives are enough faster than the 8TB drives that the risk is manageable, but as the drives get bigger architectural change will be required to manage this.

Another issue is that the servers in the Archive's racks provide both the storage and the processing needed. The CPUs are getting faster, but not fast enough to keep up with the disks getting denser. More storage per server and per rack also increases the demand for per-rack bandwidth.