Lessons From Arquivo.pt

Daniel Gomes' video

I'd like to draw your attention to Daniel Gomes excellent video entitled Improving the robustness of the Arquivo.pt web archive.

Arquivo.pt is the Portuguese Web Archive. It got started in 2007, and in 2010 was an early archive to support full-text search. In 2013 it suffered a hardware malfunction that took the service down and lost 17% of its content. This led to a complete re-think of the system architecture, implementation, and operations. Daniel describes this process and the encouraging results in detail. It is well worth the 20 minutes to watch it.

Daniel divides the re-think into 5 major sections:

1. Hardware and software architecture shifted to shared-nothing
2. Reinforced replication policies
3. Monitor the service
4. Quality assurance for software development
5. Document and test procedures

I'd agree with all these points. Many of the details correspond to things the LOCKSS Program focused on during preparation for the TRAC audit of the CLOCKSS Archive in 2014. This is especially the case for the last of Daniel's sections; the audit forced us to document our processes, which forced us to think about whether they were actually achieving their goals, which led to the discovery that in a number of cases they weren't.