The British Parliament is celebrating the 800th anniversary of Magna Carta:
On Thursday 5 February 2015, the four surviving original copies of Magna Carta were displayed in the Houses of Parliament – bringing together the documents that established the principle of the rule of law in the place where law is made in the UK today.The closing speech of the ceremony in the House of Lords was given by Sir Tim Berners-Lee, who is reported to have said:
I invented the acronym LOCKSS more than a decade and a half ago. Thank you, Sir Tim!
On October 24, 2014 Linus Torvalds added overlayfs to release 3.18 of the Linux kernel. Various Linux distributions have implemented various versions of overlayfs for some time, but now it is an official part of Linux. Overlayfs is a simplified implementation of union mounts, which allow a set of file systems to be superimposed on a single mount point. This is useful in many ways, for example to make a read-only file system such as a CD-ROM appear to be writable by mounting a read-write file system "on top" of it.
Other Unix-like systems have had union mounts for a long time. BSD systems first implemented it in 4.4BSD-Lite two decades ago. The concept traces back five years earlier to my paper for the Summer 1990 USENIX Conference Evolving the Vnode Interface which describes a prototype implementation of "stackable vnodes". Among other things, it could implement union mounts as shown in the paper's Figure 10:
About five years later Erez Zadok at Stony Brook led the FiST project, a major development of stackable file systems including two successive major releases of unionfs, a unioning file system for Linux.
About the same time I tried to use OpenBSD's implementation of union mounts early in the boot sequence to construct the root directory by mounting a RAM file system over a read-only root file system on a CD, but gave up on encountering deadlocks.
In 2009 Valerie Aurora published a truly excellent series of articles going into great detail about the difficult architectural and implementation issues that arise when implementing union mounts in Unix kernels. It includes the following statement, with which I concur:
The consensus at the 2009 Linux file systems workshop was that stackable file systems are conceptually elegant, but difficult or impossible to implement in a maintainable manner with the current VFS structure. My own experience writing a stacked file system (an in-kernel chunkfs prototype) leads me to agree with these criticisms.Note that my original paper was only incidentally about union mounts, it was a critique of the then-current VFS structure, and a suggestion that stackable vnodes might be a better way to go. It was such a seductive suggestion that it took nearly two decades to refute it! My apologies for pointing down a blind alley.
The overlayfs implementation in 3.18 is minimal:
Overlayfs allows one, usually read-write, directory tree to be overlaid onto another, read-only directory tree. All modifications go to the upper, writable layer.But given the architectural issues doing one thing really well has a lot to recommend itself over doing many things fairly well. This is, after all, the use case from my paper.
It took a quarter of a century, but the idea has finally been accepted. And, even though I had to build a custom 3.18 kernel to do so, I am using it on a Raspberry Pi serving as part of the CLOCKSS Archive.
Thank you, Linus! And everyone else who worked on the idea during all that time!
References (date order):
- David Hendricks, The Translucent File Service, pp. 87-93, Proceedings of the Autumn 1988 EUUG Conference, Vienna, Austria, October 1988.
- David S. H. Rosenthal, Evolving the Vnode Interface, Proceedings of the Summer 1990 USENIX Conference, Anaheim, 1990.
- Rob Pike, Dave Presotto, Sean Dorward, Bob Flandrena, Ken Thompson, Howard Trickey & Phil Winterbottom, Plan 9 from Bell Labs, Computing Systems Vol. 8, No. 3, Summer 1995.
- Jan-Simon Pendry & Marshall Kirk McKusick, Union Mounts in 4.4BSD-Lite. Proceedings of the USENIX Technical Conference on UNIX and Advanced Computing Systems: pp. 25–33 December 1995.
- David S. H. Rosenthal, A Digital Preservation Network Appliance Based on OpenBSD, BSDCon, 2003.
- Charles P. Wright, Jay Dave, Puja Gupta, Harikesavan Krishnan, Erez Zadok, and Mohammad Nayyer Zubair, Versatility and Unix Semantics in a Fan-Out Unification File System, Stony Brook University Technical Report FSL-04-01b, November 2004.
- Valerie Aurora, Unioning file systems: Architecture, features, and design choices, lwn.net, March 2009
- Valerie Aurora, Union file systems: Implementations, part I. lwn.net, March 2009.
- Valeria Aurora, Unioning file systems: Implementations, part 2. lwn.net April 2009.
Kirk McKusick's fascinating FAST15 keynote A Brief History of the BSD Fast File System corrects my memory. BSD acquired stacking vnodes in 1987, according to Kirk after discussions with me. I think Steve Kleiman was responsible for much of the content of those discussions. For sure, Kirk's implementation of stacking in BSD was much cleaner than my prototype in SunOS.
ReplyDeleteTim Anderson reports that systemd has taken up overlayfs:
ReplyDelete"As developer Lennart Poettering explained: "When a system extension image is activated, its /usr/ and /opt/ hierarchies and os-release information are combined via overlayfs with the file system hierarchy of the host OS."
The primary use case for system extension images is for immutable operating systems like Red Hat's Silverblue and Kinoite. In these OSes, the file system is read-only and is updated by replacing it with a new image rather than being patched, which is better both for security and stability.
It does cause compatibility issues for applications that need updated system files, and is difficult for developers who need more flexibility. Typically, these problems are overcome by running virtual machines or containers, but system extension images let users and developers update or add system files without actually modifying the immutable file system."