Using the official Linux overlayfs

I realize that it may not be obvious exactly how to use the newly-official Linux overlayfs implementation. Below the fold, some example shell scripts that may help clarify things.

An overlayfs mount involves four directories:

• below, a directory that, accessed via the mount, will be read-only.
• above, a directory that, accessed via the mount, will be read-write.
• work, a read-write directory in the same file system as the above directory.
• mount, the directory at which the union of the entries in below and above will appear.

Lets create an example of each of these directories:

% mkdir below above work mount
%

Now we populate below and above with some files:

% for A in one two three
> do
>     echo "Content of below/${A}" >below/${A}
> done
% chmod a-w below/* below
% for A in four five six
> do
>     echo "Content of above/${A}" >above/${A}
> done
% ls -la below above work mount
above:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six

below:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

mount:
total 8
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..

work:
total 8
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
%

Now we create the overlay mount and see what happens:

% OPTS="-o lowerdir=below,upperdir=above,workdir=work"
% sudo mount -t overlay ${OPTS} overlay mount
% ls -la below above work mount
above:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six

below:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

mount:
total 32
drwxr-xr-x 1 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

work:
total 12
drwxr-xr-x 3 pi   pi   4096 Feb 18 10:42 .
drwxr-xr-x 6 pi   pi   4096 Feb 18 10:42 ..
d--------- 2 root root 4096 Feb 18 10:42 work
%

The union of the files in the below and above directories has appeared in the mount directory. Now we add a file to the mount directory:

% echo "Content of mount/seven" >mount/seven
% ls -la below above work mount
above:
total 24
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six

below:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

mount:
total 36
drwxr-xr-x 1 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

work:
total 12
drwxr-xr-x 3 pi   pi   4096 Feb 18 10:42 .
drwxr-xr-x 6 pi   pi   4096 Feb 18 10:42 ..
d--------- 2 root root 4096 Feb 18 10:42 work
%

The file seven has appeared in the mount directory, and also in the above directory. They have the same content:

% cat mount/seven
Content of mount/seven
% cat above/seven
Content of mount/seven
%

Now we write to a file that is in the below directory:

% cat mount/two
Content of below/two
% echo "New content of mount/two" >mount/two
% cat mount/two
New content of mount/two
% ls -la below above work mount
above:
total 28
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six
-rw-r--r-- 1 pi pi   25 Feb 18 10:42 two

below:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

mount:
total 36
drwxr-xr-x 1 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   25 Feb 18 10:42 two

work: 
total 12
drwxr-xr-x 3 pi   pi   4096 Feb 18 10:42 .
drwxr-xr-x 6 pi   pi   4096 Feb 18 10:42 ..
d--------- 2 root root 4096 Feb 18 10:42 work
%

A file two has appeared in the above directory which, when viewed through the overlay mount, obscures the file two in the below directory, which is still there with its original content:

% cat above/two
New content of mount/two
% cat below/two
Content of below/two
%

Now we remove a file from overlay mount directory that is in the below directory:

% rm mount/three
% ls -la below above work mount
above:
total 28
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six
c--------- 1 pi pi 0, 0 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   25 Feb 18 10:42 two

below:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

mount:
total 32
drwxr-xr-x 1 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six
-rw-r--r-- 1 pi pi   25 Feb 18 10:42 two

work: 
total 12
drwxr-xr-x 3 pi   pi   4096 Feb 18 10:42 .
drwxr-xr-x 6 pi   pi   4096 Feb 18 10:42 ..
d--------- 2 root root 4096 Feb 18 10:42 work
%

The file three is still in the below directory, but a character special file three has appeared in the above directory that makes the file in the below directory inaccessible, a "whiteout". Now we remove the file in the below directory that we wrote to earlier:

% rm mount/two
% ls -la below above work mount
above:
total 24
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six
c--------- 1 pi pi 0, 0 Feb 18 10:42 three
c--------- 1 pi pi 0, 0 Feb 18 10:42 two

below:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

mount:
total 28
drwxr-xr-x 1 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six

work:
total 12
drwxr-xr-x 3 pi   pi   4096 Feb 18 10:42 .
drwxr-xr-x 6 pi   pi   4096 Feb 18 10:42 ..
d--------- 2 root root 4096 Feb 18 10:42 work
%

The file two in the above directory has been replaced with a whiteout. Now we undo the overlayfs mount:

% sudo umount mount
% ls -la below above work mount
above:
total 24
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six
c--------- 1 pi pi 0, 0 Feb 18 10:42 three
c--------- 1 pi pi 0, 0 Feb 18 10:42 two

below:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

mount:
total 8
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..

work:
total 12
drwxr-xr-x 3 pi   pi   4096 Feb 18 10:42 .
drwxr-xr-x 6 pi   pi   4096 Feb 18 10:42 ..
d--------- 2 root root 4096 Feb 18 10:42 work
% cat below/two
Content of below/two
% cat above/seven
Content of mount/seven
%

The content of the read-only below directory is, as expected, unchanged. The data that was written to the overlay mount and not removed remains in the above directory. The data that was written to the overlay mount and subsequently removed is gone. The whiteouts remain.

Now we re-assemble the mount:

% sudo mount -t overlay ${OPTS} overlay mount
% ls -la below above work mount
above:
total 24
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six
c--------- 1 pi pi 0, 0 Feb 18 10:42 three
c--------- 1 pi pi 0, 0 Feb 18 10:42 two

below:
total 20
drwxr-xr-x 2 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 three
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 two

mount:
total 28
drwxr-xr-x 1 pi pi 4096 Feb 18 10:42 .
drwxr-xr-x 6 pi pi 4096 Feb 18 10:42 ..
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 five
-rw-r--r-- 1 pi pi   22 Feb 18 10:42 four
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 one
-rw-r--r-- 1 pi pi   23 Feb 18 10:42 seven
-rw-r--r-- 1 pi pi   21 Feb 18 10:42 six 

work:
total 12
drwxr-xr-x 3 pi   pi   4096 Feb 18 10:42 .
drwxr-xr-x 6 pi   pi   4096 Feb 18 10:42 ..
d--------- 2 root root 4096 Feb 18 10:42 work
%

The state of the mount returns.