Understanding OverlayFS: The Union Filesystem Behind Every Docker Container

1. Why every container sits on a union filesystem

A Docker image consists of several read only layers stacked on top of each other, and every running container additionally gets its own writable layer on top of that stack. For applications inside the container to see this as one perfectly ordinary, unified filesystem, a union filesystem transparently merges several directory trees into a single view.

OverlayFS has been part of the mainline Linux kernel since version 3.18 and has become the standard storage driver for Docker and containerd, since it is considerably simpler to implement than older approaches like AUFS and is maintained directly in the mainline kernel. Understanding how OverlayFS works internally also explains why containers start so quickly, and why certain workloads inside containers can be surprisingly slow.

2. The concept: lowerdir, upperdir, and workdir

OverlayFS combines at least two directories into a single view: lowerdir holds one or more read only layers, while upperdir is the single writable layer that every actual change gets written to. From the perspective of a process entering the combined mount point, only one merged file tree exists, in which files from upper layers shadow same named files from lower layers.

The third required directory, workdir, is purely internal scratch space the kernel uses for atomic operations such as renaming files and must never be written to manually or used for anything else. It has to live on the same filesystem as upperdir, because OverlayFS relies internally on rename system calls for certain operations that are only atomic within the same filesystem.

# Prepare the directory structure for a manual OverlayFS mount
mkdir -p /overlay/lower /overlay/upper /overlay/work /overlay/merged

# Mount OverlayFS with a single lower layer
mount -t overlay overlay \
  -o lowerdir=/overlay/lower,upperdir=/overlay/upper,workdir=/overlay/work \
  /overlay/merged

3. How Docker builds image layers on top of it

Every instruction in a Dockerfile that changes the filesystem state typically produces its own read only layer, stored as its own directory in the storage backend. When starting a container, Docker stacks every layer of an image as a lowerdir chain, with the order determined by creation time and dependency chain, and creates a fresh, empty layer as upperdir for the container itself.

The overlay2 storage driver has supported multiple lowerdir entries at once since kernel 4.0, which removed the earlier two layer limitation of the original OverlayFS design and explains why modern Docker installations handle images with many layers without noticeable performance loss.

# Inspect the layers of a running container at the host level
docker inspect --format '{{ .GraphDriver.Data.LowerDir }}' mycontainer
docker inspect --format '{{ .GraphDriver.Data.UpperDir }}' mycontainer

# Check the storage driver and backing filesystem of the Docker daemon
docker info --format '{{ .Driver }}'

4. Mounting and testing OverlayFS manually

For troubleshooting and understanding, a manual mount outside of Docker is worthwhile, since it makes it possible to trace, in isolation, which file from which layer is actually visible in the merged directory. Changes made inside the merged directory land exclusively in upperdir, while lowerdir stays completely untouched, even if a file appears to have been overwritten there.

When a file from a lower layer is deleted inside the merged directory, OverlayFS creates a so called whiteout file in upperdir, a special character device with major and minor numbers of 0, which tells the kernel to ignore the same named file from the lower layers when assembling the view, without the original file in the lower layer actually being deleted.

# Example: create a file only in the lower layer
echo "original" > /overlay/lower/config.txt

# Delete it in the merged directory and inspect the whiteout file
rm /overlay/merged/config.txt
ls -la /overlay/upper/config.txt
stat /overlay/upper/config.txt | grep 'character special'

5. The copy up mechanism and what it costs

As soon as a file that only exists in a lower layer needs to be modified inside the merged directory, the kernel first copies it fully into upperdir before the actual write operation executes. This process is called copy up and is barely noticeable for small configuration files, but becomes a real source of latency for large files such as database files or log archives in the double digit megabyte or gigabyte range, since even changing a single byte copies the entire file.

For workloads that frequently modify large files inside a container, for example a database running directly on the container filesystem instead of a mounted volume, this exact effect produces unexpected latency spikes on the first write after container startup. The standard recommendation is therefore to always keep mutable, write heavy data in bind mounts or named volumes outside the OverlayFS layer stack.

6. Multi layer lowerdir stacks in detail

With multiple lowerdir entries, they are specified separated by colons, and order matters: the first path listed has the highest priority and shadows same named files from every following path. Docker relies on this property to stack layers in the correct historical order, so a layer created later in the Dockerfile always takes precedence over an earlier one.

For deep layer stacks with many levels, as they appear in images built from dozens of Dockerfile instructions, the number of lookups needed to open a file grows linearly with the number of layers, since the kernel may in the worst case have to search every layer from top to bottom until the file is found. In practice this effect stays negligible for typical image sizes, but can become measurable with an extreme number of layers.

# Combine multiple lower layers, the first one has highest priority
mount -t overlay overlay \
  -o lowerdir=/overlay/layer3:/overlay/layer2:/overlay/layer1,\
upperdir=/overlay/upper,workdir=/overlay/work \
  /overlay/merged

7. Performance characteristics compared to the native filesystem

Read access to files already residing in upperdir, or read unchanged from the topmost matching lowerdir layer, is nearly as fast as direct access to the underlying native filesystem, since OverlayFS needs no extra copy operation for that case. The noticeable overhead comes almost exclusively from the copy up mechanism mentioned above, plus metadata operations that span many layers.

For database workloads with many small, randomly distributed writes, a bind mount or volume is generally the better choice over the OverlayFS layer stack, since neither copy up nor whiteout bookkeeping occurs there, and the I/O path is passed straight through to the host's native filesystem.

8. Best practices for running container hosts

On the host, a regular docker system prune or the equivalent for the relevant container runtime is worth running, since unused layers otherwise permanently occupy space in the storage backend, even once no container references them anymore. Production hosts also benefit from a dedicated filesystem for the Docker storage directory, so a filled up layer stack does not endanger the entire root filesystem and the system's operation along with it.

Images should be kept deliberately lean, for example through multi stage builds that leave build dependencies in a discarded intermediate layer and only carry the actually needed artifacts into the final layer stack. Fewer and smaller layers mean fewer lookups on file access and overall faster container startup, which becomes noticeable especially with frequent deployments.

9. Common pitfalls in production operation

The most frequent trap is storing mutable application data directly on the container filesystem instead of in a volume, which not only causes the copy up latency already mentioned, but also means all that data is lost irrecoverably once the container is removed, since upperdir gets deleted along with the container.

A second trap involves filesystem limits: since OverlayFS potentially creates a full copy of every changed file in upperdir, a container that modifies many large files can unexpectedly hit the host filesystem's inode or space limits, even when the Docker image itself looks comparatively small. Nesting OverlayFS on top of OverlayFS, for example in certain Docker in Docker scenarios, also causes compatibility issues depending on kernel version and should be avoided.

10. Summary

11. FAQ: OverlayFS