Distributed Filesystems Overview: GlusterFS, Ceph and NFS Compared

1. Why distributed storage is needed for HA clusters

A distributed filesystem solves a problem that pure application redundancy alone cannot solve: when multiple servers need to see the same data, or data loss on node failure must be prevented, local storage on a single machine is no longer sufficient. Without a distributed filesystem, every application server would need to keep its own copy of the data, immediately leading to inconsistencies between nodes for uploads, sessions or generated content.

In high availability scenarios with Pacemaker and Corosync, a distributed filesystem or at least a shared block device is almost always part of the resource group, because a database or application server must continue working with the same data state left behind by the previous active node after a failover. The choice of the right distributed filesystem directly influences how fast a failover proceeds and how robust the overall system is against network issues.

The landscape of distributed filesystems on Linux ranges from simple NFS through block based replication with DRBD to full cluster filesystems like GlusterFS and object based systems like Ceph. Each of these distributed filesystem solutions makes different tradeoffs between simplicity, performance and actual failure resistance, compared in detail in the following sections.

2. NFS: the simple entry point with limits

Network File System, or NFS, is the oldest and most widely used method to expose a directory from a central server over the network to multiple clients. For many simple shared storage cases, NFS is the most pragmatic choice, since it is already built into every Linux kernel and requires no additional software on the client side.

The decisive drawback of NFS in an HA context: the NFS server itself is a single point of failure. If it fails, all clients simultaneously lose access to their data, regardless of how many redundant application servers sit behind it. For true high availability, the NFS server itself must in turn be made redundant, for example through a Pacemaker managed block device underneath, turning NFS more into an access layer than the actual redundancy solution.

# NFS server: export a shared directory
sudo apt install -y nfs-kernel-server
echo "/srv/shared 10.0.0.0/24(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports
sudo exportfs -ra
sudo systemctl restart nfs-kernel-server

# NFS client: mount the shared directory
sudo apt install -y nfs-common
sudo mount -t nfs 10.0.0.5:/srv/shared /mnt/shared

# Persistent mount in /etc/fstab
echo "10.0.0.5:/srv/shared /mnt/shared nfs defaults,_netdev 0 0" | sudo tee -a /etc/fstab

3. DRBD: block replication between two nodes

Distributed Replicated Block Device, or DRBD, works fundamentally differently from NFS: instead of sharing files over the network, DRBD mirrors an entire block device in real time to a second node, comparable to a network based RAID 1. A normal filesystem such as ext4 or XFS sits on top of the replicated block device, mounted only on the currently active node.

DRBD is the classic storage foundation for two node Pacemaker clusters, since it integrates seamlessly as a cluster resource, and on a failover the filesystem is immediately available on the taking over node with fully replicated data. The downside: DRBD conceptually scales only to a few nodes, typically two to three, and is not suited for storage that needs to be spread across many servers.

# /etc/drbd.d/shared.res — minimal DRBD resource definition
resource shared {
    protocol C;
    device /dev/drbd0;
    disk   /dev/sdb1;
    meta-disk internal;

    on node1 {
        address 10.0.1.11:7789;
    }
    on node2 {
        address 10.0.1.12:7789;
    }
}

4. GlusterFS: a distributed filesystem without a metadata server

GlusterFS is a genuine distributed filesystem that combines multiple storage nodes into a shared namespace without requiring a central metadata server. This architecture, called elastic hashing, computes a file's location directly from its name, meaning there is no single server whose failure would block access to metadata. This makes GlusterFS significantly more robust against partial failures than classic client server architectures.

A replicated volume in GlusterFS stores every file redundantly on multiple storage nodes at the same time, so that if one node fails, data remains available on the remaining nodes. In Magento and PHP environments, GlusterFS is frequently used for shared media and session storage across multiple application servers when a plain NFS solution is not sufficient due to its single point of failure.

# Install GlusterFS server on both storage nodes
sudo apt install -y glusterfs-server
sudo systemctl enable --now glusterd

# Peer the two storage nodes together (run once, from node1)
sudo gluster peer probe node2

# Create a replicated volume across both nodes
sudo gluster volume create shared_vol replica 2 \
  node1:/data/brick1/shared_vol \
  node2:/data/brick1/shared_vol

sudo gluster volume start shared_vol

# Mount the GlusterFS volume on an application server
sudo apt install -y glusterfs-client
sudo mount -t glusterfs node1:/shared_vol /mnt/shared

5. Ceph: object storage as a universal storage base

Ceph takes a fundamentally different approach than GlusterFS: at its core, Ceph is a distributed object store on which three different access layers sit, an object storage gateway compatible with S3, a block device called RBD, and a POSIX compatible filesystem called CephFS. This flexibility makes Ceph the most universal of the systems presented here, though also the most complex to operate and configure.

The basic unit of Ceph is the Object Storage Device, or OSD, one process per physical disk that stores and replicates data. The CRUSH algorithm deterministically distributes data across OSDs without needing a central lookup server, similar to GlusterFS's elastic hashing but with significantly finer grained control over replication factors, failure domains and placement rules. For environments needing block, object and filesystem access on the same storage foundation, Ceph is often the only solution covering all three requirements simultaneously.

6. Consistency models and split brain risks

Every distributed filesystem has to make a fundamental decision: how is it ensured that all nodes have the same view of the data when a network problem interrupts communication between them? DRBD uses synchronous replication in protocol C, where a write is only considered complete once both nodes have acknowledged it, guaranteeing consistency but increasing latency. Asynchronous protocols like protocol A reduce latency but risk data loss if the primary node fails shortly after a write that has not yet been replicated.

GlusterFS and Ceph use quorum mechanisms, similar to Pacemaker, to decide which node majority is considered authoritative in case of doubt when the network gets partitioned. Without correctly configured quorum, distributed filesystems too can end up in a split brain situation, where two network partitions independently make conflicting changes to the same files. The subsequent conflict resolution, called split brain resolution in GlusterFS, often requires manual intervention and should be avoided from the outset through correct quorum configuration.

7. Performance characteristics compared in practice

Performance characteristics differ significantly between the presented solutions. NFS offers the lowest latency for simple read and write operations, as long as the server itself is not overloaded, but suffers from metadata overhead through the network protocol with many small files. DRBD achieves near native block device performance with synchronous replication, though network latency between the two nodes directly feeds into the application's write latency.

GlusterFS shows good performance for large, sequential files such as video or backup data, but has traditionally shown weaknesses with very many small files, which can matter for PHP applications with thousands of session or cache files. Ceph typically requires a larger minimum cluster size to realize its performance advantages through parallelization across many OSDs, and rarely pays off performance wise with only two or three storage nodes.

8. Practical example: shared media storage for Magento

In a horizontally scaled Magento environment with multiple application servers behind an HAProxy load balancer, every server needs the same access to uploaded product images and generated cache files in the pub/media directory. A common solution is a GlusterFS replicated volume mounted as a local directory on every application server, so uploads through one server are instantly visible on all others.

For smaller setups with only two application servers, a simpler combination of NFS with a Pacemaker managed, DRBD replicated block device underneath is often sufficient and much less maintenance intensive than a full GlusterFS or Ceph cluster. The decision depends heavily on the number of application servers and expected future scaling, not on a blanket preference for the technically more complex solution.

9. Making the right choice: decision table

Choosing the right distributed filesystem depends on three factors: the number of nodes involved, the dominant file type (many small versus few large files), and the degree of operational complexity a team is willing to carry. There is no universally best solution, only the fitting one for a concrete scenario.

10. Summary

A distributed filesystem is the necessary storage foundation as soon as multiple servers need to see the same data, or a failover must happen without data loss. NFS is the simplest starting point but remains a single point of failure without additional HA measures. DRBD suits two node database failover with near native block device performance, while GlusterFS as a genuine distributed filesystem without a metadata server works for shared media storage across multiple application servers.

Ceph covers the broadest functionality with block, object and file access from a shared base, but requires a larger minimum cluster size and more operational knowledge. The choice between these options depends on node count, dominant file type and accepted complexity, not on a blanket recommendation for the technically most advanced solution.

11. FAQ: Distributed Filesystems Overview