Pacemaker and Corosync: High Availability Cluster Basics

1. Division of roles: Corosync and Pacemaker

Pacemaker and Corosync together form the established cluster stack for high availability on Linux, with both components handling different, clearly separated tasks. Corosync is the communication layer that manages cluster membership, exchanges heartbeat messages between nodes and determines which nodes are currently reachable. Without this reliable communication layer, no cluster manager could even know whether a node has failed or is merely slow.

Pacemaker builds on top of this communication layer and handles the actual resource orchestration: it starts, stops and monitors arbitrary resources such as IP addresses, filesystems, database services or application servers, relocates them to a healthy node on failure, and respects complex dependencies between multiple resources along the way. This combination makes Pacemaker significantly more powerful than keepalived, but also correspondingly more complex to configure and operate.

The typical use case for Pacemaker and Corosync is a cluster where several related resources must be managed together, for example a shared filesystem that may only be mounted on one node at a time, a database service accessing it, and a virtual IP that must point to the same node. This is exactly the kind of coordinated dependency keepalived is not designed for, while Pacemaker was built precisely for it.

2. Installation and the cluster skeleton

On most modern distributions the cluster is managed through the command line tool pcs, which simplifies Corosync and Pacemaker configuration behind a unified interface. After installation, the system user hacluster must be authenticated with a shared password on every node before the nodes register each other as cluster members.

# Install the cluster stack on RHEL/AlmaLinux (both nodes)
sudo dnf install -y pacemaker pcs pcs-cli resource-agents

# Set a shared password for the hacluster system user (both nodes)
sudo passwd hacluster

# Start and enable the pcs daemon (both nodes)
sudo systemctl enable --now pcsd

# Authenticate node1 against node2 (run once, from node1)
sudo pcs host auth node1 node2 -u hacluster

# Create the cluster with both nodes as members
sudo pcs cluster setup my_cluster node1 node2

# Start the cluster on all nodes and enable it on boot
sudo pcs cluster start --all
sudo pcs cluster enable --all

After these steps, pcs status shows the current cluster state, including every node and its reachability. At this point the cluster is technically active but not yet managing a single resource. The actual configuration of the services to be protected happens in the next step.

3. Defining cluster resources

A cluster resource is anything Pacemaker is supposed to start, stop and monitor, accessed through so called resource agents. These agents are standardized scripts implementing a uniform interface for start, stop and monitor operations, regardless of whether it is an IP address, a filesystem or a database service. The best known agent class is ocf:heartbeat, which covers a wide range of common services.

Every resource is registered with a unique name plus resource specific parameters. For a virtual IP address that is the IP itself and the netmask, for a filesystem the device path and mount point. Pacemaker then monitors every resource at a configurable interval and automatically reacts to failures by restarting the resource or relocating it to another node.

# Define a virtual IP resource
sudo pcs resource create cluster_vip ocf:heartbeat:IPaddr2 \
  ip=192.168.10.100 cidr_netmask=24 \
  op monitor interval=10s

# Define a filesystem resource (shared storage, e.g. iSCSI or DRBD device)
sudo pcs resource create shared_fs Filesystem \
  device="/dev/sdb1" directory="/mnt/shared" fstype="xfs" \
  op monitor interval=20s

# Define a database service resource
sudo pcs resource create mysql_db systemd:mariadb \
  op monitor interval=15s

4. Fencing with STONITH: why it is mandatory

Fencing, referred to in Pacemaker as STONITH, short for Shoot The Other Node In The Head, is the mechanism by which a cluster actively disconnects or restarts a node deemed faulty, rather than relying only on that node's own self reported status. Without fencing there is always the risk that a supposedly failed node is actually still running, still writing to a shared filesystem, causing data corruption while another node has taken over the same resource.

STONITH devices range from IPMI based fencing via the server management interface to API based fencing with cloud providers that can hard power off a VM via API call. A cluster without configured fencing is considered unsafe for production, even if it works flawlessly during normal operation, because exactly the failure case the cluster was built for can lead to data inconsistency without fencing.

# Configure IPMI-based fencing for node1 (adjust IP/credentials per environment)
sudo pcs stonith create fence_node1 fence_ipmilan \
  pcmk_host_list="node1" ipaddr="192.168.20.11" \
  login="admin" passwd="s3cr3tIpmiPass" lanplus=1

sudo pcs stonith create fence_node2 fence_ipmilan \
  pcmk_host_list="node2" ipaddr="192.168.20.12" \
  login="admin" passwd="s3cr3tIpmiPass" lanplus=1

# Verify STONITH is enabled and configured (never disable in production)
sudo pcs property show stonith-enabled

5. Constraints: order and colocation

Once several resources are managed together, Pacemaker needs to know in which order they must start and whether they are required to run on the same node. These rules are called constraints. An order constraint defines, for example, that the filesystem must be mounted before the database starts, while a colocation constraint enforces that filesystem, database and virtual IP always run on the same node, since otherwise they would not be functional.

Without constraints, Pacemaker could theoretically place the database on node A and the corresponding filesystem on node B, which would render the application instantly unusable. Constraints are therefore not an optimization but a necessary correctness condition for any cluster with more than one resource.

# Filesystem must be mounted before the database starts
sudo pcs constraint order shared_fs then mysql_db

# Database must start before the virtual IP becomes active
# (clients should only see the IP once the DB is reachable)
sudo pcs constraint order mysql_db then cluster_vip

# All three resources must always run on the same node
sudo pcs constraint colocation add mysql_db with shared_fs INFINITY
sudo pcs constraint colocation add cluster_vip with mysql_db INFINITY

# Group them for simpler management (equivalent to the constraints above)
sudo pcs resource group add app_group shared_fs mysql_db cluster_vip

6. A complete two node example

A realistic two node setup for a resilient database service combines all the building blocks shown so far: shared storage via a DRBD or iSCSI device, a filesystem on top of it, the database service itself, and a virtual IP, all grouped as a resource group. If the active node fails, Pacemaker does not just manually unmount the filesystem, it reliably detects via fencing that the old node is truly powered off before the resource group is fully started on the remaining node.

This process differs fundamentally from keepalived, where only a single IP address migrates. With Pacemaker, an entire chain of dependent services is shut down in a controlled way and brought back up in the correct order on the new node, which is the only safe way to prevent data corruption from simultaneous write access by two nodes when database failover involves shared storage.

In practice, such a setup is thoroughly tested before going to production, by controlledly disconnecting a node from the network or hard powering it off while continuous load runs against the database. This is the only way to verify that fencing, constraints and resource ordering actually kick in within the expected time window and no resource gets stuck in an inconsistent intermediate state.

7. Quorum and the two node problem

Quorum is the concept a cluster uses to decide whether enough nodes are reachable to safely continue operating. In a classic majority quorum, a cluster needs more than half of all configured nodes to make decisions. With exactly two nodes, this is structurally problematic: if the connection between the two fails, neither has a majority, and without an additional measure, neither could safely decide whether it is allowed to keep working alone.

For two node clusters, Corosync offers the two_node: 1 option, which adjusts this behavior and, combined with correctly configured fencing, still enables safe operation, because in case of doubt the other node gets powered off via STONITH instead of both staying active simultaneously. For clusters with three or more nodes, real majority quorum is possible without such special rules and is generally the more robust choice when the infrastructure allows it.

8. Monitoring, maintenance and troubleshooting

The central command for day to day work with a Pacemaker cluster is pcs status, which provides a complete overview of nodes, resources and their current state. When troubleshooting, pcs status --full additionally shows failed operations with timestamps, usually the fastest way to identify the cause of an unexpected failover.

For planned maintenance work, such as a kernel update, a node is first put into standby mode with pcs node standby, causing Pacemaker to controlledly relocate all resources to the remaining node before the maintenance node is restarted. This controlled transition avoids an unplanned failover during maintenance and can be reversed afterward with pcs node unstandby.

# Full cluster overview: nodes, resources, failed actions
sudo pcs status --full

# Put a node into maintenance before planned work (e.g. kernel update)
sudo pcs node standby node2

# Resume normal operation after maintenance
sudo pcs node unstandby node2

# Show detailed resource failure history
sudo pcs resource failcount show mysql_db

# Live-follow the Pacemaker/Corosync logs
sudo journalctl -u pacemaker -u corosync -f

9. Pacemaker compared to simpler solutions

For teams already using a simpler HA mechanism such as keepalived, the question of when switching to Pacemaker makes sense inevitably comes up. The decision depends almost entirely on how many interdependent resources need to be coordinated and how critical correct ordering and fencing are for data integrity.

Anyone who only wants to make a single IP address highly available between two servers is usually better off with keepalived, since it is simpler to operate and debug. As soon as multiple resources with strict dependencies need coordination, for example with shared storage or multi tier application services, Pacemaker is the only reliable solution.

10. Summary

Pacemaker and Corosync together form the standard Linux cluster stack whenever more than a single IP address must be made highly available. Corosync handles reliable communication between nodes, while Pacemaker coordinates starting, monitoring and relocating arbitrary resources such as filesystems, database services and virtual IPs on failure, controlled through order and colocation constraints.

Fencing via STONITH is not an optional extra but a necessary prerequisite for any production cluster, because it is the only way to prevent a supposedly failed node from continuing to write to shared storage. With exactly two nodes, quorum behavior requires special attention via the two_node option, while larger clusters benefit from real majority quorum. For simple single IP scenarios, keepalived remains the more pragmatic choice, but for coordinated multi resource clusters, Pacemaker is the only reliable tool.

11. FAQ: Pacemaker and Corosync Basics