Bare Metal Recovery Planning: Fully Restoring a Server

1. What bare metal recovery means and why it is different

A regular backup protects files inside a running system: databases, configuration, web directories. Bare metal recovery goes a step further and answers the question of what happens when nothing is left at all, no bootloader, no partition table, no installed operating system. That exact scenario occurs after a failed disk, a destroyed RAID controller, or a physically damaged server. Without a documented bare metal recovery plan, restoration starts with guesswork: what partition sizes, which bootloader, which kernel parameters were configured.

The difference between a file restore and bare metal recovery lies in the order of dependencies. A file restore assumes the operating system, network, and storage already work. Bare metal recovery has to rebuild that layer itself before any file restore is even possible. Teams that only back up database dumps and application data, but have no bare metal recovery strategy, discover during a total failure that the real obstacle is not the data, but a working system to restore that data onto in the first place.

2. Anatomy of a total failure: what is really lost

When planning bare metal recovery it helps to picture each component of a server individually. What can be lost includes the partition table itself (MBR or GPT), the bootloader (GRUB2 with its configuration under /boot/grub), the kernel along with its initramfs, every system configuration under /etc, the installed packages with their exact versions, and of course the actual payload data in databases and filesystems. Each of these layers needs its own answer in a bare metal recovery plan.

The partition table and disk layout are particularly underestimated. A server with LVM volumes, a separate /boot partition, and custom mount options in /etc/fstab cannot simply be reconstructed from memory. Anyone who has not documented, as part of bare metal recovery planning, how the disks were partitioned loses valuable time in an emergency piecing things together from log files, which in the worst case are no longer accessible either.

3. Building an inventory: disk layout, bootloader, and kernel

The first concrete step in any bare metal recovery planning effort is a current, machine readable inventory of the system. That includes the output of lsblk, blkid, and parted --list for the disk layout, the installed bootloader version along with /boot/grub/grub.cfg, the loaded kernel modules from lsmod, and the package list from dpkg --get-selections or rpm -qa. These files should be generated automatically and written to a location independent of the server itself, because an inventory that only lives on the affected server is worthless in an emergency.

A good bare metal recovery inventory is versioned so changes to the disk layout stay traceable. If a new partition is created or an LVM volume group is extended, the inventory must update automatically, otherwise the documentation drifts from the real state and the recovery plan becomes worthless the next time it is actually needed. A daily cron job that generates the inventory and copies it via rsync to another host is the most pragmatic way to prevent that drift.

#!/usr/bin/env bash
# bare-metal-inventory.sh — capture disk layout and boot config for recovery
set -euo pipefail

readonly OUT_DIR="/var/backup/inventory/$(hostname)"
mkdir -p "$OUT_DIR"

# Disk layout: partitions, filesystems, UUIDs
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,UUID > "$OUT_DIR/lsblk.txt"
blkid > "$OUT_DIR/blkid.txt"
parted --list > "$OUT_DIR/parted.txt" 2>/dev/null || true

# LVM layout, if in use
if command -v vgs &>/dev/null; then
  vgs -o+vg_uuid > "$OUT_DIR/lvm-vgs.txt"
  lvs -o+lv_uuid,segtype > "$OUT_DIR/lvm-lvs.txt"
fi

# Bootloader configuration
cp /boot/grub/grub.cfg "$OUT_DIR/grub.cfg" 2>/dev/null || true
grub-install --version > "$OUT_DIR/grub-version.txt" 2>/dev/null || true

# Kernel and modules
uname -a > "$OUT_DIR/kernel.txt"
lsmod > "$OUT_DIR/modules.txt"

# Package inventory (Debian/Ubuntu)
dpkg --get-selections > "$OUT_DIR/packages.txt" 2>/dev/null || \
  rpm -qa > "$OUT_DIR/packages.txt"

# fstab and network config are critical for boot
cp /etc/fstab "$OUT_DIR/fstab"
cp -r /etc/netplan "$OUT_DIR/netplan" 2>/dev/null || true

# Ship to an off-server target — never keep the only copy on the affected host
rsync -az "$OUT_DIR/" backup-host:/inventory/$(hostname)/

4. Image-based full backups with dd and partclone

The classic method for bare metal recovery is block level image backup of entire partitions or disks. dd copies every block regardless of the filesystem, which offers maximum compatibility but also copies empty blocks along with everything else, making it slow and storage hungry. partclone, on the other hand, understands the given filesystem and skips unused blocks, which noticeably reduces backup time and storage needs, especially on mostly empty partitions. For production bare metal recovery strategies, partclone is usually the more practical choice.

What matters with any image based backup is that the source partition is not actively written to while the backup runs. That is achieved either by backing up from a rescue system while the target system is shut down, or by taking an LVM snapshot that freezes a consistent state while the system keeps running. Clonezilla automates exactly this workflow and provides a ready made rescue environment from which entire disks or individual partitions can be backed up and restored, bootloader included.

#!/usr/bin/env bash
# image-backup.sh — block-level backup for bare metal recovery
# Run from a rescue environment while the target disk is unmounted
set -euo pipefail

readonly DEVICE="/dev/sda"
readonly IMAGE_DIR="/mnt/backup-nas/images/$(date +%Y%m%d)"
mkdir -p "$IMAGE_DIR"

# partclone: filesystem-aware, skips unused blocks, much faster than dd
for part in /dev/sda1 /dev/sda2; do
  name=$(basename "$part")
  fstype=$(blkid -o value -s TYPE "$part")

  case "$fstype" in
    ext4) partclone.ext4 -c -s "$part" -o "$IMAGE_DIR/${name}.pc.gz" ;;
    xfs)  partclone.xfs  -c -s "$part" -o "$IMAGE_DIR/${name}.pc.gz" ;;
    *)    partclone.dd   -c -s "$part" -o "$IMAGE_DIR/${name}.pc.gz" ;;
  esac
done

# Bootloader and partition table are not covered by partclone — capture separately
dd if="$DEVICE" of="$IMAGE_DIR/mbr-partition-table.img" bs=512 count=2048
sfdisk -d "$DEVICE" > "$IMAGE_DIR/partition-table.sfdisk"

echo "[OK] Bare metal image saved to $IMAGE_DIR"

5. Automated rebuild as an alternative to imaging

Not every bare metal recovery strategy has to rely on an image. For servers reproducibly set up through configuration management such as Ansible, Puppet, or cloud-init, an automated rebuild is often faster than restoring a multi-gigabyte image. The idea: instead of freezing an exact state, the same end state is recreated from a base install and a declarative playbook, while only the actual payload data is restored from a separate backup.

This strategy has one decisive advantage for bare metal recovery: it also works when the target hardware is not identical to the original, for instance because a replacement server had to be sourced with a different disk size. An image, by contrast, generally expects a compatible or larger target disk. In practice many teams combine both approaches: an image for the fastest possible restart on identical hardware, and a playbook as a fallback for when the hardware no longer matches.

# rebuild-playbook.yml — reproducible server rebuild for bare metal recovery
# Fallback path when identical replacement hardware is not available
- name: Rebuild server from scratch after total failure
  hosts: recovery_target
  become: true
  vars:
    php_version: "8.4"

  tasks:
    - name: Install base packages matching production inventory
      apt:
        name: "{{ item }}"
        state: present
      loop:
        - nginx
        - "php{{ php_version }}-fpm"
        - mariadb-server
        - fail2ban

    - name: Restore /etc configuration from last known-good backup
      synchronize:
        src: "rsync://backup-host/config/{{ inventory_hostname }}/etc/"
        dest: /etc/
      delegate_to: localhost

    - name: Restore application data volume
      command: >
        restic -r s3:s3.example.com/backups restore latest
        --target /var/www --path /var/www
      environment:
        RESTIC_PASSWORD_FILE: /root/.restic-pass

    - name: Reinstall bootloader on the new disk
      command: grub-install /dev/sda

6. Preparing the recovery environment: rescue systems and network boot

An image or a playbook is useless if no environment exists from which the restore can even be started in an emergency. Bare metal recovery planning therefore requires a prepared rescue environment: a bootable USB stick with Clonezilla or a minimal Debian live environment, or, for dedicated servers in a data center, a PXE network boot that works without physical access to the machine.

For root servers without physical access, PXE boot or the rescue system provided by the hosting company is often the only option. These rescue systems should be tested beforehand: does network access to the backup target work from within the rescue system? Are the needed tools such as partclone, rsync, or restic even installed, or do they have to be installed on the spot? Answering these questions for the first time during an actual emergency reliably adds hours to bare metal recovery.

7. Calculating RTO and RPO for bare metal recovery realistically

Recovery Time Objective (RTO) and Recovery Point Objective (RPO) are the central metrics of any bare metal recovery planning effort. RTO describes how long the restart may take at most, RPO how much data loss between the last backup and the failure is acceptable. For bare metal recovery, the RTO calculation must realistically include every step: sourcing or provisioning replacement hardware, booting a rescue system, applying an image or playbook, database restore, functional testing.

A common planning mistake is measuring only the raw restore time of the backup tool while ignoring the steps that come before it. Restoring an image from a terabyte volume can take two hours by itself, but if a rescue stick still has to be found, a replacement disk installed, and a DNS failover triggered manually on top of that, the real RTO quickly turns into a full day. Bare metal recovery planning should therefore time every single step and reconcile the result with the RTO the business actually requires, not the other way around.

8. Testing without production downtime: simulating recovery in a VM

A bare metal recovery plan that has never been tested is a guess, not a plan. The good news: the entire process can be simulated risk free in a virtual machine without touching the production server. The generated image is restored into a QEMU/KVM VM with a comparable virtual disk size, and it is then checked whether the VM actually boots, the network is reachable, and the application starts.

These tests belong on a fixed schedule, at least quarterly, and always after any significant change to the disk layout or bootloader. A bare metal recovery test that fails on the first attempt, for instance because the VM does not boot or a kernel module is missing, is more valuable than any untested plan, because it exposes exactly the gap that would turn into a disaster in a real emergency. Results and measured times should be documented so the RTO calculation from section seven is based on real measurements rather than estimates.

#!/usr/bin/env bash
# recovery-drill.sh — validate a bare metal image in an isolated VM
set -euo pipefail

readonly IMAGE="/mnt/backup-nas/images/20260730/sda1.pc.gz"
readonly VM_DISK="/tmp/recovery-test.qcow2"
readonly VM_NAME="recovery-drill-$(date +%s)"

qemu-img create -f qcow2 "$VM_DISK" 40G

# Restore the partclone image onto a loopback-mapped disk
partclone.ext4 -r -s "$IMAGE" -o "$VM_DISK"

# Boot the restored disk in an isolated, host-only network
virt-install \
  --name "$VM_NAME" \
  --memory 2048 \
  --disk "$VM_DISK" \
  --network network=isolated-drill-net \
  --import \
  --noautoconsole

echo "[INFO] Waiting for boot..."
sleep 60

# Confirm the recovered VM answers on SSH — proof the boot chain actually works
if ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 recovery-vm "systemctl is-system-running" ; then
  echo "[OK] Recovery drill succeeded: $VM_NAME booted and is reachable"
else
  echo "[FAIL] Recovery drill failed — image or bootloader is broken" >&2
  exit 1
fi

9. Recovery methods compared

A solid bare metal recovery plan benefits from a direct comparison of the available methods, because none of them is the best choice in every situation.

In practice a combination is the most robust option: partclone images for the fastest possible restart on identical hardware, an Ansible playbook as a fallback when the target hardware differs, and a prepared rescue system or PXE boot so both paths can even be started. That combination covers both the common case, identical hardware, and the rarer but critical case, completely new hardware.

10. Summary

Bare metal recovery is the answer to a server not just losing data, but failing completely, partition table, bootloader, and operating system included. A solid plan starts with a versioned inventory of disk layout, bootloader configuration, and package list, followed by a regular image backup with partclone or Clonezilla. An Ansible playbook as a fallback covers the case where replacement hardware is not identical. A prepared rescue environment, whether a USB stick or PXE boot, is the precondition for the image or playbook to be applied at all in an emergency.

Realistic RTO and RPO calculation together with regular recovery drills in isolated VMs turn a theoretical bare metal recovery plan into a proven, reliable process. Testing this process once a year, or after every major change, turns the worst case scenario from a crisis into a calculable, well documented event.

11. FAQ: Bare Metal Recovery Planning