Bash for Backups: Rotating Snapshots and Restore Tests

1. Backup strategy: what a good Bash backup script actually does

A professional Bash backup script solves four core problems: it reliably creates backups without silent failures, rotates old backups according to a configurable retention policy, verifies the integrity of the backups it creates, and regularly tests whether a restore actually works. Most simple backup scripts only solve the first problem, and they fail exactly at the moment a restore is truly needed.

The foundation of every Bash backup script is set -euo pipefail combined with a trap cleanup EXIT handler. If the backup process aborts mid run, because of a full filesystem, a network error, or a signal, no incomplete backup should ever appear as complete in the backup directory. The pattern: write backups to a temporary directory and only rename them after successful completion. That way, every file in the backup target directory is either fully present or not present at all.

The 3-2-1 backup rule is easy to implement with Bash backup scripts: three copies, on two different media, one of them off site. The script creates the local backup with rsync or tar, transfers it to a remote host via rsync over SSH, and reports the status of all three steps. Each step has its own exit code check and logging entry. A missing off site backup is a known, escalated failure, not a silent one.

2. rsync for incremental snapshots

rsync is the ideal tool for incremental Bash backups: it transfers only changed data, preserves permissions and symlinks, supports SSH as a transport, and offers extensive filter configuration for excluded directories. The core pattern for an incremental rsync backup with hard links is rsync --link-dest: the previous backup serves as a reference, and unchanged files are linked in as hard links instead of being copied. Every snapshot looks complete, yet it only consumes storage for the files that actually changed.

The most important rsync flags for Bash backup scripts: -a (archive mode: recursive, symlinks, permissions, timestamps, owner, group), --delete (also remove files in the target that were deleted from the source), --exclude-from (read the exclude list from a file, more maintainable than inline flags for large exclude lists), --stats (write transfer statistics to the log), --checksum (use real checksums instead of timestamp comparison for maximum integrity), and --bwlimit (bandwidth limit so production systems are not overloaded). Combining these flags in a configuration file makes the backup script adjustable without any code changes.

#!/usr/bin/env bash
# rsync_backup.sh: incremental backup with hard-link snapshots
set -euo pipefail
IFS=$'\n\t'

BACKUP_ROOT="${BACKUP_ROOT:-/backups}"
SOURCE_DIRS=("${@:-/var/www /etc /home}")
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
LATEST_LINK="$BACKUP_ROOT/latest"
CURRENT_BACKUP="$BACKUP_ROOT/snapshot-$TIMESTAMP"
INCOMPLETE="$BACKUP_ROOT/incomplete-$TIMESTAMP"

EXCLUDES=(
  "*.tmp" "*.swp" ".git/objects"
  "var/cache" "var/log" "node_modules"
  ".Trash*" "lost+found"
)

# Build rsync exclude arguments
rsync_exclude_args=()
for pattern in "${EXCLUDES[@]}"; do
  rsync_exclude_args+=(--exclude="$pattern")
done

cleanup() {
  # Remove incomplete backup directory on failure
  [[ -d "$INCOMPLETE" ]] && rm -rf "$INCOMPLETE"
}
trap cleanup EXIT

# Write to incomplete dir first, rename only on success
mkdir -p "$INCOMPLETE"

# Use previous snapshot as hard-link reference if it exists
link_dest_args=()
[[ -d "$LATEST_LINK" ]] && link_dest_args=(--link-dest="$LATEST_LINK")

rsync -aH --delete --stats --numeric-ids \
  "${rsync_exclude_args[@]}" \
  "${link_dest_args[@]}" \
  "${SOURCE_DIRS[@]}" \
  "$INCOMPLETE/"

# Atomic rename: only now is the backup "complete"
mv "$INCOMPLETE" "$CURRENT_BACKUP"

# Update latest symlink atomically
ln -sfn "$CURRENT_BACKUP" "${LATEST_LINK}.new"
mv -Tf "${LATEST_LINK}.new" "$LATEST_LINK"

echo "[OK] Backup completed: $CURRENT_BACKUP"
du -sh "$CURRENT_BACKUP"

3. tar snapshots with timestamps and compression

tar archives are the classic method for Bash backups with complete snapshots: a single archive per backup run, compressed, verifiable with a checksum, and easy to transport. The standard pattern for a tar backup script: filename with an ISO timestamp, compression with gzip, bzip2 or xz depending on the tradeoff between compression ratio and speed, then calculate a SHA256 checksum and store it in a companion file.

A critical detail with tar backup scripts in Bash: tar returns exit code 1, not 2, for certain warnings (changed files, cannot open). With set -e the script would abort on these harmless warnings. The correct pattern: tar_exit=0; tar czf archive.tgz /source || tar_exit=$?; (( tar_exit > 1 )) && { echo "tar fatal error"; exit 1; }. Exit code 1 means "files changed during the backup", which is acceptable for incremental live backups. Exit code 2 means a fatal error and must cause the script to abort.

4. Rotating backups: retention and cleanup

Rotating Bash backups implement a retention policy that keeps storage under control while still preserving enough history. The simplest rotation pattern: delete all backups older than N days. find "$BACKUP_ROOT" -maxdepth 1 -name "snapshot-*" -mtime +7 -exec rm -rf {} \; deletes every snapshot directory older than 7 days. That is simple, but not very granular.

A more refined rotation pattern in Bash follows the grandfather-father-son principle: daily backups for the last 7 days, weekly backups for the last 4 weeks, monthly backups for the last 12 months. This is implementable in Bash through naming conventions: daily with daily-YYYYMMDD, weekly with weekly-YYYY-Wnn, monthly with monthly-YYYY-MM. The rotation script checks which weekly or monthly backups are still missing, copies the best available daily backup into the matching category, and deletes daily backups outside the retention window. This pattern results in at most 7+4+12=23 stored backups instead of an uncontrolled accumulation.

#!/usr/bin/env bash
# backup_rotation.sh: grandfather-father-son rotation for Bash backups
set -euo pipefail

BACKUP_ROOT="${BACKUP_ROOT:-/backups}"
DAILY_RETAIN=7
WEEKLY_RETAIN=4
MONTHLY_RETAIN=12

# Sort backups by date, keep only the newest N of each category
rotate_category() {
  local pattern="$1" keep="$2"
  local -a all=()
  mapfile -t all < <(find "$BACKUP_ROOT" -maxdepth 1 -name "$pattern" -type d | sort -r)

  local count=${#all[@]}
  if (( count > keep )); then
    local to_delete=("${all[@]:$keep}")
    for old in "${to_delete[@]}"; do
      echo "[ROTATE] Removing old backup: $old"
      rm -rf "$old"
    done
    echo "[OK] Rotated $pattern: kept $keep, deleted $((count - keep))"
  else
    echo "[OK] $pattern: $count backups (within limit of $keep)"
  fi
}

# Copy latest daily to weekly/monthly if not yet present
promote_to_weekly() {
  local week_stamp
  week_stamp=$(date +%Y-W%V)
  local weekly_dir="$BACKUP_ROOT/weekly-$week_stamp"
  local latest="$BACKUP_ROOT/latest"

  if [[ ! -d "$weekly_dir" ]] && [[ -L "$latest" ]]; then
    echo "[INFO] Creating weekly backup: $weekly_dir"
    cp -al "$latest" "$weekly_dir"
  fi
}

promote_to_monthly() {
  local month_stamp
  month_stamp=$(date +%Y-%m)
  local monthly_dir="$BACKUP_ROOT/monthly-$month_stamp"
  local latest="$BACKUP_ROOT/latest"

  if [[ ! -d "$monthly_dir" ]] && [[ -L "$latest" ]]; then
    echo "[INFO] Creating monthly backup: $monthly_dir"
    cp -al "$latest" "$monthly_dir"
  fi
}

promote_to_weekly
promote_to_monthly

rotate_category "snapshot-*"  "$DAILY_RETAIN"
rotate_category "weekly-*"    "$WEEKLY_RETAIN"
rotate_category "monthly-*"   "$MONTHLY_RETAIN"

# Report remaining backup count and total size
total_size=$(du -sh "$BACKUP_ROOT" 2>/dev/null | cut -f1)
echo "[REPORT] Total backup size: $total_size"

5. Orchestrating database backups with Bash

Database backups in Bash require special attention, because a database is a running state that can be captured inconsistently without transaction isolation. For MySQL/MariaDB, mysqldump --single-transaction is the key to consistent snapshots of InnoDB tables without exclusive locks. For PostgreSQL, pg_dump -Fc (custom format) is the recommended format because it enables parallel restores with pg_restore -j and is compressed. The Bash backup script orchestrates the dump, the compression, the checksum and the transfer as a single atomic operation.

An often overlooked aspect of database backup scripts in Bash: the difference between a logical dump and a physical snapshot. Logical dumps (mysqldump) are slower but more portable and easier to verify. Physical snapshots via LVM or ZFS are faster and enable point in time recovery, but require an identical database version on restore. The Bash backup script should explicitly document which strategy is used and tag the dump with metadata: database version, timestamp, size, and row counts for the most critical tables.

6. Backup verification: checksums and integrity tests

Backup verification in a Bash backup script goes beyond simply checking the exit code. A tar file can return a correct exit code and still be corrupted, because of a disk error, a network problem, or a lack of storage space that only appeared after the last write. The standard pattern: calculate a SHA256 checksum immediately after creation and store it in a companion file. Before every restore, and in a separate verification job, check the checksum against the stored value.

For rsync based backups in Bash, verification looks different: instead of a checksum over an archive, you check whether the file count and total size are plausible. A backup that suddenly contains 90% fewer files than the previous one is a signal of a failure, regardless of whether the exit code was 0. The Bash backup script should compare the file count of the new backup against the previous one after every rsync run and raise an error and escalate if the deviation exceeds a configurable threshold.

#!/usr/bin/env bash
# backup_verify.sh: checksum and sanity verification for Bash backups
set -euo pipefail

BACKUP_ROOT="${BACKUP_ROOT:-/backups}"
VARIANCE_THRESHOLD=20  # alert if file count drops more than 20%

verify_archive() {
  local archive="$1"
  local checksum_file="${archive}.sha256"

  if [[ ! -f "$checksum_file" ]]; then
    echo "[ERROR] No checksum file for: $archive" >&2; return 1
  fi

  echo "[INFO] Verifying checksum: $archive"
  if sha256sum --check "$checksum_file" --quiet; then
    echo "[OK] Checksum verified: $archive"
  else
    echo "[ERROR] Checksum MISMATCH: $archive, backup may be corrupted" >&2
    return 1
  fi

  # Test tar integrity without extracting (--list reads the table of contents)
  echo "[INFO] Testing archive integrity: $archive"
  if tar --list --file="$archive" >/dev/null 2>&1; then
    local file_count
    file_count=$(tar --list --file="$archive" 2>/dev/null | wc -l)
    echo "[OK] Archive readable: $file_count entries"
  else
    echo "[ERROR] Archive is corrupted or truncated: $archive" >&2
    return 1
  fi
}

verify_rsync_snapshot() {
  local current="$1" previous="$2"
  [[ ! -d "$current" ]] && { echo "[ERROR] Backup dir not found: $current" >&2; return 1; }
  [[ ! -d "$previous" ]] && { echo "[WARN] No previous backup for comparison"; return 0; }

  local current_count previous_count
  current_count=$(find "$current" -type f | wc -l)
  previous_count=$(find "$previous" -type f | wc -l)

  local drop_pct=0
  (( previous_count > 0 )) && drop_pct=$(( (previous_count - current_count) * 100 / previous_count ))

  if (( drop_pct > VARIANCE_THRESHOLD )); then
    echo "[ERROR] File count dropped $drop_pct% (prev=$previous_count, curr=$current_count)" >&2
    return 1
  fi
  echo "[OK] File count OK: $current_count files (${drop_pct}% change from $previous_count)"
}

# Run verification on the latest backup
latest="$BACKUP_ROOT/latest"
[[ -L "$latest" ]] || { echo "[ERROR] No 'latest' symlink found" >&2; exit 1; }
resolved=$(readlink -f "$latest")

# Find the backup before the latest for comparison
prev_backup=$(find "$BACKUP_ROOT" -maxdepth 1 -name "snapshot-*" -type d | sort | tail -2 | head -1)
verify_rsync_snapshot "$resolved" "$prev_backup"
echo "[DONE] Backup verification completed for: $resolved"

7. Automated restore tests

Automated restore tests in Bash are the only way to actually know that a backup can be restored. A backup without a restore test is an unverified hypothesis. The pattern for an automated restore test in a Bash backup script: periodically (weekly or monthly) restore the latest backup into an isolated test environment, run a set of smoke tests, and log the result. An "isolated test environment" can be a Docker container, a temporary directory structure, or a VM snapshot.

For database restore tests in Bash, the pattern is: import the dump into a temporary database, check critical tables for row count and consistency, then drop the temporary database. The entire Bash script for the restore test runs automatically after every backup run or as a separate cron job. Failures are escalated. On success, the result is recorded with a timestamp in a restore test log. This log is the proof that the backup system works, not the backup files themselves.

8. Notifications and monitoring

A Bash backup script without notifications is a script that can fail silently. The monitoring pattern: on success, send a success message with metadata (backup size, duration, file count) to a central monitoring system; on failure, trigger an alert immediately. For simple notifications, a curl call to a Slack webhook or an email via sendmail is enough. For structured monitoring, a heartbeat service like Healthchecks.io can be used: send an HTTP GET on successful backup, and if the heartbeat fails to arrive, the service triggers an alert.

The superior pattern for backup monitoring in Bash: don't just report failures, actively confirm success too. A backup script that stays quiet on success and only writes on failure gives no guarantee that it is running at all. If the cron job fails to trigger for any reason, a system crash, a cron configuration error, a full crontab mail queue, you only notice this without a success heartbeat when you actually need the backup. An active heartbeat to an external system is the safest implementation.

9. Backup strategies compared

The choice of Bash backup strategy depends on the requirements for Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

In practice, a robust Bash backup system combines several strategies: rsync for daily incremental file backups, mysqldump for daily database dumps, tar for weekly full snapshots, and grandfather-father-son rotation across all categories. The Bash script orchestrates all three, performs verification, and automatically tests the restore every month.

10. Summary

Bash for backups, rotating snapshots and restore tests means building a backup routine that goes beyond simply creating archives. rsync with --link-dest for storage efficient incremental snapshots. Atomic backups by writing to temporary directories and renaming only after successful completion. Grandfather-father-son rotation for controlled retention. SHA256 checksums and file count plausibility checks for verification. Automated restore tests in isolated environments as the only proof that a backup actually works.

The most critical insight: a Bash backup script without a restore test is not a backup system, it is an archiving process with unverified recoverability. The effort for an automated restore test, even if it only covers a file count check and a mysqldump import into a test database, is the most important step from "backup exists" to "backup verified".

11. FAQ: Bash for Backups, Rotating Snapshots and Restore Tests