Automating tar and gzip in Bash Scripts: Robust Backups with Incremental Runs

1. The basic structure of a robust backup script

A backup script differs from most other automation scripts in that an unnoticed failure often only becomes visible months later, precisely when the backup is actually needed. That is why every backup script should start with set -euo pipefail, so a failing command aborts the script immediately instead of continuing with a silently incomplete archive that later looks like a valid backup.

Equally important is a meaningful exit code and structured log output, so an overarching monitoring system can immediately tell whether a backup run succeeded without having to interpret the full log text. A backup script that only misbehaves visibly in an interactive terminal but silently fails in an unattended cron job defeats its own purpose.

#!/usr/bin/env bash
set -euo pipefail

readonly BACKUP_SRC="/var/www/app"
readonly BACKUP_DST="/backups/app-$(date +%Y%m%d-%H%M%S).tar.gz"
readonly LOG_FILE="/var/log/backup-app.log"

log() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"
}

trap 'log "FAILED: backup aborted (exit code $?)"' ERR

log "Starting backup: $BACKUP_SRC -> $BACKUP_DST"
tar -czf "$BACKUP_DST" -C "$(dirname "$BACKUP_SRC")" "$(basename "$BACKUP_SRC")"
log "Backup completed successfully: $(du -h "$BACKUP_DST" | cut -f1)"

2. Incremental backups with --listed-incremental

For large data sets, a daily full backup is often neither necessary nor practical, since usually only a small fraction of files actually change between two runs. tar supports exactly this case with the --listed-incremental option, which records the state of every file at the last run in a separate snapshot file and, on the next invocation, only includes files that have changed or are new since then.

The decisive difference from a plain full backup lies in this snapshot file: it must persist across every run and must never accidentally be deleted along with the archive itself, otherwise tar loses track on the next run of what was already backed up and silently falls back to a full backup. A robust script therefore stores the snapshot file at a permanent location, separate from the actual archive destination.

#!/usr/bin/env bash
set -euo pipefail

readonly SRC="/var/www/app"
readonly SNAPSHOT="/backups/state/app.snapshot"
readonly ARCHIVE="/backups/app-incr-$(date +%Y%m%d).tar.gz"

mkdir -p "$(dirname "$SNAPSHOT")"

# First run: SNAPSHOT does not exist yet -> full backup, snapshot is created
# Subsequent runs: only files changed since the last run are included
tar --listed-incremental="$SNAPSHOT" -czf "$ARCHIVE" -C "$(dirname "$SRC")" "$(basename "$SRC")"

echo "Incremental backup written: $ARCHIVE ($(du -h "$ARCHIVE" | cut -f1))"

3. Combining full and incremental backups sensibly

Pure incremental backups spanning many months make restoration cumbersome, because every single increment has to be replayed in the correct order to reach the current state. A rotating scheme has proven itself instead: a full backup with a fresh snapshot file once a week, and an increment against that full backup on the remaining days.

This rotation can be driven directly in the script based on the day of the week, by deliberately deleting the snapshot file on full-backup day before tar runs again. That automatically forces a new full backup on that day, without needing a separate code path branching between full and incremental backup, tar itself detects the missing snapshot file and starts fresh.

#!/usr/bin/env bash
set -euo pipefail

readonly SNAPSHOT="/backups/state/app.snapshot"
readonly ARCHIVE="/backups/app-$(date +%Y%m%d).tar.gz"

# Force a fresh full backup every Sunday by resetting the snapshot file
if [[ "$(date +%u)" == "7" ]]; then
  rm -f "$SNAPSHOT"
  echo "Sunday: forcing a full backup"
fi

tar --listed-incremental="$SNAPSHOT" -czf "$ARCHIVE" -C /var/www app

4. Handling a full disk: detect early instead of failing late

A full disk while an archive file is being written is one of the most dangerous failure modes, because while tar with set -e does set the exit code correctly, the already partially written archive file stays behind at the destination and, depending on when the abort happened, can even look deceptively valid. A robust script therefore checks upfront whether enough free disk space exists, rather than discovering the problem mid-write.

In addition, every backup script needs a post-write check: the finished archive should be verified for readability with tar -tzf before it is marked as a valid backup, and an incomplete file from an aborted run must be deleted consistently rather than left at the destination, so later restore attempts do not run into an archive that looks present but is actually broken.

#!/usr/bin/env bash
set -euo pipefail

readonly ARCHIVE="/backups/app-$(date +%Y%m%d).tar.gz"
readonly DEST_DIR="/backups"
readonly MIN_FREE_MB=2048

cleanup_on_failure() {
  [[ -f "$ARCHIVE" ]] && rm -f "$ARCHIVE"
  echo "Backup FAILED, removed incomplete archive: $ARCHIVE" >&2
}
trap cleanup_on_failure ERR

# Fail fast if there is not enough free space before starting
free_mb=$(df --output=avail -m "$DEST_DIR" | tail -n1 | tr -d ' ')
if (( free_mb < MIN_FREE_MB )); then
  echo "Not enough free space: ${free_mb}MB available, ${MIN_FREE_MB}MB required" >&2
  exit 1
fi

tar -czf "$ARCHIVE" -C /var/www app

# Verify the archive is actually readable before trusting it
if ! tar -tzf "$ARCHIVE" > /dev/null; then
  echo "Archive verification FAILED: $ARCHIVE is corrupt" >&2
  exit 1
fi
echo "Backup verified OK: $ARCHIVE"

5. Compression level vs. runtime: choosing the right tradeoff

gzip offers direct control over compression through the -1 to -9 options, where -1 is the fastest but least compressing level and -9 delivers the highest compression at noticeably higher computational cost. The default sits at -6 and is already a reasonable compromise for most use cases, but for a backup script it is worth making a deliberate choice instead of relying on the default.

For very large data volumes where the backup runtime has to fit into a tight maintenance window, a lower compression level like -3 is often the better choice, because the saved compute time usually outweighs the extra storage needed for the somewhat larger archive. For archives written rarely but transferred frequently over a slow network connection or stored for long periods, the benefit of a high compression level outweighs the cost instead.

#!/usr/bin/env bash
set -euo pipefail

# Fast backup within a tight maintenance window: lower compression
tar -cf - -C /var/www app | gzip -3 > /backups/app-fast.tar.gz

# Long-term archive storage: higher compression, more CPU time accepted
tar -cf - -C /var/www app | gzip -9 > /backups/app-archive.tar.gz

6. Parallelizing compression to soften the tradeoff

A commonly overlooked way out of the runtime-versus-compression tradeoff is to spread the compression itself across multiple CPU cores, instead of having to choose between a lower level and a long runtime. Tools like pigz are a direct drop-in replacement for gzip with identical command-line syntax, but use all available cores in parallel and reach the same compression level in a fraction of the time.

It is important to write a script so it automatically falls back to the always-available gzip when pigz is missing, so the same script works identically on a minimal server without extra packages and on a well-equipped backup host. That fallback logic costs only a few lines but prevents a script from simply failing on a system without pigz.

#!/usr/bin/env bash
set -euo pipefail

# Use pigz (parallel gzip) if available, fall back to plain gzip otherwise
compressor() {
  if command -v pigz &> /dev/null; then
    pigz -9
  else
    gzip -9
  fi
}

tar -cf - -C /var/www app | compressor > /backups/app-parallel.tar.gz

7. Regularly test-restoring backups

An archive verified as readable with tar -tzf only guarantees the structure is intact, not that every single file can actually be restored without error and that the application genuinely works with the restored state. The only check that truly justifies that confidence is a regular, real restore into an isolated test environment, followed by a simple functional check.

Such a restore test can easily be automated and run weekly or monthly as its own cron job, unpacking a randomly selected backup, checking it against an expected file list, and reporting the result to the same monitoring system as the actual backup run. A team that restores a backup for the first time only during an actual emergency always discovers problems at the worst possible moment.

8. Enforcing retention periods automatically

Without automated cleanup, backup directories grow indefinitely and sooner or later run into the very full-disk situation the script is supposed to handle. A simple retention rule, for example keeping daily backups for seven days and weekly full backups for twelve weeks, can be reliably enforced with find based on the file's modification time.

This cleanup should always run after successfully verifying the new backup, never before, so that if a new run fails, the last working backup is not accidentally deleted along with it. That ordering is the single most important difference between a cleanup step that helps in an emergency and one that causes additional damage in an emergency.

#!/usr/bin/env bash
set -euo pipefail

readonly BACKUP_DIR="/backups"
readonly KEEP_DAYS=7

# Only prune AFTER the new backup was verified successfully
find "$BACKUP_DIR" -maxdepth 1 -name '*.tar.gz' -mtime "+${KEEP_DAYS}" -print -delete

9. Backup strategies compared

The choice between a pure full backup, a pure incremental backup, and a combination of both mostly depends on the data volume, how often it changes, and the required restoration time. For small to medium data volumes with a comfortable time window, a daily full backup often remains the simplest and most reliable solution, since every restoration only needs a single archive.

10. Summary

11. FAQ: Automating tar and gzip in Bash: The Essentials at a Glance