File Sizes, Checksums and Integrity: Automated Verification

1. Why verify file integrity automatically?

Wanting to verify file integrity automatically is not paranoia, it is an operational necessity. In production environments, files change due to faulty deployments, failing storage media, unnoticed intrusions or simply transfer errors when copying over the network. Anyone who does not systematically detect these changes lives with a blind spot in their infrastructure. The result: a broken backup that only surfaces when it is needed, or a tampered configuration file that stays undetected for weeks.

Checksums are the standard tool for being able to verify file integrity automatically. A cryptographic hash value is a digital fingerprint of a file: even the smallest change to the content produces a completely different hash. Once the hash value is stored or transmitted alongside the file, a single shell command can verify at any later point whether the file is unchanged. Bash provides everything needed to automate this process with md5sum, sha256sum and sha512sum.

The value lies not in a single check, but in automating it. A Bash script that compares checksums after every deployment, verifies archived files after every backup run, and reports on directory changes daily via cron creates continuous observability of the filesystem. That is the foundation for traceable, reliable automation.

2. md5sum, sha256sum and sha512sum compared

The three most common tools to verify file integrity automatically are md5sum, sha256sum and sha512sum. MD5 produces a 128 bit hash and is extremely fast, but is considered cryptographically broken since collisions can be computed. For checking accidental corruption or simple transfer errors, MD5 is still practical because no attacker is involved. For security relevant checks, such as verifying deployment artifacts or configuration files that must not have been tampered with, SHA-256 is the current standard.

SHA-512 is computationally a bit heavier than SHA-256, but on 64 bit systems it often delivers comparable or even better performance because it is natively implemented with 64 bit operations. For the daily practice of automated file integrity verification, SHA-256 is a sensible compromise between security, speed and broad toolchain support. All three tools follow the same interface: input is a list of file paths, output is one line per file with the hash and filename, separated by two spaces.

#!/usr/bin/env bash
# integrity-check.sh: Generate and verify file checksums
set -euo pipefail
IFS=$'\n\t'

readonly CHECKSUM_DIR="/var/lib/integrity-checks"
readonly TARGET_DIR="${1:?Usage: $0 <target-directory>}"

# Generate SHA-256 checksums for all files in directory
generate_checksums() {
  local dir="$1"
  local output_file="$2"
  # find handles filenames with spaces and special characters safely
  find "$dir" -type f -print0 \
    | sort -z \
    | xargs -0 sha256sum \
    > "$output_file"
  echo "[OK] Generated $(wc -l < "$output_file") checksums -> $output_file"
}

# Verify checksums against previously stored baseline
verify_checksums() {
  local baseline="$1"
  local failed=0
  while IFS= read -r line; do
    expected_hash="${line%% *}"
    filepath="${line#*  }"
    if [[ -f "$filepath" ]]; then
      actual_hash="$(sha256sum "$filepath" | cut -d' ' -f1)"
      if [[ "$actual_hash" != "$expected_hash" ]]; then
        echo "[FAIL] CHANGED: $filepath" >&2
        (( failed++ )) || true
      fi
    else
      echo "[FAIL] MISSING: $filepath" >&2
      (( failed++ )) || true
    fi
  done < "$baseline"
  return "$failed"
}

mkdir -p "$CHECKSUM_DIR"
baseline="$CHECKSUM_DIR/$(basename "$TARGET_DIR")-$(date +%Y%m%d).sha256"
generate_checksums "$TARGET_DIR" "$baseline"

3. Creating and verifying checksum files

Creating a checksum file with sha256sum is a two liner, wrapping the automated verification in a robust Bash script is the real work. The built in verify option sha256sum --check checksums.sha256 reads the checksum file, checks every file and prints a compact summary. This option is ideal for interactive checks, but for script driven automation it lacks control over exit codes and error details. A custom script that processes the checksum file line by line gives granular control over how missing, changed or new files are handled.

A proven pattern for automated file integrity verification: at the first deployment, or after a deliberate update, a baseline checksum file is created and version controlled. On every subsequent deployment, the CI system compares the current state against this baseline. Discrepancies are treated as errors and the deploy is aborted. The checksum file thus acts as an immutable reference, similar to a lock file, but for filesystem contents instead of package versions.

#!/usr/bin/env bash
# deploy-verify.sh: Verify deployment artifacts before go-live
set -euo pipefail

readonly BASELINE="/etc/deploy/checksums-baseline.sha256"
readonly DEPLOY_DIR="/var/www/html"
readonly REPORT_FILE="/tmp/integrity-report-$(date +%Y%m%d-%H%M%S).txt"

declare -i changed=0 missing=0 extra=0

check_integrity() {
  # Verify all files in baseline still match
  while IFS="  " read -r expected_hash filepath; do
    if [[ ! -f "$filepath" ]]; then
      echo "MISSING: $filepath" | tee -a "$REPORT_FILE"
      (( missing++ )) || true
    else
      actual_hash="$(sha256sum "$filepath" | awk '{print $1}')"
      if [[ "$actual_hash" != "$expected_hash" ]]; then
        echo "CHANGED: $filepath (expected: ${expected_hash:0:12}... got: ${actual_hash:0:12}...)" \
          | tee -a "$REPORT_FILE"
        (( changed++ )) || true
      fi
    fi
  done < "$BASELINE"

  # Detect files not in baseline (potentially injected)
  while IFS= read -r -d '' f; do
    if ! grep -qF "  $f" "$BASELINE"; then
      echo "EXTRA: $f" | tee -a "$REPORT_FILE"
      (( extra++ )) || true
    fi
  done < <(find "$DEPLOY_DIR" -type f -print0)

  echo "--- Summary: changed=$changed missing=$missing extra=$extra ---" \
    | tee -a "$REPORT_FILE"
}

check_integrity
if (( changed + missing > 0 )); then
  echo "[ERROR] Integrity check failed, aborting deploy" >&2
  exit 1
fi
echo "[OK] All checksums verified"

4. Filtering and monitoring file sizes with find

Besides checksums, monitoring file sizes is its own aspect of verifying file integrity automatically. Unexpected size changes can indicate file corruption, a stalled writer process or accidentally truncated files. With -size, find offers a direct filter for file sizes: find /var/log -size +100M finds all log files over 100 megabytes. The units are c for bytes, k for kilobytes, M for megabytes and G for gigabytes. Without a sign, find searches for an exact size, with + for greater than, with - for less than.

Combining size filters with integrity checks is especially valuable in backup scenarios. A backup file that is smaller than expected is suspicious, it may have been cut off while writing. A Bash script can check after a backup run whether the resulting archives meet a minimum size, whether the size sits in the expected range relative to the previous backup, and whether the checksum is correct. This three stage check reliably catches the most common types of backup failure.

5. Detecting and cleaning up duplicates

Duplicate files are a common problem on servers that have grown over years: backup copies in the wrong place, deployment artifacts copied multiple times, or import pipelines that process the same file repeatedly. The most efficient pattern to verify file integrity automatically while also detecting duplicates uses checksums as a grouping key. All files with an identical hash are duplicates, regardless of name and path.

The workflow in Bash: compute checksums for all files, sort by hash, isolate groups with more than one entry. This is done with sort, uniq -d on the hash column and a subsequent filter stage. For large directories, it pays off to first group by file size, duplicates always share the same size, and comparing hashes only within the size group saves a considerable amount of compute time. This two stage pattern (size first, then hash) is the standard used in professional duplicate scanner tools.

#!/usr/bin/env bash
# find-duplicates.sh: Detect duplicate files by content hash
set -euo pipefail

readonly SCAN_DIR="${1:?Usage: $0 <directory>}"
readonly MIN_SIZE="${2:-1k}"  # ignore files smaller than this

declare -i dup_groups=0 dup_files=0 freed_bytes=0

echo "[INFO] Scanning $SCAN_DIR for duplicates (min size: $MIN_SIZE)..."

# Step 1: group by size first (fast pre-filter)
declare -A size_groups
while IFS= read -r -d '' f; do
  size="$(stat -c '%s' "$f")"
  size_groups["$size"]+="$f"$'\0'
done < <(find "$SCAN_DIR" -type f -size +"$MIN_SIZE" -print0)

# Step 2: for each size group with multiple files, compare by hash
tmpfile="$(mktemp)"
trap 'rm -f "$tmpfile"' EXIT

for size in "${!size_groups[@]}"; do
  group_files=()
  while IFS= read -r -d '' f; do
    group_files+=("$f")
  done <<< "${size_groups[$size]}"

  if (( ${#group_files[@]} < 2 )); then continue; fi

  # Hash all files in this size group
  for f in "${group_files[@]}"; do
    sha256sum "$f" >> "$tmpfile"
  done
done

# Step 3: find duplicate hashes
sort "$tmpfile" | awk '{print $1}' | sort | uniq -d | while read -r dup_hash; do
  echo "=== Duplicate group (SHA-256: ${dup_hash:0:16}...) ==="
  grep "^$dup_hash" "$tmpfile" | awk '{print $2}' | while IFS= read -r f; do
    size_bytes="$(stat -c '%s' "$f")"
    echo "  $f  ($(numfmt --to=iec "$size_bytes"))"
    (( dup_files++ )) || true
    (( freed_bytes += size_bytes )) || true
  done
  (( dup_groups++ )) || true
done

echo ""
echo "[RESULT] $dup_groups duplicate groups, $dup_files files"
echo "[RESULT] Potential space saving: $(numfmt --to=iec "$freed_bytes")"

6. Continuous integrity monitoring

A one off checksum check at deployment is not enough to verify file integrity automatically during ongoing operation. Production systems need continuous monitoring that reports filesystem changes promptly. The classic tool for this is a cron job that compares the checksums of critical directories against a baseline hourly or daily and sends a notification on any discrepancy.

For efficient continuous monitoring, it is important to sensibly prioritize which directories to watch. Configuration files under /etc, web server document roots, deployment artifacts and binaries under /usr/local/bin are typical candidates. Temporary directories, logs and caches should be excluded, they change by design, and false positives cause real alerts to get overlooked. The baseline file itself must be read only or stored outside the monitored system, so an attacker cannot tamper with it together with the files it tracks.

#!/usr/bin/env bash
# integrity-monitor.sh: Continuous file integrity monitoring via cron
# Cron: */30 * * * * /usr/local/bin/integrity-monitor.sh >> /var/log/integrity.log 2>&1
set -euo pipefail

readonly BASELINE_DIR="/etc/integrity-baselines"
readonly ALERT_EMAIL="${ALERT_EMAIL:-admin@mironsoft.de}"
readonly STATE_DIR="/var/lib/integrity-monitor"
mkdir -p "$STATE_DIR"

declare -a WATCH_DIRS=(
  "/etc/nginx"
  "/etc/php"
  "/var/www/html/app/code"
  "/usr/local/bin"
)

declare -a ALERTS=()

check_directory() {
  local dir="$1"
  local safe_name
  safe_name="$(echo "$dir" | tr '/' '_' | tr -s '_' | sed 's/^_//')"
  local baseline="$BASELINE_DIR/${safe_name}.sha256"
  local current_checksums
  current_checksums="$(mktemp)"
  trap 'rm -f "$current_checksums"' RETURN

  # Build current state
  find "$dir" -type f -not -name "*.log" -not -name "*.tmp" -print0 \
    | sort -z \
    | xargs -0 sha256sum 2>/dev/null \
    > "$current_checksums"

  # First run: just store baseline
  if [[ ! -f "$baseline" ]]; then
    cp "$current_checksums" "$baseline"
    echo "[INFO] Baseline created for $dir"
    return
  fi

  # Compare against baseline
  local diff_output
  diff_output="$(diff "$baseline" "$current_checksums" 2>&1 || true)"
  if [[ -n "$diff_output" ]]; then
    ALERTS+=("[ALERT] Changes detected in $dir:"$'\n'"$diff_output")
  fi
}

for dir in "${WATCH_DIRS[@]}"; do
  [[ -d "$dir" ]] && check_directory "$dir"
done

if (( ${#ALERTS[@]} > 0 )); then
  report="$(printf '%s\n\n' "${ALERTS[@]}")"
  echo "$report"
  # Send alert email if mail command is available
  if command -v mail &>/dev/null; then
    echo "$report" | mail -s "[INTEGRITY] Changes detected on $(hostname)" "$ALERT_EMAIL"
  fi
  exit 1
fi

echo "[OK] Integrity check passed at $(date)"

7. Validating backups with checksums

Validating backup files is one of the most critical use cases for implementing automated file integrity verification. A backup whose integrity is never verified is not a reliable backup, it is an unchecked hope. The classic weak spot: the backup script runs to completion and reports success, but the resulting file is corrupted by a write error, a full disk or a crash mid process. That only comes to light when the restore is needed.

A robust backup script that implements automated file integrity verification combines three stages: first, a pre backup snapshot of the source files is taken. Second, the backup file is created and a checksum of it is computed. Third, the archive is test extracted and the checksums of the extracted contents are compared against the pre backup snapshot. Only when all three stages match is the backup considered valid. This three part check reliably catches errors in writing, compressing and archiving.

8. Performance: checksums on large directories

On servers with hundreds of thousands of files, computing checksums in full can take minutes and generate significant I/O pressure. There are several optimization strategies for a performant automated file integrity verification approach. The first is incremental hashing: only files modified since the last check (identifiable by mtime or ctime) get rehashed. Unchanged files keep their cached hash. This reduces the effort for regular checks to a small fraction of the original cost.

The second approach is parallelization. xargs -P 8 sha256sum distributes the hash computation across eight parallel processes and makes use of multi core systems. On SSDs, this parallelization is almost always a win, since the bottleneck is the CPU, not the storage medium. On mechanical hard drives, too much parallelism can be counterproductive due to random access, serial processing is often faster there. The third approach is prioritization: critical system files get checked hourly, large media files daily, archive data weekly. This creates a tiered safety net that avoids I/O pressure during sensitive peak times.

9. Checksum tools compared side by side

Choosing the right tool for automated file integrity verification depends on security requirements, performance and compatibility. This table shows the key differences between the common tools.

For everyday automated file integrity verification practice, SHA-256 is the industry standard. It is available everywhere, fast enough, cryptographically secure and supported by every build system, package manager and CI platform. BLAKE2 (b2sum) is a modern alternative that is not yet preinstalled on some systems, but for new projects without legacy requirements it is an excellent choice.

10. Summary

Automated file integrity verification with Bash combines three tool classes: checksum tools like sha256sum for cryptographic fingerprints, find with size filters for structural anomalies, and hash based duplicate detection for redundant storage problems. Together they form a complete system for monitoring filesystem state that fits entirely into existing shell based automation.

The key to effectiveness is not the tool, it is the discipline behind it. Checksums without a baseline are worthless. A baseline without regular comparison is a one off snapshot with no monitoring attached. And monitoring without a clear escalation path just produces reports nobody reads. Only when all three components, baseline, regular checking and alerting, work together does real observability of file integrity emerge in production.

11. FAQ: File Sizes, Checksums and Integrity: Automated Verification