Production Bash Checklist: Security, Robustness, Logging, and Tests

1. Why a Bash Checklist Is Necessary

Shell scripts are treated differently from application code in most projects. There is no code review, no automated testing, no consistent standards. The result: production environments run scripts that fail silently on certain inputs, write files without permission checks, leak sensitive data into logs, or leave inconsistent states behind when interrupted. A systematic Bash checklist closes this gap.

The Bash checklist in this article is organized into five dimensions: script foundation, security, robustness, logging, and tests. Each dimension contains concrete, verifiable points, not vague recommendations. The points are ordered by priority, so even a quick review under time pressure answers the most critical questions first. The checklist is meant as a basis for your own code review templates and can be used directly as a GitHub pull request checklist or integrated into a pre-commit hook.

Complementing it with automated tools is essential: ShellCheck as a static analyzer catches most quoting and syntax errors automatically, while BATS (Bash Automated Testing System) enables unit-style tests for shell functions. Both tools are trivial to integrate into CI pipelines. The manual Bash checklist remains necessary nonetheless, because it covers aspects that static analysis cannot detect, such as logging completeness, cleanup logic, and security requirements for sensitive data.

2. Checklist: Script Foundation and Header

The foundation of every production-ready shell script starts with a complete header. The #!/usr/bin/env bash shebang uses the system search mechanism for Bash, which is more portable on systems with multiple Bash versions (macOS vs. Linux) than an absolute path. The combination set -euo pipefail is the single most important item on the Bash checklist: -e aborts on errors, -u treats unset variables as errors, -o pipefail propagates errors through pipes. Without these three options, basic error handling is missing.

The Bash checklist for the header also includes: IFS=$'\n\t' removes the space from the field separator and prevents word splitting during variable expansion. SCRIPT_DIR should always be set with cd "$(dirname "${BASH_SOURCE[0]}")" && pwd, so that relative paths in the script work regardless of the current working directory. Bundle all configuration variables at the top, either from the environment (${VAR:?error}) or with default values (${VAR:-default}). That makes dependencies explicit and simplifies review.

#!/usr/bin/env bash
# deploy.sh: Production deployment script
# Checklist: header section
#
# Required environment variables:
#   DEPLOY_ENV     : target environment (dev|staging|prod)
#   DEPLOY_USER    : SSH user for remote host
#   DEPLOY_HOST    : remote host FQDN or IP
#
# Optional environment variables:
#   LOG_LEVEL      : log verbosity (DEBUG|INFO|WARN|ERROR), default: INFO
#   DRY_RUN        : set to 1 to skip destructive operations, default: 0
set -euo pipefail
IFS=$'\n\t'

# ---- Resolve script directory (portable) ----
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "$0")"
readonly TIMESTAMP="$(date +%Y%m%d-%H%M%S)"

# ---- Mandatory variables: abort with message if missing ----
DEPLOY_ENV="${DEPLOY_ENV:?Variable DEPLOY_ENV ist nicht gesetzt (dev|staging|prod)}"
DEPLOY_USER="${DEPLOY_USER:?Variable DEPLOY_USER ist nicht gesetzt}"
DEPLOY_HOST="${DEPLOY_HOST:?Variable DEPLOY_HOST ist nicht gesetzt}"

# ---- Optional variables with safe defaults ----
LOG_LEVEL="${LOG_LEVEL:-INFO}"
DRY_RUN="${DRY_RUN:-0}"
LOG_DIR="${LOG_DIR:-/var/log/deploy}"

# ---- Validate DEPLOY_ENV ----
case "$DEPLOY_ENV" in
  dev|staging|prod) ;;
  *) echo "[ERROR] DEPLOY_ENV muss dev, staging oder prod sein" >&2; exit 1 ;;
esac

readonly LOG_FILE="${LOG_DIR}/${SCRIPT_NAME%.sh}-${TIMESTAMP}.log"

3. Checklist: Security and Injection Prevention

The most critical area of the Bash checklist is security. Shell scripts are vulnerable to command injection whenever external data, such as user input, API responses, or filenames from unknown directories, is inserted into commands without quoting or validation. The basic rule: treat every external input as potentially hostile. Always use double quotes for variables that hold external data: "$variable" instead of $variable. Never insert input directly into eval, bash -c, or similar constructs.

Another item on the security Bash checklist: always create temporary files with mktemp, never with fixed paths like /tmp/script.tmp. Fixed paths are vulnerable to symlink attacks on multi-user systems. With trap 'rm -f "$tmpfile"' EXIT, temporary files are cleaned up even on error abort. Secrets such as passwords, API keys, and tokens must never be passed as command-line arguments, because they then become visible in the process list (ps aux). Use environment variables or temporary files with restricted permissions instead.

4. Checklist: Robustness and Error Handling

The robustness dimension of the Bash checklist covers all measures that protect a script against unexpected states. Besides set -euo pipefail, this includes a complete trap setup: trap cleanup EXIT for the cleanup function, trap 'echo Interrupted; exit 130' INT TERM for signal handling. The cleanup function must be idempotent: it can be called multiple times (directly and via the EXIT trap) and must not produce an error in that case.

For external dependencies, a preflight check belongs on the Bash checklist: at the start of the script, verify all required tools with command -v tool >/dev/null 2>&1 || { echo Tool missing; exit 1; }. That gives a clear error message instead of the script failing mid-execution with a cryptic "command not found". For critical operations such as database migrations or file deletions, the Bash checklist recommends explicit rollback logic: back up the state before the operation and restore it on failure.

#!/usr/bin/env bash
set -euo pipefail

# ---- Checklist: dependency verification ----
check_dependencies() {
  local -a missing=()
  local deps=("curl" "jq" "rsync" "ssh" "mktemp")
  for dep in "${deps[@]}"; do
    command -v "$dep" &>/dev/null || missing+=("$dep")
  done
  if (( ${#missing[@]} > 0 )); then
    echo "[ERROR] Fehlende Abhängigkeiten: ${missing[*]}" >&2
    echo "[ERROR] Installieren mit: apt-get install ${missing[*]}" >&2
    exit 1
  fi
}

# ---- Checklist: safe temp files ----
readonly TMP_DIR="$(mktemp -d)"
readonly TMP_CONFIG="${TMP_DIR}/config.json"

cleanup() {
  local exit_code=$?
  # Idempotent: safe to call multiple times
  rm -rf "${TMP_DIR:-}" 2>/dev/null || true
  [[ $exit_code -ne 0 ]] && echo "[ERROR] Abgebrochen mit Code $exit_code" >&2
  return 0
}
trap cleanup EXIT
trap 'echo "[ABORT] Signal erhalten, breche ab" >&2; exit 130' INT TERM

# ---- Checklist: validate inputs before use ----
validate_env() {
  local env="$1"
  [[ "$env" =~ ^(dev|staging|prod)$ ]] || {
    echo "[ERROR] Ungültige Umgebung: $env" >&2; exit 1
  }
}

# ---- Checklist: safe file operations ----
safe_write() {
  local target="$1" content="$2"
  local backup="${target}.bak.$(date +%s)"
  # Create backup before overwriting
  [[ -f "$target" ]] && cp "$target" "$backup"
  printf '%s\n' "$content" > "$target"
}

check_dependencies
validate_env "${DEPLOY_ENV:-}"

5. Checklist: Logging and Observability

The logging area of the Bash checklist ensures that scripts stay observable in production. Every production script needs structured logging with timestamps, log levels, and context. The minimum requirement: a logging function that outputs an ISO 8601 timestamp, the level, and the message. The entire stdout/stderr stream should be redirected into a log file with exec > >(tee -a "$LOG_FILE") 2>&1, without losing pipe compatibility in the process.

The logging Bash checklist also includes: sensitive data must never be logged. Passwords, tokens, and private keys must be masked before output. A redact() function is useful here: it replaces known secret environment variables with [REDACTED] before the script outputs env | sort for debugging purposes. Log rotation matters for scripts that run frequently: automatically archive or delete log files older than N days so disks do not fill up.

6. ShellCheck: Integrating Static Analysis into CI

ShellCheck is the most important automated tool on the Bash checklist. It statically finds quoting errors (SC2086), unsafe array expansion (SC2068), lost exit codes from local variable declarations (SC2155), unnecessary subshells, and many other common mistakes. Running shellcheck -S warning script.sh outputs all warnings and errors with an explanation and a suggested fix. With shellcheck --format=gcc, the output can be integrated into IDE formats.

Integrating it into CI is trivial: in GitHub Actions, a ready-made action is enough (uses: ludeeus/action-shellcheck@master), and in other CI systems, a simple shellcheck -S warning $(find . -name "*.sh") does the job. For the Bash checklist, the rule is: ShellCheck warnings are not ignored, they are fixed or deliberately suppressed with # shellcheck disable=SC2034, together with a comment explaining why the suppression is justified. That makes intentional exceptions visible and prevents ShellCheck directives from being abused as a general way to silence errors.

#!/usr/bin/env bash
# test/run-checks.sh: CI quality gate for all shell scripts
set -euo pipefail

readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"

# ---- ShellCheck all scripts ----
run_shellcheck() {
  echo "=== ShellCheck ==="
  local -a scripts=()
  while IFS= read -r -d '' f; do
    scripts+=("$f")
  done < <(find "$PROJECT_ROOT" -name "*.sh" -not -path "*/vendor/*" -print0)

  if (( ${#scripts[@]} == 0 )); then
    echo "Keine .sh-Dateien gefunden"
    return 0
  fi

  local failed=0
  for script in "${scripts[@]}"; do
    if shellcheck -S warning "$script"; then
      printf '  [OK]  %s\n' "${script#$PROJECT_ROOT/}"
    else
      printf '  [FAIL] %s\n' "${script#$PROJECT_ROOT/}"
      (( failed++ ))
    fi
  done

  if (( failed > 0 )); then
    echo "[ERROR] ShellCheck: $failed Dateien mit Warnungen" >&2
    return 1
  fi
  echo "ShellCheck: Alle Skripte bestanden"
}

# ---- Bash syntax check ----
run_syntax_check() {
  echo "=== Syntax-Check ==="
  local failed=0
  while IFS= read -r -d '' script; do
    bash -n "$script" && printf '  [OK]  %s\n' "${script#$PROJECT_ROOT/}" \
      || { printf '  [FAIL] %s\n' "${script#$PROJECT_ROOT/}"; (( failed++ )); }
  done < <(find "$PROJECT_ROOT" -name "*.sh" -not -path "*/vendor/*" -print0)
  (( failed == 0 ))
}

run_syntax_check
run_shellcheck

7. BATS: Automated Tests for Shell Scripts

BATS (Bash Automated Testing System) brings unit-style tests to the shell. It makes it possible to test individual Bash functions in isolation, and it is its own area of the Bash checklist. A BATS test defines a test case with @test "description" { ... }, which runs a function or command and checks the result with [ "$output" = "expected value" ] and [ "$status" -eq 0 ]. The special variable $output contains the combined stdout/stderr output of the last run command.

For the testing Bash checklist, the rule is: functions that live in the library (lib/) are the primary test targets. Scripts that call external commands are decoupled from real network calls with stub functions (curl() { echo '{"status":"ok"}'; }). BATS provides setup and teardown hooks (setup(), teardown()) that create and clean up temporary directories. The Bash checklist requires at least one success case, one failure case, and one edge case test for every public library function.

8. Enforcing the Bash Checklist in the CI Pipeline

Automated enforcement of the Bash checklist in CI pipelines is the decisive step from good intentions to lived practice. Every pull request that changes shell scripts must automatically run through ShellCheck, BATS, and the syntax check. In GitHub Actions, a workflow with three jobs is enough: shellcheck, bats, and an optional security-scan with detect-secrets or trufflehog for secrets in scripts.

As a pre-commit hook, the Bash checklist can be integrated directly into the local Git repository with the pre-commit framework. Relevant hooks: shellcheck from the official hook repository, check-executables-have-shebangs, and detect-private-key. That ensures ShellCheck warnings are visible locally before the commit, not only later in the CI pipeline after the push. The Bash checklist used as a template for manual reviews together with automated gates adds up to a complete quality system for shell scripts.

#!/usr/bin/env bats
# test/lib_utils.bats: BATS tests for lib/utils.sh functions

load '../lib/utils.sh'

# ---- Setup and teardown ----
setup() {
  export TMP_TEST_DIR
  TMP_TEST_DIR="$(mktemp -d)"
}

teardown() {
  rm -rf "${TMP_TEST_DIR:-/nonexistent}"
}

# ---- Tests for validate_env() ----
@test "validate_env accepts valid environments" {
  run validate_env "prod"
  [ "$status" -eq 0 ]
  run validate_env "staging"
  [ "$status" -eq 0 ]
  run validate_env "dev"
  [ "$status" -eq 0 ]
}

@test "validate_env rejects invalid environment" {
  run validate_env "production"
  [ "$status" -eq 1 ]
  [[ "$output" == *"Ungültige Umgebung"* ]]
}

# ---- Tests for safe_write() ----
@test "safe_write creates backup of existing file" {
  local target="${TMP_TEST_DIR}/config.txt"
  printf 'old content\n' > "$target"
  safe_write "$target" "new content"
  # Original content preserved in backup
  local backup
  backup="$(ls "${target}.bak."* 2>/dev/null | head -1)"
  [ -n "$backup" ]
  grep -q "old content" "$backup"
}

@test "safe_write creates new file when target does not exist" {
  local target="${TMP_TEST_DIR}/new-file.txt"
  safe_write "$target" "initial content"
  [ -f "$target" ]
  grep -q "initial content" "$target"
}

# ---- Tests for check_dependencies() ----
@test "check_dependencies passes when all tools present" {
  # Stub missing tools if needed
  run check_dependencies
  [ "$status" -eq 0 ]
}

9. Checklist Items Compared

The following table shows which items on the Bash checklist are automatically caught by ShellCheck and which require manual review work. This distinction helps estimate review effort correctly and avoids manually repeating checks that can already be automated.

ShellCheck reliably covers syntactic and quoting-related problems. BATS validates function behavior under various conditions. Manual review focuses on security aspects that no tool can detect: are secrets handled safely? Is the cleanup logic complete? Does an interruption leave a consistent state behind? Together, these three layers of the Bash checklist cover the full spectrum, from obvious syntax errors to subtle security problems.

10. Summary

A systematic Bash checklist is the most important tool for raising shell scripts to the level of application code. The five dimensions, foundation, security, robustness, logging, and tests, cover the full spectrum of typical Bash problems. ShellCheck automates the syntactic and quoting-related checks, BATS enables functional tests for library functions, and manual review focuses on the security and completeness questions that cannot be automated.

The greatest practical benefit comes from combining automated CI gates with regular manual reviews. Using the Bash checklist as a pull request template ensures that no script enters the repository without a basic check. ShellCheck and BATS as mandatory gates in the CI pipeline prevent regressions. And a shared Bash library with centralized logging, cleanup, and validation functions turns the Bash checklist into living documentation of the shell coding standard for the project.

11. FAQ: Bash Checklist for Code Reviews