Release Scripts with Checkpoints and Dry Run Mode

1. Why release scripts need dry run and checkpoints

Release scripts are the shell scripts with the greatest potential for damage: they modify databases, overwrite files, start and stop services, and flush caches, all in a production environment that handles real traffic. A release script that fails at step 7 of 12 and leaves no defined state creates extra work and downtime that a better structure could have prevented.

Dry run mode is the first safeguard: it runs all checks, validations, and logging, but skips every destructive operation. That lets a release script be fully simulated against the target environment before the real run, without changing any data. Checkpoints are the second safeguard: after each successfully completed step, a status flag is set, so that on failure and a restart, only the incomplete steps are repeated, not the entire release from scratch.

Rollback logic is the third layer: if a step fails, the release script automatically restores the state from before that step. This requires a snapshot or backup to be created before every destructive step. These three mechanisms, dry run, checkpoints, and rollback, together turn release scripts into deterministic, safe tools instead of nerve-wracking one-off operations.

2. Basic architecture of a release script

A production-ready release script follows a fixed structure: a header with dependencies and configuration, flag parsing, precondition checks, sequential step execution with checkpoint tracking, cleanup, and a final deployment report. Steps are implemented as functions that do exactly three things: capture the state before the change, apply the change, and call the stored rollback function on failure. This structure makes the release script readable, testable, and extensible without global side effects.

The central variable for dry run mode is DRY_RUN. Every function that makes changes calls its commands through a wrapper function run_cmd, which in dry run mode only prints the command instead of executing it. That is cleaner than scattering [[ $DRY_RUN -eq 1 ]] && echo checks throughout the whole release script. The wrapper function also logs every command with a timestamp to the audit log, both in dry run and in a real run.

#!/usr/bin/env bash
# release.sh: structured release script with dry-run and checkpoints
set -euo pipefail
IFS=$'\n\t'

readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly RELEASE_ID="$(date +%Y%m%d-%H%M%S)-$$"
readonly CHECKPOINT_DIR="/var/run/release-checkpoints"
readonly LOG_FILE="/var/log/releases/${RELEASE_ID}.log"
readonly ROLLBACK_STACK_FILE="${CHECKPOINT_DIR}/${RELEASE_ID}.rollback"

# ---- Flags (defaults) ----
DRY_RUN=0
START_FROM=""
SKIP_STEPS=()
FORCE=0

# ---- Core: run_cmd wraps all destructive operations ----
run_cmd() {
  local description="$1"; shift
  if [[ $DRY_RUN -eq 1 ]]; then
    printf '[DRY-RUN] %s\n  $ %s\n' "$description" "$*" >&2
    return 0
  fi
  printf '[EXEC]    %s\n  $ %s\n' "$description" "$*" >&2
  "$@"
}

# ---- Rollback stack: push commands to run on failure ----
push_rollback() {
  printf '%s\n' "$*" >> "$ROLLBACK_STACK_FILE"
}

execute_rollback() {
  if [[ ! -f "$ROLLBACK_STACK_FILE" ]]; then
    return 0
  fi
  echo "[ROLLBACK] Running rollback steps..." >&2
  # Execute in reverse order (tac = reverse cat)
  while IFS= read -r cmd; do
    eval "$cmd" || echo "[ROLLBACK WARN] Step failed: $cmd" >&2
  done < <(tac "$ROLLBACK_STACK_FILE")
  rm -f "$ROLLBACK_STACK_FILE"
}

3. Dry run mode: announcing commands instead of running them

Implementing a robust dry run mode in release scripts goes beyond simply skipping commands. A complete dry run performs all precondition checks (are the environment variables present, are dependent services reachable, are permissions correct) and for every step prints which command would run, with which arguments, and what side effects to expect. After a dry run, the operator sees exactly what the release script will do, without anything having changed.

Something important for dry run mode in release scripts: not every operation can be fully simulated. Database migrations that depend on the current database state can only print their SQL in a dry run, not their actual effects. In those cases the release script must explicitly document that the dry run is incomplete for that step. A good practice: each step function has a separate describe_step() function, called during dry run, that describes the planned step in human-readable terms.

4. Checkpoint flags: resuming aborted releases

Checkpoints in release scripts solve the problem of a partially executed release. If a release process fails at step 7 of 12 and is restarted after the fix, steps 1 through 6 should not run again. The checkpoint system writes a marker file to a dedicated directory after each successfully completed step. On startup, the release script checks which checkpoints already exist and skips the corresponding steps.

The checkpoint directory must be unique per release run, with the release ID as part of the path. That way, parallel releases on different servers cannot overwrite each other's checkpoints. The --resume option of the release script specifies the release ID whose checkpoints should be reused. With --from STEP_NAME, the release script can also start from a specific step without any stored checkpoints at all, useful for emergency deployments that only need to update a subset of components.

#!/usr/bin/env bash
set -euo pipefail

# ---- Checkpoint management ----
readonly CHECKPOINT_DIR="${CHECKPOINT_DIR:-/var/run/release-checkpoints}"
mkdir -p "$CHECKPOINT_DIR"

checkpoint_set() {
  local step="$1"
  touch "${CHECKPOINT_DIR}/${RELEASE_ID}.${step}"
  printf '[CHECKPOINT] %s completed\n' "$step" >&2
}

checkpoint_done() {
  local step="$1"
  [[ -f "${CHECKPOINT_DIR}/${RELEASE_ID}.${step}" ]]
}

checkpoint_clear() {
  rm -f "${CHECKPOINT_DIR}/${RELEASE_ID}."*
  printf '[CHECKPOINT] All checkpoints for %s cleared\n' "$RELEASE_ID" >&2
}

# ---- Step execution with checkpoint logic ----
run_step() {
  local step_name="$1"
  local step_fn="$2"

  # Skip if checkpoint already set (resume mode)
  if checkpoint_done "$step_name"; then
    printf '[SKIP] Step %s already completed (checkpoint)\n' "$step_name" >&2
    return 0
  fi

  # Skip if step is in SKIP_STEPS array
  local skip
  for skip in "${SKIP_STEPS[@]:-}"; do
    if [[ "$skip" == "$step_name" ]]; then
      printf '[SKIP] Step %s skipped (--skip flag)\n' "$step_name" >&2
      return 0
    fi
  done

  printf '\n[STEP] === %s ===\n' "$step_name" >&2
  local start_ts
  start_ts=$(date +%s)

  "$step_fn"  # Call the step function

  local duration=$(( $(date +%s) - start_ts ))
  printf '[STEP] %s completed in %ds\n' "$step_name" "$duration" >&2

  [[ $DRY_RUN -eq 0 ]] && checkpoint_set "$step_name"
}

# ---- Example step implementation ----
step_composer_install() {
  push_rollback "echo 'Composer rollback: restore vendor from backup'"
  run_cmd "Install composer dependencies" \
    composer install --no-dev --optimize-autoloader --no-interaction
}

step_db_migrate() {
  push_rollback "bin/magento setup:rollback --db-rollback-file=${DB_BACKUP_FILE:-}"
  run_cmd "Run database migrations" \
    bin/magento setup:upgrade --keep-generated
}

# ---- Main release sequence ----
main() {
  run_step "composer_install"  step_composer_install
  run_step "db_migrate"        step_db_migrate
  # Add further steps as needed
  [[ $DRY_RUN -eq 0 ]] && checkpoint_clear
  printf '\n[RELEASE] Completed: %s\n' "$RELEASE_ID" >&2
}

5. Rollback logic: rolling back automatically

Rollback in release scripts is not an optional feature, it is a basic requirement for production safety. The implementation follows the stack principle: before every destructive step, a rollback command is written to a stack. On failure, the rollback commands run in reverse order, the most recently added one first. That guarantees dependencies between steps are handled correctly: a database change that builds on a file change must be undone first, before the file is restored.

Rollback logic in release scripts must itself be fault tolerant. If a rollback step fails, the next rollback step must still run. Every error during rollback is logged, but not treated as fatal. At the end of the rollback, the release script prints a complete report: which rollback steps succeeded and which require manual intervention. This report is the basis for the post-mortem process.

6. Step-by-step logging and audit trail

A complete audit log is mandatory for release scripts in regulated environments and a valuable debugging aid everywhere else. For every step, the log records: start and end timestamps, the command that ran with all arguments, the exit code, stdout, and stderr. In dry run mode, the log records which commands would have run. That makes the audit log of a dry run and of a real release structurally identical, the only difference is a single flag value.

For release scripts in multi-server environments, a centralized log format matters. JSON-formatted log lines ({ "timestamp": "...", "release_id": "...", "step": "...", "status": "..." }) can be filtered with jq and imported into monitoring systems such as Elasticsearch or Datadog. The release ID in every log line lets you correlate all logs of a release across every server involved, essential for incident analysis after a failed deploy.

#!/usr/bin/env bash
set -euo pipefail

readonly LOG_FILE="${LOG_DIR:-/var/log/releases}/${RELEASE_ID:-test}.log"
mkdir -p "$(dirname "$LOG_FILE")"

# ---- Structured JSON audit log ----
audit_log() {
  local level="$1" step="$2" message="$3"
  local ts
  ts="$(date --iso-8601=seconds 2>/dev/null || date -u '+%Y-%m-%dT%H:%M:%SZ')"
  printf '{"timestamp":"%s","release_id":"%s","level":"%s","step":"%s","message":"%s","dry_run":%s}\n' \
    "$ts" "${RELEASE_ID:-unknown}" "$level" "$step" \
    "${message//\"/\\\"}" "$([[ ${DRY_RUN:-0} -eq 1 ]] && echo true || echo false)" \
    >> "$LOG_FILE"
}

# ---- Step report summary ----
declare -A STEP_STATUS=()
declare -A STEP_DURATION=()

record_step_result() {
  local step="$1" status="$2" duration="$3"
  STEP_STATUS["$step"]="$status"
  STEP_DURATION["$step"]="$duration"
  audit_log "$status" "$step" "Step completed in ${duration}s"
}

print_release_report() {
  local sep
  sep="$(printf '%0.s─' {1..60})"
  printf '\n%s\n' "$sep" >&2
  printf '%-30s %10s %10s\n' "RELEASE REPORT: ${RELEASE_ID:-unknown}" "STATUS" "DURATION" >&2
  printf '%s\n' "$sep" >&2
  for step in "${!STEP_STATUS[@]}"; do
    local st="${STEP_STATUS[$step]}"
    local dur="${STEP_DURATION[$step]:-?}s"
    printf '%-30s %10s %10s\n' "$step" "$st" "$dur" >&2
  done
  printf '%s\n' "$sep" >&2
  printf 'Full log: %s\n' "$LOG_FILE" >&2
}

# Ensure report is printed on exit
trap print_release_report EXIT

7. Command line flags with getopts and manual parsing

Release scripts need a complete command line interface. Bash's built-in options are getopts for single-letter flags (-d, -f, -v) and manual parsing for long flags (--dry-run, --from, --skip, --resume). getopts is portable and correct, but only for short options. For release scripts with many options, manual parsing in a while/case loop that consumes $1 and advances with shift is the better choice.

The flag-parsing module of a release script should produce a clear error message and a usage text for unknown flags. With a usage() function that lists every flag with its description and default value, the release script becomes self-documenting. --help prints usage() and exits the script with exit code 0. --version prints the version number. These conventions make release scripts usable by new team members without any separate documentation.

8. Stage-based execution with dependencies

Complex release scripts can be structured with stage dependencies. Instead of a simple sequential list of steps, you define stages with optional dependencies: stage B can only run once stage A has succeeded. That allows independent stages to run in parallel while enforcing the correct order for dependent stages. In Bash, a simple dependency check can be implemented through checkpoint flags: before a stage runs, it checks whether all dependency checkpoints are set.

A practical pattern for release scripts: different environments (dev, staging, prod) with different stage configurations. A configuration map assigns each environment which stages run and which are skipped. In dev, the database migration phase runs in a dry-run-like mode that only prints the SQL. In staging, everything runs in full. In prod, a backup is automatically created before the database migration phase and a checkpoint is set with the backup path, so the correct backup gets used on rollback.

9. Release script patterns compared

Different implementation approaches for release scripts have different strengths. The right choice depends on the release's complexity, the team size, and the requirements around rollback and auditability.

For most production environments, the checkpoint pattern is the best starting point. It stays manageably complex, can be added into existing release scripts incrementally, and immediately delivers the biggest benefit: aborted releases can resume safely without repeating steps that already completed. The stage-based pattern is worth adopting once multiple independent systems need to be coordinated within a single release, for example with microservice deployments or multi-region rollouts.

10. Summary

Production-ready release scripts need three central safety mechanisms: dry run mode for risk-free pre-testing, checkpoint flags for safely resuming after interruptions, and rollback logic for automatic recovery on failure. Together, these mechanisms turn release scripts from nerve-wracking one-off operations into deterministic, testable tools. The structural effort, the run_cmd wrapper function, the checkpoint directory, and the rollback stack, pays off the first time an aborted release can resume without any data loss.

A complete audit log with a JSON structure and a release ID is the other essential piece. It enables tracking every deployment: who deployed what and when, which steps ran, where it stopped, and how the rollback was carried out. For teams running multiple release scripts for different components, a shared library (lib/release.sh) with the checkpoint, rollback, and logging functions is worth building. That ensures every release script uses the same safety mechanisms.

11. FAQ: Release Scripts with Checkpoints and Dry Run