Error Analysis in Deploy Scripts: Reproducible and Logged

1. The problem with deploy errors that leave no trace

A failed deploy without a meaningful log is a nightmare for any team. The questions are always the same: which step failed? What value did a given variable hold? Was the server even reachable? Was the error deterministic, or was a race condition involved? Without systematic error analysis in deploy scripts, root-causing stays guesswork, and the fix is at best a patch for the symptom, not the cause.

The real problem is not a lack of information as such, but a lack of reproducible information. Bash scripts produce little output by default. No timestamp, no line numbers, no expanded variable values. What the developer sees is a cryptic fragment of an error message, if anything at all. The tools for systematic error analysis in deploy scripts already exist in Bash: set -x, trap ERR, LINENO, BASH_LINENO and structured log files. They just need to be used consistently.

The difference between a team that handles deploys analytically and one that guesses anew at every outage lies exactly here: in the quality of the logging and the rigor of the error analysis in deploy scripts. A deploy script that, on failure, states precisely which command failed on which line with which exit code, which environment variables were active, and what output the failed command produced, is the foundation for an analysis measured in minutes instead of hours.

2. set -x: seeing every command with expanded values

set -x is the most powerful debugging tool for error analysis in deploy scripts. It enables trace mode: every command that runs is printed to stderr before execution, with variables fully expanded, globs resolved and subshell results included. The prefix + (or ++ for commands inside subshells) marks each trace line. This makes it immediately visible which variables held which concrete values, and whether a path or argument expanded as expected.

The pattern for selective tracing: check ${DEBUG:-0} at the start of the script and only enable set -x when DEBUG=1. That way the script runs in production without trace output, but can be switched into debug mode at any time by setting the environment variable, with no code changes required. For even more precise tracing, set -x and set +x can be placed around specific sections inside the script, so only the suspect code block gets traced. BASH_XTRACEFD redirects trace output to its own file so it does not get mixed in with regular stdout/stderr.

#!/usr/bin/env bash
# deploy-with-tracing.sh: Deploy script with full debug capabilities
set -euo pipefail
IFS=$'\n\t'

readonly SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
readonly LOG_DIR="/var/log/deploy"
readonly TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
readonly LOG_FILE="${LOG_DIR}/${SCRIPT_NAME%.sh}-${TIMESTAMP}.log"
readonly TRACE_FILE="${LOG_DIR}/${SCRIPT_NAME%.sh}-${TIMESTAMP}.trace"

mkdir -p "$LOG_DIR"

# Redirect stdout and stderr to log file while also showing on terminal
exec > >(tee -a "$LOG_FILE") 2>&1

# Enable tracing to separate file if DEBUG=1
if [[ "${DEBUG:-0}" == "1" ]]; then
  exec {BASH_XTRACEFD}>>"$TRACE_FILE"
  set -x
  echo "[DEBUG] Trace output: $TRACE_FILE"
fi

log() {
  local level="$1"; shift
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] [$SCRIPT_NAME:${BASH_LINENO[0]}] $*"
}

log INFO "Deploy started: PID=$$, USER=$USER, PWD=$PWD"
log INFO "Environment: DEPLOY_ENV=${DEPLOY_ENV:-unset}"

# Trace selectively around the critical section only
set +x  # Ensure tracing is off here
log INFO "Starting database migration..."
# Temporarily enable tracing for this specific step
{ set -x; bin/magento setup:upgrade; set +x; } 2>>"$TRACE_FILE"
log INFO "Database migration completed"

3. trap ERR: catching and logging errors immediately

trap ERR registers a function that runs on every command that returns a non-zero exit code, before the script is terminated by set -e. This is the ideal point for error analysis in deploy scripts: the ERR handler can print the exit code, line number, function name and stack trace before the process ends. That way the log file always contains a precise error message that points directly to the failing spot.

The ERR handler gets the line number via ${BASH_LINENO[0]}, the current function call stack via ${FUNCNAME[*]}, and the current file via ${BASH_SOURCE[0]}. Together these three form a complete stack trace, similar to what higher-level programming languages provide. Combined with printing the most recent log lines, this enables error analysis in deploy scripts without interactive debugging: the log file alone is enough to locate and understand the error.

#!/usr/bin/env bash
# error-handler.sh: Comprehensive error handling for deploy scripts
set -euo pipefail

readonly LOG_FILE="/var/log/deploy/current.log"
readonly NOTIFY_EMAIL="${NOTIFY_EMAIL:-}"

# Full stack trace on any error
err_handler() {
  local exit_code=$?
  local line_number="${BASH_LINENO[0]}"
  local command="${BASH_COMMAND}"

  echo "" >&2
  echo "════════════════════════════════════════════════════════════" >&2
  echo "[ERROR] Command failed with exit code $exit_code" >&2
  echo "[ERROR] Failed command: $command" >&2
  echo "[ERROR] Line: $line_number in ${BASH_SOURCE[1]:-$0}" >&2
  echo "" >&2
  echo "Call stack (most recent first):" >&2
  local i
  for (( i=1; i<${#FUNCNAME[@]}; i++ )); do
    local func="${FUNCNAME[$i]}"
    local src="${BASH_SOURCE[$i]:-main}"
    local lineno="${BASH_LINENO[$((i-1))]}"
    echo "  #$((i-1))  $func() at $src:$lineno" >&2
  done
  echo "" >&2

  # Capture last 20 lines of log for context
  echo "Last log entries:" >&2
  tail -20 "$LOG_FILE" 2>/dev/null | sed 's/^/  /' >&2
  echo "════════════════════════════════════════════════════════════" >&2

  # Send alert if email is configured
  if [[ -n "$NOTIFY_EMAIL" ]] && command -v mail &>/dev/null; then
    {
      echo "Deploy failed at line $line_number"
      echo "Command: $command"
      echo "Exit code: $exit_code"
      echo ""
      echo "Last 50 log lines:"
      tail -50 "$LOG_FILE" 2>/dev/null
    } | mail -s "[DEPLOY FAIL] $(hostname) - exit $exit_code" "$NOTIFY_EMAIL"
  fi
}

trap err_handler ERR

# Cleanup on any exit
cleanup() {
  local exit_code=$?
  if [[ $exit_code -eq 0 ]]; then
    echo "[OK] Deploy completed successfully"
  else
    echo "[FAIL] Deploy aborted with exit code $exit_code"
  fi
}
trap cleanup EXIT

4. LINENO and BASH_LINENO: locating the error in the script

LINENO is a variable automatically set by Bash that holds the current line number in the script. It gets updated with every command and is indispensable in logging functions for precise error analysis in deploy scripts. The pattern: log() { echo "[${BASH_LINENO[0]}] $*"; }, the caller implicitly passes on LINENO via the BASH_LINENO array, which holds every line number in the call stack.

BASH_LINENO is an array where ${BASH_LINENO[0]} holds the line number of the caller of the current function. Inside an ERR handler, ${BASH_LINENO[0]} points to the line that caused the error. ${BASH_LINENO[1]} points to the line from which the failing function was called. This array enables a complete call-stack trace, which is essential for reliable error analysis in deploy scripts: you see not only where, but also how the script arrived at the failing spot.

5. Structured log files for deploy scripts

Structured log files are the foundation of any serious error analysis in deploy scripts. A structured log file holds, for every line: timestamp, log level, script name, line number and the actual message. These five fields make it possible to compare logs across multiple deployments, correlate points in time and detect error patterns. An unstructured log file, a pile of echo calls with no consistent format, is worthless for automated analysis.

The most efficient pattern: exec > >(tee -a "$LOG_FILE") 2>&1 at the start of the script redirects all output (stdout and stderr) to both the terminal and the log file simultaneously. A logging function wraps all echo calls and adds metadata. The log file gets a timestamp in its name, so every deploy run has its own log file and logs are never overwritten. Log rotation via a separate cron job or a logrotate configuration keeps the log directory from growing unchecked.

#!/usr/bin/env bash
# logging-library.sh: Structured logging for deploy scripts
# Source this file: source /usr/local/lib/deploy/logging.sh

readonly LOG_LEVELS=([DEBUG]=0 [INFO]=1 [WARN]=2 [ERROR]=3 [FATAL]=4)
LOG_LEVEL="${LOG_LEVEL:-INFO}"
LOG_FILE="${LOG_FILE:-/var/log/deploy/deploy.log}"
LOG_FORMAT="${LOG_FORMAT:-text}"  # text or json

_log_write() {
  local level="$1"; shift
  local message="$*"
  local timestamp
  timestamp="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
  local caller_line="${BASH_LINENO[1]}"
  local caller_func="${FUNCNAME[2]:-main}"
  local caller_file="${BASH_SOURCE[2]:-unknown}"

  # Filter by log level
  if [[ "${LOG_LEVELS[$level]:-0}" -lt "${LOG_LEVELS[$LOG_LEVEL]:-1}" ]]; then
    return 0
  fi

  if [[ "$LOG_FORMAT" == "json" ]]; then
    printf '{"timestamp":"%s","level":"%s","file":"%s","line":%d,"func":"%s","msg":%s}\n' \
      "$timestamp" "$level" "$(basename "$caller_file")" \
      "$caller_line" "$caller_func" \
      "$(printf '%s' "$message" | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))')"
  else
    printf '[%s] [%-5s] [%s:%d] %s\n' \
      "$timestamp" "$level" "$(basename "$caller_file")" "$caller_line" "$message"
  fi | tee -a "$LOG_FILE"
}

log_debug() { _log_write DEBUG "$@"; }
log_info()  { _log_write INFO  "$@"; }
log_warn()  { _log_write WARN  "$@" >&2; }
log_error() { _log_write ERROR "$@" >&2; }
log_fatal() { _log_write FATAL "$@" >&2; exit 1; }

# Usage example:
# LOG_FORMAT=json LOG_LEVEL=DEBUG source logging.sh
# log_info "Starting deployment of version $VERSION"
# log_error "Failed to connect to database: $db_error"

6. Collecting context information when an error occurs

A stack trace and an error message are necessary for error analysis in deploy scripts, but not sufficient. To truly understand an error you need context: which environment variables were set? Which services were reachable? What was the memory and CPU state at the time of the error? Which other processes were running in parallel? This information must be collected automatically when an error occurs so it is available for post-mortem analysis.

An error analysis in deploy scripts context collector is a function inside the ERR handler that runs automatically on failure: env | sort for all environment variables, df -h for disk usage, free -h for memory, netstat -tn or ss -tn for open connections, ps aux for running processes. Writing all of this into a context file that is kept alongside the log file makes later analyses far more efficient.

7. Making errors reproducible: environment snapshots

Reproducibility is the core goal of any error analysis in deploy scripts. An error that cannot be reproduced cannot be reliably fixed. The most common reason for non-reproducibility: the exact environment at the time of the error is unknown. Which dependency version was installed? Which configuration was active? Was a particular service down that is running again by the time you retest?

Environment snapshots solve this problem for error analysis in deploy scripts. At the start of the deployment, a snapshot of the relevant environment information is saved: installed package versions (dpkg --get-selections or rpm -qa), active service status (systemctl list-units --state=running), configuration file checksums, the network routing table, and active connections. When a deploy error occurs, the snapshot taken at the time of the error can be compared with the snapshot of a successful deployment. Differences between the snapshots are potential root causes.

8. Post-mortem process for deploy errors

A post-mortem is not a blame record, it is a structured analysis process that ensures a deploy error does not happen twice. The foundation for every post-mortem on error analysis in deploy scripts: the complete log file of the failed deployment, the ERR handler stack trace, the environment snapshot, and the timeline of events. Without these artifacts, a post-mortem is speculation.

An effective post-mortem process for error analysis in deploy scripts follows five steps. First: isolate the exact moment of failure from the log file, timestamp, failed command, exit code, line number. Second: evaluate the context snapshot, what was different from the last successful deploy? Third: reproduce the error on a test server with an identical environment. Fourth: implement the fix and verify it in a staging deploy. Fifth: extend the deploy script with an automatic check that catches exactly the condition that caused the failure, so the fix gets built structurally into the script instead of just being documented.

9. Debugging approaches compared

Several approaches are available for error analysis in deploy scripts, and they differ in information density, intrusiveness and suitability for different scenarios.

The table makes the point clear: complete error analysis in deploy scripts requires all four approaches combined. ShellCheck and static analysis catch many errors before deployment. Structured logging and trap ERR stay active at all times. set -x and environment snapshots get switched on as needed, or generated as a standard artifact of every deployment. The combination ensures that no deploy error ever goes without a complete analysis.

10. Summary

Reproducible error analysis in deploy scripts is not a luxury, it is the foundation of reliable automation. set -x makes every executed command visible with fully expanded values. trap ERR catches every error and logs the stack trace, line number and context. LINENO and BASH_LINENO show precisely where in the script an error occurred. Structured log files make deploys comparable over time and automatically analyzable.

The decisive value of these tools lies not in any single tool, but in their combination and consistent application. A deploy script with a complete error analysis infrastructure turns post-mortems into a matter of minutes instead of hours. That lowers the barrier to more frequent deployments and increases confidence in the automation process, because everyone knows: when something goes wrong, they will know exactly what and why.

11. FAQ: Error Analysis in Deploy Scripts, Reproducible and Logged