Signal Handling and Cleanup with trap

1. Signals in Linux and the role of trap

Signals are the Unix mechanism by which the operating system or other processes notify a running process of events. The best known signal is SIGINT (signal 2), generated by Ctrl+C, which by default terminates the process. SIGTERM (signal 15) is the "polite" termination signal that kill sends without an explicit argument, giving the process a chance to shut down cleanly. SIGKILL (signal 9) cannot be caught or ignored, it terminates the process immediately. The Bash builtin trap lets you register custom handlers for most signals, which run before the process terminates.

Besides the classic process signals, Bash also knows two special pseudo-signals: EXIT and ERR. A trap on EXIT runs on every termination of the script: normal completion, an exit call, a set -e triggered abort, or a caught signal. A trap on ERR runs whenever a command returns a non-zero exit code (subject to the set -e rules). These two pseudo-signals are the most important tool for robust cleanup and error logging in shell scripts.

The syntax of trap is trap 'command' SIGNAL [SIGNAL ...] or trap function_name SIGNAL. Multiple signals can be assigned to a single handler. trap '' SIGNAL ignores the signal entirely. trap - SIGNAL resets the handler to its default. These three variants cover every practical use case, from a simple cleanup function to fine-grained signal handling with different exit codes.

2. trap EXIT and trap ERR: the foundation

The trap cleanup EXIT handler is the single most important tool for robust shell scripts. It runs no matter how the script ends, and that is its defining property. Without this handler, every possible termination path has to be handled explicitly: normal completion, exit calls scattered across the script, errors triggered by set -e, and every caught signal. That is error prone and produces duplicated cleanup logic. A trap on EXIT centralizes that logic in one place. Important: at the start of the cleanup function, capture the current exit code with local exit_code=$?, because the cleanup operations themselves can overwrite the exit code.

The trap 'on_error' ERR handler runs on every error, before the script is aborted by set -e. It is ideal for error logging with context: which command failed, on which line (${BASH_LINENO[0]}), in which function (${FUNCNAME[0]}), with which exit code ($?). That information is indispensable in production: instead of a silent abort, the log contains exactly the context an operator needs to diagnose the failure. The ERR handler is not triggered when a failing command sits inside an if condition, after ||, or after &&; this is the same behavior as set -e.

#!/usr/bin/env bash
# trap-foundation.sh: EXIT and ERR traps as script foundation
set -euo pipefail
IFS=$'\n\t'

readonly SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
readonly LOG_FILE="/var/log/${SCRIPT_NAME%.sh}.log"
declare -a CLEANUP_FILES=()
declare -a CLEANUP_DIRS=()

# ERR trap: log detailed context before set -e kills the script
on_error() {
  local exit_code=$?
  local line="${BASH_LINENO[0]}"
  local func="${FUNCNAME[1]:-main}"
  local cmd="${BASH_COMMAND}"
  printf '[%s] ERROR in %s:%s (%s), command: %s, exit: %d\n' \
    "$(date '+%Y-%m-%dT%H:%M:%S')" "$SCRIPT_NAME" "$line" "$func" "$cmd" "$exit_code" \
    | tee -a "$LOG_FILE" >&2
}

# EXIT trap: ordered cleanup regardless of how script ends
cleanup() {
  local exit_code=$?
  # Disable ERR trap during cleanup to avoid noise
  trap - ERR
  for f in "${CLEANUP_FILES[@]:-}"; do
    [[ -f "$f" ]] && rm -f -- "$f"
  done
  for d in "${CLEANUP_DIRS[@]:-}"; do
    [[ -d "$d" ]] && rm -rf -- "$d"
  done
  exit "$exit_code"
}

trap 'on_error' ERR
trap 'cleanup' EXIT

# Register temp resources for cleanup
tmpfile="$(mktemp)"
CLEANUP_FILES+=("$tmpfile")
workdir="$(mktemp -d)"
CLEANUP_DIRS+=("$workdir")

echo "Working in: $workdir"

3. SIGINT and SIGTERM: user interruption and kill

SIGINT (Ctrl+C) and SIGTERM (kill PID) are the signals a running script most commonly has to react to. Bash's default behavior on SIGINT: the signal is forwarded to the currently running foreground process, and once that process ends, Bash checks whether it should terminate itself too. On SIGTERM: Bash terminates after the current command. A trap handler can override this behavior. The typical pattern: catch SIGINT and SIGTERM with a handler that initiates an orderly shutdown, aborting running operations, running cleanup, then exiting with the correct exit code.

The correct exit code after a signal matters to the calling process. The Unix convention: a process terminated by signal N should return exit code 128 + N. For SIGINT (2) that is 130, for SIGTERM (15) that is 143. That lets the calling process or a CI pipeline distinguish whether the script ended because of an error (exit code 1) or because of a signal (130 or 143). In the trap handler function, call exit $((128 + signal_number)). For SIGTERM it is also advisable to forward the signal to all child processes so they can shut down cleanly too.

4. SIGPIPE: silent kills in pipes

SIGPIPE (signal 13) is the signal sent to a process when it writes to a pipe whose read end has already been closed. The classic example: long_running_command | head -20. head reads 20 lines and closes its stdin. On the next write attempt, long_running_command receives SIGPIPE and is terminated. Normally that is the desired behavior, but with set -e and pipefail, the non-zero exit code produced by SIGPIPE causes the entire script to abort even though nothing actually went wrong.

The solution for SIGPIPE in Bash is to explicitly ignore it with trap: trap '' PIPE. That prevents SIGPIPE from terminating the process, and the write command instead returns exit code 1, which can be handled with an explicit || true. Alternatively, you can avoid SIGPIPE errors in pipes through clever structuring: where possible, replace head with an argument based approach (command | awk 'NR<=20' with an early exit), or temporarily disable pipefail for specific pipe expressions. Understanding SIGPIPE is essential for any script that combines set -o pipefail with pipes feeding into limiting consumer tools.

#!/usr/bin/env bash
# signal-handling.sh: complete signal handling with proper exit codes
set -euo pipefail
IFS=$'\n\t'

# Ignore SIGPIPE, handle write-to-closed-pipe gracefully
trap '' PIPE

declare -i _CAUGHT_SIGNAL=0

handle_signal() {
  local signum="$1"
  local signame="$2"
  _CAUGHT_SIGNAL="$signum"
  printf '\n[SIGNAL] Received %s, initiating graceful shutdown...\n' "$signame" >&2
  # Forward signal to all child processes in our process group
  kill -"$signum" -- -$$ 2>/dev/null || true
}

trap 'handle_signal 2 SIGINT'  INT
trap 'handle_signal 15 SIGTERM' TERM
trap 'handle_signal 1 SIGHUP'  HUP

cleanup() {
  local exit_code=$?
  trap - INT TERM HUP  # Prevent re-entry during cleanup
  # Determine final exit code: signal takes precedence
  if (( _CAUGHT_SIGNAL > 0 )); then
    exit_code=$(( 128 + _CAUGHT_SIGNAL ))
  fi
  rm -f -- /tmp/myscript.lock 2>/dev/null || true
  printf '[CLEANUP] Exiting with code %d\n' "$exit_code" >&2
  exit "$exit_code"
}

trap 'cleanup' EXIT

# Example: SIGPIPE-safe pipeline with head
generate_lines() {
  local i=0
  while (( i < 10000 )); do
    printf 'Line %d\n' "$i"
    (( i++ ))
  done
}
# Without 'trap "" PIPE', this would trigger pipefail on SIGPIPE
generate_lines | head -5 || true

5. Structuring the cleanup order correctly

The order of cleanup operations in a trap EXIT handler is critical for correctness. The general rule: clean up in reverse order of resource allocation, releasing what was created last first. In practice that means: stop background processes first, then close database connections, then undo filesystem operations (unmount mounts), then delete temporary directories, then delete temporary files, and release lockfiles last. Releasing the lockfile last matters: as long as the lockfile exists, a new instance startup knows that cleanup is still in progress.

Inside the cleanup function, errors should not be allowed to abort execution via set -e: a cleanup operation that fails should not prevent the rest of the cleanup sequence from running. It is therefore advisable to start the cleanup function with trap - ERR EXIT (deregistering all traps) and then guard every cleanup operation with || true or explicit error handling. Alternatively, the entire cleanup block can run inside a subshell with (set +e; cleanup_ops). The goal: cleanup always runs to completion, even if individual steps fail, while the exit code of the original error is preserved.

6. Subshells and background processes: forwarding signals

A common misunderstanding about Bash trap: subshells and background processes do not automatically inherit the parent process's trap handlers. Subshells (via $() or ()) reset all trap handlers to their defaults. Functions, on the other hand, do inherit trap handlers. That means a trap on EXIT registered in the main shell will not run inside a subshell. For background processes: they receive no signals sent to the parent unless the parent explicitly forwards them.

The correct pattern for forwarding signals to background processes: store the PIDs of every started background process in an array, and in the signal handler iterate over that array and send each process the signal. Afterwards, use wait to wait for all child processes to finish before the cleanup function completes the exit sequence. Without this pattern, background processes become orphans, processes with no active parent, that keep running after the main script has already terminated. That is especially problematic for resource-intensive background processes or processes that hold lockfiles.

#!/usr/bin/env bash
# subshell-signals.sh: forward signals to background children
set -euo pipefail
IFS=$'\n\t'

declare -a CHILD_PIDS=()

# Signal forwarding to all registered children
forward_signal() {
  local sig="$1"
  local pid
  for pid in "${CHILD_PIDS[@]:-}"; do
    kill -"$sig" "$pid" 2>/dev/null || true
  done
}

cleanup() {
  local exit_code=$?
  trap - INT TERM EXIT
  forward_signal TERM
  # Wait for all children to finish cleanup
  local pid
  for pid in "${CHILD_PIDS[@]:-}"; do
    wait "$pid" 2>/dev/null || true
  done
  exit "$exit_code"
}

trap 'forward_signal INT; exit 130'  INT
trap 'forward_signal TERM; exit 143' TERM
trap 'cleanup' EXIT

# Start background workers and track their PIDs
worker() {
  local id="$1"
  trap 'echo "[WORKER $id] Caught signal, exiting cleanly" >&2; exit' INT TERM
  while true; do
    sleep 1
    echo "[WORKER $id] tick"
  done
}

worker 1 & CHILD_PIDS+=($!)
worker 2 & CHILD_PIDS+=($!)

echo "Main: workers started. PIDs: ${CHILD_PIDS[*]}"
wait  # Wait for all children (or until signal)

7. trap in functions and libraries

The behavior of trap in functions has an important quirk: a trap set inside a function applies to the entire shell session, not just the function. If a library function sets a trap, it overwrites the calling script's trap. That is a common bug in shell libraries: the library internally sets a trap on EXIT for its own resource release and thereby overwrites the calling script's EXIT trap.

The correct solution for libraries: query the existing trap handler before setting a new one (trap -p EXIT), combine the new handler with the existing one, and register both. That is more complex than a plain trap 'fn' EXIT, but it is the only way to enable trap composition inside libraries. Alternatively, libraries can maintain their own cleanup lists that the calling script registers into, and use a single central trap handler in the main script that iterates over those lists. That makes trap management explicit and prevents conflicts between library code and application code.

8. Practical example: a robust deployment script

A deployment script is the typical use case where trap and signal handling make the difference between a safe script and a dangerous one. If a deployment script is interrupted without signal handling, it can leave a database in a mid-migration state, leave maintenance mode enabled, leave symbolic links pointing at unfinished releases, or leave lockfiles on the production system. With correctly structured signal handling and an orderly cleanup function, all of these states get cleaned up properly even on Ctrl+C or kill.

The deployment script demonstrates the full integration: ERR logging with context, EXIT cleanup with a resource list, SIGINT/SIGTERM handlers with signal forwarding, SIGPIPE suppression, and the correct exit code convention. In practice such a script would also include rollback logic: if certain deployment steps fail, previous steps get undone. That rollback logic naturally also lives in the trap EXIT handler, or in a dedicated rollback function called from the ERR handler.

9. Signal comparison: behavior and recommendation

The various signals and pseudo-signals that Bash trap can catch have different semantics and require different handling strategies. An overview of the most important ones:

The key takeaway from this table: EXIT is the universal cleanup signal, it always runs regardless of what other event ends the script. ERR is the precise logging signal, it runs before set -e aborts and has access to the full error context. SIGINT and SIGTERM require actively forwarding the signal to child processes. SIGPIPE should be ignored in most production scripts to avoid unexpected pipeline aborts caused by head, grep, or other limiting consumers.

10. Summary

Robust signal handling with trap in Bash consists of four layers: trap 'on_error' ERR for detailed error logging with line number and function name, trap 'cleanup' EXIT as the universal cleanup handler for every termination scenario, explicit handlers for SIGINT and SIGTERM that forward the signal to child processes and exit with code 128 + N, and trap '' PIPE to prevent unexpected pipeline aborts caused by limiting consumer tools. These four layers cover every relevant termination scenario and ensure resources get released even under unexpected conditions.

The cleanup order follows the principle of "reverse allocation order": processes first, then connections, then mounts, then files, locks last. Inside the cleanup function, all trap handlers are deregistered to prevent re-entrancy, and every cleanup operation is given error handling so cleanup runs to completion even after partial failures. The lockfile is released last: as long as it exists, it signals that cleanup has not yet finished.

11. FAQ: Signal Handling and Cleanup with trap