Locking in Bash: flock, PID Files and Race Conditions

1. Why locking is essential in Bash scripts

Locking in Bash is not an academic topic. It is a daily practical problem in any infrastructure that uses cron jobs, parallel CI jobs, or deployment scripts. If a backup script starts at 2 a.m. and the previous run has not finished yet, both instances produce inconsistent backup states. If two deployment runs start at the same time because a manual trigger collides with a still running cron trigger, you risk corrupted deployments. The core problem is always the same: a script that has no way of knowing whether another instance is already running.

Naive locking in Bash with if [ -f lockfile ]; then exit; fi; touch lockfile does not solve the problem. Between the check and the creation of the file there is a time gap in which a second instance performs the same check and also finds no lock. This classic race condition is reproducible under high load or when CI jobs start simultaneously. Kernel backed locking with flock closes this gap completely, because the atomicity lives in the kernel, not in the shell.

2. Understanding race conditions in shell scripts

A race condition in a shell script occurs when the outcome of an operation depends on the timing of other processes. For locking, the classic scenario is this: process A checks whether a lock file exists (it does not), gets interrupted by the scheduler in between, process B also checks (it does not exist either), process A creates the lock file and starts, process B also creates the lock file and starts. Both now run at the same time and no locking has actually taken place. In practice this window is often only milliseconds wide, but with sufficiently frequent execution or under load it occurs reliably.

The severity of the consequence depends heavily on the script. A backup script produces duplicate backups with inconsistent states. A deployment script can run database migrations twice when two deployments happen at once, resulting in a corrupted schema. A cleanup script can delete files that a second instance is still using. Race conditions in locking are particularly insidious because they occur intermittently and are never visible in tests that run sequentially.

3. PID files: the common but error prone pattern

The PID file pattern for locking in Bash writes the process ID of the running instance to a file and checks on startup whether that file exists and whether the stored process is still running. It is widespread but has structural weaknesses. The main weakness: checking whether the PID file exists and creating the file are not one atomic operation, so the race condition is only narrower, not eliminated. On top of that, stale PID files left behind by crashed scripts must be detected and removed manually.

A stale PID file appears when a script is terminated by a kill signal or a system crash without running its cleanup handler. On the next run, the script finds the PID file and checks with kill -0 $PID whether the process is still alive, which can give the wrong answer if the PID has been recycled: that PID could belong to a completely different process started later. This PID recycling problem makes plain PID file locking implementations unreliable for critical workflows.

#!/usr/bin/env bash
# pid-file-locking.sh - PID file locking with stale detection (better than naive)
set -euo pipefail

PID_FILE="/var/run/myapp.pid"

acquire_pid_lock() {
  if [[ -f "$PID_FILE" ]]; then
    local old_pid
    old_pid=$(< "$PID_FILE")

    # Check if PID is still alive AND belongs to this script
    if kill -0 "$old_pid" 2>/dev/null; then
      # Extra check: verify it's actually our script type
      local proc_name
      proc_name=$(ps -p "$old_pid" -o comm= 2>/dev/null || echo "")
      if [[ "$proc_name" == "bash" ]]; then
        echo "[ERROR] Another instance is running (PID $old_pid)" >&2
        return 1
      fi
    fi
    # Stale PID file, previous run crashed
    echo "[WARN] Removing stale PID file (PID $old_pid is gone)" >&2
    rm -f "$PID_FILE"
  fi

  # Write current PID, still not 100% atomic!
  echo $$ > "$PID_FILE"
}

release_pid_lock() {
  [[ -f "$PID_FILE" ]] && rm -f "$PID_FILE"
}

trap release_pid_lock EXIT

acquire_pid_lock || exit 1
echo "[INFO] Lock acquired (PID $$)"
# ... main logic here ...

4. flock: atomic locking with kernel support

flock is a Linux command that implements kernel backed locking over file descriptors. Unlike the PID file pattern, flock is fully atomic: the lock is set in the kernel, and two processes can never hold the same exclusive lock on the same file descriptor at the same time. This brings two further decisive advantages over PID files. The lock is released automatically when the process ends, whether normally, through an error, or through a crash, without any cleanup handler having to run. And stale locks are physically impossible, because the kernel ties the release of the lock to the process lifecycle.

The usage form exec 9>lockfile; flock -n 9 opens a file descriptor on the lock file and tries to set an exclusive, non blocking lock. If the lock is not available because another process holds it, flock -n returns immediately with exit code 1. This non blocking pattern suits scripts that should not wait but abort right away if an instance is already running. The lock file itself does not matter, it is never read or written, it only serves as an anchor for the kernel level lock. The file can stay empty.

5. flock patterns: non blocking, timeout and FD based

There are three essential flock usage patterns in Bash scripts: the non blocking pattern, the timeout pattern, and the FD based wrapper pattern. The non blocking pattern (flock -n 9) suits cron jobs that should simply be skipped if an instance is already running. The timeout pattern (flock -w 30 9) waits at most 30 seconds for the lock and then aborts, suitable for scripts that can afford to wait for a running predecessor. The FD based wrapper pattern encapsulates the whole lock lifecycle in a function that opens the FD, holds the lock, and releases it automatically once the function ends.

A particularly elegant flock pattern uses the subshell variant: flock -n 9 bash -c "..." or (flock -n 9; your_function) 9>lockfile. Here the entire locked block of code runs inside a subshell that owns the FD, and once the subshell exits the lock is released automatically. This pattern avoids having to manage the FD explicitly and makes the lock scope visible in the code. For Bash scripts where several sections need different locks, this pattern is the clearest solution.

#!/usr/bin/env bash
# flock-patterns.sh - All flock locking patterns for production use
set -euo pipefail

LOCK_DIR="/var/lock"
SCRIPT_NAME="$(basename "$0" .sh)"
LOCK_FILE="${LOCK_DIR}/${SCRIPT_NAME}.lock"

# Pattern 1: Non-blocking, exit immediately if already running
exec 9>"$LOCK_FILE"
if ! flock -n 9; then
  echo "[INFO] Another instance of $SCRIPT_NAME is already running. Exiting." >&2
  exit 0
fi
echo "[INFO] Lock acquired (PID $$, FD 9)"

# Pattern 2: Timeout, wait up to 30 seconds
acquire_with_timeout() {
  local timeout="${1:-30}"
  exec 8>"${LOCK_FILE}.secondary"
  if ! flock -w "$timeout" 8; then
    echo "[ERROR] Could not acquire lock within ${timeout}s" >&2
    return 1
  fi
  echo "[INFO] Lock acquired after waiting"
}

# Pattern 3: Subshell scope, lock is released when subshell exits
run_exclusive() {
  local lock_file="$1"
  shift
  (
    exec 7>"$lock_file"
    flock -n 7 || { echo "[ERROR] Resource is locked" >&2; exit 1; }
    "$@"
  )  # lock automatically released here
}

# Example usage
run_exclusive "${LOCK_FILE}.db" bash -c 'echo "Exclusive DB operation"'

# Lock info: show who holds a lock
show_lock_info() {
  local file="$1"
  if [[ -f "$file" ]]; then
    local holder
    holder=$(fuser "$file" 2>/dev/null || echo "none")
    echo "Lock file: $file | Held by PID: $holder"
  fi
}

show_lock_info "$LOCK_FILE"

6. Detecting and cleaning up stale locks automatically

Stale locks, meaning outdated locks that were not cleaned up after a process ended, are structurally impossible with the flock approach, because the kernel manages the lock together with the process lifecycle. With the PID file pattern, however, stale locks are a real problem. The reason: if a process is terminated with kill -9 or the system restarts abruptly, the cleanup handler (trap) cannot run, and the PID file is left behind.

A robust detection pattern for stale locks checks three things: does the PID file exist? Is the PID stored in it still an active process? Does that process belong to a script with the expected name? Only when all three conditions hold is the lock considered valid. For automatic cleanup of stale PID files, an age check is useful: a PID file older than the maximum expected runtime of the script can be treated as stale. With find "$PID_FILE" -mmin +120 you can check whether the file is older than 120 minutes.

#!/usr/bin/env bash
# stale-lock-cleanup.sh - Stale PID file detection and cleanup
set -euo pipefail

PID_FILE="/var/run/backup.pid"
MAX_AGE_MINUTES=120  # Maximum expected runtime

is_stale_lock() {
  local pid_file="$1"

  # No file, no lock
  [[ -f "$pid_file" ]] || return 1

  local stored_pid
  stored_pid=$(< "$pid_file") 2>/dev/null || return 1

  # Validate: must be a number
  [[ "$stored_pid" =~ ^[0-9]+$ ]] || { echo "[WARN] Invalid PID in lock file" >&2; return 0; }

  # Check 1: Is the process alive?
  if ! kill -0 "$stored_pid" 2>/dev/null; then
    echo "[WARN] Stale lock: PID $stored_pid is gone" >&2
    return 0  # stale
  fi

  # Check 2: Is it too old? (could be a stuck process)
  if find "$pid_file" -mmin +"$MAX_AGE_MINUTES" | grep -q .; then
    echo "[WARN] Lock file older than ${MAX_AGE_MINUTES}min, possible stuck process" >&2
    return 0  # treat as stale
  fi

  # Check 3: Process name check
  local proc_cmd
  proc_cmd=$(ps -p "$stored_pid" -o args= 2>/dev/null || echo "")
  if ! echo "$proc_cmd" | grep -q "backup"; then
    echo "[WARN] PID $stored_pid is not our process (PID recycled?)" >&2
    return 0  # stale
  fi

  return 1  # Lock is valid
}

if is_stale_lock "$PID_FILE"; then
  echo "[INFO] Removing stale lock file" >&2
  rm -f "$PID_FILE"
fi

# Now try to acquire with flock for real atomicity
exec 9>"${PID_FILE%.pid}.flock"
flock -n 9 || { echo "[ERROR] Lock held by active process" >&2; exit 1; }
echo $$ > "$PID_FILE"
trap 'rm -f "$PID_FILE"' EXIT

7. mkdir as an atomic locking mechanism

The mkdir command is a lesser known but portable alternative to flock for atomic locking in Bash. The operating system guarantees that mkdir is atomic: two processes that run mkdir /tmp/mylock at the same time are guaranteed to have only one of them succeed with exit code 0, the other fails. This atomicity is anchored in the POSIX standard and is available on every system, even where flock is not installed (for example some container images or NFS mounts that do not support flock).

The downside of the mkdir locking pattern is that, unlike flock, it is not crash safe. If the script ends without a cleanup handler, the directory remains and blocks every subsequent run. The way out is the PID in directory method: the lock directory contains a file with the PID of the lock owner, which enables stale lock detection. This pattern is especially suited to portable scripts that must run on various systems without guaranteed flock access, or on NFS shares where flock does not work reliably due to NFS limitations.

8. Locking in cron jobs and deployment pipelines

Locking in Bash is especially critical in cron jobs, because the cron daemon offers no inherent serialization. If a job runs longer than its execution interval, cron starts a new instance without waiting for the running one to finish. The pattern for cron safe locking is a non blocking flock at the start of the script that, on a failed lock, aborts immediately with exit code 0 (not 1, to avoid generating cron error emails) and writes a short message to a log file. That way the skipped run is documented without alerting the cron daemon.

CI/CD pipelines are a different context: here it is often desirable for a second deployment run to wait until the first one has finished, instead of simply aborting. The timeout pattern of flock is suited for that: flock -w 300 9 waits up to 5 minutes. Combined with a progress indicator (a periodic echo running in the background while waiting), it prevents CI systems from marking the job as hung due to a lack of output. This pattern is complete and addresses both the requirement for atomic locking and CI specific behavior requirements.

9. Locking methods compared directly

Every locking method in Bash has specific strengths and limitations. Choosing the right pattern depends on the environment, the required crash safety, and the desired behavior for competing instances.

The recommendation for production use is clear: flock on Linux systems for all local file systems. mkdir as a portable fallback for systems without flock or for NFS shares. PID files only combined with stale detection and an additional flock for race condition free creation. In Docker containers and CI environments, flock is available in most base images and is always the first choice.

10. Summary

Locking in Bash with flock solves the fundamental race condition problem of double execution through atomic, kernel backed locking. The PID file pattern is widespread but has structural weaknesses: no atomicity at creation time, no automatic cleanup after crashes, and PID recycling can make a stale lock falsely appear valid. flock -n 9 on an open file descriptor is the reliable alternative: atomic, crash safe, no manual cleanup required. The lock file only serves as an anchor, its content is irrelevant.

For cron jobs, the non blocking pattern with exit code 0 on a failed lock is the right choice: no error emails, but still logging. For CI/CD pipelines, the timeout pattern offers an elegant way to wait. On NFS or in environments without flock, mkdir is the atomic POSIX fallback. Anyone who needs to cover all three scenarios in deployment workflows should implement a locking function that switches between flock and mkdir depending on what the system supports.

11. FAQ: Locking in Bash, flock, PID Files and Race Conditions