Preventing Overlapping Cron Jobs: Locking With flock in Practice

1. Why overlapping cron jobs are a real operational risk

Cron implicitly assumes a job finishes within its scheduled interval. In practice, this assumption does not always hold: a daily database export that normally takes ten minutes can, as the data set grows, eventually take longer than the time until the next scheduled run. Cron starts the next run right on time anyway, without checking whether the previous one is still active. The result is two instances of the same job running in parallel, competing for the same resources.

Overlap becomes especially critical for jobs that write to shared resources, such as backup scripts writing to the same target file, or import processes operating on the same database tables. In the best case, both instances just slow each other down. In the worse case, inconsistent data, duplicate entries, or a deadlock that permanently blocks both processes until an administrator intervenes manually can result. Locking is therefore not cosmetic, it is a basic requirement for robust cron jobs with variable runtime.

2. How overlap happens: runtimes, load spikes, hung processes

Three scenarios most commonly lead to overlapping cron jobs in practice. First, growing data volumes: a job that finished in a few minutes when introduced takes considerably longer months later, without anyone adjusting the cron line accordingly. Second, external dependencies: a script waiting on a slow API or an overloaded database server can unpredictably block far longer than its normal runtime.

Third, hung processes: a script that ends up in an infinite loop due to a bug, or waits for a network response that never arrives, blocks indefinitely while cron faithfully keeps starting new instances. After a few days, dozens of zombie like processes can be running in parallel this way, each with its own memory consumption and open database connections, until the server hits its resource limits. This exact scenario is what makes locking mandatory, not optional, as soon as a cron job goes into production.

3. flock basics: exclusive and shared locks

The flock command uses advisory locks at the filesystem level, meaning locks that cooperating processes respect but that do not enforce access at the operating system level. An exclusive lock, the default setting of flock, allows only a single process at a time to hold access to a particular file used as the lock object. A second process trying to acquire the same lock either waits until the first lock is released or aborts immediately, depending on the chosen option.

The -n or --nonblock option is the decisive setting for cron jobs: instead of waiting, flock aborts immediately if the lock is already held, and returns a non zero exit code. This exact behavior prevents overlap, because the second, premature cron run recognizes right away that the previous one is still active and terminates instead of queuing up and making the situation worse.

# Basic flock usage: exclusive, non-blocking lock on a lock file
flock -n /var/lock/backup.lock -c "/usr/local/bin/backup.sh"

# Exit code 1 means the lock was already held — the job simply skips this run
echo "Exit code: $?"

4. Using flock directly in crontab

The simplest and most robust use of flock happens directly in the crontab line, without having to modify the actual script. You prefix the actual command with a flock call, specifying a lock file and the -n option. If the previous job is still running, flock aborts immediately, cron logs the failed attempt, but no second, competing instance is created.

Choosing a unique path for the lock file is important, typically under /var/lock/ or /run/lock/, with a name clearly belonging to the specific job. With several similar cron jobs on the same server, it is a common mistake to accidentally reuse the same lock file for two unrelated jobs, which causes independent tasks to block each other even though they have nothing to do with each other in terms of content.

# /etc/cron.d/nightly-export — flock prevents overlapping runs directly in crontab
# Runs every 10 minutes, but only one instance can ever be active
*/10 * * * * appuser flock -n /var/lock/nightly-export.lock /usr/local/bin/export.sh

# Explicit per-job lock names avoid accidental cross-locking between unrelated jobs
0 2 * * * appuser flock -n /var/lock/db-backup.lock /usr/local/bin/db-backup.sh
0 3 * * * appuser flock -n /var/lock/log-cleanup.lock /usr/local/bin/log-cleanup.sh

5. flock inside a script using exec and a file descriptor

For more complex cases, for example when a script itself should decide how to react to an already held lock, flock can also be used inside a Bash script, through the combination of exec and a free file descriptor. This variant opens the lock file once on a descriptor, typically 200, and holds the lock for the entire runtime of the script, without needing an external flock process as a wrapper.

The advantage of this variant: the script can run its own logic in case the lock is already held, for example writing a specific log message or notifying a monitoring system, instead of simply ending with a generic exit code. The lock is automatically released as soon as the file descriptor closes at script exit, even if the script terminates early due to an error or a signal.

#!/usr/bin/env bash
set -euo pipefail

LOCK_FILE="/var/lock/report-generation.lock"

# Open the lock file on file descriptor 200 and acquire an exclusive, non-blocking lock
exec 200>"$LOCK_FILE"
if ! flock -n 200; then
  echo "[INFO] Previous run still active, skipping this execution" >&2
  exit 0
fi

echo "[INFO] Lock acquired, starting report generation"
# ... actual job logic goes here ...

# Lock is released automatically when fd 200 closes at script exit

6. The systemd alternative: automatic overlap protection

Anyone using systemd timers instead of cron already gets most of this problem solved for free. A service of type oneshot, triggered by a timer, does not start a second instance by default as long as the first one is still running, because systemd centrally manages the activation state of the associated service unit and does not start an already active service twice. This replaces a manual locking mechanism almost completely, without having to use flock yourself.

For cases where several different timers must not trigger the same underlying operation at the same time, Conflicts= between service units can additionally be used to enforce mutual exclusion across unit boundaries. For new automation projects, switching from cron to systemd timers is therefore often the more pragmatic way to architecturally rule out overlap from the start, instead of patching it in afterward with flock.

7. Detecting overlapping jobs: monitoring and alerting

Even with working locking in place, you should monitor how often a job actually gets aborted because the lock was already held. A single skipped run is usually harmless, but repeated occurrences indicate that the scheduled runtime is systematically too tight and that the underlying interval or script performance needs reconsidering.

In practice this can be implemented via a simple extension to the script that writes every skipped run to a dedicated log file or increments a counter in a monitoring system like Prometheus. An additional watchdog check that verifies how long a lock file has already been held further helps to detect hung processes early, instead of only noticing them days later during the next manual look at the server.

#!/usr/bin/env bash
# Watchdog: alert if a lock has been held suspiciously long (possible hung process)
LOCK_FILE="/var/lock/nightly-export.lock"
MAX_AGE_MINUTES=60

if [[ -f "$LOCK_FILE" ]]; then
  age_minutes=$(( ($(date +%s) - $(stat -c %Y "$LOCK_FILE")) / 60 ))
  if (( age_minutes > MAX_AGE_MINUTES )); then
    echo "[ALERT] Lock held for ${age_minutes} minutes, possible hung process" >&2
    exit 1
  fi
fi

8. Common locking mistakes and how to avoid them

The most common mistake is using PID files instead of flock. A script writes its own process ID to a file at startup and checks on the next run whether that process still exists. The problem: after a hard crash or a server restart, an old PID can coincidentally be reused by a completely different, new process, producing a false positive and permanently blocking the job, even though the original process ended long ago.

flock avoids this problem fundamentally, because the operating system automatically releases the lock as soon as the holding process ends, regardless of whether it ends normally, crashes, or is terminated by a signal. A second common mistake is manually deleting the lock file before the next run, assuming that is necessary. That is not necessary with flock and can even be dangerous if the file is deleted while another process is still actively waiting on it.

9. Locking strategies compared

Different approaches to preventing overlap differ significantly in reliability and implementation effort.

For existing cron based infrastructure, flock is the most pragmatic way to achieve immediate protection with minimal change effort. For new automation, switching directly to systemd timers with oneshot services is often worthwhile, because overlap protection is architecturally built in there and does not need to be added afterward.

10. Summary

Overlapping cron jobs occur as soon as the actual runtime of a job exceeds its scheduled interval, whether due to growing data volumes, slow dependencies, or hung processes. flock reliably solves this problem by using advisory locks at the filesystem level, which the operating system automatically releases as soon as the holding process ends, regardless of how it ends. The -n option ensures a job aborts immediately when the lock is already held, instead of waiting and making the situation worse.

Used directly in crontab, flock protects without any script changes. Inside a script with exec and a file descriptor, you can additionally implement custom reactions to an already held lock. PID files are the outdated, error prone predecessor and should no longer be used in new code. Anyone switching to systemd timers anyway gets overlap protection architecturally built in through oneshot services.

11. FAQ: Preventing Overlapping Cron Jobs With flock