Building Reliable Cron Jobs: Logging, Locking, Monitoring and Exit Codes

1. The Most Common Cron Job Problems in Practice

A typical cron job in production looks like this: a one-line crontab entry calls a shell script, the script's output is never read, nobody cares about the exit code, and whether the job actually runs on schedule only becomes apparent once a backup restore fails. This scenario is not an edge case, it is the norm in many infrastructures. Cron jobs get set up once and are never actively monitored afterward, because everything seems to be running fine.

The first classic problem is overlapping executions. When a cron job runs longer than its interval, for example because the data volume has grown or a network hiccup causes delays, cron starts a second instance while the first is still running. Two parallel backup jobs write to the same file. Two parallel database migration jobs create race conditions. The result is data loss or data corruption, and nobody knows why. The second problem is disappearing errors: cron mails the output of cron jobs to the local system user, which on production servers is usually never read, or is silently dropped due to missing mail configuration.

The third problem concerns the execution environment. The cron daemon starts cron jobs with a minimal environment: no PATH beyond the system default directories, no SSH environment variables, no user .bashrc. A script that runs fine in an interactive terminal with the full user environment can silently fail in a cron context because mysql, php or another tool cannot be found in the minimal PATH. This class of error is particularly insidious because it cannot be reproduced in a manual test.

2. Locking with flock: Preventing Parallel Execution

The most reliable tool against overlapping cron job executions is flock, a POSIX-compliant advisory locking tool. The principle: the script opens a lock file as a file descriptor and calls flock -n on it. If no other instance holds the lock, it is acquired and the job runs. If another instance is already running, flock -n returns immediately with exit code 1, and the new invocation exits cleanly. The decisive advantage over PID file patterns is that the operating system releases the lock automatically when the process ends, whether through a normal exit, a signal, or a crash. There are no stale lock files that require manual cleanup.

An elegant variant is calling flock directly in the crontab, without touching the script's code at all: flock -n /var/lock/backup.lock /usr/local/bin/backup.sh. This secures any existing cron job with a single word. If the script itself should emit an error message when it fails to acquire the lock, the internal variant with a dedicated file descriptor is the better choice. With the optional timeout parameter flock -w 10, flock waits up to ten seconds for the lock to be released before failing, useful when short overlaps are tolerable but long parallel runs are not.

#!/usr/bin/env bash
# backup.sh: cron job with flock locking and structured logging
set -euo pipefail

readonly SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
readonly LOCK_FILE="/var/lock/${SCRIPT_NAME%.sh}.lock"
readonly LOG_DIR="/var/log/cronjobs"
readonly LOG_FILE="${LOG_DIR}/${SCRIPT_NAME%.sh}-$(date +%Y%m%d).log"

# Logging function: timestamp + level + message to log file and stderr
log() {
  local level="$1"; shift
  local msg="$*"
  local ts; ts="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
  printf '%s [%s] [%s] %s\n' "$ts" "$level" "$SCRIPT_NAME" "$msg" \
    | tee -a "$LOG_FILE" >&2
}

# Ensure log directory exists
mkdir -p "$LOG_DIR"

# Acquire exclusive lock, exit silently if already running
exec 200>"$LOCK_FILE"
if ! flock -n 200; then
  log WARN "Another instance is already running, skipping this run"
  exit 0
fi

log INFO "Job started (PID $$)"
trap 'log INFO "Job finished (exit $?)"' EXIT

# --- Main job logic here ---
log INFO "Starting database backup"
# pg_dump ... | gzip > /backups/db-$(date +%Y%m%d).sql.gz
log INFO "Database backup complete"

3. Structured Logging for Cron Jobs

The most minimal form of cron job logging is redirecting the output to a file: 0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1. That is better than nothing, but it is not enough for production-grade cron jobs. Without timestamps in the output, it is hard to say exactly when something happened once an error occurs. Without log rotation, the log file fills up disk space over weeks. Without structured log levels, it is hard to quickly distinguish normal informational messages from error entries with grep or a log aggregator.

The professional solution is a dedicated logging function in the script that embeds timestamp, level and script name into every line, combined with daily log rotation via logrotate. The pattern tee -a "$LOG_FILE" >&2 writes every log line to the log file and to stderr at the same time. Cron then forwards stderr by mail, so error messages are not lost. For cron jobs that have been converted to systemd timer units, the journal automatically captures all output, making logrotate unnecessary.

4. Setting and Evaluating Exit Codes Correctly

Exit codes are the primary communication interface between a cron job and the monitoring system. Exit code 0 means success, any value from 1 to 255 signals an error. Cron itself does nothing with exit codes, but monitoring tools such as Nagios, Icinga, Prometheus Alertmanager and healthcheck services evaluate them. For this to work, every cron job must set its exit code explicitly: exit 0 at the end of a successful run, exit 1 for general errors, and specific codes for different error categories when the monitoring needs to react differently depending on the failure type.

The biggest risk when setting exit codes is that set -e aborts the script on any error, but without a trap the exit code is never communicated cleanly to the outside. The correct pattern: register a cleanup function via trap cleanup EXIT that stores the exit code in a variable, logs it before cleanup, and calls monitoring webhooks. This way the monitoring always receives the actual exit code, even when the script was aborted by set -e.

#!/usr/bin/env bash
# cron_wrapper.sh: generic cron job wrapper with exit code monitoring
set -euo pipefail

readonly JOB_NAME="${1:?Usage: $0 <job-name> <command...>}"
shift
readonly JOB_CMD=("$@")
readonly HEALTHCHECK_URL="${HEALTHCHECK_URL:-}"  # Optional: healthchecks.io URL
readonly SLACK_WEBHOOK="${SLACK_WEBHOOK:-}"

START_TS=$(date +%s)
EXIT_CODE=0

cleanup() {
  EXIT_CODE=$?
  local duration=$(( $(date +%s) - START_TS ))

  if [[ $EXIT_CODE -eq 0 ]]; then
    echo "[OK] ${JOB_NAME} finished in ${duration}s"
    # Signal success to healthchecks.io
    [[ -n "$HEALTHCHECK_URL" ]] && \
      curl -fsS -m 5 "${HEALTHCHECK_URL}" -d "OK: ${JOB_NAME} (${duration}s)" || true
  else
    echo "[FAIL] ${JOB_NAME} failed with exit code ${EXIT_CODE} after ${duration}s" >&2
    # Notify Slack on failure
    if [[ -n "$SLACK_WEBHOOK" ]]; then
      curl -fsS -m 10 -X POST "$SLACK_WEBHOOK" \
        -H 'Content-type: application/json' \
        -d "{\"text\":\"Cronjob FAILED: ${JOB_NAME} (exit ${EXIT_CODE}, ${duration}s)\"}" || true
    fi
    # Signal failure to healthchecks.io
    [[ -n "$HEALTHCHECK_URL" ]] && \
      curl -fsS -m 5 "${HEALTHCHECK_URL}/fail" -d "FAIL: exit ${EXIT_CODE}" || true
  fi
}

trap cleanup EXIT

# Run the actual job
"${JOB_CMD[@]}"

5. Mail Notifications and Alerting

Cron mails the output of a job to the user it runs as, provided a local MTA is configured. That is no longer the case on many modern servers, and even when it works, the mails land in a mailbox nobody checks regularly. Professional cron job alerting means actively sending errors to the people responsible: by email through a configured SMTP relay, by Slack webhook, by PagerDuty alert, or through a healthcheck service like healthchecks.io.

The pattern for reliable notifications: suppress the cron job output completely (MAILTO="" in the crontab), but have the script itself send a structured alert on failure. This way no notifications are lost to non-existent mailboxes, and the alerts contain exactly the information needed for diagnosis: job name, exit code, last log lines, timestamp. Healthcheck services add dead man's switch monitoring to the active alerting: if a cron job does not send its ping within the expected window, the service automatically triggers an alert, even when the cron job never started at all.

6. Crontab Hygiene and Environment Variables

A well-maintained crontab is as self-documenting as possible. Every entry contains a comment explaining why that cron job exists and what it does. The environment is set explicitly: PATH, SHELL, MAILTO and all application-specific variables sit at the top of the crontab and apply to all subsequent entries. Alternatively, environment variables are set inside the script itself, which makes the script independent of the calling environment.

A typical trap: in the terminal, the full PATH is set through .bashrc or .profile. In the cron context there is only /usr/bin:/bin. Commands such as php, composer, mysql, node or self-compiled tools often live in /usr/local/bin or another directory missing from the minimal cron PATH. Every professional cron job script therefore starts with an explicit PATH declaration. The trick for debugging: env -i /bin/sh -c 'set' shows the minimal environment cron uses to start scripts.

7. systemd Timers as a Modern Cron Alternative

On all modern Linux systems running systemd, systemd timers offer a significantly more powerful alternative to classic cron jobs. Every timer consists of two units: a .timer file that defines the schedule, and a .service file that describes the command to run. The systemd journal automatically records the start, end, output and exit code of every timer run, without any manual logging configuration. journalctl -u backup.timer shows the complete execution history.

systemd timers support dependencies via the After= directive: a cron job that needs the database waits until postgresql.service is ready. That is not possible with classic cron jobs. OnCalendar=daily is equivalent to 0 0 * * *, but is more readable and also supports more complex expressions such as Mon,Wed,Fri *-*-* 08:00:00. With Persistent=true in the timer unit, a missed run (due to a reboot or a powered-off server) is caught up at the next system start, a capability that classic cron jobs do not offer.

# /etc/systemd/system/db-backup.service
[Unit]
Description=Database backup job
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=oneshot
User=backup
Group=backup
# Explicit PATH, not inherited from user environment
Environment=PATH=/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/local/bin/backup.sh --type database
# Capture output to journal (automatic, no logging config needed)
StandardOutput=journal
StandardError=journal
SyslogIdentifier=db-backup
# Fail the job if script exits non-zero
SuccessExitStatus=0

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run database backup daily at 02:00
Requires=db-backup.service

[Timer]
# Run every day at 02:00 local time
OnCalendar=*-*-* 02:00:00
# Catch up missed runs after reboot
Persistent=true
# Random delay up to 5 min to spread load on multiple servers
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

8. cron vs. systemd Timers Compared

The decision between classic cron and systemd timers depends on the infrastructure and the requirements. Both approaches have their strengths.

On modern Linux servers running systemd, systemd timers are the better choice for new cron jobs. The question is not whether to migrate, but when. Migrating existing cron jobs is worthwhile whenever logging, catching up missed runs, or service dependencies cause problems. For portable scripts that also need to run on macOS or BSD systems, cron remains the only option.

9. Monitoring: Watching Cron Job Execution Externally

The most reliable form of cron job monitoring is the dead man's switch principle: the cron job sends an HTTP request to a healthcheck service upon successful completion. If the ping fails to arrive within the expected time window, the service automatically sends an alert. This detects not only a failed cron job, but also a cron job that never started in the first place, for example because the server went down or a configuration error corrupted the crontab. Services such as healthchecks.io, Cronitor and Dead Man's Snitch offer exactly this pattern as a hosted service.

For teams already using Prometheus, the prometheus-pushgateway is an elegant solution: after the run, the cron job pushes metrics (duration, exit code, number of records processed) to the pushgateway, from where Prometheus scrapes them. Alertmanager can then react to an exit code other than 0 or to missing metrics (no push within the expected window). The pattern scales from a handful of cron jobs to hundreds, and enables a dashboard-based overview of all scheduled jobs across an infrastructure.

#!/usr/bin/env bash
# monitor_wrapper.sh: healthchecks.io + Prometheus pushgateway integration
set -euo pipefail

readonly HC_URL="${HC_URL:-}"            # healthchecks.io ping URL
readonly PUSHGW="${PUSHGW:-}"           # Prometheus pushgateway URL
readonly JOB_NAME="${1:?job name required}"
shift

START=$(date +%s%N)  # nanoseconds for precision timing

# Signal job start to healthchecks.io
[[ -n "$HC_URL" ]] && curl -fsS -m 5 "${HC_URL}/start" || true

run_status=0
"$@" || run_status=$?

DURATION_MS=$(( ($(date +%s%N) - START) / 1000000 ))

# Push metrics to Prometheus pushgateway
if [[ -n "$PUSHGW" ]]; then
  cat <<METRICS | curl -fsS -m 10 --data-binary @- "${PUSHGW}/metrics/job/${JOB_NAME}"
# HELP cronjob_last_success_timestamp Unix timestamp of last successful run
# TYPE cronjob_last_success_timestamp gauge
cronjob_last_success_timestamp{job="${JOB_NAME}"} $(date +%s)
# HELP cronjob_duration_milliseconds Duration of last run in milliseconds
# TYPE cronjob_duration_milliseconds gauge
cronjob_duration_milliseconds{job="${JOB_NAME}"} ${DURATION_MS}
# HELP cronjob_exit_code Exit code of last run
# TYPE cronjob_exit_code gauge
cronjob_exit_code{job="${JOB_NAME}"} ${run_status}
METRICS
fi

# Signal result to healthchecks.io
if [[ $run_status -eq 0 ]]; then
  [[ -n "$HC_URL" ]] && curl -fsS -m 5 "$HC_URL" || true
else
  [[ -n "$HC_URL" ]] && curl -fsS -m 5 "${HC_URL}/fail" || true
  exit "$run_status"
fi

10. Summary

Professional cron jobs need more than a crontab entry. flock reliably prevents parallel executions without manual cleanup logic. Structured logging with timestamps and levels makes errors visible without piling up disconnected log files. Exit codes must be set explicitly and evaluated by a monitoring system: dead man's switch services even detect cron jobs that never start in the first place. systemd timers solve many structural weaknesses of cron: automatic journaling, catching up missed runs and service dependencies are all built in.

The most important step is to stop treating cron jobs as background processes that simply run themselves, and to start treating them as critical system components deserving the same care as production services. Locking, logging, error notification and monitoring are not a luxury, they are the prerequisite for automated jobs actually being reliable, rather than merely looking reliable until a backup restore reveals they never were.

11. FAQ: Building Reliable Cron Jobs