Monitoring Cron Jobs: Reliably Detecting Failed Jobs on Linux

1. Why Cron Jobs Fail Unnoticed

A cron job runs without a terminal, without a visible console, and usually without anyone happening to watch. That is exactly what makes cron job monitoring its own discipline: a script that would immediately be noticed when run interactively, because an error message pops up in the terminal, can fail as a cron job for weeks without anyone noticing at all. The standard output of a cron job disappears by default when no MAILTO is set, or when local mail delivery has never worked, which is the normal state on freshly set up containers and minimal server installations.

The result is often only visible once the damage is already done: a backup that has not produced new files for three weeks, a database export that fails because a disk is full, or a certificate that was never renewed because the renewal cron job silently failed at a changed path. No operating system reports such failures on its own, because from cron's own perspective a failed job is not a special event, merely a nonzero exit code that nobody evaluates by default.

Effective cron job monitoring therefore needs several cooperating building blocks: working delivery of error messages, deliberate evaluation of exit codes, a mechanism that also catches the complete absence of an execution, and an alerting chain that actually reaches people instead of disappearing into an unread inbox. The following sections build these blocks step by step.

2. Configuring MAILTO and Local Mail Delivery Correctly

The classic cron variable MAILTO controls which address cron sends the standard output and standard error of a job to, provided the job produces any output at all. It is set as its own line at the top of a crontab: MAILTO=admin@example.com. It is important to understand that cron only sends a mail if the job actually writes something to stdout or stderr. A script that ends silently with a nonzero exit code without writing a single line of text produces no mail despite the failure, since for cron job monitoring via MAILTO only text output counts, not the exit code itself.

The second, and in practice more common, pitfall: MAILTO is set correctly, but the local mail transport agent is missing entirely. Minimally installed servers and most Docker images contain no sendmail compatible MTA, causing cron to fail on delivery without this being logged anywhere visibly. For production systems it is therefore worth either installing a lightweight relay like msmtp that forwards local mail directly to an external SMTP server, or dropping MAILTO entirely in favor of healthcheck pings and external alerting, as described in the following sections.

# /etc/msmtprc: minimal relay so cron mail actually leaves the host
# Cron itself just calls "sendmail" internally, msmtp intercepts that call

defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/log/msmtp.log

account        default
host           smtp.example.com
port           587
from           cron-alerts@mironsoft.de
user           cron-alerts@mironsoft.de
password       CHANGE_ME_use_a_secrets_manager

# Make msmtp the system-wide sendmail replacement
# ln -sf /usr/bin/msmtp /usr/sbin/sendmail

# In the crontab, set MAILTO explicitly and test with:
# echo "test body" | mail -s "cron mail test" admin@example.com

Anyone who deliberately does not want mail delivery should set MAILTO to an explicit empty string: MAILTO="". This suppresses delivery attempts entirely and avoids cryptic error messages in the system log that would otherwise be generated on every failed send attempt. This explicit decision is part of a deliberate cron job monitoring concept in which alerting responsibility is handed over entirely to healthchecks and structured logging.

3. Evaluating Exit Codes in Cron Scripts Correctly

Every cron job ends with an exit code between zero and 255. Zero means success, any other value signals a failure state defined by the calling script itself. The problem in practice: many cron scripts consist of several consecutive commands, and only the exit code of the last command determines what cron sees overall. A backup script that produces a database dump and then compresses it successfully reports full success even if the dump command itself aborted with an error, as long as the subsequent compression of the empty or incomplete result runs error free.

The reliable solution for this cron job monitoring problem is set -euo pipefail at the top of every script, combined with an explicit check of the exit code after every critical step. It is also worth adopting a consistent wrapper pattern: each cron job does not call the actual script directly, but a small wrapper function that evaluates the final exit code, writes a structured log line, and triggers a healthcheck ping if needed. That way, exit code evaluation becomes a reusable piece of infrastructure instead of an ad hoc solution in every single script.

#!/usr/bin/env bash
# cron-wrapper.sh — wraps any cron job, evaluates the real exit code
set -euo pipefail

readonly JOB_NAME="${1:?Usage: cron-wrapper.sh <job-name> <command...>}"
shift
readonly LOG_FILE="/var/log/cron-jobs/${JOB_NAME}.log"
readonly START_TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

mkdir -p "$(dirname "$LOG_FILE")"

# Run the actual command, capture combined output and exit code
set +e
output="$("$@" 2>&1)"
exit_code=$?
set -e

end_ts="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

{
  echo "[${START_TS}] job=${JOB_NAME} start"
  echo "$output"
  echo "[${end_ts}] job=${JOB_NAME} exit_code=${exit_code}"
} >> "$LOG_FILE"

if [[ $exit_code -ne 0 ]]; then
  logger -t "cron-${JOB_NAME}" "FAILED with exit code ${exit_code}"
  exit "$exit_code"
fi

logger -t "cron-${JOB_NAME}" "OK"

4. Healthcheck Pings as a Dead Man Switch

Exit codes and MAILTO only solve half of the problem: they detect when a cron job runs and fails. They do not detect when a cron job stops running altogether, because the crontab was accidentally deleted, because the cron daemon did not come back up after a reboot, or because a deployment made the crontab line ineffective through a typo. This exact scenario is what the concept of a dead man switch exists for: an external service expects a signal from the job at regular intervals and raises an alarm as soon as that signal fails to arrive, instead of only reacting to explicit error messages.

Services like Healthchecks.io or a self hosted alternative work on the same principle: the cron job calls a unique URL via curl at the successful end of its run, the service remembers the timestamp and compares it against an expected interval. If the ping stays absent longer than expected, for example because the job is no longer executed at all, the service triggers a notification via email, Slack, or webhook. This pattern completes cron job monitoring by covering exactly the gap that plain exit code checking cannot: the complete absence of an execution.

#!/usr/bin/env bash
# backup-with-healthcheck.sh — ping a dead man switch on success and start
set -euo pipefail

readonly HEALTHCHECK_URL="https://hc-ping.com/CHANGE-ME-uuid"
readonly BACKUP_DIR="/var/backups/mysql"

# Signal "job started" — lets the healthcheck service flag long-running jobs
curl -fsS -m 10 --retry 3 "${HEALTHCHECK_URL}/start" > /dev/null || true

if mysqldump --all-databases | gzip > "${BACKUP_DIR}/dump-$(date +%F).sql.gz"; then
  # Success ping — resets the "missed check-in" timer on the healthcheck side
  curl -fsS -m 10 --retry 3 "${HEALTHCHECK_URL}" > /dev/null || true
else
  # Failure ping with exit code — triggers an immediate alert
  curl -fsS -m 10 --retry 3 "${HEALTHCHECK_URL}/fail" > /dev/null || true
  exit 1
fi

5. Structured Logging for Cron Jobs

A single missing mail or a single missing healthcheck ping rarely explains why a job failed. For a real diagnosis, cron job monitoring needs traceable log lines with timestamp, job name, runtime, and exit code, consistent across every cron job on a server. A uniform format such as timestamp job_name exit_code duration_ms can be evaluated with simple text tools like grep and awk, without necessarily needing a separate log aggregation system, especially on smaller servers with a manageable number of jobs.

What matters is that every cron job runs through the same wrapper, instead of every script inventing its own logging format. A central wrapper, as shown in the previous section, guarantees consistent log lines and makes it possible to find every job that failed in the last week across all cron entries with a single command, regardless of who originally wrote which script.

6. Connecting Cron Output to the systemd Journal

On systemd based distributions, cron job monitoring can additionally be tied into the central journal instead of maintaining separate log files under /var/log/cron-jobs. The command logger -t cron-jobname "message" writes a line directly into the journal, tagged with a dedicated identifier, so individual jobs can be filtered specifically with journalctl -t cron-jobname. This approach unifies cron output with the rest of the system logs in one place and benefits from the automatic rotation and size limits the journal already provides.

A practical advantage over plain text files: journalctl allows time range filters like --since "1 hour ago" and combination with -p err to show only error messages, without having to grep for specific text patterns yourself. For servers that already run centralized log shipping, for example via systemd-journal-remote or a separate log collector, cron failures automatically end up in the same pipeline as every other system event, without requiring a dedicated integration just for cron.

7. From Log Line to Notification

A log line alone alerts nobody as long as it is not actively evaluated. The final building block of a complete cron job monitoring setup is therefore an alerting pipeline that actively forwards failed jobs to a channel people actually pay attention to, typically a team chat like Slack or Mattermost rather than an email address that gets lost in the daily flood. A simple webhook call directly from the cron wrapper achieves this goal without installing any additional software.

For larger environments with many servers, integration into a central monitoring system like Prometheus with the Node Exporter textfile collector is worth the effort instead: the cron wrapper writes a metrics file with timestamp and exit code after every run, Prometheus reads this file, and Alertmanager triggers a notification for missing or failed jobs according to the team's already established escalation rules. This way cron job monitoring integrates into an existing monitoring landscape rather than forming an isolated special path.

{
  "text": "Cron job failed: nightly-backup",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Cron job failed*\nJob: `nightly-backup`\nExit code: `1`\nHost: `db-primary-01`\nTimestamp: `2026-07-31T02:15:03Z`"
      }
    }
  ]
}

8. Monitoring Runtimes and Catching Silent Hangs

A job that neither crashes nor reports an error, but has been running for hours without finishing, escapes both exit code checking and a classic healthcheck ping, as long as that ping is only sent at the very end. The timeout command enforces a hard upper limit on a job's runtime and terminates it with a defined exit code once that limit is exceeded, so a hanging process does not tie up resources indefinitely and instead shows up as a regular failure in cron job monitoring.

It is also worth comparing the actual runtime against an expected value. A nightly database export that normally takes five minutes and suddenly needs forty minutes usually points to growing data volume, a missing index, or another creeping problem, long before the job actually fails outright. The start ping from section four, combined with a configured time window in the healthcheck service, covers exactly this scenario without needing any custom timing logic inside the script itself.

9. Monitoring Approaches Compared

No single approach covers every failure scenario a cron job can run into. The following table compares the building blocks and shows which class of failure each one catches.

In practice, several approaches are usually combined: a wrapper for consistent logging and exit code evaluation, a healthcheck ping to detect missing executions, and, as server count grows, a central monitoring solution on top. The individual building blocks do not exclude each other, they complement each other into a cron job monitoring setup that reliably reports both silent and loud failures.

10. Summary

Cron job monitoring needs more than a set MAILTO variable, since it detects neither missing executions nor silent failures without text output. An exit code wrapper with structured logging makes failure states traceable and consistent across every job on a server. Healthcheck pings close the crucial gap of the dead man switch by also detecting the complete absence of a job, not just its failure.

The timeout command prevents hanging jobs from tying up resources indefinitely, while connecting to the systemd journal or a central monitoring system like Prometheus unifies evaluation across many servers. Combining these building blocks results in cron job monitoring that reliably reports both loud errors and the far more dangerous silent failures to the right people.

11. FAQ: Cron Job Monitoring