Exporting Bash Script Metrics to Prometheus: The Node Exporter Textfile Collector

1. Why cron job metrics stay invisible without the textfile collector

Prometheus works on a pull model: a Prometheus server periodically scrapes an HTTP endpoint that returns current metric values in the Prometheus text format. A Bash cron job that runs a backup once a night and then exits, however, has no running process left to serve such an endpoint by the time Prometheus would ever scrape it. Without an extra tool, the success or failure of that job stays completely invisible to the monitoring system.

The Node Exporter solves that structural problem with what is called the textfile collector: on every regular scrape, it simply reads all files with the .prom extension from a configured directory and exposes their content as additional metrics, alongside the usual system metrics like CPU and memory. A Bash script does not need to implement its own HTTP server for that, just drop a simple text file in the right format at the right location.

2. The .prom file format and how the textfile collector reads it

A .prom file follows exactly the same text format a regular Prometheus metrics endpoint serves: one line per metric, with a metric name, optional labels in curly braces, and a numeric value, separated by a space. Each metric should be preceded by two comment lines, # HELP and # TYPE, documenting the metric name and declaring its type (gauge, counter, or summary) so Prometheus and downstream tools like Grafana interpret the values correctly.

The Node Exporter must be started with the --collector.textfile.directory=/var/lib/node_exporter/textfile_collector flag, so it even knows where to look for .prom files. That directory must be writable for the cron job's user, but the Node Exporter itself needs no root privileges for this, which considerably simplifies the security model compared to a dedicated HTTP endpoint per script.

# /var/lib/node_exporter/textfile_collector/backup_job.prom
# HELP backup_job_last_success_timestamp_seconds Unix timestamp of the last successful backup
# TYPE backup_job_last_success_timestamp_seconds gauge
backup_job_last_success_timestamp_seconds 1754470800
# HELP backup_job_duration_seconds Duration of the last backup run in seconds
# TYPE backup_job_duration_seconds gauge
backup_job_duration_seconds 342.6
# HELP backup_job_size_bytes Size of the last backup archive in bytes
# TYPE backup_job_size_bytes gauge
backup_job_size_bytes 1073741824

3. Writing custom metrics from a cron job script atomically

Because the Node Exporter reads every file in the textfile directory at periodic intervals, a race condition risk emerges if a script writes directly into the target file: if the Node Exporter happens to read the file exactly when the script has written only half its lines, Prometheus receives an incomplete or broken file and discards the whole metric group with a parse error visible in the Node Exporter log.

The correct fix is an atomic write: the script first writes to a temporary file in the same directory and then renames it to the final filename with mv. Since mv is atomic within the same filesystem, the Node Exporter always sees either the old, complete file or the new, complete file, never an incomplete intermediate state.

#!/usr/bin/env bash
set -euo pipefail

readonly METRICS_DIR="/var/lib/node_exporter/textfile_collector"
readonly METRICS_FILE="${METRICS_DIR}/backup_job.prom"
readonly TMP_FILE="${METRICS_FILE}.$$.tmp"

start_ts=$(date +%s)
./run-backup.sh
exit_code=$?
end_ts=$(date +%s)
duration=$(( end_ts - start_ts ))

{
  echo "# HELP backup_job_last_run_timestamp_seconds Unix timestamp of the last run"
  echo "# TYPE backup_job_last_run_timestamp_seconds gauge"
  echo "backup_job_last_run_timestamp_seconds ${end_ts}"
  echo "# HELP backup_job_duration_seconds Duration of the last run in seconds"
  echo "# TYPE backup_job_duration_seconds gauge"
  echo "backup_job_duration_seconds ${duration}"
  echo "# HELP backup_job_success Whether the last run succeeded (1) or failed (0)"
  echo "# TYPE backup_job_success gauge"
  echo "backup_job_success $([[ $exit_code -eq 0 ]] && echo 1 || echo 0)"
} > "$TMP_FILE"

mv "$TMP_FILE" "$METRICS_FILE"  # atomic within the same filesystem

4. Choosing the right metric type: gauge, counter, and HELP/TYPE comments

A gauge is the right type for values that can freely rise and fall, for example the duration of the last run or the current fill level of a directory. A counter only fits values that never decrease except on an explicit reset, for example the total number of backup runs ever performed. But since a cron job script typically rewrites the file completely on every run instead of continuing a persistent counter, gauge is in practice the right and simpler choice for most textfile collector metrics.

The # HELP and # TYPE comments are not optional decoration, they are part of the official format: without # TYPE, Prometheus interprets the metric as untyped by default, which several PromQL functions like rate() reject. A consistent, clearly documented metric with a correct type saves a lot of guesswork later in a Grafana dashboard about which mathematical operation on the value even makes sense.

5. Naming conventions: snake_case, unit suffixes, and base units

Prometheus has an established naming convention that also applies to hand-written textfile metrics: metric names consist of snake_case words, carry the script or service name as a prefix, and end with a suffix that clearly names the unit, for example _seconds, _bytes, or _total for counters. A name like backup_job_duration_seconds is self-explanatory, while backup_time reveals neither the unit nor the origin and quickly becomes ambiguous in a shared dashboard with hundreds of other metrics.

For units, the Prometheus convention is to consistently use the base unit, so seconds instead of milliseconds and bytes instead of kilobytes, even if that feels less intuitive to humans at first glance. The reason is that PromQL queries and Grafana dashboards can automatically recognize and convert unit suffixes, but only if the same base unit is used consistently across all metrics, instead of exporting milliseconds in one script and seconds in the next.

6. Label cardinality pitfalls in cron job metrics

Labels make a metric filterable across multiple dimensions in Prometheus, for example backup_job_duration_seconds{database="orders"}, but every unique combination of label values creates a fully separate time series in Prometheus's memory. A cron job script that accidentally uses a file ID, a timestamp, or a generated UUID as a label value creates a brand new, never-reused time series on every run, something called a cardinality explosion, which in the worst case can bring the Prometheus server to its knees.

The rule of thumb: a label value must come from a small, well-understood set that barely changes over time, for example a database name, an environment (prod, staging), or a hostname. Values with high variability like IDs, filenames, or timestamps never belong in a label, at most in the metric value itself, or in a separate logging system built for high-cardinality data.

7. Timestamps and staleness: detecting outdated metrics

A frequently overlooked problem is that a .prom file simply stays behind after a successful run, even if the corresponding cron job later stops running permanently, for example because a server was migrated or the cron entry was accidentally deleted. The value in the file then stays constant and still looks 'green' on a simple dashboard, even though in reality no new run has happened for days.

The fix is to always emit an explicit _last_run_timestamp_seconds value and define a Prometheus alerting rule that checks time() - backup_job_last_run_timestamp_seconds > 90000, in other words alerting when the last run happened more than roughly 25 hours ago. This kind of alert reliably catches both failed runs and runs that never happened at all, while a plain success metric without a timestamp never flags a missing cron job.

# Example alerting rule (prometheus.rules.yml)
- alert: BackupJobStale
  expr: time() - backup_job_last_run_timestamp_seconds > 90000
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "Backup job has not run successfully in over 25 hours"

8. Error handling: exporting failed runs as metrics too

A cron job script that aborts immediately on error with set -e before it writes its metrics file leaves behind exactly no information at the moment monitoring would matter most. The right pattern is to write the metrics file inside a trap on EXIT, so it is guaranteed to be created even on a failed run, with the correct exit code and the duration measured up to that point, instead of being missing entirely.

That reliably lets Prometheus distinguish between three states: a successful run, a failed run with a visible failure metric, and a run that never happened at all, detectable via the stale timestamp from the previous section. Only all three states together give a complete picture of a cron job's actual reliability.

#!/usr/bin/env bash
set -euo pipefail

readonly METRICS_FILE="/var/lib/node_exporter/textfile_collector/backup_job.prom"
readonly TMP_FILE="${METRICS_FILE}.$$.tmp"
start_ts=$(date +%s)
exit_code=0

write_metrics() {
  local end_ts duration
  end_ts=$(date +%s)
  duration=$(( end_ts - start_ts ))
  {
    echo "# TYPE backup_job_success gauge"
    echo "backup_job_success $([[ $exit_code -eq 0 ]] && echo 1 || echo 0)"
    echo "# TYPE backup_job_duration_seconds gauge"
    echo "backup_job_duration_seconds ${duration}"
    echo "# TYPE backup_job_last_run_timestamp_seconds gauge"
    echo "backup_job_last_run_timestamp_seconds ${end_ts}"
  } > "$TMP_FILE"
  mv "$TMP_FILE" "$METRICS_FILE"
}
trap write_metrics EXIT

./run-backup.sh || exit_code=$?

9. The textfile collector compared to other export paths

Besides the textfile collector, Prometheus offers the Pushgateway as a second path for short-lived jobs, where the script actively pushes its metrics over HTTP to a central service instead of writing them locally to a file. The textfile collector fits better for cron jobs that already run on the same host as a Node Exporter, while the Pushgateway makes sense when short-lived jobs run in an environment without a local Node Exporter, for example in ephemeral CI runners or serverless functions.

10. Summary

11. FAQ: Exporting Bash Script Metrics to Prometheus: The Essentials at a Glance