Correlation IDs for Distributed Tracing in Shell Pipelines

1. What a correlation ID is and why shell pipelines need one

A correlation ID is a unique identifier, usually a UUID, that accompanies a single logical operation from start to finish, regardless of how many individual processes, scripts, or programs take part in that operation. Unlike a process ID, which changes with every new script invocation, the correlation ID stays constant across the entire chain and gets passed explicitly, instead of implicitly falling out of the system environment.

In a typical Bash deployment pipeline that runs migrations, builds assets, and performs a health check, each of those steps runs as its own process invocation with its own process ID and its own log window. Without a shared correlation ID, a developer debugging an issue has to painstakingly reconstruct from timestamps alone which log lines belong to which deployment run, something that becomes practically impossible once several deployments run in parallel.

2. Generating the correlation ID once at the start of the pipeline

The correlation ID must be created at exactly one place, the pipeline's entry point, and must never be regenerated by any downstream script afterward. The script that starts the pipeline first checks whether a correlation ID was already passed in from outside, for example because it was itself called by a higher-level process like a CI job, and only generates a new one if none exists.

For the generation itself, uuidgen is enough on most Linux distributions; alternatively, $(date +%s%N)-$$ also produces a sufficiently unique identifier without that extra package if uuidgen is unavailable. What matters is that the generation stays idempotent against an already-existing ID, so nested calls to the pipeline do not accidentally create several independent IDs for the same logical operation.

#!/usr/bin/env bash
set -euo pipefail

# Reuse an inherited correlation ID, or generate a new one if this is the entry point
export CORRELATION_ID="${CORRELATION_ID:-$(uuidgen 2>/dev/null || echo "$(date +%s%N)-$$")}"

echo "Starting deployment pipeline, correlation_id=${CORRELATION_ID}"

3. Passing it via environment variable to called scripts

An environment variable set with export automatically propagates to every child process a Bash script starts, whether that is another Bash script, a Python program, or any other command. That exact behavior makes an environment variable the ideal carrier for a correlation ID: as long as every script in the chain picks up the variable at the start instead of overwriting it, it survives across as many levels as needed without extra wiring effort.

A common mistake is setting the correlation ID only as a local shell variable instead of exporting it. Without export, the variable is visible in the current script but invisible to every child process, so the chain breaks exactly at the point where a new script or program starts, a mistake that often does not show up immediately when testing interactively in a terminal.

#!/usr/bin/env bash
set -euo pipefail
# entrypoint.sh

export CORRELATION_ID="${CORRELATION_ID:-$(uuidgen)}"

./bin/run-migrations.sh    # inherits CORRELATION_ID
./bin/build-assets.sh      # inherits CORRELATION_ID
./bin/health-check.sh      # inherits CORRELATION_ID

echo "Pipeline finished, correlation_id=${CORRELATION_ID}"

4. Crossing process boundaries: SSH, cron, and background jobs

Simple inheritance via export only works within a single connected process tree. Once the pipeline runs a command on another host over ssh, the correlation ID needs to be passed explicitly as an argument or environment variable to the remote command, because SSH does not forward local environment variables to the remote session by default, unless the SSH server was explicitly configured to do so.

For a script started by cron, there is no parent shell session at all whose environment variables could be inherited, so such a script must always generate a fresh correlation ID at its entry point. For background processes started with &, inheritance works normally as long as the variable was already exported before the &, something that is easy to overlook in practice when the order in the script gets swapped.

#!/usr/bin/env bash
set -euo pipefail

export CORRELATION_ID="${CORRELATION_ID:-$(uuidgen)}"

# SSH does not forward local env vars by default -- pass explicitly
ssh deploy@remote-host "CORRELATION_ID='${CORRELATION_ID}' /opt/scripts/restart-service.sh"

# Background job started AFTER export still inherits the variable correctly
long_running_cleanup_task &

5. Integrating with structured logging

A correlation ID only reaches its full value combined with structured logging, because structured fields can be filtered directly in a log aggregator. Every logging function in a pipeline should therefore automatically insert the correlation_id field from the environment variable of the same name into every logged line, without the caller having to pass that value manually on every log call.

In practice, a small jq-based logging function that reads the environment variable directly is enough, since environment variables are globally visible within a Bash process anyway. A log aggregator like ELK or Loki can then show all lines across every involved script and host of a single deployment run with a single query for correlation_id="...", in correct chronological order.

#!/usr/bin/env bash
set -euo pipefail

log_json() {
  local level="$1" message="$2"
  jq -nc \
    --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
    --arg level "$level" \
    --arg msg "$message" \
    --arg correlation_id "${CORRELATION_ID:-unknown}" \
    '{timestamp: $ts, level: $level, message: $msg, correlation_id: $correlation_id}' >&2
}

log_json "info" "Running database migrations"

6. Passing correlation IDs on to HTTP calls

If a Bash script calls an HTTP API as part of the pipeline, for example to report a deployment status to an external system, the correlation ID should be sent along as a custom header like X-Correlation-ID. That allows the trace to continue across the boundary between shell script and web application, as long as the receiving application also picks up that header into its own structured logging.

This practice follows the same principle long established in distributed web applications, where correlation IDs or trace IDs get set by reverse proxies like nginx and passed through every downstream microservice. A Bash script that communicates via curl with such a header slots seamlessly into an existing observability chain, instead of remaining an isolated island disconnected from the rest of the system.

#!/usr/bin/env bash
set -euo pipefail

curl -sS -X POST "https://status.internal/api/deployments" \
  -H "X-Correlation-ID: ${CORRELATION_ID}" \
  -H "Content-Type: application/json" \
  -d "{\"status\": \"in_progress\"}"

7. Correlation ID versus OpenTelemetry trace ID and span ID

Modern tracing standards like OpenTelemetry distinguish between a trace ID, which identifies an entire distributed operation, and a span ID, which marks a single sub-step within that operation, together with a parent span ID for the hierarchical structure. A simple correlation ID in a Bash script usually only covers the role of the trace ID in practice, without reproducing the fine-grained span hierarchy of a full tracing system.

For most Bash automation tasks, that simplified variant is entirely sufficient, because the goal is usually not a detailed latency analysis of individual sub-steps, but simply the ability to find all related log lines of an operation. Anyone who needs real distributed tracing with span hierarchy and latency measurement should check whether an OpenTelemetry-compatible shell instrumentation tool is the better fit, rather than rebuilding a simplified solution from scratch.

8. Pitfalls: nested IDs, overwritten variables, and subshells

The most common mistake is that a called script accidentally generates its own new correlation ID instead of checking for and reusing the inherited one, usually because the script is also meant to run standalone outside the pipeline. The fix is consistently applying the pattern CORRELATION_ID="${CORRELATION_ID:-$(uuidgen)}", which always prefers an already-existing ID and only generates a new one when running standalone.

A second pitfall involves subshells created by parentheses ( ... ) or pipes: variables changed inside a subshell do not affect the parent environment, which, if the correlation ID is accidentally reassigned inside such a subshell, leads to confusing, seemingly random gaps in the log chain that are hard to reproduce because they depend on the exact structure of the pipe chain.

9. Correlation IDs compared to other tracing approaches

Choosing the right tracing approach for shell automation depends on how complex the involved systems are: for a single Bash-script deployment pipeline, a simple correlation ID is usually entirely sufficient, while a system with many microservices and a complex request fan-out structure benefits considerably more from a full OpenTelemetry setup with span hierarchy.

10. Summary

11. FAQ: Correlation IDs for Distributed Tracing: The Essentials at a Glance