set -x Tracing in Bash: Understanding and Using It Precisely

1. What set -x really shows and why that is often not enough

set -x enables Bash's so called xtrace mode: every executed command is printed to standard error before execution, with all its variables expanded. Anyone using Bash tracing with set -x for the first time is usually surprised how much information becomes visible: not the source code itself, but the command actually executed after parameter expansion, globbing and command substitution. That is exactly what makes set -x so valuable for debugging, because it shows no interpretation, only the real execution.

In its default configuration, however, Bash tracing with set -x has a significant weakness: every line starts with a plain +, with no timestamp, no filename, no line number. In a longer script with hundreds of lines of trace output, it becomes tedious to match a specific trace line back to a specific line in the source code. That exact problem is solved by customizing the PS4 variable, covered in detail in the next section.

Another point often overlooked with set -x: xtrace can not only be enabled globally with set -x for the whole script, but also for individual code sections only, by switching it on and off deliberately. This granularity is the key to turning set -x from a blunt debugging hammer into a precise tool that can also be sensibly used in production pipelines without flooding the output with irrelevant detail.

#!/usr/bin/env bash
set -euo pipefail

target_dir="/var/www/html"
files_count=3

# Enable xtrace
set -x
echo "Deploying to $target_dir with $files_count files"
mkdir -p "$target_dir/releases"
# Output on stderr:
# + echo 'Deploying to /var/www/html with 3 files'
# + mkdir -p /var/www/html/releases

# Disable xtrace again
set +x
echo "This line is not traced"

2. Customizing PS4: timestamps, line numbers and function names

The PS4 variable controls the prefix of every trace line printed by set -x. By default, PS4 is set to a simple +, but Bash provides a range of built in variables that can be embedded in PS4: $LINENO for the current line number, ${BASH_SOURCE[0]} for the filename, and ${FUNCNAME[0]:-main} for the name of the currently executing function. That makes every trace line immediately attributable to a concrete spot in the source code, without manual counting.

Bash tracing becomes even more valuable when you embed a timestamp in PS4. With $(date '+%s.%N'), every trace line can carry a Unix timestamp down to nanoseconds, which is invaluable when analyzing performance problems: you see directly which command took an unusually long time, without building separate timing code into the script. This technique is frequently used in production close deployment scripts to identify bottlenecks that would otherwise only be visible through elaborate profiling.

#!/usr/bin/env bash
set -euo pipefail

# PS4 with timestamp, file, line number and function name
export PS4='+ [$(date "+%H:%M:%S.%N")] ${BASH_SOURCE[0]}:${LINENO} ${FUNCNAME[0]:-main}(): '

deploy_release() {
  local version="$1"
  mkdir -p "/releases/$version"
  cp -r ./build/* "/releases/$version/"
}

set -x
deploy_release "v2.4.1"
set +x

# Sample trace output:
# + [14:32:07.812445123] deploy.sh:9 deploy_release(): mkdir -p /releases/v2.4.1
# + [14:32:07.819881456] deploy.sh:10 deploy_release(): cp -r ./build/* /releases/v2.4.1/

A detail that easily causes bugs: PS4 must be set with single quotes so that the command substitution $(date ...) is only evaluated freshly on every trace line, not once when the variable is set. With double quotes, the timestamp would only be computed once and stay static for the whole script run, which defeats the whole purpose of the timing measurement.

3. Turning tracing on and off deliberately

In production scripts you rarely want to trace the entire script, only a specifically suspicious section. The Bash tracing pattern for that: set -x right before the code block of interest, set +x right after it. That keeps the trace output limited to what matters, and you do not have to scroll through hundreds of irrelevant lines to find the one relevant spot.

This pattern becomes even more flexible with an environment variable as a switch: DEBUG=${DEBUG:-0} at the top of the script, followed by [[ "$DEBUG" -eq 1 ]] && set -x. That keeps the script silent in normal operation, but lets you switch it into tracing mode on demand with DEBUG=1 ./script.sh, with no code changes. This technique is especially handy when a script runs in production and a bug only occurs occasionally, so that constant tracing would produce too much irrelevant noise.

#!/usr/bin/env bash
set -euo pipefail

# Toggle xtrace via environment variable, no code changes needed
DEBUG="${DEBUG:-0}"
if [[ "$DEBUG" -eq 1 ]]; then
  set -x
fi

process_batch() {
  local batch_id="$1"
  echo "Processing batch $batch_id"
}

# Trace only this specific function call, regardless of DEBUG setting
set -x
process_batch "batch-42"
set +x

echo "Batch complete"

# Usage:
#   ./process.sh            — silent, no trace output
#   DEBUG=1 ./process.sh    — full xtrace from the start

For even finer control, you combine both techniques: the global DEBUG flag for the general mode and local set -x/set +x blocks for sections that should not be traced even in debug mode, for example handling of credentials. With { set +x; } 2>/dev/null you can even make the disabling itself invisible, so the trace output does not even show the set +x line itself.

4. Redirecting trace output with BASH_XTRACEFD

By default, set -x writes its output to file descriptor 2, standard error, which means trace lines and the script's actual error output get mixed together. Since Bash 4.1, the BASH_XTRACEFD variable solves this problem: set it to an open file descriptor, and Bash writes the complete trace output there, separate from regular stdout and stderr.

In practice, you open a dedicated descriptor to a log file, assign it to BASH_XTRACEFD, and only then enable set -x. The result: the actual script output stays clean on stdout and stderr, while the complete trace history lands in a separate file that can be analyzed after a failure without having disturbed normal operation.

#!/usr/bin/env bash
set -euo pipefail

# Open a dedicated file descriptor for trace output
exec 5>"/var/log/deploy-trace-$(date +%Y%m%d-%H%M%S).log"
BASH_XTRACEFD=5

set -x
deploy_step_one() { echo "Step one"; }
deploy_step_two() { echo "Step two"; }

deploy_step_one
deploy_step_two
set +x

# Close the trace file descriptor when done
exec 5>&-

echo "Deployment finished, trace saved separately from stdout"

A common mistake: BASH_XTRACEFD must be set before set -x is enabled, otherwise the first trace output still lands on stderr before the redirection takes effect. The descriptor should also be closed again after the trace block, to avoid unnecessarily leaving file handles open, especially in long running scripts with many such blocks.

5. Following nested function calls in the trace

With deeply nested function calls, the standard trace output quickly becomes hard to read, since all lines appear at the same indentation level regardless of the actual nesting depth. Bash provides a built in mechanism for this via the PS4 variable: repeat a character like + in PS4, and Bash automatically multiplies it by the current call depth, producing a visual indentation in the trace.

That makes it immediately visible how deep you currently are in the call stack, without needing a debugger's full backtrace. Combined with ${FUNCNAME[0]} in PS4, every line additionally shows which function it executes in, so the complete control flow can be reconstructed directly from the trace output even with multiply nested calls.

#!/usr/bin/env bash
set -euo pipefail

# The repeated '+' automatically indents by call depth
export PS4='+${FUNCNAME[0]:+${FUNCNAME[0]}(): }'

level_three() { echo "deepest level"; }
level_two()   { level_three; echo "middle level"; }
level_one()   { level_two; echo "top level"; }

set -x
level_one
set +x

# Sample output shows nesting through repeated '+':
# ++++level_three(): echo deepest level
# +++level_two(): echo middle level
# ++level_one(): echo top level

This technique is especially valuable for recursive functions where the nesting depth varies at runtime and cannot be read statically from the source code. Without PS4's built in depth multiplication, it would be nearly impossible to tell from the pure trace output at which recursion level a particular command was actually executed.

6. set -x combined with set -euo pipefail

Set -x unfolds its greatest benefit combined with set -euo pipefail, the standard hardening of robust Bash scripts. If a script aborts with set -e, the last visible trace entry before the abort shows exactly the command that triggered the error, along with all variable values expanded at that point. That massively shortens debugging, since you do not have to guess which line was responsible for the exit code.

An additional trick for this combination: register a trap on ERR that automatically prints $LINENO and $BASH_COMMAND on error abort. BASH_COMMAND holds the last command executed, regardless of whether xtrace is active. Combined with set -x, this gives double coverage: the trap points directly at the error line, and the running trace shows the full context leading up to it.

#!/usr/bin/env bash
set -euo pipefail

# Trap ERR to show exactly which command failed, even without full tracing
trap 'echo "[ERROR] Line $LINENO: command \"$BASH_COMMAND\" failed" >&2' ERR

export PS4='+ ${BASH_SOURCE[0]}:${LINENO}: '
set -x

backup_database() {
  local db_name="$1"
  mysqldump "$db_name" > "/backups/${db_name}.sql"
}

backup_database "shop_production"
set +x

In practice, the ERR trap alone is often enough to identify the error line, while full set -x tracing is only enabled additionally when needed to reconstruct the state before the failure. This two tier strategy avoids the flood of information from continuous tracing while still delivering full precision when an error actually occurs.

7. Performance implications of set -x

Set -x is not free. Every traced line requires evaluating PS4, which causes noticeable overhead especially with complex PS4 definitions involving command substitution, such as the timestamp example above. In a loop with thousands of iterations, continuous set -x tracing can measurably slow down a script's runtime, sometimes by a significant multiple compared to untraced execution.

The rule of thumb is therefore: never keep set -x permanently enabled in production scripts with a high iteration count, but switch it on deliberately for short, suspicious sections. For longer traces during debugging, it is recommended to reduce PS4 to a minimal form, for example just the line number without a timestamp, as long as performance analysis is not the focus. The combination of selective tracing and a minimal PS4 keeps overhead negligible in most cases.

8. Tracing in CI pipelines without leaking secrets

set -x is a double edged sword in CI pipelines: it helps enormously in understanding failing build steps, but carries the risk that environment variables holding credentials, API keys or tokens appear in plain text in the build log as soon as they are expanded in a traced command. A trace log that accidentally contains a database password or a deploy key is a serious security risk, especially when CI logs are visible to multiple team members or archived.

The safe practice: always run sensitive commands that take secrets as arguments with set +x, then resume with set -x immediately after, if the rest of the script should be traced. Many CI systems such as GitLab CI and GitHub Actions automatically mask known secret variables in log output, but that masking does not reliably apply when a secret is part of a composed string, for example a URL with an embedded token. Manually disabling set -x around such spots therefore remains the most reliable safeguard.

#!/usr/bin/env bash
set -euo pipefail

export PS4='+ ${BASH_SOURCE[0]}:${LINENO}: '
set -x

echo "Starting deployment pipeline"
mkdir -p /tmp/build

# Disable tracing before handling secrets
set +x
curl -H "Authorization: Bearer ${DEPLOY_TOKEN}" \
     -X POST "https://api.mironsoft.de/deploy" > /tmp/build/response.json
set -x

echo "Deployment request sent"
set +x

An additional protective measure is to read critical secrets only inside a function from a separate, untraced area and to never call that function under set -x. That keeps the risk of a secret accidentally showing up in the trace log to a minimum, even if a colleague later enables set -x globally for the whole script without knowing the original structure in detail.

9. set -x compared to other tracing techniques

Set -x is the built in, immediately available tracing solution in Bash, but not the only way to follow a script's execution. Depending on the use case, other techniques deliver more precise or more performant results.

In practice a combination proves most effective: set -x with a customized PS4 for spontaneous debugging, BASH_XTRACEFD when trace output and regular logs need to stay cleanly separated, and structured logging for permanent production operation, where set -x would produce too much noise. bashdb comes into play once set -x tracing shows in which area an error occurs, but the exact cause remains unclear.

10. Summary

set -x is far more than a simple debug switch once you go beyond the default settings. A customized PS4 with line number, filename and timestamp makes every trace line immediately attributable to a concrete spot in the code. Selective switching on and off keeps the output limited to what matters, instead of producing hundreds of irrelevant lines. BASH_XTRACEFD cleanly separates trace output from regular stdout and stderr, and the repetitive structure of PS4 makes even deeply nested function calls traceable at a glance.

In CI pipelines, using set -x demands special caution, since carelessly traced secrets in build logs pose a serious security risk. The combination of deliberately switching on and off around sensitive commands and an ERR trap for quick error localization delivers the right balance between traceability and safety. Anyone using set -x this way has a precise, free Bash tracing tool that in most cases is already enough, before a full featured debugger like bashdb is even needed.

11. FAQ: set -x Tracing in Bash