bashdb Debugger: Step by Step Bash Script Debugging

1. Why a debugger for Bash scripts makes sense

Most Bash developers debug with echo lines inserted before and after suspicious spots and later removed again. That works for ten lines of code, but quickly becomes impractical for a grown deployment or backup script with several functions, loops and external calls. This is exactly where bashdb comes in, a full featured Bash debugger built on GNU debugger infrastructure that brings concepts such as breakpoints, step execution and variable display directly into the shell.

bashdb replaces guessing with controlled observation. Instead of assuming at which line a script takes a wrong turn, you halt execution at exactly the spot of interest and look at the actual state of every variable. For scripts that run in production environments and are hard to reproduce when they fail, that is a substantial time saving compared to classic trial and error with temporary output lines.

A Bash debugger such as bashdb becomes especially valuable when a script is handed to colleagues who did not write the code themselves. Instead of reading through hundreds of lines of shell code, you start the script under bashdb, walk through it step by step and observe in real time how variables change and which functions are actually called. The following sections show the complete workflow, from installation to analyzing a call stack in a multi file script.

2. Installing and first starting bashdb

Installing bashdb is straightforward on most Linux distributions through the package manager. On Debian and Ubuntu the package is simply called bashdb, on Arch Linux it lives in the AUR, and on macOS Homebrew installs the current version. After installation, the bashdb command is available as a standalone tool that takes a Bash script as an argument and starts it in an interactive debug session instead of running it directly.

The first start immediately shows the central difference from normal script execution: bashdb halts before the first line and waits for input. From there you control execution through short commands that strongly resemble gdb, the classic GNU debugger for C programs. Anyone who has already worked with gdb or similar debuggers will feel right at home in bashdb, since the command structure was deliberately kept analogous.

# Install bashdb on Debian/Ubuntu
sudo apt-get install bashdb

# Install on macOS via Homebrew
brew install bashdb

# Start a debugging session for a script
bashdb ./deploy.sh

# Inside the bashdb prompt (bashdb<1>):
#   l          list source lines around current position
#   n          step to next line (does not enter functions)
#   s          step into the next line (enters functions)
#   c          continue execution until next breakpoint
#   q          quit the debugger session

A common misunderstanding when getting started: bashdb does not run the script in a separate sandbox, it starts a real Bash instance with an attached debug hook. That means all side effects of the script, such as writing files or making curl requests, actually happen as soon as you step past the corresponding line with c or n. For scripts with destructive operations, it is recommended to use a copy of the target environment or a dry run mode before the first debug session.

3. Setting breakpoints and controlling script execution

The core of every debug session with bashdb is the breakpoint. With break FILE:LINE or short b LINE, execution halts exactly at the given spot as soon as it is reached, no matter how many times the line was passed before in loops. That is the decisive advantage over echo based debugging: a breakpoint inside a function called a hundred times in a loop can additionally carry a condition, so it only triggers on the fiftieth iteration.

Besides line numbers, bashdb also accepts function names as a breakpoint target, which is especially practical for longer scripts with many small helper functions. With break functionname, execution halts on entering the function, regardless of where in the script it is called from. The commands n (next) and s (step) then control the granularity: n skips function calls as a whole, s actually steps into them.

# bashdb ./backup.sh

# Set a breakpoint at a specific line
bashdb<1> break 42

# Set a breakpoint on a function by name
bashdb<2> break create_backup_archive

# Set a conditional breakpoint (only triggers when condition is true)
bashdb<3> break 42 if $retry_count -ge 3

# List all currently active breakpoints
bashdb<4> info breakpoints

# Continue execution until the next breakpoint is hit
bashdb<5> continue

# Remove a breakpoint by its number
bashdb<6> delete 1

A detail that is often overlooked: breakpoints in bashdb stay active across multiple continue calls until explicitly removed with delete. In a loop with many iterations, that can mean confirming the same breakpoint dozens of times. The conditional variant with if solves exactly this problem, only letting the breakpoint fire when the given Bash condition is true, for example a counter above a certain threshold.

4. Inspecting variables and checking program state

Once execution has halted at a breakpoint, the next question is almost always: what value does this variable actually hold right now? bashdb answers that with the print command, short p, followed by the variable name. Unlike an inserted echo, you do not need to modify or restart the script for this, you can directly query any number of variables while halted and then resume execution.

For more complex cases, bashdb also supports eval, which evaluates arbitrary Bash expressions in the current context, including array access and parameter expansions. This is especially helpful to check whether a condition used later in the script actually evaluates as expected, before the corresponding code path is even reached.

bashdb<1> break 58
bashdb<2> continue

# Print a simple variable
bashdb<3> print $target_dir
/var/www/html/current

# Print an array element and array length
bashdb<4> print ${backup_files[2]}
bashdb<5> print ${#backup_files[@]}

# Evaluate an arbitrary expression in the current scope
bashdb<6> eval [[ $retry_count -ge $MAX_RETRIES ]] && echo "would abort here"

# Show all local variables in the current function
bashdb<7> info locals

# Change a variable value on the fly to test a code path
bashdb<8> set variable DEBUG=1

The set variable command even lets you change a variable's value during the running debug session to force a specific code path without restarting the script. This is especially useful for testing error handling code that is normally only reached under rare conditions, such as a retry counter exceeding a certain threshold.

5. Watchpoints and conditional breakpoints

While a normal breakpoint is bound to a line, a watchpoint in bashdb binds the interruption to a variable. With watch variablename, execution automatically halts as soon as this variable's value changes, no matter at which line that happens. That is invaluable when a variable somewhere in the script takes on an unexpected value and you do not know which of the twenty lines that could theoretically write it is actually responsible.

Combining watchpoints with conditions creates a very precise debugging tool: watch status_code if status_code != 0 only halts when the variable is actually set to an error value, not on every assignment. That drastically reduces the number of interruptions and leads directly to the line that causes the actual bug, instead of iterating through harmless intermediate states.

bashdb<1> break main
bashdb<2> continue

# Break whenever this variable changes value, anywhere in the script
bashdb<3> watch upload_status

# Break only when the variable changes to a non-zero value
bashdb<4> watch upload_status if upload_status != 0

# Continue until the watched variable actually changes
bashdb<5> continue
Watchpoint 1: upload_status changed
Old value: 0
New value: 22
deploy.sh:87  upload_status=$?

# Inspect the call stack at the moment of the change
bashdb<6> backtrace

A watchpoint causes noticeable overhead in bashdb, since after every line it has to check whether the observed value changed. For short debug sessions that is negligible, but for very long scripts with many iterations, watchpoints should be used deliberately and removed with delete again after finding the responsible line, so as not to unnecessarily slow down the rest of the execution.

6. The call stack: backtrace and following function calls

When an error occurs deep inside a nested function, the decisive question is often not what goes wrong in that function, but from where and with which arguments it was called. The backtrace command, short bt, shows the full call stack in bashdb: every function currently active, in the order it was called, together with the line number at which each call happened.

This is especially valuable for recursive functions or library functions called from multiple places in the script. Without bashdb, you would have to insert a separate debug output at every possible call site to figure out which path actually led to the observed error. With backtrace, this information is immediately and completely available, including the option to switch to a specific stack level with frame N and inspect local variables there.

bashdb<1> break validate_input
bashdb<2> continue

# Show the full call stack at the current breakpoint
bashdb<3> backtrace
#0  validate_input (config.sh:15)
#1  load_configuration (config.sh:34)
#2  main (deploy.sh:120)

# Switch to a specific stack frame to inspect its local variables
bashdb<4> frame 1
bashdb<5> info locals

# Return to the innermost frame
bashdb<6> frame 0

# Step out of the current function back to the caller
bashdb<7> finish

The finish command runs the current function to completion and halts right after it in the calling function, with the return value of the completed function immediately available. That saves several next calls and is especially efficient when you know a function works correctly but want to see what the caller does with its result.

7. bashdb in scripts with multiple files

Larger Bash projects rarely consist of a single file. Instead, a main entry point is extended with source lib/utils.sh to include library functions maintained in separate files. bashdb follows source calls automatically and allows breakpoints in every included file, just like in the main script itself. That is a clear advantage over echo based debugging, where you would have to manually insert output statements in every affected file.

To set a breakpoint specifically in a particular file, the filename is written before the line number: break lib/utils.sh:23. That matters as soon as multiple included files have overlapping line numbers, or when function names in different libraries happen to be identical. bashdb always shows the full path of the current file when halting, so it is always clear which part of the project you are currently in.

# Project structure:
# deploy.sh          — main entry point, sources lib/*.sh
# lib/utils.sh        — helper functions
# lib/notifications.sh — Slack/email notification functions

# Start debugging the main entry point
bashdb<1> bashdb ./deploy.sh

# Set a breakpoint inside a sourced library file
bashdb<2> break lib/utils.sh:23

# Set a breakpoint by function name, regardless of which file defines it
bashdb<3> break send_slack_notification

# Show which file and line the debugger is currently stopped at
bashdb<4> info line

# List all sourced files known to the current session
bashdb<5> info sources

A common pitfall in multi file scripts: if a library file is included via source with a relative path, bashdb must be started in the same working directory as the actual script, otherwise sourcing already fails before the first breakpoint is reached. It is recommended to always use $(dirname "${BASH_SOURCE[0]}") in the main script to resolve paths independent of the current working directory, also under bashdb.

8. Typical debugging scenarios with bashdb

A classic scenario: a deployment script fails in production with an unclear error but cannot be reproduced locally. With bashdb, you start the script in a staging environment, set a breakpoint shortly before the suspected error area and step through line by line until the actual state surfaces that leads to the crash. Often it turns out that an environment variable in production has a different value than locally assumed, an error that is easily overlooked with echo debugging because you do not know what to search for.

A second typical scenario involves race conditions in scripts with background processes. Since bashdb halts execution of the main process, timing dependent bugs cannot be directly reproduced with it, but the state before background processes start can be checked exactly, which is often enough to rule in or out incorrectly initialized variables or missing locks as the cause. Combined with targeted watchpoints on shared variables, the order of concurrent write accesses can also be reconstructed.

A third scenario is analyzing scripts taken over from third parties with little or no documentation. Instead of reading through the entire code line by line, you start the script under bashdb, set a breakpoint on main and work through the actual control flow with next and step, while the debug output automatically shows which functions are actually called and which are dead code.

9. bashdb compared to other debugging methods

Choosing the right debugging method depends heavily on context: a one off bug in a short script rarely justifies the effort of a full bashdb session, while a complex, multi file automation project remains barely maintainable without structured debugging.

In practice these methods do not exclude each other, they complement one another. A rough set -x trace quickly shows in which area of the script the error even occurs, and only then does the more targeted use of bashdb pay off, to narrow down the exact cause with breakpoints and watchpoints. Unit tests in turn prevent the same bug from reappearing after it was fixed with bashdb, since they permanently guard the corrected code path.

10. Summary

bashdb turns debugging Bash scripts from a guessing game with temporary echo lines into a controlled, traceable process. Breakpoints halt execution exactly at the spot of interest. Watchpoints show where a variable unexpectedly changes value, without having to manually check every possible write site in the script. The call stack with backtrace makes visible through which path a function was actually reached.

For short one liners, echo debugging remains the most pragmatic choice. But as soon as a script spans multiple files, nested functions, or production close bugs that cannot easily be reproduced locally, bashdb delivers a precision that echo output cannot offer. Combined with set -x tracing for the rough overview and unit tests for permanent regression protection, a complete debugging and quality assurance workflow emerges for Bash projects of any size.

11. FAQ: bashdb Debugger for Bash Scripts