Truly Understanding Pipes, Redirects and File Descriptors

1. Understanding the File Descriptor Model

Before you can truly understand pipes, redirects and file descriptors in Bash, you need to grasp the underlying Unix model. Every process on Unix/Linux has a table of file descriptors: positive integers that reference open files, pipes, sockets or other I/O resources. When a process calls read() on file descriptor 0, it reads from standard input. write() on FD 1 writes to standard output. That is all the process knows; whether a terminal, a file or a pipe sits behind it is transparent to it. The shell uses this model to redirect I/O however it needs to.

When the shell spawns a new process via fork(), the child inherits the entire file descriptor table of the parent. The shell can modify the child's FD table before the exec() call that loads the child program: opening files, duplicating FDs, closing FDs. That is exactly what redirects do. They manipulate the file descriptor table of the child process before the actual program starts. The program itself still reads from FD 0 and writes to FD 1 as always, unaware that these FDs now point to files or pipes instead of a terminal.

This understanding makes many confusing redirect constructs make sense. 2>&1 means "duplicate file descriptor 1 onto file descriptor 2", making FD 2 an alias for the same target as FD 1. The order of redirects is decisive here: command > file 2>&1 means "redirect FD 1 to a file, then redirect FD 2 to the same target as FD 1 (the file)". command 2>&1 >file means "redirect FD 2 to the same target as FD 1 (still the terminal), then redirect FD 1 to the file", a common mistake.

2. stdin, stdout and stderr: the Three Standard Descriptors

The three standard file descriptors are fixed on Unix/Linux: FD 0 is stdin (standard input), FD 1 is stdout (standard output) and FD 2 is stderr (standard error). Every new process inherits all three from its parent process, which is typically the shell. When an interactive terminal starts, all three point at the terminal pseudo device. The semantic separation of stdout and stderr is a convention: status messages, errors and logs belong on stderr, so that stdout stays clean for machine-readable output.

In shell scripts, following this convention consistently matters for composability, the ability to chain scripts together in pipes. A script that writes status messages to stdout makes its output useless in a pipe: the next command receives both the payload and the log lines. The correct pattern: real output (file names, computed values, JSON) goes to stdout, everything else (echo "[INFO] ...", echo "[ERROR] ...") goes to stderr via >&2. That way a script works both as a filter in a pipe and standalone with meaningful console output.

#!/usr/bin/env bash
# fd-basics.sh: File descriptor fundamentals and redirect ordering
set -euo pipefail

# FD 1 = stdout, FD 2 = stderr: always write status to stderr
log_info()  { echo "[INFO]  $*" >&2; }
log_error() { echo "[ERROR] $*" >&2; }

# Correct: stdout for data, stderr for diagnostics
process_files() {
  local dir="$1"
  log_info "Processing directory: $dir"

  while IFS= read -r -d '' f; do
    if [[ -r "$f" ]]; then
      echo "$f"  # stdout: the actual result list
    else
      log_error "Cannot read: $f"  # stderr: diagnostic
    fi
  done < <(find "$dir" -name "*.txt" -print0)
}

# Correct redirect order: FD1 -> file, then FD2 -> same target as FD1
# command > file 2>&1   : both stdout and stderr go to file
# command 2>&1 > file   : stderr goes to TERMINAL (old FD1), stdout to file
process_files /var/data > /tmp/results.txt 2>&1    # both to file
process_files /var/data > /tmp/results.txt 2>/dev/null  # only stdout

# FD duplication: save and restore stdout
exec 3>&1           # Save FD1 in FD3
exec > /tmp/log.txt # Redirect all stdout to file
echo "This goes to file"
exec 1>&3           # Restore FD1 from FD3
exec 3>&-           # Close FD3
echo "This goes to terminal again"

3. Redirects in Detail: >, >>, <, 2>, &>

The basic redirect operators in Bash are > (stdout to file, overwriting), >> (stdout to file, appending), < (stdin from file), 2> (stderr to file) and &> (stdout and stderr to the same file, a Bash-specific shortcut, not POSIX). Prefixing with an FD number, 3>file, opens a dedicated file descriptor. N>&M duplicates FD M onto FD N. N>&- closes FD N.

One difference deserves particular attention: &>file (Bash) versus >file 2>&1 (POSIX-compatible). Both redirect stdout and stderr to the same file, but only the second form works in a POSIX shell (sh). For scripts that only ever run under Bash, &> is shorter and clearer. For portable scripts, prefer >file 2>&1. The pattern >/dev/null 2>&1 is the classic way to suppress all output of a command; &/dev/null is the shorter Bash equivalent.

4. Opening and Closing Custom File Descriptors

Bash lets you open your own file descriptors with exec N>file, exec N<file or exec N<>file (read and write). FD numbers 0 through 2 are reserved for the standard streams; FDs from 3 upward are free to use. One important use case: opening a logging FD that writes into a log file alongside the normal output. exec 9>/var/log/script.log opens FD 9 for the whole script; echo "message" >&9 writes to the log file without touching stdout or stderr.

Custom file descriptors are also the foundation of flock for process locking. exec 9>/var/lock/script.lock; flock -n 9 opens FD 9 as a lock file and tries to acquire an exclusive lock. The operating system releases the lock automatically when the FD is closed, whether the process exits normally or crashes. That is considerably more reliable than PID-file-based locking mechanisms, which need to be cleaned up manually. When opening custom FDs, always run exec N>&- at the end of their scope to close the FD and free the resource.

5. exec for Persistent Redirects in a Script

The exec builtin has two uses in Bash. With a command as its argument, exec command replaces the current shell process with that command. Without a command, but with redirects, exec changes the file descriptors of the current shell itself, permanently, for every command that follows in the script. That is the key difference from a one-off redirect on a single command. exec > /var/log/script.log 2>&1 at the top of a script redirects all stdout and stderr output of the entire script into the log file; every echo call, command output and error message that follows lands automatically in the file.

The classic pattern for scripts that need to write to the terminal and to a log file at the same time combines exec with tee: exec > >(tee -a /var/log/script.log) 2>&1. The process substitution >(tee -a ...) creates a pipe that forwards all output to tee. tee writes it simultaneously to its own stdout (the terminal) and to the given file. The result: a complete log in the file, plus interactive output on the terminal, without changing a single command in the script. This is a powerful production pattern for deployment scripts that need to be both observable and auditable.

#!/usr/bin/env bash
# exec-redirect.sh: Persistent redirects with exec + tee logging
set -euo pipefail

LOG_FILE="/var/log/deploy/$(date +%Y%m%d-%H%M%S).log"
mkdir -p "$(dirname "$LOG_FILE")"

# Redirect all stdout and stderr to file AND terminal simultaneously
exec > >(tee -a "$LOG_FILE") 2>&1

echo "[$(date)] Deployment started"

# Custom file descriptors for structured logging
exec 3>>"${LOG_FILE%.log}.debug.log"  # FD3 for debug log

debug() { echo "[DEBUG] $*" >&3; }   # Write to debug FD only
info()  { echo "[INFO]  $*"; }        # Write to stdout (-> tee -> terminal+log)
error() { echo "[ERROR] $*" >&2; }    # Write to stderr (-> also tee'd)

info "Checking dependencies..."
debug "PATH=$PATH"
debug "User=$(id -un)"

# Read from custom FD (useful for reading configuration)
exec 4< /etc/deploy/config.env
while IFS='=' read -r -u 4 key value; do
  [[ "$key" =~ ^#.*$ ]] && continue  # Skip comments
  [[ -z "$key" ]] && continue
  export "${key}=${value}"
  debug "Config loaded: ${key}=***"
done
exec 4<&-  # Close FD4

info "Deployment complete"

# Close debug log FD
exec 3>&-

6. Pipes: How They Work and What They Cannot Do

A pipe in Bash is a unidirectional buffer in the kernel: data that the left-hand process writes to stdout can be read by the right-hand process through stdin. Both processes run at the same time; the pipe mechanism is a synchronization mechanism, not a sequential call. When the buffer fills up (typically 64 KB on Linux), the writing process blocks until the reading process consumes data. This allows processing of data volumes that exceed available RAM, as long as the consumer reads fast enough.

What pipes cannot do: send data backward. Every pipe is unidirectional. Anyone who needs bidirectional communication between two processes needs two pipes (one per direction) or a named pipe (mkfifo). Another important limitation: variables set inside one element of a pipeline are not visible after the pipe, because each pipeline element runs in a subshell. That explains why command | while read line; do count=$((count+1)); done; echo $count always prints 0: the while body runs in a subshell, and the count variable in the parent process stays unchanged. The fix is process substitution or another accumulation pattern.

7. tee: Writing to Multiple Streams at Once

The tee command is a T-piece for data streams: it reads from stdin and writes the same data simultaneously to stdout and to one or more files. Combined with process substitution, command | tee >(further-processing), tee becomes a powerful fan-out operator that splits a data stream into multiple pipelines at once. That enables patterns such as: reading a file once while simultaneously computing its SHA256 and compressing it, without reading the file twice or creating a temporary file.

Important flags: tee -a appends to an existing file instead of overwriting it, the correct behavior for log files that should accumulate across multiple script runs. tee /dev/stderr duplicates stdout onto stderr, useful for debugging inside pipes. tee /dev/fd/3 writes into a custom file descriptor. Combining exec > >(tee) with several >() process substitutions lets a single command write in parallel to the terminal, a log file, and another processing step, a capability that would need considerably more code in most scripting languages.

#!/usr/bin/env bash
# tee-fanout.sh: Fan-out patterns with tee and process substitution
set -euo pipefail

SOURCE="/var/backup/database.sql.gz"
DEST_DIR="/var/backup/encrypted"
HASH_FILE="${DEST_DIR}/database.sha256"
COMPRESSED_FILE="${DEST_DIR}/database.sql.gz.enc"
LOG_FILE="/var/log/backup-verify.log"

# Fan-out: read source once, simultaneously:
#   1. Decrypt and decompress for verification
#   2. Calculate SHA256 of encrypted file
#   3. Log transfer stats
openssl enc -aes-256-gcm -d \
  -in "$SOURCE" \
  -pass file:/etc/backup/.passphrase \
  -pbkdf2 | \
tee \
  >(sha256sum | awk '{print $1}' > "$HASH_FILE") \
  >(gzip -9 | openssl enc -aes-256-gcm \
      -out "$COMPRESSED_FILE" \
      -pass file:/etc/backup/.passphrase \
      -pbkdf2) | \
wc -c | \
tee -a "$LOG_FILE" | \
awk '{printf "[VERIFY] Processed %d bytes\n", $1}'

# Here-doc as stdin for a command (no temporary file needed)
mysql --defaults-file=/etc/mysql/backup.cnf <<'SQL'
  SELECT COUNT(*) AS total_rows FROM information_schema.tables
  WHERE table_schema = 'myapp';
SQL

# Here-string: single-line stdin without echo subprocess
read -r hostname port <<< "db.example.com:5432"
echo "Host: $hostname, Port: $port"

# Named pipe for bidirectional shell coordination
FIFO="$(mktemp -u)"
mkfifo "$FIFO"
trap 'rm -f "$FIFO"' EXIT

producer() { for i in {1..5}; do echo "item-$i"; done; } > "$FIFO" &
consumer() { while read -r item; do echo "Processing: $item"; done; } < "$FIFO"
wait

8. Mastering Here-Docs and Here-Strings for Inline Input

Here-docs (<<DELIMITER) let you define multi-line text directly in a script as stdin input for a command, without a temporary file. This is especially useful for SQL queries, configuration files and API payloads that need to be passed directly to commands. The important difference: with <<DELIMITER (delimiter unquoted), variables and backticks are expanded. With <<'DELIMITER' (delimiter in single quotes), the entire text stays literal, no variable expansion happens. The second form is the right choice for SQL queries and shell scripts that themselves contain $ variables.

Here-strings (<<< "value") are the single-line variant: they pass a string directly as stdin, without starting a subshell. read -r host port <<< "${address//:/ }" splits a "host:port" string into two variables via read, with no fork and no echo subshell. grep "pattern" <<< "$variable" searches inside a string without echo "$variable" | grep. Indenting here-doc contents with tabs (not spaces) is possible with <<-DELIMITER, which keeps the source code tidy without affecting the content.

9. Redirect Forms Compared

The sheer number of redirect forms in Bash is confusing at first glance. The table below sorts the most important constructs and explains when each one is the right choice.

The right choice often depends on context: for simple logging to a file, > is enough. For writing to the terminal and a file in parallel, exec > >(tee -a log) is the most elegant pattern. For while loops that set variables in the body, < <(command) is the correct choice over a pipe. Here-strings avoid unnecessary echo subshells and make the data flow explicit.

10. Summary

A complete understanding of pipes, redirects and file descriptors starts with the Unix FD model: every process has an FD table, and the shell modifies it before the child process's exec() call. 2>&1 duplicates FD 1 onto FD 2; the order determines the result. exec without a command changes the FDs of the current shell permanently. tee combined with process substitution enables fan-out to multiple processing paths. Here-docs and here-strings replace temporary files and echo subshells for inline input.

The practical consequence of this knowledge: scripts that consistently use stdout for data and stderr for diagnostics are composable and can be chained in pipes. Custom file descriptors for logging and locking are more robust than temporary files. Process substitution < <() solves the variable loss problem in pipe-based while loops. With these tools you can write shell scripts that behave like well-designed Unix tools: composable, observable and predictable.

11. FAQ: Pipes, Redirects and File Descriptors