Streaming Logs and Files in Real Time

1. Why Streaming Logs in Real Time Matters

Streaming logs in real time is the difference between reactive and proactive system monitoring. When a web server suddenly starts producing hundreds of 500 errors per minute, reaction time determines whether an incident lasts minutes or hours. Classic monitoring tools with pull intervals of 30 seconds or longer fall short here. The solution lies directly in Bash, using tools available on every Linux system.

The core mechanism behind streaming logs is remarkably simple: a process continuously reads new lines from a file or a file descriptor and pipes them through a processing pipeline. What gets complex are the edge cases: log rotation that changes the file's inode, multiple parallel log sources that need to be aggregated, buffering in pipes that introduces latency, and wasted resources from busy waiting. The following sections cover all of these aspects systematically.

Bash is not just sufficient for streaming logs in real time, it is often the most pragmatic choice: no extra dependencies, no running services, direct access to system resources. Anyone who masters these tools can build an alerting system in just a few lines that works equally well in a CI pipeline or on a dedicated monitoring host.

2. tail -f and tail --follow: Basics and Pitfalls

tail -f is the best known tool for streaming logs. It opens a file and prints new lines as soon as they are written. The difference between -f (follow by file descriptor) and --follow=name (follow by filename) is critical during log rotation: -f keeps following the original file descriptor even if the file is renamed. After a rotation, tail -f then points at the old, emptied file and stops receiving new entries. --follow=name re-resolves the filename on every check and follows the new file after a rotation.

Another pitfall when streaming logs with tail: the -F option (uppercase) is an alias for --follow=name --retry and automatically tries to reopen the file if it no longer exists or a permission error occurs. In production environments with regular log rotation, tail -F is the more robust choice. Combined with --pid=PID, tail automatically exits once the monitored process dies, which prevents orphaned tail processes in monitoring scripts.

#!/usr/bin/env bash
# log-stream.sh: real-time log streaming with automatic rotation handling
set -euo pipefail

LOG_FILE="${1:-/var/log/nginx/error.log}"
ALERT_PATTERN="${2:-ERROR|CRITICAL|emerg}"
ALERT_CMD="${3:-/usr/local/bin/send-alert.sh}"

# -F: follow by name + retry, survives log rotation
# --line-buffered: flush each line immediately, no blocking buffering
tail -F --line-buffered "$LOG_FILE" | \
  grep --line-buffered -E "$ALERT_PATTERN" | \
  while IFS= read -r line; do
    ts="$(date '+%Y-%m-%dT%H:%M:%S')"
    echo "[$ts] ALERT: $line" >&2
    # Fire alert command asynchronously, do not block the stream
    "$ALERT_CMD" "$line" &
  done

# Cleanup dangling background alert processes on exit
trap 'wait' EXIT

The --line-buffered flag is indispensable when streaming logs through pipes. Without it, grep buffers its output in 4 KB blocks, which causes latency spikes in real-time alerts. The same applies to awk and sed in pipelines: always use --line-buffered with grep and stdbuf -oL with other tools to force line buffering. Real-time behavior in log streaming is only as good as the weakest buffering setting in the chain.

3. while read: Safely Processing Lines from Streams

The pattern while IFS= read -r line; do ... done is the fundamental Bash construct for streaming logs line by line. IFS= (empty IFS) prevents leading and trailing whitespace from being trimmed. -r prevents backslash sequences from being interpreted. Without these two options, indentation, IP addresses with certain patterns, or paths with backslashes come out corrupted after processing.

A common problem when streaming logs with while read is losing the exit code from a pipe: in cmd | while read, the while loop runs in a subshell. Variables set inside the loop become invisible after the pipe. The Bash pattern for this: use process substitution instead of a pipe: while IFS= read -r line; do ...; done < <(cmd). The loop then runs in the current shell context, variables remain visible after the loop, and the exit code of cmd can be evaluated with ${PIPESTATUS[0]}.

#!/usr/bin/env bash
# stream-counter.sh: count error classes from a log stream in real-time
set -euo pipefail

declare -A error_counts
declare -i total=0
declare -i last_report=0
REPORT_INTERVAL=60  # seconds

report_stats() {
  echo "--- Stats at $(date '+%H:%M:%S') ---"
  for key in "${!error_counts[@]}"; do
    printf "  %-20s %d\n" "$key" "${error_counts[$key]}"
  done
  echo "  Total: $total"
}

trap report_stats EXIT

# Process substitution keeps the while loop in the current shell context
# variables are accessible after the loop ends
while IFS= read -r line; do
  total+=1

  # Extract severity level from structured log (e.g. "[ERROR]", "[WARN]")
  if [[ "$line" =~ \[([A-Z]+)\] ]]; then
    level="${BASH_REMATCH[1]}"
    error_counts["$level"]=$(( ${error_counts["$level"]:-0} + 1 ))
  fi

  # Periodic reporting without blocking the stream
  now=$(date +%s)
  if (( now - last_report >= REPORT_INTERVAL )); then
    report_stats
    last_report=$now
  fi
done < <(tail -F --line-buffered /var/log/app/application.log)

4. inotifywait: Filesystem Events as a Stream Source

inotifywait from the inotify-tools package offers an alternative to tail -f for streaming logs: instead of reading file content, it monitors kernel events at the file and directory level. This makes it possible to react to new files being created in a directory, to alert on permission changes, or to kick off a full processing workflow as soon as a file has been fully written. Polling is eliminated entirely: inotifywait blocks until a kernel event arrives.

The --monitor mode of inotifywait is the crucial one for lasting log streaming: without --monitor, inotifywait exits after the first event. With --monitor, it runs continuously and prints each event as one line. The format --format '%w%f %e %T' delivers path, event type, and timestamp in a structured form that can be fed directly into while read. For streaming logs in log directories, the CLOSE_WRITE event type is recommended, since it signals that a file has been fully written, rather than MODIFY, which fires on every write operation.

#!/usr/bin/env bash
# dir-watcher.sh: process new log files as soon as they are fully written
set -euo pipefail

WATCH_DIR="${1:-/var/log/uploads}"
PROCESS_CMD="${2:-/usr/local/bin/process-log.sh}"

if ! command -v inotifywait &>/dev/null; then
  echo "[ERROR] inotify-tools not installed: apt install inotify-tools" >&2
  exit 1
fi

echo "[INFO] Watching $WATCH_DIR for new complete files..."

# --monitor: run continuously (not one-shot)
# CLOSE_WRITE: file was written and closed, safe to process
# --format: structured output for reliable parsing
while IFS=' ' read -r filepath event _ts; do
  [[ -f "$filepath" ]] || continue
  echo "[INFO] Processing $filepath (event: $event)"
  # Run processing in background, do not block the event loop
  "$PROCESS_CMD" "$filepath" &
done < <(
  inotifywait --monitor --quiet \
    --event CLOSE_WRITE \
    --format '%w%f %e %T' \
    --timefmt '%Y-%m-%dT%H:%M:%S' \
    "$WATCH_DIR"
)

wait  # drain background jobs on exit

5. Real-Time Alerting: Detecting Patterns and Reacting Instantly

Real-time alerting for streaming logs means more than simple matching with grep. Production-relevant alerts require thresholds (more than N errors in M seconds), deduplication (not alerting on every single error when hundreds occur at once), and cooldown periods (no second alert while the first one is still active). Bash is fully capable of implementing this logic with associative arrays and arithmetic, without any external dependencies.

The core pattern for threshold alerting in log streaming: a sliding counter tallies events within a time window. Once the counter exceeds a threshold, it triggers the alert and starts a cooldown timer. New events keep incrementing the counter but no longer trigger a second alert until the cooldown has expired. The counter resets once the time window elapses without a new event. This logic can be implemented in about 20 lines of Bash and covers the most common alerting requirements in production systems with no overhead.

6. Log Aggregation: Merging Multiple Sources

When streaming logs from multiple sources at once, aggregation is the central challenge. The simplest pattern: merge several tail -F processes into a shared pipe. tail -F /var/log/app1.log /var/log/app2.log follows several files simultaneously and prefixes each line with the filename. That works for a handful of files but does not scale to dozens of sources, because tail keeps a file descriptor open for every file and limits can be hit.

For larger-scale log aggregation in log streaming, the named pipe pattern is the cleaner solution: each log source writes through its own subshell into a shared named pipe (FIFO). A single reader process reads from the FIFO and processes the aggregated stream. Named pipes solve the problem of multiple writer processes writing to a regular file and overwriting each other: write operations to a FIFO are atomic up to the buffer size (PIPE_BUF, typically 4096 bytes), which is sufficient for individual log lines.

#!/usr/bin/env bash
# aggregate-logs.sh: merge multiple log streams into one annotated stream
set -euo pipefail

FIFO="/tmp/log-aggregator-$$"
declare -a TAIL_PIDS=()

cleanup() {
  for pid in "${TAIL_PIDS[@]:-}"; do
    kill "$pid" 2>/dev/null || true
  done
  rm -f "$FIFO"
}
trap cleanup EXIT

mkfifo "$FIFO"

# Start one tail per source, each prefixing its lines with the source name
for logfile in /var/log/nginx/error.log /var/log/app/application.log /var/log/mysql/error.log; do
  source_name="$(basename "${logfile%.log}")"
  # Each tail writes to the shared FIFO
  tail -F --line-buffered "$logfile" \
    | sed --unbuffered "s/^/[$source_name] /" \
    > "$FIFO" &
  TAIL_PIDS+=($!)
done

# Single reader processes the merged stream
while IFS= read -r line; do
  ts="$(date '+%Y-%m-%dT%H:%M:%S')"
  echo "$ts $line"
  # Route to alert if CRITICAL found
  if [[ "$line" =~ CRITICAL|emerg|panic ]]; then
    logger -t log-aggregator -p user.crit "$line"
  fi
done < "$FIFO"

7. Log Rotation and Ensuring Stream Continuity

Log rotation is the most common reason log streaming with tail -f (without -F) goes silent after a while. Logrotate renames the current file and creates a new one; the file descriptor held by tail -f keeps pointing at the old, renamed file and receives no new entries. tail -F detects this: it periodically checks whether the filename now points to a different inode, and if so, reopens the new file. The check frequency defaults to 1 second and can be adjusted with --sleep-interval.

For log streaming with inotifywait, log rotation is more transparent because monitoring happens at the inode level. On the MOVED_TO event in the directory, the operating system signals that a new file has been created, and the watch process opens it immediately. A robust strategy for continuous streaming in production is periodically restarting the tail process with a position marker: the last processed byte offset is stored in a file, and on restart, tail -c +OFFSET reads the new file starting from that position. This guarantees no lost lines even across rotation boundaries.

8. Performance and Backpressure in Long Pipelines

When streaming logs at high throughput, buffering and backpressure are critical concerns. If processing in the while read loop is slower than the log is being written, the pipe buffer fills up. Linux pipes have a default buffer size of 64 KB. Once the buffer is full, the writer process, in this case tail, blocks, and the whole system gets backed up. For high-frequency log streams, processing must be fast enough, or the stream must be throttled deliberately.

The pattern for fast, backup-free log streaming: offload heavy processing steps asynchronously. The while read loop writes each line into a fast queue (another FIFO or a simple ring buffer script), and a separate worker process reads from the queue and performs the actual processing. This way, the read loop always stays fast while processing runs at its own pace. Another trick: apply grep --line-buffered early in the pipeline to filter out irrelevant lines before running expensive parsing operations.

9. Streaming Approaches Compared

Different tools suit streaming logs better depending on the scenario. The choice depends on log rotation behavior, the number of sources, real-time requirements, and the packages available.

The combination of tail -F for individual files, named FIFOs for aggregation, and inotifywait for directory monitoring covers nearly every requirement in log streaming. Important: on systems with very high log throughput (over 10 MB/s), pure Bash solutions reach their limits, and specialized tools like Vector, Fluent Bit, or Promtail make more sense there. For most production systems with moderate log volume, however, Bash is fully sufficient and has the advantage of introducing no additional runtime dependencies.

10. Summary

Streaming logs in real time with Bash is a powerful tool that requires no external dependencies. tail -F should always be preferred over tail -f for production use, since it survives log rotation. while IFS= read -r line; do ... done < <(cmd) keeps variables in the current shell context and enables stateful processing across the stream. inotifywait --monitor is the right choice for event-driven processing of new files without polling. Named FIFOs solve aggregation from multiple log sources. Buffering in pipes must be controlled with --line-buffered and stdbuf -oL.

The most critical property of log streaming is robustness against edge cases: log rotation, backpressure under heavy load, orphaned processes on failure. With trap cleanup EXIT, careful PID management, and an explicit wait at the end of the script, these edge cases can be handled systematically. The result is an alerting system that keeps working correctly even after days of continuous operation.

11. FAQ: Streaming Logs and Files in Real Time