Processing Logs in Bash: grep, awk, sed and jq Working Together

1. Logs in Bash: Choosing Tools and Strategy

Processing logs in Bash is one of the most common tasks in server administration. The tools grep, awk, sed and jq each cover a different layer: grep filters lines by pattern, awk splits and aggregates structured text lines, sed transforms and normalizes content, and jq processes modern JSON based log formats. Their real strength lies in combining them into pipelines: each tool handles its part of the transformation, and the output of one becomes the input of the next.

The strategy you choose for processing logs in Bash has a major impact on performance. The basic rule: filter early, aggregate late. Filtering the relevant lines with grep first and aggregating with awk afterward processes far less data than doing it the other way around. For gigabyte sized log files, the filtering order is decisive. On top of that, zcat and zgrep let you work directly with compressed logs without decompressing them first. For live logs, tail -f combined with grep --line-buffered is the right pattern for real time filtering.

2. grep: Filtering Strategies for Large Log Files

When processing logs with grep in Bash, there is more to it than a simple line pattern. grep -E (extended regex) allows complex patterns such as grep -E "ERROR|CRITICAL|FATAL" to filter for several severity levels at once. grep -v inverts the filter and excludes patterns, for example grep ERROR access.log | grep -v "healthcheck" removes health check noise from the error stream. grep -c only outputs the number of matches, which is useful for quick frequency checks without further processing.

For context analysis while processing logs, grep -A N -B N is indispensable: -A 5 prints 5 lines after the match (After), -B 3 prints 3 lines before the match (Before). This is essential for stack traces in application logs that span multiple lines. grep -n adds the line number, grep -h suppresses the filename when searching multiple files. The pattern grep -rl "ERROR" /var/log/ recursively finds every file containing "ERROR", useful for a first overview of which log sources are affected.

#!/usr/bin/env bash
# log-grep-strategies.sh: advanced grep filtering for Bash log processing
set -euo pipefail

LOG_DIR="/var/log/nginx"
TODAY=$(date +%Y-%m-%d)

# Strategy 1: Multi-pattern filter with context lines
echo "=== Recent errors with context ==="
grep -E "ERROR|CRITICAL|500" "${LOG_DIR}/error.log" \
  | grep "$TODAY" \
  | grep -v "healthcheck\|favicon" \
  | tail -20

# Strategy 2: Count errors per hour (grep extracts, awk aggregates)
echo "=== Errors per hour today ==="
grep "$TODAY" "${LOG_DIR}/error.log" \
  | grep -oP '\d{4}-\d{2}-\d{2}T\d{2}' \
  | sort | uniq -c | sort -rn

# Strategy 3: Compressed log search without decompression
echo "=== Errors in last 7 days (including .gz) ==="
zgrep -h "ERROR" "${LOG_DIR}"/error.log* 2>/dev/null \
  | grep -E "$(date -d '-7 days' +%Y-%m-%d)|$(date -d '-6 days' +%Y-%m-%d)" \
  | wc -l

# Strategy 4: Find all log files with recent errors
echo "=== Log files with errors in last 24h ==="
find /var/log -name "*.log" -mtime -1 -exec grep -l "ERROR\|CRITICAL" {} \;

# Strategy 5: Live stream with filtering (use in terminal, not cron)
# tail -f /var/log/app.log | grep --line-buffered -E "ERROR|WARN" | ts '[%Y-%m-%d %H:%M:%S]'

3. awk: Extracting, Aggregating and Calculating Fields

awk is the most powerful tool for structured text processing when processing logs in Bash. It automatically splits every line into fields (by default separated by whitespace, configurable with -F) and provides variables, control structures, associative arrays and mathematical operations. The classic pattern for Nginx access logs: awk '{print $7}' extracts the URL (field 7), awk '{sum += $10} END {print sum}' sums the response bytes. The Apache log format is largely consistent; Nginx log formats vary more and need to be adjusted accordingly with -F.

Associative arrays in awk enable on the fly aggregation while processing logs, without any temporary files. The pattern awk '{count[$7]++} END {for (url in count) print count[url], url}' | sort -rn | head -20 outputs the 20 most frequent URLs in a single pass over the log file. For large log files this matters a lot: instead of iterating through the same file multiple times, you extract every metric you need in one awk pass. awk's BEGIN and END blocks let you initialize state before the first record and summarize after the last one.

#!/usr/bin/env bash
# log-awk-analysis.sh: awk-based aggregation for Bash log processing
set -euo pipefail

ACCESS_LOG="/var/log/nginx/access.log"

# Extract all metrics in a single pass over the log file
awk '
BEGIN {
  total = 0; errors = 0; bytes_total = 0
}
{
  total++
  # Field layout: IP date method url protocol status bytes ...
  status = $9; bytes = $10
  url = $7; ip = $1

  # Count HTTP status codes
  status_count[status]++

  # Count errors (4xx and 5xx)
  if (status >= 400) { errors++ }

  # Sum response bytes (handle "-" for missing values)
  if (bytes ~ /^[0-9]+$/) { bytes_total += bytes }

  # Track top IPs
  ip_count[ip]++

  # Track slowest response times (field 11 if present)
  if (NF >= 11 && $11 ~ /^[0-9.]+$/) {
    total_time += $11; timed++
  }
}
END {
  printf "Total requests: %d\n", total
  printf "Error rate: %.2f%%\n", (errors/total)*100
  printf "Total bytes: %.2fMB\n", bytes_total/1024/1024

  print "\n--- Top Status Codes ---"
  for (s in status_count) print status_count[s], s | "sort -rn | head -5"

  print "\n--- Top 5 IPs ---"
  for (ip in ip_count) print ip_count[ip], ip | "sort -rn | head -5"
}' "$ACCESS_LOG"

4. sed: Transforming and Normalizing Logs

sed is the tool for line based text transformations when processing logs in Bash. While grep filters and awk aggregates, sed normalizes: it replaces patterns, strips irrelevant parts and brings inconsistent formats to a common denominator. The most common pattern: unifying timestamp formats across different log sources so that subsequent time window analyses work consistently. sed with the -n flag and the p command only prints lines that are explicitly output, an efficient filtering pattern for ranges between two markers.

The pattern sed -n '/STARTMARKER/,/ENDMARKER/p' extracts a range between two markers, useful for processing logs that mark transactions or deployments with start and end markers. Combined with timestamp filters, this allows precise extraction of a deployment window from a log that spans several days. sed is also the tool of choice for redacting passwords and API keys before forwarding logs to external systems: sed 's/password=[^&]*/password=REDACTED/g'.

5. jq: Processing Structured JSON Logs

Modern applications, container logs (Docker, Kubernetes) and structured logging frameworks like Loguru, Monolog or Winston write logs as JSON. Processing this JSON in Bash is far more efficient with jq than with regex based patterns. jq can access the semantic fields directly: jq 'select(.level == "ERROR")' filters exactly for error level entries, without matching lines that happen to contain the word "ERROR" somewhere in the message text. That is a fundamental quality difference when filtering structured logs.

For time window filtering in JSON logs, jq offers the select pattern with comparison operators: jq 'select(.timestamp >= "2026-05-09T10:00" and .timestamp < "2026-05-09T11:00")' extracts exactly one hour of logs. Aggregations like "errors per minute" are possible directly with the group_by filter once the input is sorted. For very large JSON log files, jq -c (compact output) in the pipeline is recommended to reduce memory overhead. The pattern jq -r '[.timestamp, .level, .message] | @tsv' converts JSON logs into TSV for further awk processing.

6. Time Window Filtering: Evaluating Logs by Time Range

Restricting log processing in Bash to a specific time window is one of the most common requirements: "What happened between 2pm and 3pm?" For Apache/Nginx logs in combined format, there is no built in time window option in the standard tools. The pattern relies on awk with timestamp parsing: the timestamp is extracted, compared against the start window using mktime or a string comparison, and lines outside the window are discarded. For ISO 8601 timestamps (YYYY-MM-DDTHH:MM:SS), plain string comparisons are valid because the format sorts lexicographically.

For rotated log files spread across multiple files, you need a clear strategy when processing logs in Bash: find /var/log -name "access.log*" -newer timestamp_file finds only files modified after a reference point in time. Combined with zcat for compressed files and sort -m to merge several sorted streams, this produces a complete time window analysis across rotated logs.

#!/usr/bin/env bash
# log-timewindow.sh: time window log filtering in Bash
set -euo pipefail

LOG_FILE="${1:-/var/log/nginx/access.log}"
START="${2:-$(date -d '-1 hour' '+%Y-%m-%dT%H:%M:%S')}"
END="${3:-$(date '+%Y-%m-%dT%H:%M:%S')}"

echo "[INFO] Filtering: $START to $END"

# Method 1: ISO-8601 string comparison (works for sorted timestamps)
filter_json_logs() {
  local file="$1"
  jq -c --arg s "$START" --arg e "$END" \
    'select(.timestamp >= $s and .timestamp <= $e)' "$file"
}

# Method 2: Apache/Nginx Combined Log Format time window
# [09/May/2026:14:32:01 +0200] -> convert to comparable format
filter_apache_logs() {
  local file="$1"
  local start_epoch end_epoch
  start_epoch=$(date -d "$START" +%s)
  end_epoch=$(date -d "$END" +%s)

  awk -v s="$start_epoch" -v e="$end_epoch" '
  {
    # Extract timestamp: [09/May/2026:14:32:01 +0200]
    match($0, /\[([0-9]+)\/([A-Za-z]+)\/([0-9]+):([0-9:]+)/, arr)
    if (RSTART > 0) {
      cmd = "date -d \"" arr[1] " " arr[2] " " arr[3] " " arr[4] "\" +%s"
      cmd | getline ts
      close(cmd)
      if (ts >= s && ts <= e) print
    }
  }' "$file"
}

# Method 3: grep-based quick filter for ISO dates in YYYY-MM-DD HH: format
filter_by_hour() {
  local file="$1" hour="$2"  # hour format: "2026-05-09 14:"
  grep -h "$hour" "$file" "$file".1 2>/dev/null || true
  zgrep -h "$hour" "$file"*.gz 2>/dev/null || true
}

echo "=== Last hour error count ==="
filter_by_hour "$LOG_FILE" "$(date -d '-1 hour' +'%Y-%m-%d %H:')" \
  | grep -c "HTTP/[0-9.]\" [45]" || echo "0"

7. Error Extraction: Automatically Finding ERROR, WARN and Stack Traces

Automated error extraction from logs is a core part of log processing in Bash for monitoring systems. The basic pattern has several stages: first grep -E "ERROR|CRITICAL|FATAL|Exception|Traceback" for the initial filtering, then context lines with -A for multi line stack traces, followed by deduplication with sort -u or hash based grouping of similar error patterns. The result is a list of distinct error classes with frequency counts, far more informative than a raw error list.

For Java stack traces, Python tracebacks and PHP fatal errors that span multiple lines, a multi line reading pattern in awk works well: when an error line is encountered, the script starts collecting lines into a buffer, and when a new log line appears (recognizable by the timestamp pattern), the buffer is printed and reset. This Bash log processing pattern extracts complete stack traces as individual blocks that can then be processed further.

8. Automated Reporting from Logs with Bash

Automated log reporting in Bash combines all the extraction tools into a complete analysis script that runs daily or hourly from a cron job and produces a structured report. The report script follows a clear structure: a header with the date and the files analyzed, a summary of the totals, top errors by frequency, top clients by request volume, status code distribution and the slowest requests. This report can be emailed, written to a file, or output as JSON for a monitoring dashboard.

The Bash log reporting pattern uses process substitution for parallel processing: while awk counts errors, a background process can calculate the status code distribution at the same time. With tee, the log file can be forwarded to several analysis processes simultaneously. The end result is a report script that typically stays under 100 lines yet still produces complete metrics from gigabyte sized log files in under a minute.

#!/usr/bin/env bash
# log-report.sh: automated daily log report generation
set -euo pipefail

LOG_FILE="${1:-/var/log/nginx/access.log}"
REPORT_DATE="${2:-$(date +%Y-%m-%d)}"
REPORT_FILE="/tmp/log-report-${REPORT_DATE}.txt"

{
echo "=== Log Report: $REPORT_DATE ==="
echo "Generated: $(date '+%Y-%m-%d %H:%M:%S')"
echo "Source: $LOG_FILE"
echo ""

# Filter today's lines first (reduces subsequent processing)
TODAY_LINES=$(grep "$REPORT_DATE" "$LOG_FILE" 2>/dev/null || true)

echo "--- Summary ---"
TOTAL=$(echo "$TODAY_LINES" | wc -l)
ERRORS=$(echo "$TODAY_LINES" | grep -cE '" [45][0-9]{2} ' || echo "0")
echo "Total requests : $TOTAL"
echo "Error requests : $ERRORS"
printf "Error rate     : %.2f%%\n" "$(echo "scale=4; $ERRORS/$TOTAL*100" | bc)"

echo ""
echo "--- Status Code Distribution ---"
echo "$TODAY_LINES" \
  | awk '{print $9}' \
  | grep -E '^[0-9]{3}$' \
  | sort | uniq -c | sort -rn \
  | awk '{printf "  HTTP %-4s : %d\n", $2, $1}'

echo ""
echo "--- Top 10 Error URLs ---"
echo "$TODAY_LINES" \
  | awk '$9 >= 400 {print $7}' \
  | sort | uniq -c | sort -rn | head -10 \
  | awk '{printf "  %5d  %s\n", $1, $2}'

echo ""
echo "--- Top 5 Client IPs ---"
echo "$TODAY_LINES" \
  | awk '{print $1}' \
  | sort | uniq -c | sort -rn | head -5

} > "$REPORT_FILE"

cat "$REPORT_FILE"
echo "[INFO] Report saved to $REPORT_FILE"

9. Tools in Direct Comparison

When processing logs in Bash, there is often more than one way to reach the goal. Knowing the strengths and weaknesses of each tool lets you pick the right one for every task in the log analysis pipeline.

For maximum efficiency when processing large logs in Bash, follow this pipeline hierarchy: zcat first for compressed sources, then grep for early filtering, then awk for single pass aggregation, then sed for output normalization. jq comes first for JSON sources and can fully replace grep and awk for structured logs. The goal: a single scan over the log file that produces every metric you need.

10. Summary

Processing logs in Bash with grep, awk, sed and jq working together is a highly efficient alternative to heavyweight log management systems for everyday server administration. The core strategy: filter early with grep or jq select, aggregate in a single pass with awk, normalize with sed. Use zgrep and zcat directly for compressed logs. Filter JSON logs semantically with jq instead of regex. Use ISO 8601 string comparison or epoch based awk comparisons for time window filtering.

Automated daily reporting from logs as a cron job is one of the most effective steps toward proactive server monitoring with almost no ongoing effort. A single report script that runs daily and outputs error rates, top error URLs, status code distributions and anomalies saves weekly manual log review and surfaces trends before they turn into incidents. The whole tool set is available on every Linux server with no additional installation.

11. FAQ: Processing Logs in Bash: grep, awk, sed and jq