System Diagnostics with ps, top, ss, lsof, journalctl and Bash Helpers

1. Why structured system diagnostics matters

Good system diagnostics does not start with guessing, it starts with systematic data collection. Every Linux production system provides a complete information infrastructure: the proc filesystem, the systemd journal, network stack statistics and file descriptor tables. The challenge lies in pulling this information quickly and precisely, before the pressure of an ongoing incident starts to erode your focus.

Prebuilt Bash helpers for system diagnostics solve exactly this problem: instead of reconstructing the right flags for ps, ss and lsof under stress, you run a prepared script that delivers a complete diagnostic picture in 30 seconds. These helpers should live in /usr/local/bin on every production server and be runnable at any time without root privileges. The following sections show how to put the most important tools to effective use.

System diagnostics breaks down into four domains: processes (what is running, how many resources it consumes), network (which connections exist, which ports are listening), memory (how much is used for what, are there leaks), and kernel or journal (what has the system logged, were there OOM kills or hardware errors). Each domain has its own tools, which can be combined in Bash.

2. ps: process analysis beyond ps aux

ps aux is the best known command for process level system diagnostics, but it is far from its full potential. With ps -eo pid,ppid,user,pcpu,pmem,vsz,rss,stat,start,time,cmd --sort=-pcpu you get a process list sorted by CPU usage, complete with parent PID, memory usage in kilobytes and the exact command path. The -o format is the key to targeted system diagnostics: it lets you select exactly the columns relevant to the question at hand.

For diagnosing process trees as part of system diagnostics, ps --forest is indispensable: it shows the parent child relationships of all processes as an ASCII tree and immediately reveals which processes are hanging as zombies (Z status) or spawning unexpected child processes. pstree -p is an alternative with a more compact display. For analyzing a single process, cat /proc/PID/status delivers detailed information such as memory limits, namespaces and capabilities that ps does not show.

#!/usr/bin/env bash
# proc-diag.sh - Process analysis helper for system diagnosis
set -euo pipefail

TOP_N="${1:-10}"

echo "=== Top $TOP_N CPU consumers ==="
ps -eo pid,user,pcpu,pmem,rss,vsz,cmd --sort=-pcpu | head -n $(( TOP_N + 1 ))

echo ""
echo "=== Top $TOP_N Memory consumers (by RSS) ==="
ps -eo pid,user,pcpu,pmem,rss,cmd --sort=-rss | head -n $(( TOP_N + 1 ))

echo ""
echo "=== Zombie processes ==="
zombies=$(ps -eo stat,pid,ppid,cmd | awk '$1 ~ /^Z/ {print}')
if [[ -n "$zombies" ]]; then
  echo "$zombies"
else
  echo "No zombie processes found."
fi

echo ""
echo "=== Thread counts per process (top 5) ==="
# /proc/<pid>/status contains Threads: field
for pid_dir in /proc/[0-9]*/; do
  pid="${pid_dir//[^0-9]/}"
  [[ -f "$pid_dir/status" ]] || continue
  threads=$(grep -m1 '^Threads:' "$pid_dir/status" 2>/dev/null | awk '{print $2}')
  name=$(grep -m1 '^Name:' "$pid_dir/status" 2>/dev/null | awk '{print $2}')
  echo "$threads $pid $name"
done 2>/dev/null | sort -rn | head -5

3. top and htop: watching resources in real time

top is essential for interactive system diagnostics, but its non-interactive use in Bash scripts is often overlooked. With top -b -n 1 (batch mode, single pass), top outputs a complete snapshot of the current resource distribution, ideal for diagnostic scripts that log system state to a file. top -b -n 3 -d 2 takes three measurements two seconds apart and averages them, which is more meaningful than a single snapshot.

For automated system diagnostics in Bash, /proc/loadavg is more direct than top: the file contains the 1, 5 and 15 minute load averages, the number of running processes and the last PID. Reading this file is far more efficient than parsing top output. The same applies to CPU statistics: /proc/stat contains raw values for all CPU states (user, nice, system, idle, iowait, irq, softirq, steal), from which two consecutive measurements let you calculate exact CPU utilization without any external tools.

4. ss: network diagnostics instead of legacy netstat

ss (socket statistics) is the successor to netstat and is considerably faster and more informative for network level system diagnostics. While netstat is based on /proc/net/tcp and related files, ss reads directly from the kernel via netlink sockets, without the limitations and parsing errors of text files. On systems with thousands of open connections the difference is measurable: netstat -an can take several seconds, ss -an returns its result in milliseconds.

For connection level system diagnostics, the ss filters are powerful: ss -tp state established '( dport = :443 or dport = :80 )' shows all established HTTP/HTTPS connections together with the owning PID and process name. ss -lnp --filter "sport = :8080" checks which process is listening on port 8080. The -e flag shows extended socket information such as timers, UID and inode number. ss -s delivers a compact summary of all socket types.

#!/usr/bin/env bash
# net-diag.sh - Network diagnosis helper using ss and /proc
set -euo pipefail

echo "=== Listening ports with process names ==="
ss -lntp | awk 'NR==1 || $1=="LISTEN"'

echo ""
echo "=== Established connections summary ==="
ss -tn state established | awk 'NR>1 {print $4}' \
  | sed 's/:[0-9]*$//' \
  | sort | uniq -c | sort -rn | head -10

echo ""
echo "=== Connection state counts ==="
ss -tan | awk 'NR>1 {states[$1]++} END {for (s in states) printf "%6d %s\n", states[s], s}' \
  | sort -rn

echo ""
echo "=== TIME_WAIT connections (potential exhaustion risk) ==="
tw_count=$(ss -tan state time-wait | wc -l)
echo "TIME_WAIT sockets: $tw_count"
if (( tw_count > 1000 )); then
  echo "[WARN] High TIME_WAIT count, check net.ipv4.tcp_tw_reuse" >&2
fi

echo ""
echo "=== UDP socket usage ==="
ss -uanp | head -20

5. lsof: open files, sockets and file descriptors

lsof (list open files) is the most comprehensive tool for resource level system diagnostics. On Linux almost everything is a file: regular files, sockets, pipes, devices and pseudo files under /proc. lsof can enumerate all of these and filter by process, user, filename or protocol. lsof -p PID shows all open resources of a process and is the first place to look when a process behaves unexpectedly, fails to close files, or leaks memory.

For system diagnostics around disk space problems, lsof +L1 is an indispensable tool: it shows all files that have been deleted but are still held open by a process. This is the classic case where df shows no free space while du / reports considerably less usage than expected: the gap between the two is exactly the space taken up by deleted but still referenced files. lsof -i :PORT shows which process occupies a specific port, without requiring root privileges.

6. journalctl: evaluating kernel logs and systemd units

journalctl is the central logging tool for system diagnostics on systemd systems. It combines kernel logs (dmesg), systemd unit logs and application logs into a single structured journal. journalctl -k shows only kernel messages, equivalent to dmesg, but with correct timestamps relative to system boot. For OOM diagnostics, journalctl -k | grep -i 'oom\|kill' is the first command to run: if the kernel has terminated processes due to memory pressure, it will be recorded here.

Time based filtering in journalctl is powerful for structured system diagnostics: journalctl --since "2026-05-09 10:00" --until "2026-05-09 11:00" -p err shows all errors within a time window. -p err filters on priority error and above (err, crit, alert, emerg). The --output json format produces structured JSON output that can be further processed with jq. For system diagnostics after a reboot, journalctl -b -1 shows the journal of the previous boot.

#!/usr/bin/env bash
# system-health.sh - Combined system diagnosis snapshot
set -euo pipefail

SINCE="${1:-1 hour ago}"
OUTPUT_DIR="${2:-/tmp/diag-$(date +%Y%m%d-%H%M%S)}"

mkdir -p "$OUTPUT_DIR"

echo "[INFO] Running system diagnosis snapshot to $OUTPUT_DIR"

# Process snapshot
ps -eo pid,ppid,user,pcpu,pmem,rss,stat,cmd --sort=-rss > "$OUTPUT_DIR/processes.txt"

# Network connections
ss -tanp > "$OUTPUT_DIR/sockets.txt"
ss -s    > "$OUTPUT_DIR/socket-stats.txt"

# Kernel messages since given time
journalctl -k --since "$SINCE" > "$OUTPUT_DIR/kernel-log.txt"

# Errors from all units
journalctl --since "$SINCE" -p err --no-pager > "$OUTPUT_DIR/errors.txt"

# OOM and kill events
journalctl -k --since "$SINCE" | grep -iE 'oom|killed|segfault' \
  > "$OUTPUT_DIR/oom-events.txt" || true

# Memory breakdown
cat /proc/meminfo > "$OUTPUT_DIR/meminfo.txt"

# Disk usage of open but deleted files
lsof +L1 2>/dev/null | awk 'NR==1 || $7 < 1' > "$OUTPUT_DIR/deleted-open-files.txt" || true

echo "[INFO] Diagnosis complete. Archive with:"
echo "  tar czf diag-$(date +%Y%m%d).tar.gz -C $(dirname $OUTPUT_DIR) $(basename $OUTPUT_DIR)"

7. Memory checks: /proc/meminfo, free and smaps

System diagnostics for memory problems starts with free -h, which gives a rough overview. For a detailed analysis, /proc/meminfo provides the raw values: MemAvailable is the relevant figure for the memory actually available, since it accounts for caches and buffers that the kernel can release on demand. MemFree alone is misleading: a system with low MemFree but high MemAvailable is not under memory pressure.

For system diagnostics of a single process suspected of a memory leak, /proc/PID/smaps is the most detailed source. smaps_rollup aggregates all memory map entries and shows Private_Dirty, the actual private memory consumed that is not shared with other processes. If this value grows continuously, it is a strong indicator of a memory leak. The smem tool or a custom Bash helper can record these values over time and detect trends.

8. Bash helpers: diagnostic scripts for the real emergency

The value of Bash helpers for system diagnostics lies in their preparation: in a real emergency you should not need to look up flags, you should run a prepared script instead. The most important of these scripts is the "runbook script": a Bash script that collects all relevant diagnostic information in a defined order and stores it in a compressed file. This lets you capture a complete system snapshot in 30 seconds before a restart or a change overwrites the state.

A second important Bash helper for system diagnostics is the trend script: it runs in the background and writes CPU, memory, connection count and disk I/O to a CSV file every minute. When an incident occurs, you have retrospective data from the last few hours and can pinpoint the exact moment things started degrading. Combined with journalctl filters for the same period, this produces a complete diagnostic picture without any external monitoring infrastructure.

9. System diagnostics tools compared

The four domains of system diagnostics require different tools. The following table shows which tool is the first choice for which question.

System diagnostics becomes more efficient once you wrap the tools in Bash functions callable by descriptive names. An alias like alias who-uses-port='ss -lntp | grep' or a function proc-mem() { cat /proc/"$1"/smaps_rollup; } reduces the cognitive load during a real incident. Such libraries of aliases and functions belong in ~/.bashrc or a dedicated ~/.bash_diag file present on every production server.

10. Summary

Structured system diagnostics with ps, top, ss, lsof and journalctl delivers a complete picture of system state within minutes. ps -eo with custom formats is more flexible than ps aux. ss is faster and more precise than netstat and offers powerful filters for connection analysis. lsof +L1 finds deleted files that still occupy disk space. journalctl -k shows kernel messages with correct timestamps. /proc/PID/smaps_rollup provides the actual private memory usage needed for memory leak diagnostics.

The decisive multiplier for system diagnostics is preparation: Bash helpers ready to run in a real emergency drastically cut reaction times. A runbook script that gathers all relevant data in 30 seconds is more valuable than any monitoring dashboard when a server is barely reachable under load. These helpers should be versioned, available on every server and tested regularly.

11. FAQ: System Diagnostics with ps, top, ss, lsof, journalctl and Bash Helpers