A Systematic Troubleshooting Workflow for Linux Servers

1. Why Aimless Troubleshooting Fails

In a real incident, even experienced administrators tend to try random measures under time pressure: restarting the service, rebooting the whole server, reverting a suspected configuration change, ideally all at once. Each of these actions changes the system state before the actual cause is even known. If the error disappears afterward, it remains unclear which measure actually worked, and the problem can silently return at any time. This mode, often called shotgun debugging, tends to significantly extend incidents, because every undocumented change introduces new variables that make the next diagnosis even harder.

A systematic troubleshooting workflow solves this problem by enforcing a fixed order: first verify that an outage actually exists, then check what changed most recently, then work through resources in a fixed sequence from load through memory and disk to network, and only after that search the logs in a targeted way, starting with the most recent entry. This order is not arbitrary: each step rules out or confirms an entire class of errors before the next one begins. The result is a reproducible process that every team member on call can apply the same way, regardless of their individual experience with the affected system.

2. Step 1: Is the Service Really Down?

Before anything is changed, verification comes first: is there really an outage, or is this a false alarm from a broken monitoring check, a misconfigured firewall rule that only blocks the monitor, or a single faulty data point? A health check directly on the server with curl against localhost, combined with a check from outside the network, reliably uncovers exactly these cases. Anyone who skips this step and dives straight into the depths regularly wastes time diagnosing a problem that does not even exist.

It is also important to distinguish between three completely different states: the service has crashed entirely, the service is running but responding slowly or with errors, or it is a pure false alarm. A crashed process usually leaves a clear exit code and a stack trace in the journal, while a slow but running service almost always points to resource scarcity, which only the following steps on load, memory, disk, and network will reveal. This early classification determines how much time the remaining steps will take.

#!/usr/bin/env bash
# Step 1: verify the outage is real before touching anything
systemctl status nginx --no-pager
# Active: active (running) since Fri 2026-07-12 08:14:02 UTC; 3h ago -> unit is up

curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" http://localhost/health
# 200 0.041s -> local health check succeeds

# Check from an external vantage point too, monitoring can be blind locally
curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" https://shop.example.com/health

# Confirm the process is actually running, not just the unit file loaded
pgrep -a nginx || echo "No nginx process found"

3. Step 2: What Changed?

Most incidents are not triggered by spontaneous hardware wear, but by a specific change: a deployment, a package update, a cron job, a configuration management run, or an expired certificate. Comparing the time of the outage with the time of the last change is therefore often the single most valuable diagnostic step in the entire workflow, even before a single log line has been read in detail. Anyone who performs this step before the resource analysis often finds the cause within minutes.

In practice, it is worth looking at several sources at once: journalctl shows restarts and errors of affected services in the relevant time window, /var/log/apt/history.log or /var/log/dpkg.log document package updates, crontab -l and /etc/cron.d show scheduled jobs, and git log in the deployment repository provides the exact time of the last rollout. When using Ansible or Puppet, the run history of these tools also belongs on the checklist. If the start of the outage coincides with one of these events, the cause has very likely already been found.

#!/usr/bin/env bash
# Step 2: correlate incident start time with recent changes
uptime -s                                   # last reboot time
journalctl --since "2 hours ago" -u nginx.service --no-pager | tail -30

# Package changes in the relevant time window
grep "2026-07-12" /var/log/apt/history.log
zgrep "2026-07-12" /var/log/dpkg.log* 2>/dev/null

# Cron jobs that ran around the incident
grep CRON /var/log/syslog | grep "12:1[0-5]"

# Deployment history in the application repository
cd /var/www/shop && git log --oneline --since="3 hours ago"

# Configuration management run history (if Ansible/Puppet is used)
ls -la /var/log/ansible/ | tail -5

4. Step 3: Check Load and CPU

Only after verification and change comparison does the actual resource analysis begin, and it deliberately starts with load and CPU, because this metric is available with a single uptime command and immediately gives a rough classification of the incident: compute bound or waiting. A load average well above the number of available cores, combined with high CPU usage in top, points to a compute intensive process, while a high load with low CPU usage almost always means I/O wait time, which already steers the next suspicion toward disk or network.

top or htop, sorted by the CPU column, shows within seconds whether a single process has gotten out of control, for example after a faulty deployment with an infinite loop. mpstat -P ALL 1 additionally reveals whether the load is evenly distributed across all cores or whether a single threaded process is fully utilizing one core while other cores sit idle, something the averaged load value hides. This detail often decides whether restarting the process is enough or whether a structural fix is needed.

; /etc/troubleshooting/thresholds.ini
; Load and CPU thresholds used during triage, step 3 of the workflow
[load]
warning_per_core = 1.0
critical_per_core = 2.0
sustain_minutes = 10

[cpu]
warning_iowait_percent = 20
warning_steal_percent = 5

; Rule of thumb: escalate on the sustained 15-minute value, not on a 1-minute spike

5. Step 4: Check Memory and Swap

Memory is checked right after load and CPU, because memory pressure often disguises itself as a disk or performance problem: a system that is actively swapping feels slow, even though the actual cause lies in RAM. free -h gives a quick overview of used, free, and buffer/cache memory, while vmstat 1 shows in the si and so columns whether the system is actively writing to or reading from swap right now. Continuous swapping under load is a reliable sign of genuine memory shortage, not a normal operating state.

A second, often overlooked check is looking for the OOM killer: dmesg -T | grep -i "killed process" shows whether the kernel forcibly terminated a process due to memory shortage, frequently without the process itself leaving a meaningful error message. In Magento and PHP environments, typical causes are a PHP-FPM pool size configured too large relative to available RAM, an OPcache growing without a limit, or a memory leak in a background process that only becomes visible after hours of operation.

6. Step 5: Check Disk and Filesystem

After memory comes disk, because full filesystems or exhausted inodes produce symptoms that look like application errors at first glance: failed writes, aborted log rotations, rejected database commits. df -h shows the used space per partition, but that is not enough: df -i shows the number of free inodes, and a filesystem with plenty of free space can still be completely unusable if millions of small files, for example sessions or cache entries, have used up all available inodes.

If there is enough space and enough inodes but the symptoms persist, it is worth looking at the I/O level with iostat -xz 1, which shows wait times and utilization per block device. Another important check after disk errors in the kernel log is an unintended read only remount of the filesystem, visible in dmesg after a hardware error. Deleted but still open files that continue to occupy disk space can be tracked down with lsof +L1, a common but rarely suspected reason for inexplicably full disk space.

7. Step 6: Check the Network

Network problems come last in the workflow before the logs, because they usually only become relevant once load, memory, and disk have already been ruled out. ss -tulnp shows whether the affected service is actually listening on the expected port, a surprisingly common error after a failed restart where the process is running but bound to the wrong port or interface. ss -tan state established additionally shows whether an unusually high number of open connections is exhausting the available file descriptors.

For connection problems to dependent systems such as the database, Redis, or external APIs, simple basic checks help: DNS resolution with dig or getent hosts, a direct connection test with nc -zv host port, and the current firewall rules with nft list ruleset or iptables -L -n. Packet loss or increased latency along the path is shown far more reliably by mtr than by a simple ping, because it displays the route hop by hop with loss rates and can pinpoint a single broken network segment.

8. Step 7: Read the Logs Correctly, Newest First

Logs deliberately come at the end of the workflow, not the beginning, because they are the most time consuming and least structured source of information. Anyone who starts directly with the logs often searches huge volumes of data without a hypothesis, while after the previous steps there is already a narrow suspicion that focuses the search. The reading direction also matters: journalctl -r or tail -f show the newest entries first, because the most recent entry is most likely related to the symptom currently being observed, instead of scrolling from the top through megabytes of old, irrelevant lines.

The second decisive trick is correlating across multiple log sources: application log, web server log, systemd journal, and kernel messages from dmesg must be viewed within the exact same time window already identified in the "What changed" step. journalctl --since "12:10" --until "12:20" narrows the search precisely and prevents irrelevant entries from a completely different time period from diluting the diagnosis. Structured output with journalctl -o json and jq filtering further simplifies automated processing for larger log volumes.

{
  "_comment": "journalctl -o json --since 12:10 --until 12:20 -u php8.3-fpm | jq",
  "unit": "php8.3-fpm.service",
  "timestamp": "2026-07-12T12:14:02Z",
  "priority": "err",
  "message": "WARNING: [pool www] seems busy (you may need to increase pm.max_children)",
  "pid": 28471
}

9. Documentation, Escalation, and the Workflow Compared

Documentation does not start only after the incident, it begins with the verification of the outage. A jointly visible document or a dedicated chat channel where timestamps, tested hypotheses, findings, and applied measures are continuously noted prevents duplicate checks during a handover between on call staff and later becomes, almost unchanged, the basis of the postmortem. Anyone who tries to reconstruct from memory after the incident what was checked and when regularly loses important details and timestamps.

Just as important as documentation is a clear escalation criterion, defined before the next incident starts, not during it. A fixed time window without measurable progress, for example thirty minutes, or exceeding a defined impact threshold, for example lost revenue or affected core functions, are objective triggers that allow escalation without discussion under time pressure. Escalating early with a second, fresh perspective is almost always more effective than digging alone for hours, even when one's own experience with the system is high.

The following overview directly compares the common anti-patterns with the recommended steps of this workflow.

The table shows a consistent pattern: every risky approach replaces a structured step with chance or lone wolf effort. Anyone who consistently establishes the right column as a fixed process not only shortens the downtime of the individual incident, but over time builds a runbook that makes new team members immediately effective.

# incident-log-2026-07-12-1214.yaml
incident:
  started_at: "2026-07-12T12:10:00Z"
  detected_by: "external uptime monitor, 5xx on /checkout"
  severity: "sev2"
timeline:
  - time: "12:11"
    action: "Verified outage: curl to /health returns 502 from outside and inside"
  - time: "12:14"
    action: "Checked changes: deploy at 12:05 added new PHP-FPM pool config"
  - time: "12:17"
    action: "Load average 1m 9.8 on 4 cores, mpstat shows single core at 100% sys"
  - time: "12:22"
    action: "Rolled back PHP-FPM pool config from the 12:05 deploy"
    result: "5xx rate dropped to zero within 90 seconds"
escalated: false
next_steps:
  - "Postmortem draft from this file, scheduled for 2026-07-13"

10. Summary

A systematic troubleshooting workflow for Linux servers always solves the same underlying problem: under time pressure, even experienced administrators tend toward random guessing, which extends incidents instead of shortening them. The fixed order of verification, change comparison, load, memory, disk, network, and logs rules out an entire class of errors at each step before the next one begins, and ultimately delivers a traceable chain of observations instead of a random hit.

The biggest lever lies in consistently applying this across every incident, regardless of who is on call. Live documentation during the incident turns the later postmortem into a mere formality, and clear escalation criteria prevent a single person from being stuck alone for hours. Anyone who establishes this workflow as a fixed part of incident response measurably reduces downtime while building knowledge that stays with the whole team beyond the individual incident.

11. FAQ: Troubleshooting Workflow for Linux Servers