Using journalctl Effectively: Searching systemd Logs

1. Why journalctl is the central logging tool under systemd

On every modern Linux system running systemd, the log output of services, kernel, and boot process flows by default into the binary systemd journal, managed by the systemd-journald daemon. Instead of searching twenty different log files under /var/log/ with inconsistent formats, journalctl gives you a single tool that returns structured, indexed entries. Every entry carries metadata such as PID, UID, unit name, and boot ID, information that classic text logs simply do not have.

The practical advantage shows up most clearly when debugging under time pressure: instead of running grep across several files with differing timestamp formats, journalctl filters directly by service, time range, priority, or boot session and returns consistently formatted output. Because the journal is stored in binary form, full text search and field filters are noticeably faster than plain grep over large text files, especially once several gigabytes of logs have accumulated.

2. Filtering by unit: -u and mapping to services

The -u or --unit parameter is the most commonly used filter, because most incidents can be traced to a specific service. journalctl -u nginx.service shows only entries attributed to that unit, whether they arrived via stdout, stderr, or directly through the sd_journal API. The .service suffix can usually be dropped, so journalctl -u nginx works just as well as long as the name is unambiguous.

Multiple units can be combined by repeating -u, for example journalctl -u nginx -u php8.3-fpm, which is evaluated internally as a logical OR. For container or slice setups, --user-unit works the same way for services running under a systemd --user context. If you do not know the exact unit name, find it with systemctl list-units --type=service or directly via systemctl status <process-name>, which also shows the service's most recent journal lines.

# Show all log entries for a single unit
journalctl -u nginx.service

# Combine multiple units (logical OR)
journalctl -u nginx.service -u php8.3-fpm.service

# Filter by unit and priority together
journalctl -u mysql.service -p err

# Show only entries from the current boot for a unit
journalctl -u docker.service -b

# Find the exact unit name first
systemctl list-units --type=service | grep -i redis

3. Narrowing the time window: --since and --until

The --since and --until options restrict output to a specific time range and accept both relative and absolute time expressions. journalctl --since "1 hour ago" shows the last hour, while journalctl --since "2026-07-12 08:00:00" --until "2026-07-12 09:30:00" defines an exact window. Symbolic values such as today, yesterday, and now are also understood, which saves significant time during quick lookups while an incident is unfolding.

One important pitfall: by default journalctl interprets time expressions in the system's local timezone, not UTC. On servers running in UTC while the monitoring team thinks in a different timezone, this regularly causes confusion. journalctl --utc --since "2026-07-12 06:00:00 UTC" switches both output and time filter interpretation to UTC. Combining a time window with a unit filter, for instance journalctl -u mysql.service --since "30 min ago", is by far the most common pattern during incident analysis because it immediately narrows the data down to what matters.

4. Filtering by priority: -p and understanding severity levels

The -p or --priority parameter filters by syslog severity and follows the eight classic levels: emerg, alert, crit, err, warning, notice, info, and debug. Important to know: journalctl -p err shows not just that exact level by default, but all entries at that priority and higher severity, meaning err, crit, alert, and emerg. If you want exactly one severity level, specify a range with two values, for example -p warning..err.

In practice, journalctl -p err -b usually shrinks the output of the current boot drastically, because info and debug noise disappears entirely while real error messages remain. It is the first command many administrators run after an unexplained outage. Keep in mind that many applications do not map their log levels correctly to syslog priorities, homegrown services in particular often log everything at info, which makes priority filtering less useful there than for well-maintained system services.

# Show only errors and worse from the current boot
journalctl -p err -b

# Priority range: warning up to and including err
journalctl -p warning..err

# Numeric priority levels also work (0=emerg .. 7=debug)
journalctl -p 3

# Combine priority with unit and time window
journalctl -u nginx.service -p err --since "today"

# Kernel-only messages at crit level or higher
journalctl -k -p crit

5. Following logs live: -f and how it compares to tail -f

The -f or --follow option behaves like tail -f on classic log files: journalctl prints the most recent lines and then stays active, showing new entries in real time. Particularly useful during focused debugging is the combination journalctl -u app.service -f, letting you watch live how a service reacts to a test request without searching through the noise of other units. -n additionally controls how many lines are shown initially, for example journalctl -u app.service -f -n 50 for the last 50 lines plus the live stream.

One difference from tail -f: journalctl can follow multiple units simultaneously and interleave the output by timestamp, something a classic log file could not do without an extra tool such as multitail. journalctl -f -u nginx -u php8.3-fpm shows both streams mixed together in chronological order in a single terminal. For automated processing, --output json combined with -f is a good fit, forwarding every new line as a JSON object to a monitoring script. If you want to run the live mode inside a systemd timer or cronjob, use --cursor instead of -f so the evaluation script can resume seamlessly at the last position after a restart, without processing entries twice or missing any.

6. Configuring persistent versus volatile journal storage

By default, on many distributions systemd-journald stores the journal only volatilely under /run/log/journal/, a tmpfs directory that gets wiped on every reboot. After a reboot, logs from an incident are then irretrievably lost, which can be fatal for root-cause analysis of a crash. Whether the journal is persistent can be checked with journalctl --disk-usage together with a look at the storage directive in /etc/systemd/journald.conf.

To make the journal permanent, create the directory /var/log/journal/ and restart journald, or alternatively set Storage=persistent explicitly in journald.conf. In addition, SystemMaxUse and SystemMaxFileSize should be configured so the journal does not uncontrollably fill the entire disk. Without this limit, journald can theoretically consume up to ten percent of the filesystem, which quickly causes disk space problems on small root partitions.

# /etc/systemd/journald.conf
# Make the journal persistent across reboots and limit its disk footprint

[Journal]
Storage=persistent
Compress=yes

# Hard cap on total journal size on disk
SystemMaxUse=1G

# Keep at least this much free space on the filesystem
SystemMaxFileSize=128M

# Automatically remove entries older than this
MaxRetentionSec=1month

# Rate limit to avoid a noisy service flooding the journal
RateLimitIntervalSec=30s
RateLimitBurst=10000

7. Correlating logs across multiple services during an incident

During complex incidents, looking at a single service is rarely enough, because an error in the database often propagates through PHP-FPM up to the web server. journalctl is built exactly for this, letting you combine several -u filters within a shared time window so events line up chronologically: journalctl -u nginx -u php8.3-fpm -u mysql --since "09:14" --until "09:20" shows exactly in what order the three services reacted to the incident.

For even more precise correlation across container and cgroup boundaries, --identifier for syslog identifiers or the _SYSTEMD_CGROUP field, visible with journalctl -o verbose, helps show which container an entry came from in Docker or systemd-nspawn setups. For applications with their own request ID, that ID should consistently be written into the structured log, for example via journalctl-compatible structured logging with sd_journal_send, so a single request can be traced across multiple services with one grep call.

# Correlate three services during a narrow incident window
journalctl -u nginx.service -u php8.3-fpm.service -u mysql.service \
  --since "2026-07-12 09:14:00" --until "2026-07-12 09:20:00"

# Merge with kernel messages to rule out OOM kills
journalctl -u app.service -k --since "09:14" --until "09:20"

# Show verbose output to inspect cgroup and container fields
journalctl -u app.service -o verbose | grep -E "_SYSTEMD_CGROUP|CONTAINER_NAME"

# Trace a single request ID across services (structured logging required)
journalctl --since "09:14" --until "09:20" | grep "req_id=8f3a91"

8. Exporting logs and sharing them with teammates

To pass logs on to teammates or document them in a ticket, --output json-pretty works well for readable, structured output, or --output export for the native binary exchange format, which can be re-imported on another system with journalctl --file. For a plain text export without any formatting overhead, journalctl -u app.service --since today --no-pager > incident-2026-07-12.log is enough, and the result attaches cleanly to an email or a ticket.

A frequently overlooked feature is journalctl -u app.service -o export | gzip > incident.journal.gz, which preserves the full metadata including every journal field, unlike a plain text export where fields such as _PID or _SYSTEMD_UNIT are lost. Before sharing sensitive logs, check whether environment variables or request bodies were accidentally logged, since journalctl by default outputs exactly what the service wrote, unaltered, including any credentials that ended up in error messages.

{
  "__CURSOR": "s=8f2c1e...;i=5d3a;b=91a7f4...",
  "__REALTIME_TIMESTAMP": "1752307140000000",
  "_SYSTEMD_UNIT": "app.service",
  "_PID": "48213",
  "_UID": "1001",
  "PRIORITY": "3",
  "SYSLOG_IDENTIFIER": "app",
  "MESSAGE": "Database connection timeout after 5000ms",
  "_HOSTNAME": "shop-web-02"
}

9. journalctl compared directly to classic log files

Switching from grep over text files to journalctl filters is not a matter of taste, it brings measurable gains in speed, accuracy, and traceability. The table below contrasts typical tasks with the recommended approach for each.

In practice, the biggest time savings come from combining multiple filters into a single command: where classic tooling would need several pipes with grep, awk, and sed, one line with -u, -p, and --since is enough with journalctl. That not only reduces typing, it also reduces mistakes under stress during a live incident.

10. Summary

journalctl solves the core problem of scattered text logs by bundling every systemd log into one structured place and making it precisely filterable via -u, --since/--until, -p, and -f. Combining a unit filter with a time window is by far the most common pattern during incident analysis, while priority filtering reliably hides info noise. Without a persistent journal set through Storage=persistent in journald.conf, logs are lost on every reboot, which is fatal for crash analysis in particular.

During complex incidents involving several services, combining multiple -u filters within a shared time window delivers a chronologically correct overall picture that classic text logs could only match with extra tools such as multitail. For handing logs to teammates or documenting them in tickets, --output json-pretty is a good fit for readability, and --output export when the full metadata needs to be preserved.

11. FAQ: Using journalctl Effectively