Centralized Logging with rsyslog: Consolidating Logs From Multiple Servers

1. Why distributed logs become a problem during an incident

In a typical Magento infrastructure with several application servers behind a load balancer, logs are produced in many places at once: Nginx access logs on every frontend, PHP-FPM error logs on every app server, plus database and cache logs on separate machines. When a customer reports an error, nobody knows in advance which server actually holds the relevant line, and manually searching four or five servers over SSH costs valuable time during an incident.

Centralized logging moves that effort from the search phase to the collection phase: instead of hunting for the right timestamp on every server during an incident, every log line is already sitting in one place, sorted chronologically. rsyslog is particularly well suited for this because it is already installed as the default syslog daemon on practically every Linux distribution and needs neither an extra agent deployment nor an external dependency to act as a collection point.

2. Understanding the module architecture of rsyslog

rsyslog is internally split strictly into input, parser, filter and output modules, connected through a processing pipeline. Input modules, whose names start with im, accept messages, for example imuxsock for the local Unix socket, imjournal for importing from the systemd journal, or imtcp and imudp for messages arriving over the network.

Output modules, whose names start with om, decide what happens to a message at the end of the pipeline. omfile writes to local files, omfwd forwards messages to another syslog server, and specialized modules such as omelasticsearch send directly to external systems. Between input and output sit rulesets and queues that determine which message takes which path through the pipeline.

# Validate the current rsyslog configuration without actually starting the daemon
rsyslogd -N1

# Extract active modules and their load order from the configuration
grep -r "module(load=" /etc/rsyslog.conf /etc/rsyslog.d/

3. Setting up a central rsyslog server

The central server first needs an enabled input module for incoming network connections. For TCP that is imtcp, which, unlike UDP, guarantees reliable, connection oriented delivery and is therefore almost always the right choice for production centralized logging, since individual UDP packets can silently get lost under network load.

So incoming logs do not all end up in a single file, a template directive defines the destination file path dynamically based on the hostname and program name of each message. That automatically creates a separate, clearly named directory structure per sending server, without having to configure every client individually.

# /etc/rsyslog.d/10-central-receiver.conf on the central server
module(load="imtcp")
input(type="imtcp" port="514" streamdriver.mode="0")

template(name="PerHostLog" type="string"
  string="/var/log/central/%HOSTNAME%/%programname%.log")

if $inputname == "imtcp" then {
  action(type="omfile" dynaFile="PerHostLog")
  stop
}

4. Configuring clients to forward to the central server

On every application server, a short additional configuration block forwards all locally received messages to the central server via omfwd. The order of actions matters: forwarding should sit before any local filtering, so that messages discarded locally are still preserved centrally.

By default, rsyslog buffers outgoing messages in memory. During a brief network interruption to the central server, those messages get lost once the buffer fills up. For production environments a disk assisted queue is therefore recommended, which temporarily spills messages to local disk instead of dropping them.

# /etc/rsyslog.d/60-forward-central.conf on every application server
action(type="omfwd"
  target="log-central.internal" port="514" protocol="tcp"
  queue.type="LinkedList"
  queue.filename="fwd_central"
  queue.maxdiskspace="1g"
  queue.saveonshutdown="on"
  action.resumeRetryCount="-1")

5. Enabling TLS encryption for log transport

Unencrypted syslog over the network means log content, potentially including session IDs, IP addresses or error details from PHP stack traces, travels across the network in plain text. rsyslog supports TLS through the GnuTLS or OpenSSL stream driver and fully encrypts the connection between client and central server, including optional certificate validation on both sides.

For production use, a small dedicated certificate authority is recommended, with its certificate deployed to every participating server. The central server presents a server certificate, each client validates it against the CA certificate, and optionally the server in turn requires a valid client certificate before it accepts a connection at all.

# Server side: enforce TLS and require client certificates
module(load="imtcp"
  StreamDriver.Name="gtls"
  StreamDriver.Mode="1"
  StreamDriver.AuthMode="x509/name")
global(
  DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls/ca.pem"
  DefaultNetstreamDriverCertFile="/etc/rsyslog.d/tls/server-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/tls/server-key.pem")
input(type="imtcp" port="6514")

6. Comparison to journald forwarding

As an alternative to rsyslog, systemd offers its own way to transfer journal entries to a central server over HTTP or HTTPS, using systemd-journal-upload and systemd-journal-remote. The advantage lies in native integration with structured journal fields, transferred as JSON while keeping extra metadata such as boot ID or unit name that gets lost in the classic syslog format.

The downside is lower adoption and maturity compared to rsyslog: fewer distributions enable the journal remote packages by default, configuring HTTPS certificates is less documented, and filtering capabilities at the central receiver are more limited than rsyslog's flexible ruleset system. In mixed environments with Nginx, PHP-FPM and older applications that write classic syslog, rsyslog therefore usually remains the more pragmatic choice.

7. Practical example: centralizing Nginx and PHP-FPM logs

Nginx and PHP-FPM write to local files by default rather than directly to syslog. To pull these logs into the central collection, imfile monitors the relevant files like tail -f and feeds new lines into the rsyslog pipeline as syslog messages, including state tracking so content is not read twice after a restart.

For later analysis it pays off to switch Nginx to a structured JSON log format instead of the default one. That way status code, request time and upstream response time can be processed directly as searchable fields, instead of extracting them later with error prone regular expressions from a plain text line.

# /etc/rsyslog.d/20-nginx-phpfpm-input.conf
module(load="imfile")

input(type="imfile"
  File="/var/log/nginx/access.log"
  Tag="nginx-access"
  Severity="info"
  Facility="local1")

input(type="imfile"
  File="/var/log/php-fpm/www-error.log"
  Tag="php-fpm"
  Severity="err"
  Facility="local2")

8. Performance and queue tuning under heavy log volume

With several thousand requests per second on a Magento store, log volume alone can become a noticeable load. rsyslog processes messages using several parallel worker threads per queue by default, and their count can be adjusted with queue.workerThreads whenever the default configuration cannot keep up during traffic spikes.

Another important lever is rate limiting directly at the input, to throttle misbehaving applications that generate thousands of identical error messages per second in a loop. Without a limit, a single faulty application can overwhelm the entire central logging infrastructure and thereby degrade visibility into other, genuinely relevant errors.

# Increase worker threads and enable rate limiting per sender
main_queue(queue.workerThreads="4" queue.dequeueBatchSize="1000")

input(type="imtcp" port="514"
  ratelimit.interval="10"
  ratelimit.burst="2000")

9. Best practices and everyday troubleshooting

Before any configuration change, a test run with rsyslogd -N1 is worthwhile, since it catches syntax errors without actually restarting the daemon. To test the connection between a client and the central server, logger is the right tool, sending a single test message over the local syslog socket that should immediately be traceable in the expected destination file on the central server.

Firewall rules are a common source of errors: the central server's port must be explicitly opened for the application servers, typically 514 for unencrypted connections and a separate port such as 6514 for TLS, so both modes stay cleanly separated and an accidentally unencrypted client does not silently send plain text over the network.

Documentation pays off especially here: a short overview of which server uses which facility and which tag saves new team members the tedious work of reverse engineering the configuration from the already collected logs.

10. Summary

11. FAQ: Centralized rsyslog Logging