Debugging "Too Many Open Files" Errors Systematically

1. What "too many open files" really means

The error message "Too many open files" corresponds to the errno code EMFILE or, less often, ENFILE. EMFILE means that a single process has reached its own limit of open file descriptors, while ENFILE means the system-wide cap across all processes combined has been exhausted. On Linux, practically everything counts as a file descriptor: regular files, network sockets, pipes, Unix domain sockets, and even epoll instances. A web server holding many concurrent connections, or a PHP process opening database connections, can therefore quickly hit a limit that was originally intended for classic file access.

The decisive mistake with this error is treating it purely as a capacity problem and simply raising the limit. In most production cases, the actual cause is a file descriptor leak: a code path opens a file, a socket, or a database connection but forgets to close it on an error path or an early return. A higher limit then only delays the inevitable next crash instead of fixing the actual leak. Just like with memory leaks, "too many open files" therefore needs systematic investigation before touching configuration values.

2. Understanding limits: ulimit, /proc/sys/fs/file-max and systemd

Before searching for a leak, you need to know which limit actually applies. ulimit -n in an interactive shell shows the soft limit for the current process tree, while ulimit -Hn shows the hard limit that an unprivileged process cannot exceed on its own. For services started via systemd, however, it is not the shell limit that applies, but the value from LimitNOFILE in the respective unit file or, if not set there, the system-wide default from /etc/systemd/system.conf. A common mistake is setting ulimit -n 65536 in /etc/security/limits.conf and being surprised that a service started via systemd still fails with EMFILE, because PAM limits and systemd limits are two separate mechanisms.

The system-wide cap /proc/sys/fs/file-max additionally limits how many file descriptors may be open in total across all processes. On modern servers with sufficient memory this value is rarely the limiting factor, but it should still be checked with cat /proc/sys/fs/file-nr, which shows the currently used, allocated, and maximum descriptors. Only once both levels, the per-process limit and the system-wide limit, are known can you judge whether an error is really due to too low a configuration or an actual leak.

# Soft and hard limit for the current shell session
ulimit -n
ulimit -Hn

# Effective limit for a running process, read directly from /proc
cat /proc/$(pgrep -f php-fpm | head -1)/limits | grep "Max open files"

# System-wide file descriptor usage: allocated, free, maximum
cat /proc/sys/fs/file-nr

# Limit currently configured for a systemd-managed service
systemctl show php8.4-fpm.service --property=LimitNOFILE

3. Counting and categorizing open file descriptors with lsof

lsof (list open files) is the central tool for seeing what a process actually holds open. lsof -p $PID lists every single descriptor with type, size, and target, while lsof -p $PID | wc -l gives a quick total that can be compared against the configured limit. What matters for diagnosing a file descriptor leak, though, is not the plain count but the distribution by type: many entries of type REG (regular file) hint at unclosed file handles, many entries of type IPv4/IPv6 at unclosed network connections, for example to the database or an external API.

The path column itself is particularly revealing. If it shows a striking number of identical paths, for example always the same log file or the same socket to Redis, that points to a code path that opens a new connection on every call instead of reusing an existing one or closing it correctly. With lsof -p $PID | awk '{print $5}' | sort | uniq -c | sort -rn this distribution can be produced in seconds and often already provides the decisive clue before a debugger or strace is even used.

# Total number of open file descriptors for one process
lsof -p 4823 | wc -l

# Breakdown by file descriptor type (REG, IPv4, sock, pipe, ...)
lsof -p 4823 | awk '{print $5}' | tail -n +2 | sort | uniq -c | sort -rn

# Which specific paths/sockets appear suspiciously often?
lsof -p 4823 | awk '{print $9}' | tail -n +2 | sort | uniq -c | sort -rn | head -10

# Aggregate open FD count across all PHP-FPM worker processes
for pid in $(pgrep -f "php-fpm: pool"); do
  echo "$pid: $(lsof -p "$pid" 2>/dev/null | wc -l) fds"
done

4. /proc/[pid]/fd in detail: sockets, pipes and deleted files

When lsof cannot or should not be installed, the /proc/[pid]/fd directory provides the same information without any extra software. Every entry in it is a symlink whose target can be read with readlink, and ls -la /proc/[pid]/fd | wc -l yields the same total as lsof. Particularly valuable for a file descriptor leak is spotting entries whose target is marked (deleted): these are files that have already been removed from the file system, but because a process still holds them open, the underlying disk space remains occupied. This pattern often occurs during log rotation, when logrotate renames or deletes a file but the writing process never receives a SIGHUP or equivalent to reopen the file.

A second common pattern is entries of type socket:[inode] without a recognizable path. To find out which connection such a socket belongs to, matching the inode number against the output of ss -tp, which shows the associated inode and process for TCP connections, helps. This combination of /proc/[pid]/fd and ss allows a complete built-in diagnosis, even on minimal containers without lsof.

# List all file descriptors with their symlink targets
ls -la /proc/4823/fd

# Find descriptors pointing to files already deleted from disk
# (classic logrotate-without-reopen symptom)
for fd in /proc/4823/fd/*; do
  target=$(readlink "$fd")
  [[ "$target" == *"(deleted)"* ]] && echo "$fd -> $target"
done

# Match a socket inode from /proc/fd back to its TCP connection
ls -la /proc/4823/fd | grep socket
ss -tp | grep "pid=4823"

5. Watching growth over time instead of counting once

As with any resource leak, a single count proves nothing. A normal web server can briefly have hundreds of open descriptors without any problem, as long as the number drops again after load subsides. A file descriptor leak only shows up when the number of open descriptors keeps growing monotonically over hours, regardless of the current request rate. A simple cron job writing the output of lsof -p $PID | wc -l to a log file every minute provides an unambiguous curve within a few hours.

Correlating this curve with the number of processed requests from the access log or the PHP-FPM status endpoint is particularly revealing. If the number of open descriptors grows proportionally to the number of requests and clearly exceeds the number of simultaneously active connections, that is clear proof of a leak: every request leaves behind, on average, one additional, never-closed descriptor. Without this correlation, any discussion about the cause remains pure speculation.

#!/usr/bin/env bash
# fd-watch.sh — log open file descriptor count for a PID over time
set -euo pipefail

PID="${1:?Usage: fd-watch.sh <pid> <log-file>}"
LOG_FILE="${2:?Usage: fd-watch.sh <pid> <log-file>}"

echo "timestamp,open_fds" > "$LOG_FILE"

while kill -0 "$PID" 2>/dev/null; do
  count=$(ls "/proc/$PID/fd" 2>/dev/null | wc -l)
  echo "$(date -Iseconds),$count" >> "$LOG_FILE"
  sleep 60
done

6. Typical causes in PHP-FPM and Nginx setups

In PHP environments, file descriptor leaks mostly arise from unclosed database connections, especially when persistent PDO connections (PDO::ATTR_PERSISTENT) are managed incorrectly and accumulate per worker process over time instead of being reused. Another classic case is cURL handles opened with curl_init() but never released before curl_close() when an exception occurs. Writing to log files through a once-opened file handle, combined with log rotation without a proper reopen signal, also leads to growing, effectively wasted disk usage from deleted but still open files.

On the Nginx side, the most common trigger is a worker_rlimit_nofile directive configured too low in combination with many long-running upstream connections, for example keepalive to PHP-FPM or to a reverse proxy target. If worker_connections in events {} is configured higher than worker_rlimit_nofile allows, Nginx logs exactly the situation that leads to "too many open files" as soon as the actual connection count reaches the internal limit in its error log. This configuration mismatch is quickly fixed but should not be confused with an actual leak.

# nginx.conf — worker_rlimit_nofile must be >= worker_connections
worker_rlimit_nofile: 65535
events:
  worker_connections: 32768
  # If worker_connections exceeds worker_rlimit_nofile, upstream keepalive
  # connections will fail with "too many open files" under sustained load.

7. Identifying the responsible syscall with strace

When lsof and /proc/[pid]/fd show that a specific connection type is growing but it remains unclear which code path is responsible, strace -e trace=open,openat,socket,close -p $PID provides the chronological sequence of all relevant syscalls. A leak typically shows up here as a series of open() or socket() calls without a matching 1:1 ratio of close() calls. The output can be filtered with grep -c for both syscall types to quantify the difference over an observation period without having to go through every single line manually.

For PHP applications, strace -f with the -f flag for child processes additionally provides important detail when PHP-FPM creates workers via fork(). Combined with a timestamp (-tt), it becomes possible to determine exactly which request a never-closed descriptor belongs to, which greatly simplifies matching it against the access log and thus identifying the triggering URL or function.

# Trace file and socket related syscalls with timestamps
strace -tt -f -e trace=open,openat,socket,close -p 4823 2>&1 | tee /tmp/fdtrace.log

# Count opens vs. closes — a growing gap confirms a leak
grep -cE '^\[pid [0-9]+\] (open|openat|socket)\(' /tmp/fdtrace.log
grep -cE '^\[pid [0-9]+\] close\(' /tmp/fdtrace.log

# Correlate a specific leaked fd number back to its open() call
grep "= 87$" /tmp/fdtrace.log

8. Configuring limits correctly without masking the leak

After the root cause analysis, the question remains how to set limits sensibly. The rule of thumb: the limit should be generously above expected peak load, but low enough that an actual leak visibly hits its limit within a manageable time instead of growing unnoticed for days. For PHP-FPM you set LimitNOFILE in the systemd unit or via php_admin_value[rlimit_files] per pool, for Nginx worker_rlimit_nofile matching worker_connections.

In addition, a monitoring alert should trigger at a percentage of the configured limit, for example at 80 percent of open descriptors relative to the limit from /proc/[pid]/limits. This gives the team time to investigate a file descriptor leak before the process actually crashes with EMFILE. It is important to always treat the raised limit as a temporary safeguard and to document in a ticket that the actual root cause in the code is still open, so the problem does not fall into oblivion.

; /etc/systemd/system/php8.4-fpm.service.d/limits.conf
[Service]
LimitNOFILE=16384

; /etc/php/8.4/fpm/pool.d/www.conf — per-pool override if needed
[www]
rlimit_files = 16384

9. File descriptor diagnostic approaches compared

Depending on the situation, different tools are better or worse suited to narrowing down a file descriptor leak. The overview below arranges the approaches presented by use case.

In practice, the investigation starts with lsof or /proc/[pid]/fd for quick categorization by type and path, followed by ss -tp for exact matching of network sockets. Only once these two steps have narrowed down the suspicious resource type does it pay off to use strace in a targeted way to confirm the exact code path. This order prevents starting with the most expensive tool when simpler means already provide sufficient clues.

10. Summary

The "too many open files" error is almost always a symptom of an underlying file descriptor leak and rarely a pure capacity problem. The systematic investigation starts with understanding the limits involved, ulimit, systemd LimitNOFILE, and the system-wide cap from /proc/sys/fs/file-max, followed by counting and categorizing open descriptors with lsof or /proc/[pid]/fd. A time series over hours confirms whether a leak is actually present, while strace identifies the exact code path behind missing close() calls.

In PHP-based setups, unclosed database connections and cURL handles are the most common causes, while on Nginx a mismatch between worker_connections and worker_rlimit_nofile is often mistaken for a real leak. Anyone who combines limits as a safeguard with monitoring alerts and tracks down the actual cause in the code in a documented way prevents "too many open files" from becoming a recurring, never truly resolved incident.

11. FAQ: Debugging "Too Many Open Files" Errors Systematically