Exploring the proc Filesystem: Kernel State Without Extra Tools

1. What the proc filesystem really is

The proc filesystem, usually just called /proc, is not a real filesystem stored on disk, it is a virtual filesystem the kernel generates dynamically on every access. When a process reads /proc/1234/status, the kernel does not access stored data, it generates the file content at the exact moment it is read, straight from its internal data structures. That makes the proc filesystem one of the most reliable sources of system state, because it can never be stale. It always shows the exact kernel state at the moment of reading.

Historically, the proc filesystem was introduced to expose process information without special system calls, which classic Unix tools such as ps previously had to obtain through direct memory access to the kernel. Today /proc covers far more than processes: CPU information, memory usage, network connections, loaded kernel modules and thousands of kernel parameters under /proc/sys are all reachable through the same unified interface. Understanding how the proc filesystem is structured also means understanding how almost every standard diagnostic tool on Linux works internally, since nearly all of them ultimately read from /proc.

2. /proc/[pid]: process information straight from the kernel

For every running process, the kernel automatically creates a directory under /proc/[pid]/, where [pid] is the process ID. This directory disappears the moment the process terminates, which makes it a perfect live view of running processes. Inside this directory sit dozens of files and subdirectories, each exposing a specific aspect of the process: cmdline holds the full command line the process was started with, cwd is a symlink to the current working directory, and fd/ lists every open file descriptor as a symlink to the corresponding file or socket.

Particularly useful in the proc filesystem is /proc/[pid]/environ, which shows the complete environment variables of a running process, invaluable when diagnosing configuration problems in PHP-FPM or Nginx worker processes. /proc/[pid]/maps shows the complete memory mapping table of a process, including every loaded shared library with its address range, which helps when analyzing memory leaks or segmentation faults. The special case /proc/self always refers to the process currently reading it, regardless of its actual PID.

# Full command line the process was started with (null-separated)
cat /proc/1234/cmdline | tr '\0' ' '; echo

# Current working directory of a running process
readlink /proc/1234/cwd

# List every open file descriptor as a symlink target
ls -la /proc/1234/fd/

# Full environment variables of a running process (root or same user)
sudo cat /proc/1234/environ | tr '\0' '\n'

# Memory mapping table: shared libraries and their address ranges
cat /proc/1234/maps | head -20

# /proc/self always refers to the process reading it
readlink /proc/self/exe

3. CPU and memory: /proc/cpuinfo and /proc/meminfo

/proc/cpuinfo is the canonical source for detailed CPU information on Linux and is read by practically every higher level diagnostic tool. The file lists a separate block for every logical processor core, with model name, clock speed, cache size and the supported CPU flags such as avx2 or sse4_2, which decide whether certain optimized code paths in applications like PHP Opcache or database engines can even be used. The number of blocks equals the number of logical cores including hyperthreading, not the number of physical cores.

/proc/meminfo is the counterpart for memory and provides far more detail than free -h, which internally parses exactly this file. Lines such as MemAvailable show the memory actually available for new applications including reclaimable caches, while Dirty shows the amount of memory pages that have been changed but not yet written to disk. For database servers, Cached is particularly relevant, because a high value means the kernel is holding a lot of filesystem cache, which gets released instantly when applications need it, not wasted resources.

# Count logical CPU cores (includes hyperthreading)
grep -c ^processor /proc/cpuinfo

# Show model name and clock speed for each core
grep -E "model name|MHz" /proc/cpuinfo

# Check whether the CPU supports AVX2 (relevant for optimized code paths)
grep -o avx2 /proc/cpuinfo | head -1

# Key memory figures, same source free -h parses internally
grep -E "MemTotal|MemAvailable|Cached|Dirty|SwapTotal" /proc/meminfo

# Live memory monitoring without top or free
watch -n 1 'grep -E "MemAvailable|Cached" /proc/meminfo'

4. Process status in detail: status, statm and cmdline

/proc/[pid]/status is the human readable summary of a process state and contains fields hardly any other tool delivers this compactly: State shows the current process state (running, sleeping, in interruptible or uninterruptible wait, or zombie), VmRSS the portion actually resident in physical memory, and Threads the number of threads inside the process. For PHP-FPM workers, VmRSS matters especially because it shows real memory usage per worker, unlike VmSize, which also counts virtual, unallocated address space and therefore often shows a multiple of the actual usage.

The file /proc/[pid]/statm delivers the same memory figures as plain numbers in memory pages instead of formatted text, which makes it more suitable for scripts and monitoring agents than the textual status. The difference between status and stat (without m): /proc/[pid]/stat is a single line with space separated fields in fixed order, optimized for machine parsing with minimal overhead, while status is formatted for humans. Both files describe the same kernel state, just for a different audience.

# Human-readable process status summary
cat /proc/1234/status | grep -E "State|VmRSS|VmSize|Threads"

# Real resident memory in kilobytes for a PHP-FPM worker
awk '/VmRSS/{print $2, $3}' /proc/1234/status

# Machine-parseable memory figures in pages (statm)
cat /proc/1234/statm

# Single-line stat file, fixed field order, minimal parsing overhead
cat /proc/1234/stat

# Sum resident memory of all php-fpm worker processes
for pid in $(pgrep php-fpm); do
  awk '/VmRSS/{sum+=$2} END{print sum " kB"}' /proc/"$pid"/status
done | awk '{s+=$1} END{print s " kB total"}'

5. Reading network state from /proc/net

The subdirectory /proc/net/ is the data source classic tools like netstat draw their information from, before more modern alternatives such as ss switched to the kernel's netlink interface. /proc/net/tcp and /proc/net/tcp6 list every active TCP connection with local and remote address in hexadecimal notation, connection state as a hex code, and the inode number of the associated socket. This inode number can be matched against entries in /proc/[pid]/fd/ to find out which process holds a given connection, with no need for lsof or ss.

For aggregated statistics, /proc/net/dev is the central source: it shows cumulative byte and packet counters for incoming and outgoing traffic since the last boot for every network interface, including error and drop counters. A steadily rising drop counter on a production interface usually indicates overloaded ring buffers or a receive queue that is too small, a classic signal that network tuning is needed. The proc filesystem makes these raw numbers accessible without installing a separate monitoring tool.

# Active TCP connections in raw kernel format (hex addresses, hex state)
cat /proc/net/tcp | head -5

# Cumulative interface statistics since last boot
cat /proc/net/dev

# Watch for a rising error/drop counter, a sign of NIC or buffer pressure
watch -n 2 'cat /proc/net/dev | grep eth0'

# Match a socket inode from /proc/net/tcp to the owning process
# 1. Find the inode for a connection in /proc/net/tcp
# 2. Grep for that inode across all /proc/[pid]/fd/ symlinks
for pid in /proc/[0-9]*; do
  ls -la "$pid"/fd 2>/dev/null | grep -q "socket:\[12345\]" && echo "$pid"
done

6. /proc/sys: the kernel's sysctl interface

The directory /proc/sys/ is the direct interface to thousands of kernel parameters, commonly managed through the sysctl command line tool. In fact, sysctl is just a convenient wrapper around exactly these same files, which can also be read and written directly. The parameter net.ipv4.ip_forward, for example, corresponds to the file /proc/sys/net/ipv4/ip_forward, and both access paths change the same kernel state with no difference in outcome. Every dot in a sysctl name corresponds to one directory level in the path under /proc/sys.

Direct access through the proc filesystem matters especially in minimal container environments where sysctl is not installed, but /proc stays available as long as it has not been explicitly masked. Changes via echo value > /proc/sys/path take effect immediately and behave identically to sysctl -w, but like all direct sysctl changes are not persistent across a reboot. For permanent changes, /etc/sysctl.d/ remains the right place, but the proc filesystem always shows the actually active value, regardless of what a configuration file says.

# Read a kernel parameter directly, equivalent to sysctl net.ipv4.ip_forward
cat /proc/sys/net/ipv4/ip_forward

# Same value via sysctl, just a convenient wrapper around the same file
sysctl net.ipv4.ip_forward

# Write directly, takes effect immediately, not persistent across reboot
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

# Every dot in a sysctl name is a directory level under /proc/sys
ls /proc/sys/net/ipv4/ | grep tcp_

# Find every writable tunable under a given subsystem
find /proc/sys/vm -type f

7. Tracking file descriptors and limits through /proc

A common production problem is hitting the maximum number of open file descriptors, which crashes applications with the error Too many open files. The proc filesystem makes this state directly visible: /proc/[pid]/limits shows both the soft and hard limit for every resource type of a specific process, while ls /proc/[pid]/fd/ | wc -l gives the actually used count right now. If this value approaches the limit shown in limits, a crash from resource exhaustion is only a matter of time.

System wide, /proc/sys/fs/file-nr shows three values on one line: the number of currently allocated file descriptors, the number free and currently unused, and the system wide maximum from /proc/sys/fs/file-max. This system wide limit is independent of the per process limits from ulimit and affects all processes combined. On heavily loaded database or web servers with many concurrent connections, regularly checking both levels, per process and system wide, is part of solid capacity planning.

# Soft and hard limits for a specific running process
cat /proc/1234/limits | grep -i "open files"

# Currently open file descriptors for that same process
ls /proc/1234/fd/ | wc -l

# System-wide: allocated, free, and maximum file descriptors
cat /proc/sys/fs/file-nr

# System-wide hard ceiling for all processes combined
cat /proc/sys/fs/file-max

# Find the process closest to exhausting its own file descriptor limit
for pid in /proc/[0-9]*; do
  p=$(basename "$pid")
  used=$(ls "$pid"/fd 2>/dev/null | wc -l)
  max=$(awk '/Max open files/{print $4}' "$pid"/limits 2>/dev/null)
  [ -n "$max" ] && [ "$used" -gt 0 ] && echo "$p: $used/$max"
done | sort -t/ -k1 -rn | head -5

8. Using the proc filesystem in your own scripts

Because the proc filesystem consists of plain text files, it can be read directly from Bash, PHP or Python without libraries or API calls, which makes it the ideal foundation for lightweight, dependency free monitoring scripts. A healthcheck script that checks whether a PHP-FPM master process is running and how much memory it uses needs no external library, it simply reads /proc/[pid]/status and parses the relevant fields with awk or grep.

Important for robust scripts: between listing a process directory and reading a file inside it, the process may have terminated, leading to a No such file or directory error. Production scripts that systematically scan the proc filesystem must handle this case, for example with 2>/dev/null and a check for empty output, rather than letting the script abort on a single vanished process. This race condition is normal and expected, not a sign of a bug in the script itself.

#!/usr/bin/env bash
# Minimal healthcheck script using only /proc, no external dependencies
set -euo pipefail

check_process_memory() {
  local pid="$1"
  local max_mb="$2"
  # Handle the race: process may vanish between listing and reading
  local rss_kb
  rss_kb=$(awk '/VmRSS/{print $2}' /proc/"$pid"/status 2>/dev/null) || return 1
  [ -z "$rss_kb" ] && return 1

  local rss_mb=$((rss_kb / 1024))
  if (( rss_mb > max_mb )); then
    echo "[WARN] PID $pid uses ${rss_mb}MB (limit: ${max_mb}MB)"
    return 1
  fi
  return 0
}

for pid in $(pgrep php-fpm); do
  check_process_memory "$pid" 256 || echo "  -> investigate $pid"
done

9. proc filesystem compared to classic tools

Almost every classic diagnostic tool on Linux is ultimately just a formatted view of files in the proc filesystem. Direct access to /proc pays off when a tool is missing, when raw data is needed for scripts, or when you want to understand exactly where a number comes from.

The practical value of the proc filesystem shows especially in minimal container images without diagnostic packages, in monitoring agents that should not spawn external processes, and when debugging requires understanding where a number in a higher level tool actually comes from. Almost every metric monitoring systems like the Prometheus Node Exporter collect ultimately originates from exactly these files under /proc.

10. Summary

The proc filesystem is perhaps the most underrated diagnostic source on Linux, since almost every standard tool is ultimately just a formatted view of exactly these virtual files. /proc/[pid]/ delivers complete process information including command line, environment variables and memory mapping. /proc/cpuinfo and /proc/meminfo show hardware and memory state in maximum detail. /proc/net/ makes connections and interface statistics accessible, and /proc/sys/ is the direct, unfiltered interface to every sysctl parameter of the kernel.

The biggest practical value of the proc filesystem lies in its universality: it works in every container, on every distribution, and without a single extra package, because it is part of the kernel itself. Anyone writing their own monitoring or healthcheck scripts saves external dependencies through direct /proc access, while gaining a more precise understanding of which kernel states sit behind every displayed metric.

11. FAQ: Exploring the proc Filesystem