Exploring debugfs: Kernel Internals Beyond /proc and /sys

1. Why debugfs exists in the first place

Both /proc and /sys follow strict stability rules: once published, files and their format count as stable kernel ABI and cannot be changed lightly, because userspace programs rely on them. That noticeably slows down kernel developers who want to export debug information quickly, without long term compatibility promises.

debugfs was created exactly for that purpose: a filesystem with no stability guarantee whatsoever, where kernel subsystems can expose arbitrary internal state as files without having to worry about format changes between kernel versions. For admins that means, in return, that whatever works under debugfs today may change or disappear in a later kernel update, without warning.

2. Mount point and setup of debugfs

The standard mount point for debugfs is /sys/kernel/debug, already mounted automatically at boot on most modern distributions. Whether debugfs is currently mounted can be checked quickly via mount or simply by looking into the directory, where an empty or nonexistent directory usually means debugfs is not active.

If the mount is missing, for instance on a minimally configured kernel or inside a container environment, it can be added manually, provided the kernel was built with CONFIG_DEBUG_FS. On most distribution kernels this option is enabled by default.

# Check whether debugfs is already mounted
mount | grep debugfs

# Mount it manually if missing
mount -t debugfs none /sys/kernel/debug

# Check kernel configuration for debugfs support
zgrep CONFIG_DEBUG_FS /proc/config.gz

3. Block layer statistics under debugfs

For storage diagnostics beyond the aggregated numbers in /proc/diskstats, debugfs offers far more detailed insight into the block layer. Under /sys/kernel/debug/block/<device> you find information about the I/O scheduler queue, outstanding requests, and, depending on the scheduler, internal queue state.

Especially with performance problems on database or Elasticsearch volumes backed by NVMe or SSD storage, these values help distinguish between a problem in the kernel I/O scheduler and a problem at the storage backend level itself, a distinction that iostat alone often cannot make clear.

# List available block devices under debugfs
ls /sys/kernel/debug/block/

# Inspect scheduler-specific statistics for a device (example: nvme0n1)
find /sys/kernel/debug/block/nvme0n1/ -type f | xargs -I{} sh -c 'echo "== {} =="; cat {}'

4. The tracing infrastructure under debugfs/tracing

The kernel's entire ftrace subsystem, the built-in function tracing facility, is traditionally controlled through files under /sys/kernel/debug/tracing, even though part of it is now also reachable via /sys/kernel/tracing. Anyone who has used perf or trace-cmd has already used this infrastructure indirectly, without touching the raw files themselves.

For a manual look, it is enough to list available tracers via available_tracers, enable one of them in current_tracer, and then read trace. This direct, file based access is considerably less convenient than bpftrace or perf, but it also works on minimal systems without any additional packages.

# List available ftrace tracers
cat /sys/kernel/debug/tracing/available_tracers

# Enable the function tracer and observe briefly
echo function > /sys/kernel/debug/tracing/current_tracer
sleep 2
cat /sys/kernel/debug/tracing/trace | head -n 40

# Disable the tracer again
echo nop > /sys/kernel/debug/tracing/current_tracer

5. Other subsystems with their own debugfs entries

Beyond the block layer and tracing, numerous other kernel subsystems expose their own debugfs directories: the ext4 filesystem, for example, under /sys/kernel/debug/ext4 with internal statistics per mount, network drivers frequently under their own directory with register dumps, and memory management adjacent subsystems such as kmemleak for finding memory leaks inside the kernel itself.

Which directories actually exist depends heavily on loaded drivers and kernel configuration, which is why a simple recursive look into /sys/kernel/debug pays off, to see what is actually available on the specific system rather than relying on a fixed list.

# List existing debugfs subdirectories as an overview
find /sys/kernel/debug -maxdepth 1 -type d | sort

# Check ext4-specific debug information for a filesystem (if present)
ls /sys/kernel/debug/ext4/ 2>/dev/null

6. Security considerations and restricting access

debugfs deliberately exposes very detailed, sometimes security relevant kernel internals, including memory addresses, internal driver state, and in some cases even cryptographic material during development. For that reason access is restricted to root by default, and that restriction should not be loosened on production servers.

Hardening guidelines such as the CIS benchmark additionally recommend disabling debugfs entirely on production systems, or at least mounting it read only when no active diagnosis is underway. The kernel boot parameter debugfs=off can disable the filesystem completely at boot, which can make sense on hardened production servers without a regular need for debugging.

Precisely because debugfs guarantees no API stability and looks deep into kernel internals, everyday access should stay limited to targeted, time boxed diagnostic sessions rather than being wired permanently into monitoring scripts that would silently start returning wrong or no data after a kernel update.

# Check the current permissions of the mount point
stat -c "%a %U:%G" /sys/kernel/debug

# Disable debugfs entirely on the next boot (GRUB kernel parameter)
# in /etc/default/grub: GRUB_CMDLINE_LINUX="... debugfs=off"

7. Comparison with /proc and /sys

The central difference lies in the stability promise: /proc and /sys follow clear API rules, file formats there are effectively part of the kernel ABI and are changed only with great caution. debugfs deliberately knows no such promise, which gives developers freedom, but means scripts that parse debugfs files should be re-verified with every major kernel update.

In practice this means for admins: for ongoing automation and monitoring integration, /proc and /sys remain the more reliable source, debugfs stays the tool for a focused, deep dive diagnosis of a specific problem where the standard interfaces do not provide enough detail.

8. A practical workflow for a debugfs diagnostic session

A sensible flow starts by checking whether debugfs is mounted at all, followed by a targeted look into the relevant subdirectory, for example block for storage problems or tracing for function call questions. Changes to tracer settings should always be reverted after diagnosis, in particular resetting current_tracer back to nop, to avoid leaving behind permanent tracing overhead.

Since many debugfs files are writable and can influence direct kernel state, such as enabling a tracer or resetting counters, every write access belongs documented and, where possible, tried on a test system first before applying it to a production Magento hosting server.

9. Common mistakes when working with debugfs

A widespread mistake is assuming an empty or missing directory under /sys/kernel/debug automatically means the corresponding kernel feature does not exist. Often the cause is simply that debugfs is not mounted at all, or the associated driver was not loaded, which can be clarified quickly with a look at lsmod and the mount status.

Another typical stumbling block is trying to read debugfs files as an unprivileged user, which usually results in Permission denied, even if the exact same information would be open to all users under /proc. That is intended behavior, not a misconfiguration, which is why diagnostic scripts should generally run with root privileges or access the relevant file explicitly via sudo.

Finally, confusing debugfs paths across kernel versions leads to silent failures in scripts: a path that existed under kernel 5.10 may have been renamed or moved into a different subdirectory under kernel 6.x. A quick find call before any major kernel migration provides clarity here, before production monitoring scripts suddenly start returning empty output.

# Check whether the associated driver is even loaded
lsmod | grep nvme

# Test access as root instead of as an unprivileged user
sudo cat /sys/kernel/debug/tracing/current_tracer

10. Summary

11. FAQ: debugfs