Exploring sysfs: Kernel and Device Attributes Under /sys

1. What sysfs is and why it exists

sysfs is a virtual filesystem the Linux kernel has provided under /sys since version 2.6, in which every internally managed kernel object becomes visible as a directory or file. Instead of querying hardware information through proprietary ioctl calls or specialized libraries, you simply read a text file under /sys, often with cat, and get the current state back as a readable string. This uniformity makes sysfs one of the most important introspection interfaces for system administrators who want to check hardware state without additional software.

Technically, sysfs is based on the kernel's kobject subsystem, which manages every device, driver, bus subsystem and kernel module as a structured object with attributes. These objects live in kernel memory, but sysfs exports their attributes as files, turning standard Unix tools like cat, echo and find into a fully fledged management interface. For server administration this means: whoever can read sysfs understands directly from the kernel what hardware was detected and in what state it currently is.

2. sysfs versus procfs: a clear distinction

sysfs and the older procfs under /proc are often confused, because both are virtual filesystems that present kernel state as text files. The central difference lies in focus: procfs originally emerged to expose process information, PID directories, memory mappings, open file descriptors, and was later extended with general system values such as /proc/sys for sysctl tuning. sysfs, on the other hand, was designed from the ground up for a uniform, hierarchical representation of the device and driver model.

A practical distinguishing feature: anyone examining a network card, a USB stick or a CPU core as a physical or logical device finds the relevant attributes in sysfs under /sys/class or /sys/devices. Anyone examining a running process or setting a kernel wide tuning parameter via sysctl stays in procfs under /proc. Both interfaces complement each other but cover different aspects of kernel state, and sysfs is the more modern, more clearly structured of the two for everything related to devices.

3. Systematically exploring the /sys structure

The directory structure of sysfs follows a fixed schema with several top level directories, each offering a different view of the same kernel objects. /sys/devices contains the physical device hierarchy exactly as the kernel discovered it over buses such as PCI or USB. /sys/class groups devices by function, for example all network interfaces under net or all block devices under block, regardless of the physical bus.

# Top-level layout of sysfs
ls /sys
# block  bus  class  dev  devices  firmware  fs  kernel  module  power

# Explore a network interface by function (symlink into /sys/devices)
ls -l /sys/class/net/eth0
readlink -f /sys/class/net/eth0

# Explore the same device by its physical bus location
ls /sys/devices/pci0000:00/*/net/ 2>/dev/null

# List every loaded kernel module and its exposed parameters
ls /sys/module/ | head -10
ls /sys/module/nvme/parameters/

# Find all block devices the kernel currently knows about
ls /sys/class/block/

The key to understanding sysfs is that almost every directory under /sys/class is only a symlink into the actual device hierarchy under /sys/devices. Using readlink -f, any of these symlinks can be traced back to its physical bus path, which is especially useful on servers with multiple identical network cards or storage controllers, to uniquely identify a device.

4. Reading device attributes: power, storage, network

Every device in sysfs exports a set of attribute files that describe its current state or static properties. For network interfaces, these attributes show link speed, duplex mode and operational state, without requiring a separate diagnostic tool. For storage devices, they provide size, sector size and queue depth straight from the kernel's block layer.

# Network interface attributes exposed by sysfs
cat /sys/class/net/eth0/speed          # link speed in Mbit/s
cat /sys/class/net/eth0/duplex         # full or half duplex
cat /sys/class/net/eth0/operstate      # up, down, unknown
cat /sys/class/net/eth0/address        # MAC address

# Storage device attributes
cat /sys/class/block/sda/size          # size in 512-byte sectors
cat /sys/class/block/sda/queue/rotational   # 0 = SSD/NVMe, 1 = spinning disk
cat /sys/class/block/sda/device/model  # device model string reported by firmware

# Power management state of a PCI device
cat /sys/bus/pci/devices/0000:00:1f.6/power/control   # "on" or "auto"

# CPU frequency scaling attributes per core
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq

This kind of introspection through sysfs is especially valuable when troubleshooting performance problems. A suddenly slow network interface can immediately be narrowed down to an autonegotiation failure via speed and duplex, without installing an external network diagnostic tool. Likewise, rotational shows whether a supposed SSD storage array was actually detected as solid state or is mistakenly treated as a spinning disk, which directly affects the I/O scheduler choice.

5. Kernel module parameters at runtime via sysfs

Many kernel modules accept parameters that can either be set at load time through modprobe, or, for some parameters, even changed at runtime via sysfs under /sys/module/<name>/parameters/. Whether a parameter is changeable at runtime depends on how the driver developer declared it in the source code using module_param and the corresponding permission flags.

# Inspect current parameters of a loaded module
ls /sys/module/nvme_core/parameters/
cat /sys/module/nvme_core/parameters/multipath

# Check file permissions to see which parameters are writable at runtime
ls -l /sys/module/nvme_core/parameters/
# -r--r--r-- read-only, cannot change without reload
# -rw-r--r-- writable at runtime

# Change a runtime-writable parameter (example only, verify first!)
echo Y | sudo tee /sys/module/nvme_core/parameters/multipath

# Cross-reference with modinfo to see the parameter description
modinfo -p nvme_core | grep -A2 multipath

This ability to check module parameters via sysfs without reloading the module, and in many cases to adjust them, matters especially for storage and network drivers, where restarting the service or even the server would be undesirable. It is important to check the permission bits of every attribute file, because not every parameter is actually writable at runtime, some only display the value set at load time, read only.

6. Block device tuning via sysfs: scheduler and read-ahead

One of the most practically relevant uses of sysfs is tuning block devices for database and storage workloads. A device's I/O scheduler, its read-ahead size and its queue depth can all be changed directly through files under /sys/class/block/<device>/queue/, without a reboot and without specialized tools.

# Show available and currently selected I/O scheduler
cat /sys/class/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber

# Switch to a different scheduler for a spinning disk
echo bfq | sudo tee /sys/class/block/sda/queue/scheduler

# Tune read-ahead size (in 512-byte sectors) for sequential workloads
cat /sys/class/block/sda/queue/read_ahead_kb
echo 256 | sudo tee /sys/class/block/sda/queue/read_ahead_kb

# Inspect queue depth for an NVMe device
cat /sys/class/block/nvme0n1/queue/nr_requests

# Make a sysfs tuning value persistent via a udev rule
cat <<'EOF' | sudo tee /etc/udev/rules.d/60-io-scheduler.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/scheduler}="bfq"
ACTION=="add|change", KERNEL=="nvme[0-9]n[0-9]", ATTR{queue/scheduler}="none"
EOF

Changes through sysfs are transient and are lost on the next reboot unless they are made permanent through a udev rule as in the example. For database servers with heavy, random I/O load, switching from a scheduler optimized for spinning disks to none or mq-deadline on NVMe devices often delivers measurable latency improvements, which can be validated directly through sysfs.

7. udev and sysfs: how device events work together

udev, the userspace device manager, works closely with sysfs. When the kernel detects a new device, for example a plugged in USB stick, it first creates the corresponding entries in sysfs and then sends a uevent over the netlink socket to udev. udev reads the attributes out of sysfs, applies matching rules, and generates, for example, a stable device name under /dev/disk/by-id/.

This chain makes sysfs the actual data source behind every udev rule. Anyone writing a custom udev rule to, for example, automatically assign a specific I/O scheduler to all NVMe devices ultimately accesses the same attribute paths under sysfs that are also readable and writable manually with cat and echo. Understanding sysfs is therefore a prerequisite for not just copying udev rules but actually writing and debugging them yourself.

8. Caution when writing to sysfs: risks and practice

Not every write operation to sysfs is risk free. Some attributes trigger immediate hardware actions when written, such as disabling a network interface or ejecting a storage device, without prior confirmation or an undo option. On production servers, every change to sysfs should therefore first be verified on a test system before being applied in live operation.

# Always read the current value before writing a new one
cur=$(cat /sys/class/block/sda/queue/scheduler)
echo "Current scheduler setting: $cur"

# Prefer udev rules over one-off manual writes for anything permanent
# so the setting survives reboots and device re-enumeration

# Guard against writing to a non-existent or non-writable attribute
attr="/sys/class/net/eth0/mtu"
if [[ -w "$attr" ]]; then
  echo 9000 | sudo tee "$attr"
else
  echo "Attribute not writable, check permissions or driver support" >&2
fi

# Document every manual sysfs change in a runbook, including the
# exact path, old value and new value for later rollback

Another risk: attribute paths under sysfs are not guaranteed to be stable across kernel versions. A script that hardcodes a specific path can suddenly run into nothing after a kernel version upgrade, because the structure of a driver changed. That is why every automation around sysfs should include an existence check of the path before writing, plus error handling for the case that the structure has changed.

9. sysfs compared to other introspection interfaces

sysfs does not stand alone, it competes with and complements several other ways of querying kernel and hardware state. The choice of the right interface depends on the concrete use case.

For everyday server operations, sysfs remains the most accessible and best standardized of these interfaces. It requires no special tools, works with every standard Unix command, and directly reflects the structure that udev and many monitoring tools also read in the background.

10. Summary

sysfs maps the internal object structure of the Linux kernel as a readable file hierarchy under /sys, making device attributes, driver parameters and kernel objects accessible without special tools. Unlike procfs, which primarily deals with processes and generic sysctl values, sysfs focuses on the device and driver hierarchy and thereby forms the foundation on which udev applies its rules as well.

For production use, two things matter most: changes through sysfs are transient by default and must be made permanent through udev rules, and not every write operation is risk free, which is why testing on a staging system before every production change remains mandatory. Anyone following these rules gains, with sysfs, direct, tool-free access to the actual hardware and kernel state of a server.

11. FAQ: Exploring sysfs