Why disk space is free and the server is still full
A server can appear completely full even though df -h still shows plenty of free disk space, because it is not the disk blocks but the number of available inodes that has run out. This article explains what an inode stores, how to reliably diagnose inode exhaustion with df -i, and how PHP applications generating millions of small cache and session files can avoid hitting the limit in the first place.
Table of Contents
- 1. What an inode actually stores
- 2. df -i vs. df -h: disk space is not inode count
- 3. Why a filesystem fills up despite free disk space
- 4. Diagnosing and locating inode exhaustion
- 5. Common PHP causes: sessions, cache, and tmp files
- 6. Magento- and Hyvä-specific inode traps
- 7. Setting the inode count when creating a filesystem
- 8. Immediate fixes and long-term strategies
- 9. Filesystem limits compared
- 10. Summary
- 11. FAQ
1. What an inode actually stores
Every file on a Linux filesystem consists of two separate parts: the filename, which exists only as an entry in a directory, and the inode, which stores the actual metadata. An inode holds owner, permissions, timestamps, file size, and pointers to the data blocks that contain the content, but explicitly not the filename itself. That is why a file can have multiple hard links with different names that all point to the same inode without the content being duplicated. ls -i shows the inode number of any file, and stat file returns all stored metadata in detail.
What matters for understanding capacity limits is that classic filesystems such as ext4 reserve a fixed number of inodes when the filesystem is created, independent of actual storage usage. This number is calculated from a fixed bytes-per-inode ratio and cannot be changed afterward without recreating the filesystem. Every file, every directory, and every symlink consumes exactly one inode, regardless of whether the file is one byte or ten gigabytes in size. It is precisely this property that causes a server generating mostly small files to run out of inodes long before it runs out of disk space.
2. df -i vs. df -h: disk space is not inode count
The command df -h shows how much block storage is used, while df -i shows exclusively the inode usage, meaning how many of the reserved inode slots are already allocated. Both figures are completely independent of each other because they measure different resources of the filesystem. A server can report forty percent usage in df -h and still be at one hundred percent in df -i if it has mostly been writing very small files. This exact contradiction is the most common sign of an impending or already occurring inode exhaustion.
In practice it pays to run both commands together routinely, because monitoring systems often watch only block usage, letting inode problems go unnoticed until the application halts with a cryptic error message. The IUse% column in the output of df -i directly corresponds to the Use% column of df -h, just for inodes instead of bytes. Anyone running Nagios, Zabbix, or the Prometheus Node Exporter should check whether inode metrics are already configured as their own alert channel instead of relying solely on disk space usage.
# Compare block usage and inode usage on the same filesystem
$ df -h /var
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 19G 29G 40% /var
$ df -i /var
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 3276800 3276800 0 100% /var
# 40% block usage but 100% inode usage: classic inode exhaustion signature
3. Why a filesystem fills up despite free disk space
As soon as a filesystem's inode table is completely allocated, the kernel refuses to create any further file with the error message No space left on device, even though plenty of free disk space is still available. For application developers and administrators this message is misleading, because it is the exact same one that appears when disk space is genuinely full. Without a targeted look at df -i, the real cause often stays hidden for hours while deployments fail, sessions can no longer be written, and cron jobs abort with write errors.
Systems that generate large volumes of very small files are particularly vulnerable: mail servers using the Maildir format, build systems with thousands of intermediate files, and above all PHP applications with file-based session and cache handling. A single directory with several million empty or few-byte files is enough to fully exhaust a standard ext4 filesystem with a typical inode ratio, even though the occupied disk space barely registers. The cause is almost always missing cleanup, not genuine data growth.
4. Diagnosing and locating inode exhaustion
The first step in any inode diagnosis is df -i, to determine which filesystem is affected and how close it is to its capacity limit. After that, you need to identify which directory is responsible for the bulk of the inode usage. Since there is no direct du equivalent for inode counts, you fall back on find combined with wc -l, or on specialized tools such as ncdu, which can be started in file-count mode.
A proven method is to iterate recursively over the top-level directories and count the number of files contained in each one to narrow down the main offender. This process can take several minutes with millions of files, because every count requires a full directory traversal that does not run in constant time on ext4. On production systems it is advisable to throttle the diagnosis with ionice and nice so the analysis does not add extra load to running operations.
#!/usr/bin/env bash
# Find directories with the highest inode (file) count under /var
set -euo pipefail
for dir in /var/*/; do
count=$(find "$dir" -xdev -type f 2>/dev/null | wc -l)
printf "%10d %s\n" "$count" "$dir"
done | sort -rn | head -n 10
# Drill deeper into the top offender, throttled to avoid I/O pressure
ionice -c3 nice -n19 find /var/www/html/var/session -xdev -type f | wc -l
5. Common PHP causes: sessions, cache, and tmp files
In its default configuration, PHP writes every session as an individual file to session.save_path, usually /var/lib/php/sessions or /tmp. Under heavy traffic without working garbage collection, millions of orphaned session files accumulate there quickly, because session.gc_probability and session.gc_divisor are configured conservatively on many distributions, or the cron-based cleanup simply does not run. Each of these files is often only a few hundred bytes, so it barely affects disk space, but it still consumes a full inode.
The same applies to file-based opcode and object caching, as well as log rotation schemes that create new files instead of appending. Frameworks such as Symfony and Magento generate tens of thousands of cache files in development mode or under misconfiguration, which keep growing without rotation. Composer and npm installations with deeply nested node_modules trees also generate disproportionately many small files relative to their total size and can noticeably contribute to inode usage on CI systems running many parallel builds.
; php.ini: aggressive session garbage collection to prevent inode exhaustion
[Session]
session.save_handler = files
session.save_path = "/var/lib/php/sessions"
session.gc_probability = 1
session.gc_divisor = 100
session.gc_maxlifetime = 1440
; Better: move sessions off the filesystem entirely
session.save_handler = redis
session.save_path = "tcp://127.0.0.1:6379?database=2"
6. Magento- and Hyvä-specific inode traps
In Magento stores, var/cache, var/page_cache, var/session, and var/log are the classic candidates for uncontrolled file growth. In particular, the file-based session handler on stores with high guest traffic and no Redis integration continuously creates new session files that are never deleted without active garbage collection. Configuring Redis as the session and cache backend instead removes this inode consumption entirely, because Redis manages keys in memory rather than as individual files on the filesystem.
A second common case is the Full Page Cache in filesystem mode, which creates a separate file for every cached page variant. With large catalogs offering many filter combinations, this can generate several hundred thousand files in a short time. Switching to Varnish or Redis as the cache backend via app/etc/env.php solves this structurally, while bin/magento cache:flush alone only clears the content but does not prevent the pattern from repeating.
{
"cache": {
"frontend": {
"default": {
"backend": "Cm_Cache_Backend_Redis",
"backend_options": { "server": "127.0.0.1", "port": "6379", "database": "0" }
},
"page_cache": {
"backend": "Cm_Cache_Backend_Redis",
"backend_options": { "server": "127.0.0.1", "port": "6379", "database": "1" }
}
},
"session": {
"save": "redis",
"redis": { "host": "127.0.0.1", "port": "6379", "database": "2" }
}
}
}
7. Setting the inode count when creating a filesystem
If you already know that a filesystem will hold many small files, you should plan the inode count deliberately when creating it instead of relying on the default. On ext4, mkfs.ext4 sets the inode count via the -i option, which specifies the bytes-per-inode ratio, or directly via -N, which sets the absolute number of inodes. The default is one inode per 16 kibibytes, which is clearly too little for servers with many small files such as session stores or mail directories.
XFS takes a different approach and allocates inodes dynamically on demand by default, making fixed limits less likely to become a problem. With very large numbers of small files, the maxpct option can still act as a constraint, so it should be deliberately raised when a high file count is expected. Retroactively changing the inode count of an existing ext4 filesystem is not possible without recreating it, which is why this decision is ideally made during server provisioning. Running tune2fs -l /dev/sdb1 | grep -i inode at any time reveals the current inode capacity and remaining free inodes of an existing filesystem before a migration is due.
8. Immediate fixes and long-term strategies
In an acute emergency, when a filesystem already sits at one hundred percent inode usage, the only remedy is targeted deletion of files in the affected directory until free inodes become available again. Expired session files can be safely identified by their modification date and removed with find -mtime combined with -delete, without endangering active sessions. Before any mass deletion, a dry run without -delete is recommended to verify the affected file count before anything is actually removed.
In the long run, the most sustainable solution is to avoid placing files that are inherently short-lived and numerous on the persistent filesystem at all. A tmpfs mount for session directories keeps these files in memory, where inode limits are far more generous and generally tied to available RAM size rather than a fixed number. In addition, a regular cron job or a systemd-tmpfiles entry ensures that orphaned files are automatically removed after a defined lifetime instead of accumulating indefinitely.
# docker-compose.yml: move PHP session storage off the persistent filesystem
services:
php-fpm:
image: mironsoft/php:8.4-fpm
volumes:
- app_code:/var/www/html
tmpfs:
# Sessions live in RAM-backed tmpfs: no persistent inode consumption
- /var/www/html/var/session:size=512m,uid=1000,gid=1000,mode=1777
environment:
PHP_SESSION_SAVE_PATH: /var/www/html/var/session
volumes:
app_code:
9. Filesystem limits compared
Depending on the filesystem, use case, and configuration, inodes either become a bottleneck or never draw attention at all. The following overview summarizes the most common scenarios and the recommended action for each.
| Scenario | Problem | Recommended action | Effect |
|---|---|---|---|
| ext4 with millions of small files | Inode exhaustion despite free disk space | mkfs.ext4 -N or a smaller -i value |
Sufficient inodes reserved from the start |
| PHP sessions without Redis | Millions of orphaned session files | Redis session handler instead of filesystem | Inode consumption stays constant at zero |
| Monitoring with df -h only | Inode bottleneck goes unnoticed | df -i as its own alert channel | Root cause identified within seconds |
| XFS without capacity planning | Dynamic inodes without a defined limit | Set and monitor maxpct deliberately |
Flexibility without later resizing |
| Cache directory without rotation | Unchecked file growth up to the limit | logrotate / tmpfiles.d with age-based cleanup | Predictable ceiling instead of surprise |
It is striking that almost every row in the table traces back to the same underlying principle: inodes are a limited, mostly invisible resource that only becomes visible once it is already exhausted. Anyone who routinely checks df -i and consistently decouples applications from uncontrolled small-file growth avoids the entire catalog of failures listed above.
Mironsoft
Server administration, capacity planning, and Magento infrastructure
Ready to solve and prevent inode problems for good?
We analyze your server infrastructure, identify inode traps in PHP and Magento applications, and set up Redis session handling, log rotation, and filesystem capacity so that No space left on device becomes a thing of the past.
Inode audit
df -i monitoring, diagnosis of the largest directories, and early warning
Redis migration
Move session and cache handling from files to Redis
Server provisioning
Plan filesystems with the right inode capacity from the start
10. Summary
The key insight about inodes and filesystem limits is this: disk space and inode count are two completely separate resources that must be monitored separately. df -h never shows inode exhaustion; only df -i delivers the relevant figure. Millions of small files, such as those generated by PHP sessions, file-based caches, and unrotated logs, can push a filesystem to its limit long before the actual disk space runs out. The error message No space left on device is identical whether blocks or inodes have been exhausted, which makes diagnosis harder without a targeted check.
Effective countermeasures address three areas: deliberate inode planning during filesystem creation with mkfs -N or -i, consistently moving short-lived small files to tmpfs or Redis instead of the persistent filesystem, and automated cleanup via logrotate or systemd-tmpfiles. Anyone who combines these three measures and adds df -i permanently to monitoring prevents a seemingly empty server from suddenly refusing to accept any new files.
Inodes and filesystem limits, the essentials at a glance
Inode vs. disk space
Every file consumes exactly one inode, regardless of its size. Millions of small files exhaust inodes long before they exhaust disk space.
df -i as a mandatory check
df -h shows no inode usage. Only df -i reveals inode exhaustion before it causes an outage.
PHP sessions & cache
Redis instead of file-based session and cache handling removes the biggest inode consumer entirely.
Planning & cleanup
mkfs -N for sufficient inodes, tmpfs for short-lived files, logrotate for automated cleanup.