Emergency Response for a Full Disk on Linux

1. The first seconds: no panic deletion

When a disk is full and applications start crashing with errors like No space left on device, the impulse to immediately delete something is understandable, but dangerous. Deleting files at random without first knowing what they are needed for can corrupt a running database transaction, destroy an active backup, or remove a configuration file that is then missing. The first step with a full disk is therefore always observation before action.

A second common mistake is deleting a file that is still held open by a running process. On Linux, the operating system only releases the disk space of a deleted file once no process still holds an open file handle to it. If a large log file is deleted while a daemon is still writing to it, the disk space stays occupied until the process is restarted, and the full disk problem appears to persist unchanged, causing additional confusion.

2. df -h for the quick overview

The first concrete command with a full disk is df -h, which shows the usage of all mounted filesystems in human-readable units. It is important to check every line, not just the root partition: a separate mount for /var or /var/log can be full while / itself still has capacity, and vice versa. On Magento servers with a separate mount for pub/media in particular, this distinction is critical for the further search.

df -hT additionally shows the filesystem type, which helps in mixed setups with tmpfs, overlay mounts for Docker containers and regular ext4 or XFS partitions to distinguish real physical fullness from temporary RAM-backed mounts. A tmpfs mount that fills up has a different cause and different consequences than a full physical disk and must not be mixed up in the diagnosis.

# Step 1: check every mounted filesystem, not just root
df -hT
# Filesystem     Type   Size  Used Avail Use% Mounted on
# /dev/sda1      ext4    50G   49G  512M  99% /
# /dev/sdb1      xfs    200G  198G  1.8G  99% /var/lib/mysql
# tmpfs          tmpfs  8.0G  120M  7.9G   2% /tmp

# /var/lib/mysql is the actual bottleneck here, not root

In this example the actual problem is not the root partition but the separate mount for /var/lib/mysql, which is also nearly full. Anyone looking only at / overlooks the more critical partition and risks MySQL failing shortly after with a write error, even though the disk as a whole appeared to still have reserves.

3. Finding the biggest consumers with du and ncdu

Once it is clear which mount is actually full, the search for the biggest consumer follows. du -h --max-depth=1 /var | sort -rh shows the largest subdirectories at one level, sorted by size. The approach is repeated recursively: if /var/log is the largest consumer, du -h --max-depth=1 /var/log follows to narrow it down further within that directory, until the specific file or log is found.

For a faster, interactive search, ncdu is well suited, a text-based tool that scans a directory once and then makes it navigable via arrow keys, without repeated du calls. With an acutely full disk, this saves considerable time, because you click your way interactively from the root to the largest consumer instead of running a new command manually at every step.

# Recursive drill-down with du, one level at a time
du -h --max-depth=1 /var | sort -rh | head -10
# 18G   /var/log
# 4.2G  /var/lib/docker
# 1.1G  /var/cache

du -h --max-depth=1 /var/log | sort -rh | head -10
# 14G   /var/log/mysql
# 3.1G  /var/log/nginx

# Interactive drill-down — faster for repeated exploration
ncdu /var/log

4. Deleted but still open files via lsof

A frequently overlooked special case with a full disk: du shows occupied space, but large files can no longer be found in the filesystem directory tree because they were already deleted while a process still holds them open. The kernel only releases the disk space once the last file handle is closed, regardless of whether the file still exists from the filesystem's point of view.

lsof +L1 specifically lists files with a link count below one, i.e. deleted but still open files, together with their size and the holding process. This is extremely common with log rotation failures: a logrotate run renames or deletes a log file, but the writing daemon keeps writing merrily into the already deleted, but still space-occupying file through its old file handle.

# Find deleted-but-still-open files holding disk space
lsof +L1
# COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
# mysqld   1842   mysql   5w   REG  253,0  8.9G      0 4211 /var/log/mysql/slow.log (deleted)

# Reclaim the space by restarting or signaling the process to reopen its log file
kill -HUP 1842

The fix in this case is rarely to hard-kill the process. Many daemons, including mysqld and nginx, respond to SIGHUP by reopening their log file handles, which releases the deleted but blocked file immediately, without restarting the service. This technique is at the core of any emergency workflow for a full disk, because it often frees several gigabytes within seconds without risking an outage in production.

5. Safe cleanup: logs, caches, temp files

After identifying the biggest consumer, the actual cleanup follows, but here too the rule applies: targeted rather than blanket. Old, already rotated log files under /var/log with extensions like .1, .gz or .old can generally be compressed or deleted without risk, because logrotate has already marked them as complete. Composer and npm caches, temporary Docker build layers, and outdated kernel images on apt-based systems are further common, low-risk candidates for quick cleanup with a full disk.

Caution is required with anything currently being written: active session files under /var/lib/php/sessions, running backup archives, or database binlogs still needed for replication. Before any deletion on a production system, a quick lsof check on the target directory is worthwhile to make sure no active process is currently using exactly that file.

# Safe, low-risk cleanup candidates on most Debian/Ubuntu servers
apt-get clean                                    # cached .deb packages
journalctl --vacuum-size=200M                    # cap systemd journal size
find /var/log -name "*.gz" -mtime +30 -delete    # old rotated logs
docker system prune -af --volumes                # unused Docker images/layers
composer clear-cache

6. When df still shows free space: inode exhaustion

A lesser-known special case of a full disk: df -h shows free space, yet the system still reports No space left on device. In this case it is not byte-level disk space that is exhausted, but the number of available inodes. Every filesystem reserves a fixed number of inodes at creation time, and every file, no matter how small, consumes exactly one. Directories with millions of tiny files, such as PHP session files or mail queues, exhaust inodes long before space itself runs low.

df -i shows inode usage analogous to df -h for bytes. If IUse% reads 100 percent there, the number of files must be reduced, not their size. The fix is usually to consistently clean up stale session files or queue entries; long-term, a separate filesystem for such directories with an appropriately sized inode count at reformatting time helps.

7. Database special case: binlogs and tablespaces

On MySQL and MariaDB servers, a particularly common cause of a full disk is an unboundedly growing binlog history. If expire_logs_days or binlog_expire_logs_seconds is not set, binlog files accumulate indefinitely, especially when a replication slave has been offline for a while and the master retains logs for it. PURGE BINARY LOGS BEFORE with a concrete date cleans up here in a targeted way, without endangering active replication.

A second database-specific case is InnoDB tablespace growth from deleted but not reclaimed data regions. Without innodb_file_per_table, all tables grow inside a shared ibdata1 file that does not automatically shrink after large tables are dropped. A clean dump-and-reimport into a freshly created database with the per-table file option enabled is often the only sustainable fix for a recurring full disk.

8. Prevention: monitoring, quotas, log rotation

The best way to deal with a full disk is to never let it happen in the first place. A monitoring alert at 80 percent usage gives enough lead time to act calmly instead of improvising during an acute full state. Tools like Prometheus with the node_exporter, or a simple cron script with df evaluation and email dispatch, already fulfill this purpose without much effort.

A properly configured logrotate rule with maxsize, rotate and compress prevents individual log files from growing without bound. For directories with user-generated data, for example upload folders in Magento, filesystem quotas limit the maximum possible consumption per user or directory and prevent a single misbehaving process from filling the entire disk.

9. Causes and remedies compared

The following table maps the most common causes of a full disk to the appropriate immediate remedy and the long-term prevention step, so that the right trail is followed quickly during an emergency.

The table makes it clear: a full disk rarely has a single cause. In practice, several of the listed factors often combine, for example unrotated Nginx logs together with growing MySQL binlogs, which is why a complete diagnosis should always go through every layer from filesystem to application.

10. Summary

A full disk demands a fixed order of operations: first observe with df -hT, then narrow down with du or ncdu, remembering to check deleted but open files with lsof +L1, and only then delete in a controlled manner. Special cases like inode exhaustion and growing database binlogs require their own diagnostic paths that a plain df -h look does not reveal.

The most sustainable protection against a recurring full disk is prevention: timely monitoring at 80 percent usage, consistent log rotation, and automated database housekeeping jobs. Anyone who sets up these three building blocks properly once turns a recurring emergency into a plannable maintenance topic.

11. FAQ: Full disk on Linux