Safely recovering damaged ext4 and XFS filesystems
An unclean shutdown or a power outage is enough to leave a filesystem with inconsistent metadata. fsck detects and repairs such errors, but only if you follow the workflow, unmounting, checking read-only, and repairing only afterward, in the right order.
Table of Contents
- 1. What fsck is for and when a check is needed
- 2. Why the filesystem must be unmounted first
- 3. Running fsck: read-only, interactive, automatic
- 4. Scheduling a check for the next reboot
- 5. XFS specifics: xfs_repair instead of fsck
- 6. Understanding lost+found and identifying fragments
- 7. Superblock problems and using backup superblocks
- 8. Preventing errors: journaling, UPS, clean shutdowns
- 9. fsck tools in direct comparison
- 10. Summary
- 11. FAQ
1. What fsck is for and when a check is needed
fsck (File System Check) is the central Linux tool for checking and repairing filesystem metadata. It compares a filesystem's internal bookkeeping, which blocks are allocated, which inodes belong to which files, which directory entries exist, against the actual state on disk and corrects any discrepancies. Such discrepancies almost always arise from an unclean shutdown, a power outage in the middle of a write, or a kernel crash where write operations were not fully completed.
Modern filesystems like ext4 and XFS use journaling to cushion exactly these situations, the journal logs metadata changes before they are actually written and can be automatically replayed after a crash. A full fsck run is still necessary, however, when the journal itself is damaged, when a hardware fault has left bad sectors, or when a filesystem is due for a routine scheduled check after many mount cycles anyway. Understanding the difference between automatic journal recovery and a full fsck check also tells you when manual intervention is really needed.
2. Why the filesystem must be unmounted first
The most important rule when working with fsck is: never check or repair a filesystem while it is mounted and actively being written to. If fsck runs alongside the application's write access, it sees a constantly changing state and makes repair decisions based on data that may already look different the next moment, which can make existing damage worse instead of fixing it.
# Never run fsck on a mounted, actively written filesystem
sudo umount /dev/sdb1
# For the root filesystem, boot from a rescue medium or single-user mode instead
# systemctl rescue (schedules a check-only run on next reboot, see below)
For data partitions, unmounting is unproblematic as long as no processes still access them (lsof +D /mount/point shows open handles). For the root filesystem itself, unmounting while the system is running is naturally not possible, here the only options are a rescue system, a live medium, or a check scheduled for the next reboot, as described in the section after next.
3. Running fsck: read-only, interactive, automatic
After unmounting, three useful operating modes are available. Read-only mode with -n reports found problems without changing anything, this is the safe first step to gauge how badly a filesystem is affected. Interactive mode asks for confirmation on every repair decision, fully automatic mode with -y accepts every suggested fix without asking.
# Read-only check, reports problems without changing anything
sudo fsck -n /dev/sdb1
# Automatic check-and-repair for routine cases (interactive prompts allowed)
sudo fsck -f /dev/sdb1
# Fully automatic repair, accepts all suggested fixes without asking
sudo fsck -y /dev/sdb1
# ext4-specific: force a full check even if the filesystem looks clean
sudo e2fsck -f /dev/sdb1
In practice, a three step approach almost always pays off: first -n for diagnosis, read the result carefully, and only then -y for the actual repair, ideally after a recent backup exists or at least a disk image was taken with dd. On ext4, fsck internally always calls e2fsck, calling e2fsck directly provides the same options and sometimes more detailed error messages.
4. Scheduling a check for the next reboot
For the root filesystem, which cannot be unmounted while running, a check scheduled for the next reboot is the way to go. Classically this happens through an empty file named /forcefsck in the root directory, which the init process detects on the next boot and then automatically triggers an fsck run on all relevant filesystems before they are mounted regularly.
# Schedule a filesystem check on the next reboot (systemd systems)
sudo touch /forcefsck
sudo reboot
# Alternative on systemd: request a check without a stray marker file
sudo systemctl reboot --force-fsck
# ext4: check how many mounts remain until the automatic periodic check
sudo tune2fs -l /dev/sdb1 | grep -i "mount count"
On systemd, systemctl reboot --force-fsck provides a clean, explicit way to do this without manually creating a marker file. In addition, ext4 filesystems already check automatically by default after a certain number of mounts or after a time interval has elapsed, visible via tune2fs -l, this behavior can be adjusted with tune2fs -c and -i, or disabled if needed.
5. XFS specifics: xfs_repair instead of fsck
A common misunderstanding: fsck.xfs exists as a program, but only performs a trivial pre-mount check and reports that a different tool handles actual repairs. The real checking and repairing of XFS filesystems is handled by xfs_repair, a standalone tool that works differently from fsck on ext4.
# XFS uses a dedicated repair tool instead of fsck.xfs for actual repairs
# fsck.xfs itself only performs a trivial pre-mount sanity check
# Unmount first, then run a full structural check and repair
sudo umount /dev/sdc1
sudo xfs_repair /dev/sdc1
# Dry run: report problems without writing any changes
sudo xfs_repair -n /dev/sdc1
xfs_repair fundamentally works in two phases: first it tries to bring the filesystem into a consistent state using the journal (log replay), only when that is not sufficient does a full structural check follow. Unlike ext4, XFS has no interactive mode with individual confirmations, xfs_repair makes repair decisions automatically, which simplifies the workflow but also means a recent backup is especially important before running it.
6. Understanding lost+found and identifying fragments
If fsck or xfs_repair finds file fragments that can no longer be attached to a directory entry, for example because the directory entry itself was damaged, the tool moves these fragments into the lost+found directory at the root of the respective filesystem. There, the files only carry their inode number as a name, the original filename is lost, but the content is usually still completely intact.
# Inspect recovered fragments after a repair moved them to lost+found
ls -la /mount/point/lost+found
# Identify file types of recovered, unnamed inodes
file /mount/point/lost+found/*
# Cross-check against a recent backup to restore correct filenames
diff -rq /mount/point/lost+found /backup/latest/
To meaningfully identify recovered fragments, file helps with a rough type determination (text file, image, archive), and for known formats often a look at the content itself. If a recent backup exists, diff -rq can often quickly reveal which file from the backup corresponds to a given fragment in lost+found, and the original name can be restored.
7. Superblock problems and using backup superblocks
The superblock is the most important metadata structure of a filesystem, it contains, among other things, the size, block size, and a pointer to the inode table. If the primary superblock is damaged, the kernel typically reports an unknown filesystem type when trying to mount, even though the filesystem is actually intact, ext4 creates several redundant backup superblocks at various, predictable positions for exactly this reason.
Running mke2fs -n /dev/sdb1 shows the list of all backup superblock positions of a filesystem without actually formatting anything. The command fsck -b 32768 /dev/sdb1 tells fsck to use one of the intact backup superblocks instead of the damaged primary one, in most cases the filesystem can be fully repaired this way, even if the mount attempt previously failed completely.
8. Preventing errors: journaling, UPS, clean shutdowns
The best repair is the one that is never needed. Journaling on ext4 and XFS already significantly reduces the frequency of serious filesystem errors, but does not replace an uninterruptible power supply (UPS) on physical servers, which enables a controlled shutdown during a power outage instead of letting the system crash hard.
On virtual servers and in cloud environments, the risk of an unclean shutdown due to host side maintenance or migration is real, here a properly configured systemctl poweroff or reboot instead of a hard reset helps, wherever that is possible. In addition, smartctl -a as a regular check is worthwhile, to see whether the underlying hardware already shows signs of an impending failure before it turns into a filesystem error.
9. fsck tools in direct comparison
Depending on the filesystem and situation, a different tool is used. The following overview classifies the most important options.
| Situation | Wrong approach | Correct tool | Benefit |
|---|---|---|---|
| Checking ext4 without changes | Running fsck -y directly | fsck -n /dev/sdX | Diagnosis without risk |
| Repairing XFS | Expecting fsck.xfs to repair it | xfs_repair /dev/sdX | Only tool for real XFS repair |
| Checking the root filesystem | Starting fsck while running | /forcefsck + reboot | Safe check without active mounting |
| Damaged superblock | Writing off the filesystem as lost | fsck -b 32768 /dev/sdX | Uses an intact backup superblock |
| Identifying recovered files | Ignoring lost+found | file + diff against backup | Original names can be reconstructed |
In practice, the biggest mistake is running fsck or xfs_repair unprepared on a mounted or actively written filesystem. Unmount first, check read-only, repair only afterward, that is the order that keeps a fixable problem from turning into a complete data disaster.
Mironsoft
Linux server administration, data recovery and filesystem maintenance
Damaged filesystem and no time for experiments?
We handle the safe diagnosis and repair of damaged ext4 and XFS filesystems on your production servers, including superblock recovery and identifying fragments from lost+found.
Safe diagnosis
Read-only check to assess the damage before anything gets written
Repair without data loss
Using fsck, e2fsck and xfs_repair in a controlled way, including backup superblocks
Prevention
Monitoring, UPS concepts and clean shutdown behavior against future damage
10. Summary
fsck and its XFS counterpart xfs_repair are the central tools for detecting and repairing damaged Linux filesystems. The right order matters: unmount the filesystem first, check it read-only, and only then proceed with a full repair, ideally with a recent backup as a safety net.
Superblock problems can almost always be solved via one of the redundant backup superblocks, recovered fragments end up in lost+found and can usually be correctly renamed again using file and a comparison against a backup. Prevention through journaling, a UPS, and clean shutdowns remains the best strategy nonetheless.
fsck and Filesystem Repair: The essentials at a glance
Order of operations
Unmount first, then fsck -n for diagnosis, only afterward fsck -y for the actual repair.
XFS is different
fsck.xfs only checks superficially, real repairs run exclusively through xfs_repair.
lost+found
Recovered fragments only carry their inode number, file and a backup comparison help with identification.
Rescuing the superblock
fsck -b uses an intact backup superblock when the primary superblock is damaged.