Monitoring Filesystem Events With inotify and fanotify

1. Why polling falls short

A classic approach to monitoring a directory is scanning its contents at regular intervals through a cron job or script and comparing against the previous state. This approach wastes CPU time even when nothing changed, and delivers at best a reaction time in the order of the scan interval, which is unsuitable for time critical use cases such as instantly detecting a tampered configuration file.

The Linux kernel solves this with two complementary interfaces: inotify, available since kernel 2.6.13, reports changes to individual files and directories as events to a waiting process, while the more powerful fanotify, since kernel 2.6.36 and fully since 2.6.37, can additionally make access decisions and monitor entire mounts instead of individual paths.

2. inotify basics: watches, events, and inotifywait

inotify works with so called watches, each set up for a specific file or directory, delivering events such as IN_MODIFY, IN_CREATE, IN_DELETE, or IN_MOVED_TO to the watching process. Every watch refers explicitly to one path, which means monitoring a directory tree with many subdirectories requires registering a separate watch for every single subdirectory, since inotify has no native recursion.

For shell scripts, inotifywait from the inotify-tools package is the most practical tool, wrapping the low level inotify API in a simple command line interface that can be used directly in bash scripts without writing any custom C code.

# Install inotify-tools
apt install -y inotify-tools

# Watch a directory for change events, printing one line per event
inotifywait -m -e modify,create,delete,move /etc/nginx/

3. fanotify basics and how it differs from inotify

fanotify was originally built for antivirus software and access control and, unlike inotify, can not only report events but also actively decide, via so called permission events, whether an access is allowed to happen at all before the actual operation completes. That mode, however, requires root privileges or the CAP_SYS_ADMIN capability, while plain inotify watches are usable by unprivileged processes as well.

Another key difference lies in scope: fanotify can monitor an entire mount point or a whole filesystem at once with the FAN_MARK_MOUNT or FAN_MARK_FILESYSTEM flag, instead of having to explicitly register individual paths the way inotify does, which makes fanotify considerably more practical for monitoring large, dynamically changing directory structures.

# fanotify example using the fatrace CLI tool, shows access system wide
apt install -y fatrace

# Log every file access on the system live
fatrace

4. Understanding and tuning watch limits

Every registered inotify watch occupies kernel memory, which is why the kernel enforces a limit by default via fs.inotify.max_user_watches, sitting around 8192 watches per user on many distributions. Once that limit is exceeded, for example because a development tool or build system recursively watches every directory including node_modules, the kernel rejects further watch registrations with the ENOSPC error, which in practice often shows up as a cryptic error inside IDEs or file sync tools.

For servers monitoring many directories, such as configuration management tools or log aggregators, the limit should be raised noticeably. Equally relevant are fs.inotify.max_user_instances, which caps the number of concurrently open inotify instances per user, and fs.inotify.max_queued_events, which sets the event queue size per instance and can cause event loss on very active directories if set too low.

# Show current limits
sysctl fs.inotify.max_user_watches
sysctl fs.inotify.max_user_instances
sysctl fs.inotify.max_queued_events

# Raise limits permanently
cat >> /etc/sysctl.d/99-inotify.conf <<'EOF'
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 1024
fs.inotify.max_queued_events = 65536
EOF
sysctl --system

5. Practical example: detecting config changes live

A common use case on servers is reacting instantly to changes in security relevant configuration files, for example triggering an alert right away on an unexpected change to /etc/passwd or an nginx configuration, instead of waiting for the next scheduled audit run. A simple script using inotifywait in continuous mode is already enough for this and can easily run permanently as a systemd service.

For production use, it is worth having the script emit not just the raw event, but also context such as the changed filename and a timestamp into a log or straight to a monitoring system, so later forensic analysis stays traceable.

#!/usr/bin/env bash
# scripts/watch-config-changes.sh: report config changes instantly
set -euo pipefail

inotifywait -m -e modify,attrib,move,create,delete \
  /etc/passwd /etc/nginx/nginx.conf /etc/ssh/sshd_config |
while read -r path event file; do
  echo "$(date -Iseconds) ALERT: $event on $path$file" \
    | tee -a /var/log/config-watch.log
  # An additional alert could be sent to monitoring here
done

6. Practical example: monitoring an upload directory

For applications that need to process uploaded files asynchronously, such as generating thumbnails after a product image upload, event driven processing with inotify beats a periodic scan. Instead of scanning the entire upload directory every minute, processing is triggered exactly when a new file has been fully written.

Choosing the right event matters here: IN_CLOSE_WRITE only fires once a file is actually closed after being written, making it noticeably more reliable than IN_CREATE, which already fires when the file gets created, long before the actual upload has finished, potentially triggering processing of an incomplete file. Alternatively, the daemon tool incrond takes over this job in a configuration driven way, without a custom script having to run permanently.

# Only react once a file has been fully written and closed
inotifywait -m -e close_write --format '%f' /var/www/media/upload/ |
while read -r filename; do
  /usr/local/bin/generate-thumbnail.sh "/var/www/media/upload/$filename"
done

# Alternative with incrond: entry in /etc/incron.d/media-upload
# /var/www/media/upload IN_CLOSE_WRITE /usr/local/bin/generate-thumbnail.sh $@/$#

7. fanotify for access control and malware scanning

Through permission events, fanotify can decide whether a file access is actually allowed to happen before the underlying system call returns, a capability inotify fundamentally does not offer, since inotify events are always delivered only after the operation has already completed. Antivirus solutions rely on exactly this mechanism to scan files at the moment they are opened and block access on a match, before any malicious code can execute at all.

This capability, however, makes fanotify noticeably more complex to handle than inotify and requires careful timeout handling, since a hanging scan process would otherwise block all file access on the monitored mount entirely. For most administrative monitoring tasks that do not need active access control, inotify therefore remains the simpler and sufficient choice.

8. systemd path units as an operations friendly alternative

Anyone wanting to run inotify monitoring permanently and production ready should consider whether a systemd path unit fits better than a hand written long running script. A path unit declaratively defines which path to react to and starts an associated systemd service once triggered, relying on inotify internally as well, but fully managed by the init system.

The practical benefit lies in integration: restart on crash, structured logging through journalctl, and dependencies on other services can all be expressed with systemd's existing tooling, instead of rebuilding that functionality inside a custom bash script. For simple trigger patterns, such as reloading a service after a configuration change, a path unit is often the more robust and lower maintenance choice compared to a custom inotifywait loop.

# /etc/systemd/system/nginx-config-watch.path
[Unit]
Description=Watch nginx config for changes

[Path]
PathModified=/etc/nginx/nginx.conf

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/nginx-config-watch.service
[Unit]
Description=Reload nginx after config change

[Service]
Type=oneshot
ExecStart=/usr/sbin/nginx -s reload

9. Common pitfalls with recursive monitoring

The most common pitfall is inotify's missing native recursion: a newly created subdirectory is not automatically monitored, unless the script explicitly reacts to IN_CREATE events for directories and dynamically registers new watches. Tools like inotifywait -r solve this conveniently by recursively registering watches for the whole tree at startup, though that quickly runs into the watch limit on very large trees.

Another pitfall involves directories such as node_modules or large vendor directories, whose recursive monitoring occupies thousands of watches at once, using up the limit for other applications on the same server. For such cases, targeted, non recursive watches on the actually relevant directories are usually the better choice over blanket recursive monitoring of the entire tree.

10. Summary

11. FAQ: inotify and fanotify