Troubleshooting DNS Resolution Problems on Linux

1. Why DNS problems are rarely the DNS server's fault

A DNS problem usually shows up as "the website is sometimes unreachable" or "the database connection fails sporadically", even though the target server itself is demonstrably running. The instinctive first reaction is often to suspect the DNS server itself, but in practice the cause lies in the overwhelming majority of cases in the local resolution chain on the affected Linux host: a misconfigured resolv.conf, a stuck local resolver cache, or a contradictory nsswitch.conf. The actual authoritative DNS server or upstream resolver is almost always the last place to look, not the first.

This misjudgment costs valuable time in practice, because teams reflexively test with dig @8.8.8.8 against an external resolver, get a clean result, and wrongly conclude that DNS overall is working. The problem here: a direct test against an external server bypasses exactly the local configuration layer where the actual DNS problem resides. The correct order therefore always starts with the local configuration and only works outward from there.

2. The resolution chain: nsswitch.conf, resolv.conf and NSS

Before a DNS packet even leaves the system, the Name Service Switch configuration in /etc/nsswitch.conf decides which sources are queried in which order for name resolution. The line hosts: files dns means that /etc/hosts is checked first and only then DNS. A commonly overlooked mistake: a stale, manually entered entry in /etc/hosts completely overrides any DNS answer, without any error or warning becoming visible, because files comes before dns in the chain and stops the search immediately on a hit.

If /etc/hosts looks fine, the next step is /etc/resolv.conf, which contains the actual nameservers as well as options like timeout, attempts, and search. On modern systems with systemd-resolved, this file is often just a symlink to /run/systemd/resolve/stub-resolv.conf, which points to the local stub resolver 127.0.0.53 instead of naming the actual upstream servers directly. Anyone who doesn't know this and tries to manually change the nameservers in resolv.conf will find them overwritten on the next restart of systemd-resolved and be puzzled by seemingly randomly vanishing changes.

# Which sources are consulted, and in which order?
cat /etc/nsswitch.conf | grep ^hosts

# Is resolv.conf a real file or a systemd-resolved symlink?
ls -la /etc/resolv.conf

# If it is a symlink to the stub resolver, the real upstream servers
# are configured elsewhere — check systemd-resolved directly instead
readlink -f /etc/resolv.conf

# Check for a stale manual entry in /etc/hosts overriding DNS entirely
grep -v '^#' /etc/hosts | grep -v '^$'

3. Testing resolution in isolation with dig and drill

dig is the most precise tool for testing DNS resolution in isolation from the application, but that very isolation is also its biggest trap. dig example.com without further options uses the nameservers configured in /etc/resolv.conf and thus goes through the same chain as a normal application, while dig @1.1.1.1 example.com bypasses the local resolver entirely and only tests reachability of Cloudflare's resolver. For a reliable diagnosis, you need to run both variants and compare the results, because only a difference between the two clearly shows whether the problem is local or external.

The response time in the dig output (Query time) is also informative: values above 100 milliseconds for what should be a local resolver point to a forwarding problem or an overloaded upstream connection. The +trace flag additionally shows the complete resolution path from the root servers down to the authoritative zone and is especially helpful when you suspect a broken delegation between DNS zones rather than a local problem.

# Resolve through the locally configured resolver (same path as applications)
dig example.com

# Bypass the local resolver entirely — tests only the upstream server
dig @1.1.1.1 example.com

# Full resolution path from root servers down to the authoritative zone
dig +trace example.com

# Query time and answer section side by side for quick comparison
dig +noall +answer +stats example.com

4. Understanding systemd-resolved and checking its status

On most modern distributions, systemd-resolved runs as a local caching resolver and stub server on 127.0.0.53:53, which accepts requests, caches them, and forwards them to the actually configured upstream servers. The command resolvectl status shows the DNS servers, search domains, and current mode (such as DNS-over-TLS) configured per network interface, making it the central first stop for any DNS problem on a systemd-based system. A common finding: two network interfaces with conflicting DNS servers, for example a VPN interface trying to enforce its own internal nameservers while the physical interface still points to the public resolvers.

resolvectl statistics additionally provides cache hit ratios and failure counters showing whether a high proportion of requests actually time out or get answered with SERVFAIL. For persistent problems, a targeted cache flush with resolvectl flush-caches helps remove a potentially incorrectly cached negative entry before searching further towards the network or the application.

# Per-interface DNS servers, search domains, and current mode
resolvectl status

# Cache hit ratio and failure counters since the last restart
resolvectl statistics

# Force-flush the local resolver cache — useful after fixing a
# misconfigured upstream server or a stale negative cache entry
sudo resolvectl flush-caches

# Confirm the stub resolver is actually listening
sudo ss -tulpn | grep ':53'

5. Why applications resolve differently than dig

A successful dig call does not prove that a specific application gets the same result, because many runtimes bring their own DNS implementation that only partially respects the system configuration. PHP with cURL enabled, for example, uses the glibc resolver functions by default and thus the same chain as dig, but Java applications often cache successful DNS answers for the entire process lifetime, regardless of the actual TTL, because the JVM sets networkaddress.cache.ttl to a very high or infinite value by default. A failover to a new server IP address via a DNS change then only takes effect for running Java processes after a restart.

Also relevant for PHP-FPM and Nginx setups is that Nginx upstream blocks with hostnames resolve only once at startup by default, unless resolver with an explicit TTL is configured. A DNS change of the backend, for example with a cloud load balancer with changing IP addresses, is then simply ignored by Nginx until the process is reloaded. These differences between application runtime and system resolver are one of the most common causes of "dig works, but the application doesn't" situations.

# Check what a PHP process actually resolves, bypassing the CLI
php -r 'var_dump(gethostbyname("example.com"));'

# Nginx: verify whether a resolver directive with explicit TTL is configured
grep -A2 "resolver " /etc/nginx/nginx.conf

# Confirm actual TTL returned by DNS vs. what the application might cache
dig +noall +answer example.com | awk '{print $1, $2}'

6. Caching pitfalls: negative caches and TTL ignorance

A particularly tricky pattern is the negative cache: when a resolver receives an NXDOMAIN or SERVFAIL for a query, it caches this negative result for a certain time, often based on the zone's SOA minimum TTL value. If the actual DNS problem, for example a briefly unreachable authoritative server, is fixed afterward, the local resolver still keeps returning the cached error until the negative TTL expires. This explains why a DNS problem sometimes persists for minutes after the actual fix, which, without knowing this mechanism, leads to unnecessary, repeated troubleshooting in the wrong direction.

A second common pattern is applications deliberately or unknowingly ignoring the TTL, as described in the previous section. For Magento setups with multiple servers behind a load balancer, it is therefore advisable to reduce DNS TTLs to a low value like 60 seconds well in advance of planned migrations, ideally days before, so that both resolver caches and application caches pick up the change promptly instead of using stale IP addresses for days.

7. Correctly placing DNS in containers and behind VPNs

In Docker containers, name resolution is controlled by default via a resolv.conf injected by the Docker daemon, which either references the host DNS servers or, for user-defined networks, the built-in Docker DNS server at 127.0.0.11. A DNS problem that only occurs inside a container but is not reproducible on the host almost always points to a misconfigured container network, for example a wrong --dns flag or a network mode that bypasses Docker's own resolution.

With an active VPN using split-tunneling configuration, there can additionally be a race between multiple DNS servers, where systemd-resolved sends different queries to different servers depending on the configured routing domain (~company.internal in resolvectl). If this domain-based routing is not configured correctly, internal hostnames resolve sometimes through the internal, sometimes through the public resolver, with correspondingly inconsistent results. The command resolvectl domain shows exactly this mapping per interface and is the first thing to check for VPN-related DNS problems.

# Which DNS server is configured inside a running container?
docker exec my-container cat /etc/resolv.conf

# Which routing domains are bound to which network interface?
# Reveals VPN split-tunnel DNS routing conflicts
resolvectl domain

# Force a specific interface's DNS server for one manual test query
resolvectl query --interface=tun0 internal-host.company.internal

8. Continuously monitoring DNS latency and error rates

One-off diagnostic commands resolve an acute problem but do not prevent intermittent DNS problems from recurring unnoticed until customers are affected. A simple cron job that regularly runs dig against critical hostnames and logs response time and return code makes sporadic delays or errors visible long before they cause visible outages. For production environments it is worth integrating this into existing monitoring such as Prometheus via the blackbox_exporter, which supports DNS checks as its own probe type and records latency and success rate as a time series.

Monitoring several resolution paths in parallel is particularly valuable, for example directly against the configured resolver and additionally against a known public resolver. A divergence between the two paths over a longer period is a reliable early warning sign of a developing local configuration problem before it leads to a complete name resolution outage.

9. DNS diagnostic tools compared

The tools presented test different slices of DNS resolution and should therefore be combined deliberately rather than swapped interchangeably. The overview below arranges them by scope of testing.

10. Summary

A DNS problem on Linux, in the overwhelming majority of cases, does not sit with the authoritative server but in the local resolution chain: /etc/hosts via the nsswitch.conf order, a possibly misunderstood resolv.conf, which on systemd-based systems is often just a symlink to the stub resolver, or a systemd-resolved configured contradictorily across multiple network interfaces. The correct order of checks starts locally and only works towards the external resolver afterward, instead of reflexively testing against a public server like 8.8.8.8 first.

It is also crucial to distinguish between the system resolver and the actual application runtime, since PHP, Nginx, and Java deviate from the system configuration to different degrees and bring their own caching strategies. Negative caches and ignored TTLs explain why a DNS problem can persist for minutes after being fixed. Anyone who systematically checks this chain from the inside out and monitors the results continuously, instead of starting from zero with every incident, usually resolves DNS problems within a few minutes.

11. FAQ: Troubleshooting DNS Resolution Problems on Linux