Debugging Network Connections Step by Step

1. Why a layer model speeds up diagnosis

The biggest time sink when debugging a network connection is jumping randomly between tools without following a fixed order. A systematic approach instead works from bottom to top through the network layers: first name resolution, then reachability at the IP level, then routing, then firewall rules, then the TCP handshake, then TLS, and finally the application layer itself. Every layer rules out an entire class of possible causes before the next one is considered.

In practice this approach saves considerable time, because it prevents, for example, spending minutes searching application logs when the actual problem already lies in DNS resolution. A failing network connection always has one concrete layer where it actually breaks, and the trick is to narrow down that layer as quickly as possible instead of treating symptoms at the wrong level.

An additional benefit of the layer model: it provides a shared vocabulary for the whole team. Instead of vague statements like it just doesn't work, a failing network connection can be precisely named as a DNS problem, a firewall problem, or an application problem, which considerably speeds up handoff between colleagues or to an external provider.

2. Layer 1: Does the name even resolve correctly?

The first diagnostic step for any failing network connection to a hostname is to check DNS resolution in isolation before even attempting a connection. dig api.example.com +short shows the resolved IP address directly, while dig api.example.com without +short additionally shows response time, the nameserver used, and the TTL. An empty response or a SERVFAIL means the problem already lies here, long before any TCP packet was ever sent.

A common special case: resolution works with an external nameserver like 8.8.8.8, but fails with the system's configured resolver. dig @8.8.8.8 api.example.com against dig api.example.com shows this difference directly and points to a problem with the local or internal DNS server, not with the target server itself. For the network connection this is a crucial distinction, because the fix then lies in the DNS configuration, not the target service.

# Layer 1: isolate DNS resolution before attempting any connection
dig api.example.com +short
# 203.0.113.42

# Compare system resolver against a known-good external one
dig @8.8.8.8 api.example.com +short
# 203.0.113.42

# Check /etc/resolv.conf if results differ
cat /etc/resolv.conf

3. Layer 2: Routing and reachability with ping

If the name resolves correctly, checking basic reachability at the IP level follows. ping -c 4 203.0.113.42 tests whether ICMP packets reach the target and a response comes back. Important here: many modern firewalls specifically block ICMP while TCP connections on certain ports continue to work. A failed ping therefore does not automatically mean the actual network connection is impossible, it must be interpreted in the context of the rest of the diagnosis.

More informative for TCP-based services is nc -zv 203.0.113.42 443, which specifically tests whether a TCP handshake on the concrete target port succeeds, independent of ICMP filtering. This distinction between ICMP reachability and TCP port reachability is one of the most common pitfalls when diagnosing network connections, because both tests make different statements and are easily confused.

4. Layer 3: Visualizing the path with traceroute and mtr

If both ping and the TCP connection attempt fail, traceroute shows the path of the packets through the network hops to the destination and reveals exactly where the route breaks. traceroute -T -p 443 203.0.113.42 uses TCP packets instead of the default UDP packets, which in environments with strict UDP filtering delivers more realistic results, because it tests the same protocol family as the actual network connection.

# Layer 3: trace the path using TCP instead of default UDP/ICMP
traceroute -T -p 443 203.0.113.42
#  1  gateway (10.0.0.1)  0.412 ms
#  2  10.20.0.1  1.203 ms
#  3  * * *
#  4  * * *
#  5  203.0.113.1  14.902 ms

# Hop 3-4 show no response — route breaks there or filters TCP probes

mtr 203.0.113.42 combines ping and traceroute in one continuously updated, live view and shows packet loss rates per hop over time, instead of just a single snapshot. A hop with consistently high packet loss where all following hops work normally often points to plain ICMP rate limiting at that hop and is not a real problem for the network connection. Loss that persists all the way to the destination, on the other hand, is a serious signal.

5. Layer 4: Ruling out firewalls and security groups

If routing works all the way to the destination but the network connection still fails, a firewall rule is a very likely candidate, either locally on the source server, on the target server, or in an intervening cloud security group. iptables -L -n -v, or with nftables nft list ruleset, shows the locally active rules, paying particular attention to DROP and REJECT rules for the affected port.

In cloud environments like AWS, Hetzner Cloud, or Azure, the provider-level security group or cloud firewall must additionally be checked, which exists completely independent of local iptables rules and can drop packets before they even reach the server. A common mistake when diagnosing a network connection: checking only the local firewall and overlooking the upstream cloud security group that is the actual blocker.

6. Layer 5: Observing the TCP handshake with tcpdump

tcpdump makes the TCP handshake directly visible and shows exactly at which point a network connection fails. tcpdump -i any -n host 203.0.113.42 and port 443 captures all packets between the local server and the target on the relevant port. A successful handshake shows the sequence SYN, SYN-ACK, ACK. If the SYN-ACK response is missing entirely, the SYN packet is either dropped along the way or the target server does not respond, which points to a firewall problem closer to the destination or a service that is not listening.

# Layer 5: capture the TCP handshake for the specific connection
tcpdump -i any -n host 203.0.113.42 and port 443
# 14:22:01.102341 IP 10.0.0.5.51234 > 203.0.113.42.443: Flags [S], seq 123456
# (no SYN-ACK response follows within several seconds)

# Compare to a working connection for reference
tcpdump -i any -n host 203.0.113.42 and port 443
# 14:23:10.881233 IP 10.0.0.5.51240 > 203.0.113.42.443: Flags [S], seq 998877
# 14:23:10.902104 IP 203.0.113.42.443 > 10.0.0.5.51240: Flags [S.], seq 445566, ack 998878
# 14:23:10.902411 IP 10.0.0.5.51240 > 203.0.113.42.443: Flags [.], ack 445567

A response packet with the [R] flag for reset instead of a SYN-ACK means something different: the target server is reachable and the firewall lets the packet through, but simply no service is listening on the requested port. This distinction between no response and an explicit reset is extremely valuable when diagnosing a network connection, because it distinguishes between a firewall problem and a misconfigured or stopped service.

7. Layer 6: TLS handshake and certificate problems

If the TCP handshake succeeds but the network connection still fails, for encrypted services the TLS handshake follows as the next suspect layer. openssl s_client -connect api.example.com:443 -servername api.example.com manually establishes a TLS connection and shows the full certificate chain, the expiration date, and any validation errors, for example an expired or untrusted issuer. An expired certificate or a missing intermediate certificate in the chain is one of the most common causes of failing TLS-based connections that look completely unremarkable at the TCP level.

8. Layer 7: Checking the application layer with curl -v

If DNS, routing, firewall, TCP and TLS handshake all work but the network connection still delivers an unexpected result, the problem lies in the application layer itself. curl -v https://api.example.com/endpoint shows every single phase of the request including sent and received HTTP headers, which becomes immediately visible especially with authentication errors, wrong content types, or unexpected redirect chains.

The option --resolve api.example.com:443:203.0.113.42 forces a specific IP address for the hostname without changing DNS, which is particularly helpful in load balancer setups with multiple backend IPs, isolating and testing a single faulty backend specifically while the rest of the network connection infrastructure stays unchanged.

Another common special case at this layer is a connection that suddenly dies after a longer idle period, even though the original handshake succeeded. An intervening NAT gateway or a load balancer with too short an idle timeout may have already dropped the connection table entry while client and server still consider the network connection open, which only shows up as an error on the next data exchange.

9. Symptoms and layers compared

The following table maps typical symptoms to the affected network layer and the appropriate diagnostic tool, serving as a quick entry point for the next failing network connection.

Using this table as a mental checklist, you work systematically from bottom to top through the layers for every new failing network connection, avoiding wasted time at the wrong level, for example searching application logs when the problem actually lies in DNS resolution.

Asymmetric routing, where outbound and return packets take different paths, is another special case that only becomes visible when traceroute is run from both sides of the network connection, instead of relying on a single direction of observation.

10. Summary

A failing network connection can be reliably narrowed down by strictly following the order of network layers: DNS with dig, reachability with ping and nc -zv, routing with traceroute -T and mtr, firewall rules locally and in the cloud, the TCP handshake with tcpdump, TLS with openssl s_client, and finally the application layer with curl -v. Every layer delivers an unambiguous signal that either confirms or rules out the next set of possible causes.

The biggest mistake when debugging a network connection is skipping layers or checking them in the wrong order, for example searching application logs immediately when DNS resolution is already failing. Anyone who has internalized the order described here usually finds the actual cause within minutes rather than hours.

The same systematic approach applies regardless of the concrete target system, whether the affected network connection is to a payment provider, an internal microservice interface, or an external SaaS service.

11. FAQ: Debugging network connections