Troubleshooting Path MTU Discovery: When Packets Mysteriously Vanish

1. What Path MTU Discovery actually solves

Every network link along a path can support a different maximum packet size, the Maximum Transmission Unit. While local Ethernet segments typically allow 1500 bytes, tunneling technologies such as VPNs, GRE, or PPPoE regularly reduce the usable size through additional protocol overhead, often down to values between 1400 and 1480 bytes. Without a mechanism that recognizes these differences, routers would either silently fragment packets or, worse, simply drop larger ones.

Path MTU Discovery solves this by having the sender transmit packets with the Don't Fragment bit set, explicitly forbidding routers from splitting the packet into smaller fragments if needed. When a router along the path encounters a packet larger than the next hop's MTU, it drops it and sends an ICMP type 3, code 4, Fragmentation Needed message back to the sender, which also carries the actual supported MTU of the affected hop. The sender then reduces the packet size for that connection and retries the transfer.

2. Why blocked ICMP messages become the real problem

In practice, many firewalls configure blanket rules that drop all ICMP traffic, often based on an outdated security mindset that treats ICMP as inherently dangerous. Those very rules also block the Fragmentation Needed message that Path MTU Discovery depends on, which means the sender never finds out its packet was dropped and keeps sending packets at the original, too large size.

This behavior is known as black hole PMTUD, because packets above the actual path MTU literally vanish somewhere on the network without any feedback reaching the sender or application. Since smaller packets, such as the initial TCP handshake or short SSH control messages, do not trigger the issue, the connection appears to work fine at first, right up until the application tries to send its first larger packet, which then gets silently dropped and effectively freezes the connection.

3. Typical symptoms: hanging SSH sessions and truncated HTTP responses

A classic symptom is an SSH connection that establishes without issue and works fine for simple commands like ls, but freezes completely as soon as a command with large output, such as cat on a bigger file, or an interactive program with a full terminal redraw, is run. The SSH handshake itself consists of many small packets below the problematic size threshold, while the actual data transfer is the first thing to generate packets at full MTU size.

At the HTTP and HTTPS level, the same problem shows up as pages that only partially load, TLS handshakes that complete successfully but then hang, or API responses that work fine with small JSON payloads but consistently time out on larger ones. What makes this particularly tricky is that the behavior often only appears for certain destination or source networks, while other connections through the same server work completely fine, because only specific path segments actually have a reduced MTU.

4. Diagnosis step one: finding the actual path MTU with ping

The most reliable first diagnostic step is a targeted ping with the Don't Fragment bit explicitly set and a variable packet size, iteratively narrowing down the largest packet size that still works. On Linux, the Don't Fragment bit is set via the -M do option, while -s defines the payload size excluding the 28 byte IP and ICMP header, which is why for a target MTU of 1500 bytes the largest working -s value is 1472.

If a ping with a given size fails while smaller sizes succeed, a binary search between the two values quickly narrows down the actual path MTU. If no error appears at all and packets simply vanish without any response, that is already a strong sign the necessary ICMP feedback is being blocked somewhere along the path, rather than there being no MTU restriction at all.

# Test whether a payload of 1472 bytes (1500 MTU) makes it through
ping -M do -s 1472 -c 3 target-server.example.com

# On failure, step down through smaller sizes until the actual
# path MTU is found
ping -M do -s 1400 -c 3 target-server.example.com

5. Diagnosis step two: tracepath for automated MTU discovery

Instead of manually bisecting the correct packet size, tracepath automates exactly that process while simultaneously identifying at which hop along the path the MTU actually gets reduced. The tool sends packets with increasing TTL, similar to traceroute, but combines this with Path MTU Discovery and explicitly shows in its output where the allowed packet size first shrinks.

If the tracepath output stops at a certain hop with no further pmtu messages, that points to an ICMP block right at that location, where the necessary Fragmentation Needed message is either not generated by the affected router at all, or dropped by a firewall on the way back. This information is especially valuable when several network segments are involved, for example a VPN link between two data centers, since it narrows down whether the problem sits at your own edge router or further upstream at the provider.

# Automatically determine the path MTU and the hop where it shrinks
tracepath target-server.example.com

# Example output with one hop that reduces the MTU to 1420
#  3:  vpn-gateway.example.net       1.2ms pmtu 1420
#  4:  no reply

6. Diagnosis step three: watching ICMP traffic directly with tcpdump

To confirm beyond doubt whether an ICMP Fragmentation Needed message ever reaches the original sender at all, a targeted tcpdump capture session on the affected server, run alongside a simultaneous large packet test, is invaluable. If no ICMP type 3 code 4 message arrives even though the ping or tracepath test clearly points to an MTU restriction, the fault most likely lies in a local or upstream firewall rule that drops incoming ICMP traffic outright.

Especially in cloud environments and with security groups that by default only open selected ports rather than entire protocols, a missing ICMP allowance is one of the most common causes of PMTUD problems. The capture makes it visible whether the problem actually sits on your own system or further upstream at the provider, which speeds up troubleshooting considerably instead of changing configuration in several places on a hunch.

# Capture incoming ICMP traffic on the server
tcpdump -ni eth0 icmp

# In a second terminal, trigger a large packet test during the capture
# to verify whether the Fragmentation Needed reply actually arrives
ping -M do -s 1472 -c 3 target-server.example.com

7. TCP MTU probing: PMTUD without depending on ICMP

In response to widespread ICMP blocking across the public internet, RFC 4821 defines Packetization Layer Path MTU Discovery, a mechanism that determines the actually usable packet size directly at the TCP layer without relying on incoming ICMP messages at all. Instead of waiting for a Fragmentation Needed reply, the TCP stack experimentally sends larger packets and checks whether an acknowledgment for them arrives, so missing ICMP feedback no longer causes a black hole, but is simply interpreted as a missing acknowledgment.

On Linux, this behavior can be enabled via the kernel parameter net.ipv4.tcp_mtu_probing, where a value of 1 enables probing only after a black hole has been detected, and a value of 2 forces probing by default for all TCP connections. For servers that regularly talk to networks whose ICMP configuration cannot be controlled, for example connections to customer networks or third party APIs, enabling TCP MTU probing is a valuable addition to MSS clamping and allowing ICMP, since it catches the problem even when neither of the other two measures can be enforced for organizational reasons.

# Enable TCP MTU probing permanently, value 1 = only after a detected
# black hole, value 2 = always active for all TCP connections
sysctl -w net.ipv4.tcp_mtu_probing=1

# Persist it in the sysctl configuration
echo "net.ipv4.tcp_mtu_probing = 1" >> /etc/sysctl.d/99-pmtud.conf
sysctl --system

8. Fix one: allow ICMP Fragmentation Needed deliberately instead of blanket blocking

The cleanest and most durable fix is to stop blocking ICMP outright and instead deliberately allow the message types Path MTU Discovery needs, while still restricting other ICMP types as desired. For IPv4, that specifically means ICMP type 3 code 4, and for IPv6 the structurally equivalent ICMPv6 type 2, Packet Too Big, which is even more essential, since IPv6 routers never fragment packets in transit at all and rely entirely on working PMTUD.

A correspondingly precise firewall rule can be implemented with classic iptables just as well as with the more modern nftables framework, and it should be a fixed part of every production server firewall, regardless of how restrictive the rest of the ICMP policy is otherwise.

# Allow ICMP Fragmentation Needed deliberately for IPv4
iptables -A INPUT -p icmp --icmp-type fragmentation-needed -j ACCEPT

# Equivalent for IPv6: allow Packet Too Big
ip6tables -A INPUT -p ipv6-icmp --icmpv6-type packet-too-big -j ACCEPT

9. Fix two: MSS clamping as a robust alternative on gateways and tunnels

Where blocked ICMP messages cannot be reliably fixed, for example because they get dropped by an upstream provider you have no control over, MSS clamping helps by avoiding the problem at the source instead of relying on working PMTUD at all. The technique adjusts the TCP Maximum Segment Size directly within the SYN packet during connection setup, so the resulting packets stay below the actual path MTU from the start.

Especially on VPN gateways and tunnel endpoints, where added protocol overhead regularly reduces the effective MTU, MSS clamping is the more pragmatic and robust solution compared to relying solely on ICMP based PMTUD, since it works independently of firewall configuration along the rest of the path and structurally prevents the problem instead of merely detecting it.

# MSS clamping on a VPN gateway for outgoing traffic over tun0
iptables -t mangle -A FORWARD -o tun0 -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

# Equivalent in nftables syntax
nft add rule inet mangle forward oifname tun0 tcp flags syn tcp option \
  maxseg size set rt mtu

10. Summary

11. FAQ: PMTUD Troubleshooting