Network Fundamental Across networks
MTU — the number nobody sets, and everybody has to guess right
Eight articles on this site mention MTU and none explains it. This one finds the edge with a single command, then shows why the mechanism that hunts for it breaks so often
· Part 4 · Across networks · 8 min read
The word MTU appears in 8 of the 31 articles here. The Ethernet article sends readers to ping; the firewall article says blocking ICMP breaks MTU discovery.
None of them says what it is, or why it has to be discovered at all. This one answers both.
If you have never looked, start here
Find the edge yourself. It takes under a minute. The -D flag means do not fragment.
$ ping -c1 -D -s 1472 8.8.8.8
$ ping -c1 -D -s 1473 8.8.8.8On the machine this was written on the first succeeds and the second does not. The edge is exactly between those two lines.
Why the edge is there
I walked one byte at a time across it.
-s total on wire result
1470 1498 through
1471 1499 through
1472 1500 through
1473 1501 blocked
1474 1502 blockedThe edge is at 1500 exactly. Not at 1472 — that is what is left after the headers come off.
link MTU 1500
IPv4 header -20
ICMP echo -8
----
payload 1472Where 1500 comes from is the subject of the Ethernet article: a decision made in 1980. And that is the whole point here — nobody set this number for you. It is a property of each individual link.
One machine, several values
MTU is not a property of a machine. It belongs to an interface.
$ ifconfig | grep mtuOn this machine
lo0 16384
en1 1500
gif0 1280Three values on one machine, and which one a packet meets depends on the way it leaves.
Path MTU is the smallest value along the way
RFC 1191, of 1990, defines it in a sentence.
"This datagram size is referred to as the Path MTU (PMTU), and it is equal to the minimum of the MTUs of each hop in the path."
The smallest wins — not the average, and not the first link's. And the sender cannot know it in advance, because the routing article already established that nobody knows the whole path.
RFC 8201 uses the same definition for IPv6.
"This packet size is referred to as the Path MTU, and it is equal to the minimum link MTU of all the links in a path."
The mechanism — send it and wait for a complaint
RFC 1191 describes it plainly.
"The basic idea is that a source host initially assumes that the PMTU of a path is the (known) MTU of its first hop, and sends all datagrams on that path with the DF bit set. If any of the datagrams are too large to be forwarded without fragmentation by some router along the path, that router will discard them and return ICMP Destination Unreachable messages with a code meaning 'fragmentation needed and DF set'"
Three steps.
1 assume use the first hop's MTU for now
2 send with the DF bit set, meaning do not fragment
3 learn it was discarded and ICMP came back, so lower itStep three is the whole weakness. It depends on a message coming back.
When ICMP is blocked — the black hole
The firewall article already says blocking all ICMP breaks MTU discovery. Here is how.
1 assume use 1500
2 send large packets are discarded en route
3 learn ICMP is blocked, so the source hears nothing
4 and keeps sending 1500 foreverThe symptom is one of the strangest in networking — the connection works, ping succeeds, small pages load, and then something large simply hangs.
Because small packets get through and large ones vanish in silence.
RFC 8201 warns about it for IPv6.
"Nodes implementing Path MTU Discovery and sending packets larger than the IPv6 minimum link MTU are susceptible to problematic connectivity if ICMPv6 messages are blocked or not transmitted."
The way out that does not need ICMP
RFC 4821, of 2007, offers a method that works with no ICMP at all.
"In the absence of ICMP messages, the proper MTU is determined by starting with small packets and probing with successively larger packets."
Start small and grow, rather than start large and wait to be told off.
The difference between the two
RFC 1191 start big needs ICMP to come back
RFC 4821 start small needs no ICMPAnd the document places it above IP, not at the IP layer.
"The bulk of the algorithm is implemented above IP, in the transport layer (e.g., TCP) or other 'Packetization Protocol' that is responsible for determining packet boundaries."
Which is worth reading beside the protocols and their layers article, where MTU is named as one of the places the layering leaks — layer four having to know about layer two.
IPv6 removes one of the options
IPv4 has a fallback: a router can fragment for you, as long as DF is not set.
IPv6 has none. Routers do not fragment at all, as the IPv6 article describes. The source has to handle it.
So the guaranteed minimum differs.
IPv4 576
IPv6 1280And 1280 is the number that turns up on my own machine, on the gif0 interface — which is a tunnel interface.
Why tunnels are always the culprit
Every tunnel wraps the original packet inside a new one, and the added headers eat into the payload.
outer link MTU 1500
tunnel overhead -40 an example; depends on the tunnel
----
inner usable 1460The machine inside the tunnel still believes it has 1500, because nobody tells it otherwise. That is where the familiar VPN hang comes from.
The 40 above is illustrative only. The real figure depends on the kind of tunnel, and this article does not verify the figure for each kind.
When it lies
"MTU is always 1500." That is Ethernet's value, not every link's. The machine I wrote this on carries three different values.
"Ping works, so the path is fine." An ordinary ping sends small packets, which get through even on a path with an MTU problem. It takes -D and a large size to find it.
"Block all ICMP for safety." That blinds the mechanism that finds the MTU, and produces one of the hardest symptoms to diagnose there is.
"Set the MTU small to be safe." RFC 1191 says itself that sending datagrams smaller than the path allows wastes resources and gives worse throughput than it should.
"Routers fragment for you anyway." Only IPv4, and only without DF. IPv6 does not do it at all.
Real cases from real work
Case 1 — it connects, and then large transfers hang
Situation Ping works, small pages load, and anything large stalls.
Command Find the real edge for that path.
$ ping -c1 -D -s 1472 <destination>
$ ping -c1 -D -s 1400 <destination>How to read it If 1472 fails and 1400 succeeds, the path MTU is below 1500. If no ICMP comes back at all, that is the black hole described above.
What this does not prove Finding the edge with ping describes the round trip at that moment. Paths change, and ping uses ICMP, which some places treat differently from TCP.
Case 2 — only some destinations misbehave
Situation Same machine; some sites work and some hang.
How to read it Path MTU belongs to the path, not to the machine, exactly as RFC 1191 defines it. Different destinations cross different links and so have different values. Some destinations working does not mean your machine is fine.
What this does not prove Destination-specific symptoms have other causes too, including a problem at the far end. Compare from another machine on the same segment.
Case 3 — a tunnel is configured and things slow down badly
Situation With the VPN up, some things break or crawl.
How to read it Compare the tunnel interface's MTU against the real link's.
$ ifconfig | grep -B4 mtuIf the tunnel interface is still set to 1500, the same as the real link, the tunnel's own header has not been subtracted.
What this does not prove Lowering the tunnel's MTU fixes many cases, but the right value depends on the kind of tunnel, and this article does not verify the figure for each kind.
What this number teaches
MTU is the clearest example of a value nobody owns. The source did not set it, the destination did not set it, and each router in between knows only its own link.
So the usable value has to be discovered rather than read, and the mechanism that discovers it depends on a message somebody else sends back — which may never arrive.
That is the shape the whole series keeps meeting. The system works because somebody is willing to tell you, and fails silently when nobody does.
References
Standards
- RFC 1191, November 1990, Path MTU Discovery — the PMTU definition and the DF-bit mechanism quoted here
- RFC 8201, July 2017, Path MTU Discovery for IP version 6 — the same definition for IPv6 and the warning about blocked ICMPv6
- RFC 4821, March 2007, Packetization Layer Path MTU Discovery — the method that works without ICMP
Measured on the machine this was written on
ping -Dwalked one byte at a time, putting the edge at exactly 1500 bytes on the wireifconfigshowing three MTUs on one machine: 16384, 1500 and 1280
Computed here
- The table subtracting a 20-byte IPv4 header and an 8-byte ICMP header from 1500
What could not be confirmed from the source
- The header size of each kind of tunnel. The 40 bytes used here is illustrative