NetKubeLab ไทย

Network Fundamental Out to the wider world

VPNs and tunnels — a packet wrapped inside a packet

The machine this was written on has fifteen tunnel interfaces set to five different MTUs, and one of them is larger than the wire it rides on

· Part 8 · Out to the wider world · 6 min read

The word VPN appears in 6 articles here, mostly as the cause of some odd symptom.

This one explains what it actually does. The answer is shorter than people expect — it puts a packet inside another packet. Everything else follows from that sentence.

If you have never looked, start here

Your machine may already have several tunnel interfaces.

$ ifconfig | grep -E '^(utun|gif|stf)'

The machine this was written on has fifteen, and their MTUs are

   MTU   interfaces
  1000            1
  1280            3
  1380            5
  1500            5
  2000            1

Five different values on one machine, while the real link has exactly one: 1500.

Encapsulation — the whole of it

RFC 2003, of 1996, puts it in one sentence.

"To encapsulate an IP datagram using IP in IP encapsulation, an outer IP header is inserted before the datagram's existing IP header"

The original packet in its entirety becomes the payload of a new packet, with a new header placed in front of it

The original packet is not modified. It becomes the payload of a new one, and the routers along the way see only the outer header, with no idea what is inside.

The document also says how the length is counted.

"The Total Length measures the length of the entire encapsulated IP datagram, including the outer IP header, the inner IP header, and its payload."

Two headers, one packet. And that is where every problem in this article comes from.

The price is room

The added header comes straight out of the payload.

  outer header          bytes   inner usable on a 1500 link
  IPv4 header only         20                          1480
  IPv4 + UDP               28                          1472
  IPv6 header only         40                          1460

The machine inside the tunnel knows nothing about this. It sees an interface declaring an MTU and sends that much. If that figure has not had the header subtracted, the packet is too big the moment it is wrapped.

Which is why the MTU article says tunnels are always the culprit.

What the values on this machine tell you

Back to the table at the top.

  1500   same as the real link; no tunnel header subtracted at all
  1380   120 bytes taken off
  1280   the minimum IPv6 guarantees
  1000   a generous margin
  2000   larger than the wire it rides on

The last line is the interesting one. An interface set to 2000 on a machine whose real link is 1500 means its full-size packets are over the limit the moment they are wrapped.

I cannot verify who set these values or why. These interfaces are created by installed software. What is verifiable is that the values exist, and that in some cases no two of them agree.

A tunnel has to discover its own MTU too

RFC 2003 has a section called Tunnel MTU Discovery.

"When the Don't Fragment bit is set by the originator and copied into the outer IP header, the proper MTU of the tunnel will be learned from ICMP Datagram Too Big (Type 3, Code 4) messages reported to the encapsulator."

MTU discovery runs at two levels: an inner one for the original packet and an outer one for the wrapped packet

This is the same mechanism as the MTU article, nested twice.

  inner   the source host discovers the tunnel's MTU
  outer   the encapsulator discovers the real path's MTU

Both levels depend on ICMP. If ICMP is blocked, both go blind at once — which is why VPN symptoms are harder to trace than the same symptom without a tunnel.

What a tunnel gives you, and what it does not

Wrapping means the routers along the way see only the outer header. But wrapping alone is not encryption.

  encapsulation   put inside a new packet, still readable
  encryption      unreadable without a key, a separate matter

RFC 2003 in its entirety is about wrapping. Encryption is a different document, RFC 4301. "VPN" in ordinary use means both together, but technically they are two layers that come apart.

The TLS article covers encryption in the context of the web. The key material is the same idea; what differs is the layer it runs at.

When it lies

"VPN means encrypted." Not necessarily. Wrapping and encrypting are separate; some tunnels only wrap.

"The VPN is slow because of the encryption." Maybe. A more common cause is an MTU with no header subtracted, so packets are dropped and resent.

"Setting the tunnel MTU to the link's value is fine." This machine has one set to 1500, the same as the real link, which leaves no room for the tunnel's own header.

"A tunnel makes you invisible to routers along the way." They see the outer header — who is talking to whom at the tunnel endpoints, and how much.

"Lowering the MTU always fixes VPN problems." It helps in many cases, but the right amount depends on the tunnel, and lowering it further than needed costs throughput, as RFC 1191 warns.

Real cases from real work

Case 1 — with the VPN up, some sites do not load

Situation The VPN connects, ping works, some pages hang.

Command Compare the tunnel's MTU against the real link's.

$ ifconfig | grep -E 'mtu' | sort -u

How to read it If a tunnel interface carries the same value as the real link, the header has not been subtracted, and the symptom is exactly the MTU black hole from the MTU article.

What this does not prove MTU is a common cause, not the only one. Confirm with a large ping -D through the tunnel to find where the edge really is.

Case 2 — choosing the tunnel's MTU

Situation A new tunnel is being configured and a value has to be chosen.

How to read it Start from two numbers.

  1  the real path's MTU, measured with ping -D
  2  the header size of the tunnel type in use

The value to set is the first minus the second — not a figure copied from somewhere else.

What this does not prove This article does not verify the header size of each tunnel type. Check the documentation for the one actually in use.

Case 3 — find out what tunnels a machine has

Situation A machine behaves oddly on the network and you are not sure what is running.

Command

$ ifconfig | grep -E '^(utun|gif|stf|ipsec)'

How to read it These interfaces are created by software, not hardware. An unusual number of them means several programs have made their own. The machine I wrote this on has fifteen.

What this does not prove An interface existing does not mean it is carrying traffic. Check its state and the routing table alongside.

What wrapping teaches

A tunnel solves a problem by adding a layer, which works very well. Its price is that every property of the layer beneath has to be recalculated.

The MTU changes, the hop count you can see changes, and any mechanism that depends on ICMP now has to work twice instead of once.

It is the pattern the protocols and their layers article describes. Layering makes design easier, and a leaking layer makes diagnosis harder. A tunnel adds one more set of leaks.

References

Standards

  • RFC 2003, October 1996, IP Encapsulation within IP — the definition of wrapping, how total length is counted, and the Tunnel MTU Discovery section quoted here
  • RFC 4301, December 2005, Security Architecture for the Internet Protocol — the document covering the encryption layer, which is separate from the wrapping

Measured on the machine this was written on

  • ifconfig found 15 tunnel interfaces carrying five different MTUs — 1000, 1280, 1380, 1500 and 2000 — while the real link has only 1500

Computed here

  • The table of room left after subtracting the outer header from 1500

What could not be confirmed from the source

  • Which software set each tunnel interface's MTU, and why
  • The header size of each tunnel type; this article gives only standard IP header sizes

Measured on a real machineComputed here

อ่านหน้านี้เป็นภาษาไทย