The Internet Protocol article ended with a packet arriving at the destination machine, and that is all IP undertakes to do.
The two questions left over afterwards matter as much as the one it answered.
This page is about two protocols that answer them differently, and why both are still here.
IP delivers to a machine, and stops there
The packet has arrived. That machine is running dozens of programs. The first question is which program this data belongs to.
And IP never undertook that the packet would arrive, or arrive in order, or arrive only once. The second question is whether it actually got here at all.
UDP answers the first and stops. TCP answers both, and every remaining difference between the two traces back to that sentence.
This is not about which is better. TCP costs more because it promises more, and there is a great deal of work for which that promise is not worth the price.
If you have never looked, start here
The machine reading this page has dozens of TCP connections open right now, and will tell you the state of each.
$ netstat -an -p tcp | awk 'NR>2{print $NF}' | sort | uniq -c
67 ESTABLISHED
16 LISTEN
7 CLOSE_WAIT
2 TIME_WAIT
1 SYN_SENTThese state names are not macOS terminology. They are the state names from TCP's state machine as written in the RFC in 1981, and every one of them still carries the same name today.
Two lines are worth noticing already: seven CLOSE_WAIT and two TIME_WAIT. Both come back in the section on closing a connection, and they say very different things.
Ports — what both of them share
The answer to the first question is two 16-bit numbers called the source and destination ports, and they occupy the first four bytes of both UDP and TCP identically.
16 bits means 65,536 values, 0 through 65535, divided into three ranges.
0 - 1023 well known
1024 - 49151 registered
49152 - 65535 dynamic, usually called ephemeralThe last range is what a machine draws from when it opens an outbound connection, and this machine confirms it uses exactly that range.
$ sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last
net.inet.ip.portrange.first: 49152
net.inet.ip.portrange.last: 65535A connection is identified by four values, not by one.
source IP source port destination IP destination portThat is why ten tabs open on the same site do not get mixed up. All ten share a destination address, a destination port and a source address. Only the source port column differs.
UDP — eight bytes, and a refusal to promise
RFC 768 was published on 28 August 1980 by Jon Postel. It is three pages long and has never been replaced.
Its header has four fields of 16 bits each, eight bytes in total.
0 16 31
+---------+---------+---------+---------+
| source port | destination port |
+---------+---------+---------+---------+
| length | checksum |
+---------+---------+---------+---------+The most important sentence in that document is the one saying what it does not do — "delivery and duplicate protection are not guaranteed". No undertaking that it arrives, and none that it arrives only once.
The last two fields are more interesting than they look.
Length appears redundant, because the IP header already gives a length. The RFC specifies that this value includes its own header, so the minimum is 8, which is a datagram carrying nothing at all.
The checksum is computed over things that are not in the packet. The RFC says it covers a pseudo header consisting of the source address, the destination address, the protocol and the UDP length — all of which are read from the IP header.
This is a deliberate leak in the layering. An upper layer reads values from the one below in order to catch packets delivered to the wrong machine, and it is why NAT has to recompute the checksum every time it rewrites an address.
The checksum can also be switched off. The RFC states that "An all zero transmitted checksum value means that the transmitter generated no checksum" — sending zero means none was computed.
TCP — twenty bytes, and every promise
The current TCP specification is RFC 9293, from August 2022, which replaces RFC 793 from 1981 along with six other documents.
Every field
bytes field bits
0-1 source port 16
2-3 destination port 16
4-7 sequence number 32
8-11 acknowledgment number 32
12 data offset + reserved 4 + 4
13 control bits 8
14-15 window 16
16-17 checksum 16
18-19 urgent pointer 16
---
160 bits = 20 bytesData offset counts 32-bit words, not bytes. The RFC defines it as "the number of 32-bit words in the TCP header". The field is 4 bits, so the maximum is 15 words, or 60 bytes. That is the ceiling on the whole TCP header, which leaves at most 40 bytes for options.
Those 40 bytes are a scarcer resource than people assume, and they come back as an issue in the window scaling section.
There are eight control bits, and their names are the words you see most often when reading traffic.
CWR ECE URG ACK PSH RST SYN FINSequence numbers count bytes, not packets
This is the most commonly misread field in the whole header.
The sequence number does not say which packet this is. It says which byte of the data stream the first byte of this packet is. The acknowledgment number says which byte is wanted next.
Counting in bytes is what lets TCP resegment data along the way. A sender may merge two packets into one or split one into two, as long as the byte numbering stays continuous.
There are two exceptions to remember. SYN and FIN each consume one sequence number despite carrying no data. The RFC says the SYN is considered to occur before the first actual data octet, and the FIN after the last one.
That exception explains every +1 in the next section.
The three-way handshake — why three
-> SYN seq = x
<- SYN ACK seq = y ack = x+1
-> ACK seq = x+1 ack = y+1The question worth asking is why not two. One side asks, the other agrees, and that would seem to be that.
The answer is in the numbers. There are two initial sequence numbers, not one — x from the calling side and y from the listening side. Neither has ever seen the other's, and both must be acknowledged.
The second message does two jobs at once: it acknowledges x and announces y. The third acknowledges y. Four jobs therefore compress into three messages, but not into two.
RFC 9293 states the reason directly: "A 3WHS is necessary because sequence numbers are not tied to a global clock in the network". There is no shared clock, so each machine picks its own starting number and neither can guess the other's.
The ISN must be unguessable, and that is a security matter
If the initial sequence number can be guessed, anybody who guesses it can inject data into an existing connection without ever seeing its traffic.
RFC 9293 therefore specifies how to choose it.
ISN = M + F(localip, localport, remoteip, remoteport, secretkey)M is a timer that ticks every four microseconds, and F() must be a pseudorandom function, with the RFC stating plainly that it "MUST NOT be computable from the outside".
Note that F() takes the four values identifying the connection as inputs. The result is that different connections get different starting numbers even when they are created in the same second.
Two windows people most often confuse
The word window means two entirely different things in TCP, and calling them by one name is the source of a great deal of confusion.
rwnd — how much the receiver can hold
RFC 5681 defines rwnd as "the most recently advertised receiver window". It is the window field in the TCP header, in which the far end states outright how much space its buffer has left.
This is flow control, and the only thing it prevents is a sender overwhelming a receiver.
cwnd — how much the network can hold
cwnd is "a TCP state variable that limits the amount of data a TCP can send", and the important words in that definition are state variable.
cwnd never appears on the wire. No field in the TCP header holds it. It is a number in the sender's memory alone, and it is the product of guessing what the path in between can carry.
This is congestion control, and it prevents something different: a sender overwhelming the network.
What can actually be sent is the smaller of the two. The RFC says a sender uses "the minimum of cwnd and rwnd".
Keeping the two apart matters when fixing things. Throughput limited by rwnd is fixed by more buffer; throughput limited by cwnd is not helped by any amount of buffer.
Window scaling — 65535 stopped being enough in 1992
The window field is 16 bits, so the largest value is 65,535 bytes.
That was once enough and stopped being so long ago. The reason computes directly.
1 Gbit/s x 0.1 s RTT = 12.5 MB the path holds this much at once
65,535 bytes / 0.1 s = 655,350 B/s what the old window permits
= 5.24 Mbit/s about 0.5% of the linkOn a gigabit path with 100 ms of delay, the unscaled window yields 5 Mbit/s, because the sender must stop and wait for an acknowledgment every 65,535 bytes no matter how empty the path is.
RFC 7323 puts it as "For LFN paths where the bandwidth * delay product exceeds 64 KiB, the receive window limits the maximum throughput".
The fix is an option called window scale, giving a shift count for the field.
window field 16 bits -> 65,535 bytes
shift count max 14
scaled window 2^30 = 1 GiBThe RFC caps it: "the shift count MUST be limited to 14 (which allows windows of 2^30 = 1 GiB)".
Two properties of it cause real problems.
First, the option is only sent during the handshake. The RFC says it goes in the SYN, and appears in the SYN-ACK only if it was received in the SYN. So if middleware strips it during the handshake, that connection is stuck at 64 KiB for its whole life, with no way to recover mid-stream.
Second, if either side does not support it, both set their shift to zero. There is no half-way negotiation.
This machine has buffers set at twice the old ceiling already.
$ sysctl net.inet.tcp.sendspace net.inet.tcp.recvspace
net.inet.tcp.sendspace: 131072
net.inet.tcp.recvspace: 131072131,072 bytes is 128 KiB, more than 65,535. Without window scaling this machine could not advertise its own buffer in full.
Congestion control
TCP has no way of knowing what the path in between can carry. Nobody tells it and there is no field to ask. Its only method is to keep increasing until it sees a sign of having gone too far.
And the only sign of having gone too far is a lost packet.
Slow start is not slow, it starts small
RFC 5681 says "during slow start, a TCP increments cwnd by at most SMSS bytes for each ACK received", which means cwnd doubles every round trip. The name refers to where it starts, not to the rate.
That starting point has changed.
RFC 5681 SMSS > 2190 IW = 2 * SMSS, max 2 segments
SMSS 1096 to 2190 IW = 3 * SMSS
SMSS <= 1095 IW = 4 * SMSS, max 4 segments
RFC 6928 IW = min(10*MSS, max(2*MSS, 14600))RFC 6928, from 2013, raised the initial window to ten segments, which matters enormously for short transfers, because the whole transfer may finish before slow start has had time to expand at all.
Once cwnd passes ssthresh, growth changes from doubling to linear.
cwnd += SMSS * SMSS / cwnd3 dup ACK — halve it, because news is still coming back
RFC 5681 specifies that "the fast retransmit algorithm uses the arrival of 3 duplicate ACKs as an indication that a segment has been lost".
Three is used because loss has to be distinguished from reordering. Packets that overtake each other slightly produce one or two duplicate ACKs; three or more is more convincing.
ssthresh = max(FlightSize / 2, 2 * SMSS)
cwnd = ssthresh + 3 * SMSSThis is where the graph halves rather than restarting, because still receiving ACKs is evidence that other packets are still reaching the far end. The path is not dead.
RTO — guessing how long counts as lost
If no ACK comes back at all, something has to decide when it counts as lost. That is the RTO, and RFC 6298 gives the formulas.
first measurement
SRTT <- R
RTTVAR <- R/2
every measurement after
RTTVAR <- (1-beta)*RTTVAR + beta*|SRTT - R'|
SRTT <- (1-alpha)*SRTT + alpha*R'
always
RTO <- SRTT + max(G, K*RTTVAR)
alpha = 1/8 beta = 1/4 K = 4What these formulas do is look at the variability, not only the average. SRTT is a weighted average and RTTVAR is the variability, and the RTO is set above the average by four times that variability.
A path with a steady 200 ms delay therefore gets a lower RTO than one swinging between 50 and 200 ms, even where the averages are close. That is correct: a path that swings deserves more patience.
The RFC also sets a floor — if the computed value is under a second, "the RTO SHOULD be rounded up to 1 second" — and on expiry, "set RTO <- RTO * 2".
Doubling every time makes the waits look like this.
1 2 4 8 16 32 64 ...That explains a symptom everybody has met: the network drops, and the program hangs far longer than expected without ever reporting an error.
Closing, and TIME_WAIT
Closing takes four messages rather than three, because each direction closes separately. One side saying it will send no more does not mean the other side is finished.
The gap between the first FIN and the second is the CLOSE_WAIT state on the receiving side, and the operating system does not decide how long that lasts. The application does, because the state ends only when the program calls close.
The side that closed first must then sit in TIME_WAIT. RFC 9293 requires it to "linger in the TIME-WAIT state for a time 2xMSL" and defines MSL as "For this specification the MSL is taken to be 2 minutes".
Twice two minutes is four minutes. That is what the document says.
The machine used to write this page does not do that. The way to check is to open a real connection, close it from this side, and time how long the entry stays in the table.
0.0s local .56826 -> TIME_WAIT
31.0s local .56826 -> None$ sysctl -n net.inet.tcp.msl
15000Thirty-one seconds, not four minutes. And the msl value the machine reports is 15000, which against the measurement confirms the unit is milliseconds: 15 seconds doubled is 30, matching what was measured.
Measuring beats reading a unit off a variable name. Guessing seconds would have been wrong by a factor of sixty.
Why wait at all? Because old packets from the previous connection may still be loose in the network. Release the same port pair immediately and those old packets turn up mixed into a new connection using the same four values. TIME_WAIT trades a port for a while in exchange for the guarantee that nothing stale gets mixed in.
Head-of-line blocking — the price of the promise
TCP promises to hand data to the application in order. That promise has a price, and the price is most visible when a single packet goes missing.
Suppose six segments were sent and the third is lost, while the fourth, fifth and sixth arrive intact.
The data of four, five and six is already in the destination's memory. But the application receives only one and two, and then waits — because handing over four before three would break the promise about order.
What blocks it is not the network. The network did its job. What blocks it is the promise.
RFC 9114, the HTTP/3 standard, states the effect directly: "a lost or reordered packet causes all active transactions to experience a stall regardless of whether that transaction was directly impacted by the lost packet". Every transaction in flight stalls, related to the lost packet or not.
This is why QUIC exists, and why it chose to run over UDP. RFC 9000 states that "QUIC packets are carried in UDP datagrams to better facilitate deployment in existing systems and networks".
QUIC did not abandon reliability. It moved reliability down to the level of individual streams instead of the whole connection, so streams unrelated to the lost packet keep going.
Where it lies to you
"TCP guarantees the data arrives." No. TCP guarantees that if data arrives, it arrives complete and in order. If the cable is genuinely cut, all TCP can do is report failure. It converts silent failure into reported failure, which is a large difference, but it is not a guarantee of success.
"UDP is faster than TCP." Not a meaningful sentence. Both travel the same path at the same speed. UDP does not wait for acknowledgments and does not slow itself down, which means it does not wait — not that it is fast. Work that needs complete data has to build that waiting into the application anyway.
"No ACK means the packet was lost." Not always. The ACK itself may be what was lost. The sender cannot tell the two apart and retransmits in both cases, so the far end may receive duplicates — which is why sequence numbers must also do the job of discarding them.
"A bigger window makes it faster." Only where the window is the limit. If cwnd is the limit, more buffer changes nothing at all — and too much buffer in intermediate equipment produces a different problem entirely: queues so long that latency climbs.
"Lots of TIME_WAIT means something is wrong." No. TIME_WAIT is the trace of an orderly close from our side. The one to worry about is CLOSE_WAIT piling up, because that means our own program forgot to close.
Worked examples from real work
Case 1 — CLOSE_WAIT accumulating until nothing new can connect
A server accepts less and less work until it accepts none. Restarting fixes it for a while, and then it comes back.
How to read it. Count the states before anything else.
$ netstat -an -p tcp | awk 'NR>2{print $NF}' | sort | uniq -c
4812 CLOSE_WAIT
23 ESTABLISHED
8 LISTENFour thousand CLOSE_WAIT entries is the whole answer. That state means the far end sent a FIN, the kernel replied with an ACK, and it is now waiting for this side's application to close.
The kernel will never release them on its own, however long it waits, because it has no right to decide on the program's behalf.
This is not a network problem. It is a bug in code that fails to close a socket on some path — usually an error path, which is the least tested one.
Not yet proved. We know the program does not close, but not which code path fails to. Look at which peers those sockets are talking to and work back from there.
Case 2 — throughput caps at the same number every time, on an idle link
Transferring a file between regions gives about 5 Mbit/s every time, whatever the hour, while both ends are on gigabit.
How to read it. A number that steady is the signature of a ceiling, not of congestion. Congestion fluctuates.
Take the throughput and the RTT and work backwards to the window in use.
5.24 Mbit/s / 8 = 655,000 B/s
655,000 B/s x 0.1 s = 65,500 bytes65,500 is far too close to 65,535 to be coincidence. This connection is running at the ceiling of an unscaled window field.
Then look at the handshake for a window scale option. If one side sent it and the other did not answer, suspect middleware that stripped it.
Not yet proved. The arithmetic says the window is the limit; it does not say who removed the option. Capture the handshake at successive points along the path.
Case 3 — the program hangs far longer than it should when the far end vanishes
Unplug the far end and this side reports no error for minutes.
How to read it. Look at the interval between retransmissions.
t = 0.00 first transmission
t = 1.00 retransmit
t = 3.00 retransmit
t = 7.00 retransmit
t = 15.00 retransmit
t = 31.00 retransmitThe gaps are 1, 2, 4, 8, 16 — the doubling RFC 6298 requires. This is not a malfunction; it is correct behaviour.
Summing them explains why it takes so long, and explains why reducing the timeout in the application is a more direct fix than trying to change TCP.
Not yet proved. The doubling confirms no ACK came back, but not whether the outbound packet or the returning ACK was lost. Capture at the far end too.
Case 4 — cannot open new connections although the load is low
A machine firing requests at the same destination repeatedly starts failing to open new connections in bursts.
How to read it. Compute the ceiling from the values the machine gives us.
65535 - 49152 + 1 = 16384 source ports available
16384 / 30 s = 546 new connections per second546 per second is the real ceiling to one destination on one port, because a port that has been closed is still reserved in TIME_WAIT for another thirty seconds.
Note the ceiling is per destination, not per machine, because a connection is identified by four values — a different peer can reuse the same source port.
Not yet proved. The 546 is computed from this machine's values. Another machine has a different port range and a different MSL. Read them off the machine that actually has the problem before computing.
References
The core standards
- RFC 768 — User Datagram Protocol J. Postel, 28 August 1980, three pages, and the source of the sentence that delivery and duplicate protection are not guaranteed
- RFC 9293 — Transmission Control Protocol (TCP) August 2022, Standards Track, replacing RFC 793 and six others — the source for the header layout, the reason for the three-way handshake, ISN selection, and the definition of MSL
- RFC 793, September 1981, the original, and the origin of every state name
netstatstill prints today
Control and tuning
- RFC 5681 — TCP Congestion Control September 2009 — the definitions of cwnd and rwnd, the slow start rule, the three duplicate ACK rule, and the ssthresh formula
- RFC 6298 — Computing TCP's Retransmission Timer the SRTT, RTTVAR and RTO formulas with alpha, beta and K, and the doubling rule
- RFC 7323 — TCP Extensions for High Performance September 2014 — window scaling, the shift count cap of 14, timestamps and PAWS
- RFC 6928 — Increasing TCP's Initial Window April 2013, Experimental, the source of the ten-segment initial window
What came after
- RFC 9000 — QUIC: A UDP-Based Multiplexed and Secure Transport May 2021, and its reason for running over UDP
- RFC 9114 — HTTP/3 June 2022, the clearest available statement of what head-of-line blocking costs
What is on the machine
netstat -an -p tcp— every state name comes straight from the RFCsysctl net.inet.ip.portrange.firstand.last— the range drawn from for outbound connectionssysctl net.inet.tcp.sendspaceand.recvspace— buffers set to 128 KiB, twice the ceiling of an unscaled window fieldsysctl -n net.inet.tcp.mslgives 15000, and measuring an actual TIME_WAIT gives 31 seconds, which confirms the unit is milliseconds rather than seconds
Related reading here
- Internet Protocol — the layer underneath, and the promises it does not make
- NAT — what rewrites the source port in the header this page describes
- DNS — work that runs mostly on UDP and falls back to TCP when an answer is too large