The firewall article already introduced the two kinds of filter — the one that looks at each packet alone, and the one that remembers connections — and showed how easily a stateless rule that trusts the ACK bit is fooled.
What it did not answer is the pair of questions that follow — what exactly does it remember, and what does remembering cost?
The first answer is more interesting than expected, because a firewall does not merely remember that "this connection exists". It runs its own state machine alongside both endpoints, and for a protocol with no connections at all, like UDP, it invents one.
The second answer has three parts, and the second part explains one of the most elusive failures in real work.
If you have never looked, start here
All the evidence is in a manual page on this machine.
$ man pf.conf | grep -A2 'tcp.established'The most revealing thing in it is the list of tunable timeouts, because their number and their names give away how many states the firewall thinks a connection has.
The firewall runs a state machine of its own
For TCP the manual names six states, each with its own description.
tcp.first after the first packet
tcp.opening before the destination has ever sent anything back
tcp.established the fully established state
tcp.closing after the first FIN is sent
tcp.finwait after both FINs have been exchanged
tcp.closed after one endpoint sends an RSTThat is nearly the whole of TCP's own state machine, but the firewall is not an endpoint. It merely watches packets go past and infers which state the connection is in.
And because it infers, it can infer wrongly. The manual documents a real case under tcp.finwait.
"Some hosts (notably web servers on Solaris) send TCP packets even after closing the connection. Increasing tcp.finwait (and possibly tcp.closing) can prevent blocking of such packets."
The endpoint says it has closed, then keeps sending; the firewall believes the close and drops what follows. This is the case where the firewall's opinion and the endpoint's reality disagree, and the result is that legitimate packets die.
UDP has no connections, so the firewall invents them
This is the part of the manual page I find most interesting of all.
UDP, as the TCP and UDP article explains, has no handshake, no close, and no state whatsoever. Yet the manual offers three states to tune.
udp.first after the first packet
udp.single source sent more than one, destination never replied
udp.multiple both hosts have now sent packetsNone of these three exist in the UDP protocol. No bit in the UDP header says anything of the kind. The firewall builds them out of the only thing it can observe — the direction of the traffic.
udp.single means "I have heard only one side speak", which is more suspicious than udp.multiple, meaning "both sides are talking". So the firewall grants them different lifetimes, on the reasoning that an exchange with two participants is more likely to be a real conversation.
The manual states the idea directly: "ICMP and UDP are handled in a fashion similar to TCP" — treating things without state as though they had it.
ICMP is the same, with icmp.first and icmp.error, and the manual shows that a stateful rule really does match the replies.
"allows echo requests (such as those created by ping(8)) out statefully, and matches incoming echo replies correctly to states"
That is why ping works without writing a rule for the return direction, even though ICMP has no ports to match on at all.
It checks more than four values — it checks sequence numbers
The firewall article says a stateful filter looks at four values, the addresses and ports of both ends. True, but incomplete. The manual describes more.
"For TCP connections, comparing a packet to a state involves checking its sequence numbers, as well as TCP timestamps if a scrub reassemble tcp rule applies to the connection. If these values are outside the narrow windows of expected values, the packet is dropped."
And it says plainly what that defends against.
"This prevents spoofing attacks, such as when an attacker sends packets with a fake source address/port but does not know the connection's sequence numbers."
Here is what that is worth as a number, assuming an attacker who knows both addresses but cannot see the traffic.
filter must guess odds, 1 in
stateless, ACK bit only source port 65,535
stateful, 4-tuple + seq port + sequence 4,294,967,296
... plus TCP timestamps +10 to 18 bits 1,125,899,906,842,624Checking sequence numbers makes it 65,537 times harder, and the manual states that timestamps add "by 10 to 18 bits" on top, which at 18 bits is another factor of 262,144.
This is the real answer to the ACK-bit trap the previous article demonstrated. It is not simply that a stateful filter "remembers" — it holds information the attacker cannot obtain without seeing the real traffic.
The first price — memory, and timeouts that shrink themselves
Remembering every connection means storing every connection, and as the store fills, pf has a response that few people know about.
"Timeout values can be reduced adaptively as the number of state table entries grows."
It works linearly between two values.
adaptive.start entry count at which timeouts begin shrinking
adaptive.end entry count at which timeouts reach zeroThe manual describes the far end without euphemism.
"When reaching this number of state entries, all timeout values become zero, effectively purging all state entries immediately."
The entire table is discarded at once, which means every working connection drops at the same moment. The documentation itself warns that "it should not actually be reached".
This behaviour is a good explanation for a symptom that looks exactly like a device rebooting when nothing has rebooted — everything fails together and then recovers by itself.
The second price — asymmetric paths break it while everything is correct
This is the most difficult price to diagnose, and it follows directly from the routing article.
That article points out that routes are one-directional. The path out and the path back need not be the same, which on the internet is entirely normal.
A stateless filter does not care, since it judges each packet alone. A stateful filter breaks, because:
outbound firewall A creates the state for this connection
return firewall B never saw the outbound, so holds no state
result firewall B drops a perfectly valid packetEvery setting is correct, the routing is correct, and each firewall behaves exactly as designed — and it does not work. The symptom is intermittent, because the path can shift with load.
This is why a pair of firewalls must either share state with each other, or be forced to keep both directions of a connection on the same device — and both add complexity that a stateless filter never needs.
The third price — you cannot scale out for free
Stateless filters scale trivially, because every instance decides identically without needing to know anything from the others. Add a machine, spread the traffic across it, done.
Stateful filters cannot do that, because the decision depends on what has been seen before. A newly added machine has no history of the connections currently running, so it will drop all of them.
The manual also notes an advantage in speed.
"Also, looking up states is usually faster than evaluating rules."
Which is true — once the state exists, matching it beats walking the whole rule set. But that advantage is bought with the three constraints above.
When it lies
"Stateful is always better." Better for security, clearly, by the numbers above. It is bought with memory, fragility against asymmetric paths, and harder scaling.
"The firewall knows the connection closed." It infers that from the packets it saw, and can be wrong. The Solaris case in the manual is a documented example.
"UDP has no state, so a firewall cannot filter it." It can, using state it invented from traffic direction — which is inherently less trustworthy than TCP's.
"A full state table just refuses new connections." Not necessarily. With adaptive timeouts configured, lifetimes shrink progressively, and at the far end the documentation says every entry is purged at once, which hits existing connections too.
"Traffic returns the way it left." It need not, and that assumption is precisely what breaks stateful filters in networks with more than one path.
Real cases from real work
Case 1 — a connection dies after going quiet
Situation An ssh window is left open, and typing into it later hangs even though the network is fine.
How to read it Look at tcp.established on the devices in between. If the connection was idle for longer than that value, the state was deleted, and the next packet matches nothing. Both endpoints still believe the connection is open while the firewall has forgotten it. The fix is keepalive at the endpoints rather than at the firewall, because raising the timeout grows the table in direct proportion.
What this does not prove The same hang comes from NAT forgetting its mapping, which is a different table entirely. You have to establish which device is doing what before concluding.
Case 2 — random failures after adding a second path
Situation A second circuit is added for resilience, and some connections begin failing with no discernible pattern.
How to read it Check whether the outbound and return traffic pass through the same device. If not, this is the asymmetric case described above. It looks random because path selection changes with load, not because anything is faulty.
What this does not prove Seeing asymmetry explains the symptom but does not prove the firewall is what drops the traffic. Look at the counters on both devices for packets dropped as matching no state.
Case 3 — choosing which kind to use
Situation You must design filtering for a system carrying very high traffic.
How to read it Ask three questions.
1 is return traffic guaranteed to cross the same device
2 peak concurrent connection count, and is memory sufficient
3 will machines need to be added mid-flight to absorb loadIf the first is uncertain or the third is yes, a stateless filter at the outer position with a stateful filter at the inner one is usually a more workable division than using a single kind throughout.
What this does not prove These three cover the architectural constraints, not the policy question of what should be permitted, which the firewall article addresses instead.
What remembering buys, and what it sells
Remembering connections genuinely buys security, and buys a great deal of it — from 1 in 65,535 to 1 in roughly a quadrillion, by the figures computed above.
What it sells is simplicity. A stateless filter is a pure function that returns the same answer for the same input every time. A stateful filter depends on the past, which means it has memory that can fill, an assumption about paths that can be wrong, and an opinion about connections that can differ from the truth.
Every one of the strangest firewall failures traces back to those three words — depends on the past.
References
From the machine this was written on
man pf.conf— the state names for TCP, UDP and ICMP, the adaptive timeout values, the sequence number and timestamp checks, and every sentence quoted in this article- The firewall article on this site cites the same manual page for rule ordering and default behaviour
Computed here
- The guessing table is computed from TCP's 2^32 sequence space, a 64 KB window, a 16-bit source port, and the 10 to 18 bits the manual attributes to timestamps