Recently, I set firewall-cmd --set-log-denied=all on a home server (running Fedora, which uses firewalld as an iptables/nftables frontend by default), and was trying to identify all “rogue” packets. I noticed some UDP packets from external IPs being denied by the firewall. There were a lot of problems identifying/confirming the source of these packets related to slirp4netns rootless networking in Podman, but that’s outside the scope of this post. But ultimately:

  • it seemed these UDP packets were trying to communicate with a process I was running; and
  • the fact that these external IPs made it through my router further suggests I was previously communicating with these peers.

I’m far from a networking expert, so, among other inaccuracies or errors, I may get some terminology and “how things work” wrong. But roughly, my understanding is that:

  • UDP packets have a source and destination IP and port;
  • UDP packets are stateless; and
  • firewalls usually “remember” outgoing UDP packets, to allow UDP response packets to be returned from the original destination to the original source.

So my suspicion was that my router was “remembering” connections longer than the firewall/kernel on my home server was - hence router allowed those packets to be sent to my home server, but the home server, after “forgetting” the communication from its end, denied those packets.

These two kernel parameters are relevant, which can be checked by passing them to sysctl:

  • net.netfilter.nf_conntrack_udp_timeout: how long a “new” outgoing UDP packets will be remembered for (to allow returning traffic) (in seconds).
  • net.netfilter.nf_conntrack_udp_timeout_stream: if a “new” outgoing UDP packet gets a response, the connection will be remembered for this long (in seconds).

On my system (Fedora 37), the default values are 30 seconds and 120 seconds respectively.

I tried searching for the (default) value on my router, and unfortunately it hasn’t been documented. However, for another router from the same manufacturer, I saw on thread referencing the value 180 seconds.

At this point, I could have tried setting those timeouts on the home server kernel to 180, and checking if I still saw those denied packets. However, I wanted to be sure of the timeout on my router. And we can (with best effort accuracy) find this out empirically. Note that in my case, the router (presumably) has a longer timeout than the Linux kernel. If the router had a shorter timeout than the Linux kernel, the packets would have been dropped at the router and we would never even see them on the home server.

UDP hole punching

I originally ran this test with a public-facing server I run, and opened a UDP port on the public server. I have a public-facing server, and opened a UDP port. I’ll be using port 50000 in this post. For example (as root/with sudo):

firewall-cmd --add-port=50000/udp

We’ll call my home server the “client” (behind my router which is doing NAT), and my public server the “server” (which has a firewall, but a public IP address/no NAT).

We’ll be using nc (BSD version)/ncat (nmap version) (either should work, although the BSD version only supports the short version of flags, e.g. -u instead of --udp) to send UDP packets.

First, let’s listen for packets on the server.

ncat --udp --listen 0.0.0.0 50000

I’ll be using 1.2.3.4 as the public server’s IP. Then, for example, on the client, we can run the following to send a packet to the server, using 50001 as the (source) port on the client. For later, we’ll also listen for a response.

echo hi | ncat --udp --source-port 50001 1.2.3.4 50000 && ncat --udp --listen 0.0.0.0 50001

Then, we should see hi printed on the server.

Next, on both devices, we can see the “active” connections by reading /proc/net/nf_conntrack (requires root/sudo permissions). We can run the following to monitor it (as root or using sudo on the cat command).

watch --interval 1 bash -c 'cat /proc/net/nf_conntrack | grep 'sport=50001 dport=50000'

If you run that within 30 seconds on the client (assuming the same default 30 second value for net.netfilter.nf_conntrack_udp_timeout), you should see something like this, where 10.9.8.7 is the local IP of the client (behind the NAT).

ipv4     2 udp      17 8 src=10.9.8.7 dst=1.2.3.4 sport=50001 dport=50000 [UNREPLIED] src=10.9.8.7 dst=1.2.3.4 sport=50000 dport=50001 mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=0 use=2

The 5th value 8 above is the remaining timeout for this connection (in seconds).

On the server, we should see something like this, where 5.6.7.8 is the public IP of the client’s router (as given by its ISP).

ipv4     2 udp      17 8 src=5.6.7.8 dst=1.2.3.4 sport=50001 dport=50000 [UNREPLIED] src=1.2.3.4 dst=5.6.7.8 sport=50000 dport=50001 mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=0 use=2

Again, the 5th value 8 above is the remaining timeout for the connection.

In my case, it turns out the source port the server sees for the incoming packet (i.e. the port from the client) is still 50001, but I believe in general the client’s router may rewrite this port. (I believe if you knew or optimistically assumed that the client port didn’t get rewritten by its router, you wouldn’t even need to open the firewall port on the server (but it still must be public-facing, i.e. not behind a NAT).)

Within 30s, we can respond to the client from the server.

echo bye | ncat --udp --source-port 50000 5.6.7.8 50001

We should see bye printed on the client listening process. Also, the output of the watch command should now no-longer have the [UNREPLIED], and the timeout would start at 120 seconds from the reply (assuming the same default 120 second value for net.netfilter.nf_conntrack_udp_timeout_stream).

Now, enable logging of denied packets on the client - with firewalld, this is firewall-cmd --set-log-denied=all (as root/with sudo). We can watch denied packets as follows.

dmesg --decode --kernel --follow-new --ctime

Shortly after the connection expires (the line gone from watch), try sending another packet from the server. Assuming the router is still remembering the connection, we should see a denied packet in the client’s kernel/dmesg output, where the ... are values that have been left out.

kern  :warn  : [Sun Nov 27 21:43:08 2022] filter_IN_FedoraServer_REJECT: IN=... OUT=... MAC=... SRC=1.2.3.4 DST=10.9.8.7 LEN=32 TOS=0x00 PREC=0x00 TTL=55 ID=13759 DF PROTO=UDP SPT=50000 DPT=50001 LEN=12

Now, we can keep sending packets from the server to the client, longer and longer apart. At some point (assuming the time between packets is longer and longer), the router will forget the connection and immediately drop the packet from the server, and we wouldn’t see it in the client’s kernel/dmesg output.

Note that while the router remembers the connection, every packet it receives from the server (probably) refreshes the connection’s timeout, and like the kernel, it may have multiple timeouts - e.g. for the first response, and for future responses; you have to be careful about how you are tracking the timeouts.

This is a tedious process because you have to wait 30 seconds - a few minutes between each test. In my case, I seem to have more or less narrowed it down to the following behaviour of my router:

  • 60 seconds for the first two responding packets; and
  • 180 seconds for the third (and presumably any future) packets.

Potentially, the window gets even larger for packets after the third response. The fact that, after the first response, the second response must come within 60 seconds as well seems strange and I could be incorrect, but my tests were satisfactory for me.

In my case, I ended up setting net.netfilter.nf_conntrack_udp_timeout=60 and net.netfilter.nf_conntrack_udp_timeout_stream=180. (On Fedora, one can create files in /etc/sysctl.d/ to always apply these settings on boot.) I mostly stopped seeing the dropped packets, which I think mostly confirms my hypothesis. Potentially, the dropped packets - which, again, I am only aware of it meaning the router is remembering a previous connection - are due to the router further increasing the timeout beyond 180 seconds if it sees enough communication.