iptables router failure after Debian Lenny upgrade solved by setting MTU
I recently upgraded my home router box to Debian Lenny. Everything went fairly smoothly, with a few exceptions. My NFS mounts no longer worked because apparently wildcards are no longer allowed in IP addresses in /etc/exports; the export addresses needed to be translated to subnet format (e.g., 192.168.98.* becomes 192.168.98.0/824).
But after a power failure last night, the router box rebooted and I was no longer able to access the Internet from any clients on my LAN. Strangely, I could ping or traceroute external hosts and perform DNS lookups, but web surfing and ssh timed out after an initial handshake. I noticed by telnetting to port 80 of an external host, I got an error back from an invalid HTTP request (e.g. “oeunthioues”), but if I sent a standard valid request (GET /index.html HTTP/1.0), the connection just hung with no response.
I won’t recount all the false leads I had in diagnosing this problem. It turned out that the Internet-facing NIC on my router box had been reset to a low MTU. By setting the MTU on the LAN clients to that low number, or raising the MTU on the Internet-facing NIC back to 1500, the problem was solved:
# ifconfig eth2 mtu 1500
After restarting networking on the router box, the MTU was again set back down to 576, which is apparently the default MTU for an X.25 network. I have no idea why the interface is getting that value by default (where it wasn’t before), so I just added a hack to /etc/network/interfaces to fix it:
iface eth2 inet dhcp post-up /sbin/ifconfig eth2 mtu 1500
Interestingly, pre-up didn’t work.
Hopefully I’ve included enough relevant terms in this entry that others with this problem will find it. It was hard to diagnose because no errors appeared in any log file, and I had partial but not complete connectivity from internal clients to the Internet. My first guess was that it was due to the iptables upgrade, but in fact it was entirely unrelated.
[tags]Debian, Lenny, iptables, firewall, router, MTU[/tags]
Steve Laniel Feb 25
I thought the days were long since gone when we had to think about MTUs. I think the last time I thought about them was in the WinSock 1 era.
It’s things like this that make me doubt Linux. Not for long, mind you, but it’s there.
mvw Feb 25
Ah, the PMTU discovery blackhole …
http://en.wikipedia.org/wiki/Path_MTU_discovery
Other reasons for losing Fragmentation Needed packets are RFC1918 addresses on intermediate links and plain broken router firmware.
adam Feb 25
Interesting! So it would seem, perhaps, something in Lenny is attempting to discovery the proper MTU size when the interface is brought up and getting the wrong answer (perhaps from Comcast?), whereas Etch performed no such discovery.
Ben Hutchings Feb 25
PMTU discovery doesn’t affect the interface MTU; it only affects traffic to specific addresses.
Also you want /24 in your netmask, not /8.
adam Feb 25
Oops! You’re right re netmask. So we don’t have an explanation as to why the interface is defaulting to such a small MTU size under Lenny where it didn’t before.
Shadow Panther Feb 26
http://tldp.org/HOWTO/IP-Masquerade-HOWTO/mtu-issues.html
(7.15.2. Clamping the MSS via IPTABLES)
Adam SKutt Feb 26
The MTU is being sent by the DHCP server, and your DHCP client is happily setting the interface MTU to whatever is being published. Comcast, at least, still sends out 576 as MTU in a few places.
Editing your /etc/dhclient.conf and removing “interface-mtu” as a “request” option will fix this.
However, you really should setup MSS clamping as suggested by Shadow Panther to prevent issues with other hosts on the Internet. Note, however, the documentation in that link is out of date, you need to say:
iptables -t mangle -A forward -p tcp –tcp-flags SYN,RST SYN -j TCPMSS –clamp-mss-to-pmtu
adam Mar 1
There was no interface-mtu option in dhclient.conf — it looks like that was taken out of the default configuration precisely for this reason. I uncommented the request line to specifically list options without interface-mtu, but it still resets the MTU to 576. So I just keep my post-up MTU reset even if it’s a temporary hack.
Incidentally, the MSS clamping command line didn’t work either — iptables says “Bad argument `SYN,RST'”.
Josh Apr 19
Adam,
I got the command to work. Due to formatting you can’t simply copy and paste what Adam Skutt posted. You’ll need to make sure you have two -‘s in front of ‘tcp-flags’ and ‘clamp-mss-to-pmtu’. Also, unless you capitalize FORWARD your iptables will complain about an invalid chain.
Cheers
Max Mar 16
Old post, but was a recent issue for me. After modifying /etc/dhclient.conf you must remove /var/lib/dhcp3/dhclient.leases, or else it’ll attempt to renew the old lease.
alice May 22
Thank you so, so, so much for posting this. My router died after an Ubuntu upgrade (thanks a lot Canonical) and upon rebuilding it with Debian I have been tearing my hair out for the last few hours wondering why my network was still not working. It never would have occurred to me that it was this of all things, but sure enough you solved my problems. You have my undying gratitude!