Linux dropped UDP packets
Troubleshooting UDP packets being dropped
Chris GThis image is one of the best summaries of all of the Linux troubleshooting tools I have ever seen:
linux_observability_tools.
I once had to troubleshoot an issue with UDP packets larger than MTU being dropped between hosts in different subnets. Hard proof was needed in order for the Cloud provider to admit that there was an issue.
First, I used the following command to test for fragmented packets :
iperf -s -p 7870 -u
------------------------------------------------------------
Server listening on UDP port 7870
Receiving 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.1 port 7870 connected with 192.168.1.1 port 45467
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.1 sec 494 KBytes 402 Kbits/sec 0.021 ms 0/ 344 (0%)
[ 4] local 192.168.1.1 port 7870 connected with 192.168.8.8 port 43130
[ 4] 0.0-10.1 sec 494 KBytes 402 Kbits/sec 0.023 ms 0/ 344 (0%)
[ 3] local 192.168.1.1 port 7870 connected with 192.168.8.8 port 47294
[ 3] 0.0-10.1 sec 494 KBytes 402 Kbits/sec 0.048 ms 0/ 344 (0%)I then ran a test from a "good" host by sending a UDP request larger than the allowed MTU:
iperf -c host1 -u -p 7870 -l 3832
------------------------------------------------------------
Client connecting to sd6pstg-35b5, UDP port 7870
Sending 3832 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.8.8 port 47294 connected with 192.168.1.1 port 7870
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 1.26 MBytes 1.05 Mbits/sec
[ 3] Sent 344 datagrams
[ 3] Server Report:
[ 3] 0.0-10.1 sec 494 KBytes 402 Kbits/sec 0.048 ms 0/ 344 (0%)They ran the same test on a "bad" host:
iperf -c host1 -u -p 7870 -l 3832
------------------------------------------------------------
Client connecting to sd6pstg-35b5, UDP port 7870
Sending 3832 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.9.9 port 38406 connected with 192.168.1.1 port 7870
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 1.26 MBytes 1.05 Mbits/sec
[ 3] Sent 344 datagrams
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
sysctl -a |grep udpAdditional details about that command:
https://iperf.fr/iperf-doc.php#tuningudp
sudo ifconfig eth0 mtu 800
ip route flush cache
ping -s 1520 -M dont -i 0.8 192.168.1.1
tcpdump -vv -i eth0 host 192.168.1.1 and udp -nThe following article was very useful also: Does Linux have an Equivalent of Windows PMTU Blackhole Router Discovery? to turn on tcp_mtu_probing:
echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing
Possible values
0: disabled
1: enabled when black hole detected
2: always enabled
tracepath -n 192.168.1.1
1?: [LOCALHOST] pmtu 1464
1: 169.254.154.42 1.429ms
1: 169.254.154.42 1.058ms
2: 169.254.32.144 2.134ms
3: 169.254.32.19 1.034ms
4: no reply
tcpdump -nnvvXSs 1514 -i eth0
nn = don't resolve host names or port names
vv = verbosity level (can be v, vv, or vvv)
X = Payload. Shows packets contents in both ASCII and HEX. If you need the ethernet header us XX instead of just X
S = prints absolute sequence numbers
s = allows you to set snaplen (in this case 1514) so we capture the whole packet.
cat /proc/sys/net/ipv4/ip_default_ttl
64
sudo sysctl net.ipv4.ip_default_ttl=129
Posted Comments: 0
Tagged with:
troubleshooting performance networking