As I’ve already noticed last year, there is quite a bit of packet loss over Starlink connections that has a significant impact on TCP performance. Also, I was wondering if and how much packet loss occurs in cellular networks during a handover. Recently, I picked up this topic again and was looking for a way to detect, measure and analyze packet loss. One idea I came up with was to run a continuous iperf3 UDP stream at a relatively slow 1 Mbps and see what would happen. While iperf is running, it shows when and how many packets were lost, but I wanted to dig a bit deeper. So I used Wireshark / tcpdump to record the data transfers and then wrote a Python script to analyze the iperf packet counters to get more details. Here’s a link to the source code.
So let’s look at one of my results. The first output below is from iperf3 and shows the data transfer rate and packet loss rate once a second of a 1 Mbps iperf3 UDP downlink data stream over Starlink. I significantly abbreviated the output for readability by removing most lines in which no packet loss occurred:
$ iperf3 -u -c x.x.x.x -p yyyy -R -t 6000
#Connecting to host 1x.x.x.x, port yyyy
Reverse mode, remote host x.x.x.x is sending
[ 5] local 192.168.1.22 port 59049 connected to x.x.x.x port yyyy
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 134 KBytes 1.10 Mbits/sec 4.981 ms 0/95 (0%)
[ 5] 1.00-2.00 sec 129 KBytes 1.05 Mbits/sec 2.562 ms 0/91 (0%)
[ 5] 2.00-3.00 sec 126 KBytes 1.03 Mbits/sec 3.852 ms 0/89 (0%)
[ 5] 3.00-4.00 sec 129 KBytes 1.05 Mbits/sec 5.142 ms 0/91 (0%)
[ 5] 4.00-5.00 sec 126 KBytes 1.03 Mbits/sec 4.845 ms 1/90 (1.1%)
[...]
[ 5] 27.00-28.00 sec 127 KBytes 1.04 Mbits/sec 5.722 ms 0/90 (0%)
[ 5] 28.00-29.00 sec 129 KBytes 1.05 Mbits/sec 3.425 ms 0/91 (0%)
[ 5] 29.00-30.00 sec 129 KBytes 1.05 Mbits/sec 4.990 ms 0/91 (0%)
[ 5] 30.00-31.00 sec 126 KBytes 1.03 Mbits/sec 3.800 ms 1/90 (1.1%)
[ 5] 31.00-32.00 sec 129 KBytes 1.05 Mbits/sec 4.511 ms 0/91 (0%)
[...]
[ 5] 48.00-49.00 sec 129 KBytes 1.05 Mbits/sec 2.909 ms 0/91 (0%)
[ 5] 49.00-50.00 sec 127 KBytes 1.04 Mbits/sec 4.012 ms 0/90 (0%)
[ 5] 50.00-51.00 sec 129 KBytes 1.05 Mbits/sec 2.781 ms 0/91 (0%)
[ 5] 51.00-52.00 sec 124 KBytes 1.02 Mbits/sec 2.612 ms 2/90 (2.2%)
[ 5] 52.00-53.00 sec 129 KBytes 1.05 Mbits/sec 4.373 ms 0/91 (0%)
[ 5] 53.00-54.00 sec 126 KBytes 1.03 Mbits/sec 5.730 ms 0/89 (0%)
[ 5] 54.00-55.00 sec 130 KBytes 1.07 Mbits/sec 2.632 ms 0/92 (0%)
[ 5] 55.00-56.00 sec 127 KBytes 1.04 Mbits/sec 4.618 ms 0/90 (0%)
[ 5] 56.00-57.00 sec 129 KBytes 1.05 Mbits/sec 4.211 ms 0/91 (0%)
[ 5] 57.00-58.00 sec 126 KBytes 1.03 Mbits/sec 9.358 ms 1/90 (1.1%)
[ 5] 58.00-59.00 sec 129 KBytes 1.05 Mbits/sec 13.319 ms 0/91 (0%)
[ 5] 59.00-60.00 sec 127 KBytes 1.04 Mbits/sec 10.610 ms 0/90 (0%)
[ 5] 66.00-67.00 sec 127 KBytes 1.04 Mbits/sec 13.993 ms 0/90 (0%)
[...]
[ 5] 73.00-74.00 sec 129 KBytes 1.05 Mbits/sec 3.456 ms 0/91 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-182.29 sec 0.00 Bytes 0.00 bits/sec 0.000 ms 0/0 (0%) sender
[ 5] 0.00-182.29 sec 22.5 MBytes 1.04 Mbits/sec 4.341 ms 22/16326 (0.13%) receiver
In total, 22 out of 16326 packets were lost over a period of 3 minutes (180 seconds). For comparison purposes: Over a VDSL link or a LTE/5G stationary link, 0 packets are lost during such a test.
22 packets doesn’t sound like much, but it does have a significant impact on TCP throughput in practice. Not sure how much impact this would have on a real time video stream, but in an audio stream, such packet loss is not audible.
And here’s how the output of my Python script that focuses on packet loss, and hence is much more compact:
Extracting iperf3 UDP counter IDs...
................
Sorting counter values...
Looking for missing counter IDs...
................
Now grouping missing counter IDs...
Missing IDs in the sequence:
446 (size: 1)
2757 (size: 1)
4673 (size: 1)
4683 (size: 1)
5252 (size: 1)
6388 (size: 1)
6593 (size: 1)
7981 (size: 1)
12024 (size: 1)
12076 (size: 1)
12134 (size: 1)
12549 (size: 1)
13622 (size: 1)
14393 (size: 1)
14740 (size: 1)
14743-14744 (size: 2)
14754-14756 (size: 3)
16096 (size: 1)
16123 (size: 1)
The output of the script matches iperf’s shell output and hence, the outputs confirms each other. So as far as Starlink’s packet loss is concerned, it happens irregularly and in most of the cases, exactly one packet at a time is lost. Only in few instances, two or three packets were lost in a row. Why packet loss occurs is quite unclear to me, however. Perhaps during satellite handover? But that’s just a wild guess and perhaps not even likely, as I assume handovers occur much less frequently. But there is no documentation of the air interface protocol, so any other guesses are welcome in the comments below.