In the previous post, I’ve been looking at Wondershaper, a great script that uses the Traffic Control (tc) command to counter buffer bloat situations. The main mechanism to do that is to set the maximum transmission and reception rate of an interface slightly below the available line rate, and thus prevent large transmission buffers to fill up and create unacceptable delay. But Wondershaper does a lot more, such as preferring packets of interactive SSH session over other data packets. When I had a look at how this is done, I was quite surprised that the packets were not preferred based on their TCP port number.
A IP Header Field With Many Names and Purposes: TOS, DSCP…
Here’s the tc command in the script that configures a preference for ssh:
tc filter add dev "$IFACE" parent 1: protocol ip prio 10 u32 \
match ip tos 0x10 0xff flowid 1:10;
Instead of looking at the TCP source or destination port, this filter analyzes the Type of Service (TOS) field of the IP header. In ordinary IP packets, the eight bits of the TOS field are all set to 0, which represents ‘background’ traffic. This rule, however, looks for the value 0x10 (16) in the TOS field and puts packets with this value in a special transmission queue (flowid 1:10), which gets preferential treatment.
According to the Wikipedia page on the topic, 0x10 represents IP precedence 2 (=immediate). But what does this have to do with ssh? It turns out that for interactive sessions, ssh sets the TOS field to 0x10 right after the authentication phase of the session. A quick look with Wireshark confirmed that this is actually done which is why ssh sessions remain snappy even if the line is fully saturated.
More SSH Surprises
But ssh has even more interesting surprises in store! In practice, I use a number of ssh tunnels from the same host for TCP port forwardings, so I can access the device via ssh, run a remotely triggered periodic backup via rsync, and access a VNC server for screen control. Using ssh for reverse tunneling is necessary in this case, because the machine is behind a NAT, so direct access is not possible. The thing that had me puzzled for a while was that while the rsync backup is running, my ssh terminal sessions became a little bit slow to respond while the VNC GUI session remained quite snappy. So I had a closer look with Wireshark and noticed that the SSH tunnel that was used for rsync and ssh (i.e. ssh through ssh) had the TOS field set to 0x00 (=background). The ssh tunnel for VNC, however, had the TOS field properly set to 0x10 (=immediate).
It took me a bit to find out why there was a difference. It turned out that the first reverse ssh tunnel was started with the “-N” option for non-interactive use, i.e. the ssh tunnel was only established for TCP port forwarding. The tunnel for forwarding the TCP port of the VNC server, however, did not use this option and thus opened an (unused) interactive shell. And this ssh connection had the TOS field properly set to 0x10.
This means that when using the “-N” option with ssh, the TOS field is not set, as only ports are forwarded without an interactive shell session. This makes complete sense and inadvertently kept my VNC sessions snappy while large bulk data transfers over another (non-interactive) ssh tunnel saturated the line. On the Internet at large, the TOS field is of course ignored and even set to 0 on purpose again. That’s quite appreciated, as all my packets should be treated equally on the Internet, while they should be preferred while they are in my local network due to the low line rate of the backhaul link.
I’m glad I made the extra effort to take a look at the Wondershaper code, I’ve learnt a lot and for the first time ever, I’ve seen the TOS field of the IP header actually being honored in a setup.