A Firewall for Online Workshops

Over the past months, I’ve been experimenting a lot with Docker and Kubernetes and decided to pass on the knowledge I have gained in occasional cloud-based hands-on workshops rather than doing presentations or videos. The advantage: Instead of talking ‘against a wall’ with virtually no feedback whatsoever during and after a presentation, hands-on workshops are much more interactive and fun to do in the virtual world for everybody involved. So while this is fun, it comes with a certain personal liability risk that can, however, be mitigated to a large degree with a firewall. So here’s the story:

Some Background

To have a common baseline for all participants, I have come up with the idea to provide every participant with a workstation in the cloud with a virtual desktop to which they can connect via VNC. In case a participant gets stuck during the workshop, I can also use VNC to access the participant’s individual workstation screen to see where the problem lies. I’ve done such sessions a couple of times now and I very much like the experience and the interesting technical feedback I get. A small thing that has been nagging me, however, is that the virtual workstations are of course connected to the Internet and participants could do things on them that the law would not approve of. As the workstations are only online for a few hours and I basically trust the workshop participants, I consider it an acceptable risk. Trust is good, but minimizing my exposure is even better, so I started experimenting with limiting traffic to and from the workstations with a virtual firewall.

Incoming Connections

Luckily, Hetzner, my VM cloud provider of choice, has recently added a firewall feature to their offering. It’s not a unique feature, AWS, for example, has had for many years now. The basic idea of the firewall is to define a set of rules for incoming and outgoing TCP connections (or UDP ports). This firewall configuration can then be used for several virtual machines and changes made later-on immediately take effect for all VMs. For incoming directions, the rules are quite straight forward in my use-case: Reject all incoming UDP packets and all TCP connection attempts unless they are to the port used by the VNC desktop service. Access to the SSH port can be limited to a single IP address, i.e. the IP address of my home connection.

Outgoing Connections

Limiting incoming connectivity to the bare minimum already mitigates my exposure quite a bit, but outgoing TCP connections are of course still a concern. Fortunately, these can be limited as well. In my case, I only need outgoing connectivity during the workshops to Docker hub for downloading images, and to an Etherpad service I host on my own. Defining a rule for my own Etherpad service is quite straight forward, as the DNS request only returns a single IP address. Interaction with Docker Hub, however, is not quite as straight forward:

Multiple IP Addresses of Servers

For one thing, several servers are contacted during an image download. As that happens under the hood, I used tcpdump and a remote Wireshark to trace which domain names are queried during a ‘docker pull‘ command. Here’s the command line to run tcpdump remotely and forward all traffic to Wireshark running on my PC:

ssh root@xxx.xxx.xxx.xxx 'sudo tcpdump -n -i eth0 -s 65535 -w - port not XXXXX and port not XXXXX' |  wireshark -k -S -i -

The second challenge that follows is that the DNS responses always return several IP addresses which can be contacted as shown in the screenshot above. One could of course make a firewall rule for each of them but that’s quite a bit of work. So instead, I decided to add the domain names used for a ‘docker pull‘ in /etc/hosts on each virtual workstation and use one of the returned IP addresses. This way, I get a 1:1 relationship between domain name and IP address and hence an easier firewall configuration. This is of course at the risk that the server behind this particular IP address becomes unreachable just before or during the workshop. However, if that happens, I could still reconfigure the firewall and modify the /etc/hosts file on each workstation over ssh. Looping over the following command which pipes the content of the ‘hosts-add.txt‘ on my notebook into /etc/hosts of a target machine does the trick. This is also helpful to initially write content to the hosts file when a new VM is instantiated from an image file, as the hosts file of the image gets overwritten by the defaults.:

# Use > to overwrite or >> to add
#

cat hosts-add.txt | ssh root@xxx.xxx.xxx.xxx -T "cat >> /etc/hosts"

Fortunately, this was not required so far during a workshop. But better be prepared.

By limiting all incoming and also all outgoing connections this way, I am pretty confident that all non-intended interactions with the network that could be a liability for me can be prevented. But if you have an idea how that could be circumvented, please let me know, I’m always interested to learn and improve 🙂