Building on the previous post on this topic, I will have a closer look today how to use my new and shiny bare metal server in the data center for some virtual machine fun. As the server acts as a warm standby fallback for my cloud server at home, I’ve decided to use a setup that is as close as possible to main setup. Giving Proxmox a go would certainly have been interesting, but it’s only a single server and I would have strayed too far away from my already existing setup. So I decided to go for KVM/Qemu, as it is straight forward to set-up. Also, I could just use copies of my already existing virtual machines by modifying their configurations slightly, as they will obviously run in a different IP subnet. And this is where it starts to become interesting.
The Cloud At Home With NAT
At home, my cloud runs behind a DSL line with a single public IPv4 address, and I use NAT port forwarding on the DSL router to forward particular TCP and UDP ports to different virtual machines on the same bare metal server. Each VM uses its own IP address from my home network over a macvtap network interface, so NAT-ing on the router to the virtual machines is easy. To the router, each VM looks like a separate server, as each has its own MAC address and its own IP address.
The slight downside of this approach is that I can only have one VM serving important ports such as TCP ports 80 and 443 for web services. Web servers that I use for services in other virtual machines have to use non standard ports on the NAT gateway. I could of course run a reverse proxy in a dedicated virtual machine and then distribute incoming http/https requests to other VMs based on the domain name they are for, but that’s a project for another day.
NAT in the Data Center
My bare metal host in the cloud has a similar setup. By default, it also has a single public IP address. One can buy additional public IPv4 addresses, and I will have a closer look at that in the next post. But for my ‘ordinary’ virtual machines, a single IP address is just what I need. There’s no NAT gateway in the data center, however, so this needs to be set-up on the host operating system, i.e. on the bar metal server.
Fortunately, KVM/Qemu already does that job when selecting “NAT” as network adapter type when creating a new VM or when importing a qemu2 disk image. For virtual machines using a NAT network adapter, KVM creates a virtual bridge interface and a dedicated local IP subnet, 192.168.122.x. The physical Ethernet interface with the public IPv4 address is then connected to the bridge interface over a NAT. Virtual machines can then use DHCP to get a local IP address or use a fixed IP address as configured in /etc/netplan. The virtual machines I copied over from my home cloud already had a fixed IP addresses configured, so I only had to change the IP address, default gateway and DNS server address in /etc/netplan, and also the network adapter name (e.g. enp1s0), which, for reasons I don’t quite understand, keep changing when a VM image is reused on another machine. After a netplan apply, IP connectivity starts to work right away and the VM can talk to the Internet.
Port Forwarding to the Virtual Machines
The tricky part of the game is to get incoming TCP connection requests forwarded over the NAT to individual VMs. At home, the router was doing that job. In the data center, the operating system of the bare metal server needs to be configured for this. As port forwarding through a NAT is quite an everyday thing to set-up, I searched for a long time in KVM’s Virtual Machine Manager (VMM) GUI for a way to configure this. But it seems there is no way to do it in the GUI, it has to be done with a script.
The Internet knows that iptables rules can be used for the purpose, but the first hits I found had a slight shortcoming that I subsequently had to correct as shown below. The Internet suggests to put the iptables rules for incoming port forwarding in /etc/libvirt/hooks/qemu. An already corrected rule, e.g. to forward an incoming TCP connection request to port 443 to a VM on the local subnet on the bridge looks as follows:
#!/bin/bash
HOST_IP=XXX.XXX.XXX.XXX
if [ "${1}" = "NAME-OF-THE-VM" ]; then
# Update the following variables to fit your setup
GUEST_IP=192.168.122.224
GUEST_PORT=443
HOST_PORT=443
if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
/sbin/iptables -D FORWARD -o virbr0 -p tcp \
-d $GUEST_IP --dport $GUEST_PORT -j ACCEPT \
/sbin/iptables -t nat -D PREROUTING -p tcp \
-d $HOST_IP --dport $HOST_PORT -j DNAT \
--to $GUEST_IP:$GUEST_PORT
fi
if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
/sbin/iptables -I FORWARD -o virbr0 -p tcp \
-d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
/sbin/iptables -t nat -I PREROUTING -p tcp \
-d $HOST_IP --dport $HOST_PORT -j DNAT \
--to $GUEST_IP:$GUEST_PORT
fi
fi
For each incoming port, the same if-clause with different guest ip, port, etc. values has to be added to the file. The thing most people who gave advice on the net missed, is to include the host’s IP address in the prerouting rule (see blue addition). Things will work great without the host IP at first, until you try, from any VM, including the one to which port 443 is forwarded, to send an https request to a server on the Internet. Without the host’s IP address in the rule, iptables will always redirect a TCP syn to port 443 for any IP address to that virtual machine, even for outgoing connections from this or any virtual machine on the subnet. In other words, without giving the host’s IP address in the prerouting rule, all https requests to the internet would be redirected into the virtual machine on the subnet that serves port 443. This took me a long while to figure out, so take my advice and add the hosts’s IP address to the rule to save yourself the same trouble.
And finally, here’s the command to check which NAT port forwarding rules are currently active:
sudo iptables -t nat -L -n -v
To delete an active rule manually, run the commands from the ‘stopped‘ section above and use the desired IP addresses and port numbers instead of the variables. The same goes for experimentally adding forwarding rules.
And that’s all I had to do to get my warm standby VMs up and running in the data center. Next up: How to buy additional public IPv4 addresses and bind them to virtual machines, so I can run more than just one virtual machine that uses ports 80 and 443.