Two years ago, I migrated a lot of my services running on virtual machines in the cloud to VMs on a bare metal server, which was also running in the cloud. The main reason: A significant cost reduction while at the same time having significantly more dedicated processing power and storage available. While I put most of those VMs behind a NAT and a single IP address, I had a BBB video conferencing server running in a VM that required it’s own IP address. Fortunately, my hoster could assign more than one IP address to a bare metal server. In the meantime, I have 3 VMs on that bare metal server that use their own public IP address.
For those VMs, I don’t use a NAT interface, but a macvtap interface. At the time, I thought that macvtap just maps a public IP address to a VM and that’s it. For the details have a look at my blog entry that describes the setup. However, it turned out that macvtap can do a bit more, which is particularly useful when a bare metal server hosts several VMs with their own public IP addresses. Read on for the details.
When I only had a single VM requiring it’s own IP address, I pretty much copied the content of the netplan file of the bare metal server to the virtual machine and changed the IP address. Here is how that looks like:
network:
version: 2
renderer: networkd
ethernets:
enp9s0:
addresses:
- 65.22.33.157/32
routes:
- on-link: true
to: 0.0.0.0/0
via: 65.22.33.129
gateway6: fe80::1
nameservers:
addresses:
- 185.12.64.2
- 185.12.64.1
Note the /32 behind the IP address, which signifies that there is only a single IP address in the subnet. This works pretty well.
In the meantime, however, I have 3 VMs running on that server that require their own IP address. Things are always changing, and Ubuntu 24.04 server no longer allows a configuration in it’s setup program in which the default gateway IP address is not part of the subnet. Fortunately, the three IP addresses I bought for my VMs are in the same /26 subnet (255.255.255.192). After configuring 65.22.33.161/26 as subnet and 65.22.33.161 as IP address, the installer would run and created up with a working server. The netplan file of the VM now looked as follows:
network:
ethernets:
enp1s0:
addresses:
- 65.22.33.161/26
nameservers:
addresses:
- 185.12.64.2
- 185.12.64.1
search: []
routes:
- to: default
via: 65.22.33.129
version: 2
OK, not much gained you might say. Well, that depends. If the VMs have to communicate between each other (which is the case in my setup), then using a /26 instead of /32 setup means that IP packets can go straight from one virtual interface to the other. When a /32 subnet is used on both sides, the IP packet has to go to the external physical switch / route and back to the bare metal server and from there into the destination virtual machine. A very nice shortcut!
But there’s a catch: In the /32 setup, the VM can communicate with the bare metal server underneath and hence also with the VMs behind the NAT. The /26 setup unfortunately prevents this and KVM even makes one aware while instantiating a new VM with a macvtap interface that the VM will probably be unable to communicate with the host directly.
For follow up reading, this page contains some more info on how packets are forwarded between macvtap interfaces.