When I recently read this article about vulnerabilities to break out of virtual machines, I became aware again of the concept to run virtual machines inside virtual machines. This is also called nested virtualization and it seems most modern Intel and AMD CPUs support this feature. But does it really work ‘just like that and out of the box’ and if so, what’s the performance penalty?
I don’t have a use for this in mind straight away. However, if nested virtualization should ever come in handy, it will be on my bare metal server I rent in the cloud. As described in the 5 episodes on the topic, the server is based on an AMD Ryzen 5 3600 processor with 6 cores, 12 threads, and 64 GB of RAM. To put this to the test, I spun-up a VM to which I assigned 6 cores and 8 GB of RAM. Inside the VM, I installed KVM in exactly the same way as outside the VM on the bare metal server. Once done, I then created a virtual machine in the virtual machine, assigned it 6 GB of RAM and all 6 processor cores that the outer virtual machine has available. Like on the bare metal server and the first virtual machine, I then installed ‘Ubuntu 20.04 server’ in the nested virtual machine. Worked like a charm!
The nested VM looks and feels like any other virtual machine or bare metal server. But what’s the speed penalty compared to the bare metal server it runs on and to the virtual machine it runs in? I put this to the test by running my ffmpeg video transcoding benchmark and compared it to the the values I got when I ran my benchmark in the “normal” virtual machine a few weeks back. There, my ffmpeg benchmark ran for 5:05 minutes and had a speed-up value of 9,34. In the nested virtual machine, ffmpeg ran for 5:18 minutes, which resulted in a speedup of 9,01. That’s only 4% slower compared to the ‘normal’ VM, which itself is around 10% slower than the bare metal machine.
Note that my benchmark focuses on the CPU, as the input file’s size is only 200 MB, so there’s little IO activity. I’m not sure what kind of performance penalty the simulated disk on the simulated disk creates, perhaps this is something I will chase up in a follow up post.
In any case, a CPU performance penalty of only 4% of the nested VM compared to the ‘normal’ VM sounds like a very good value to me. And a final note: If you rent a VM in the cloud, it’s likely that nested virtualization is disabled in your VM. At least it was in my VMs I rent at Hetzner.
So, let’s come back to the question what nested VMs could be useful for!? I could think of experimenting with Kubernetes that uses a couple of nested VMs in one big VM as master- and worker nodes. Or it could be a nice playground for network configuration and routing tests that involve several servers, all in a single big VM with a simulated network inside that connects the nested VMs, so any network changes on the inside will in no way touch the bare metal server. Hm, I’m beginning to see interesting possibilities here!