VM Liberation – Downloading the Disk of a Running VM from Hetzner

In a previous post I’ve had a look at how I could download the disk image of a VM running on a Netcup server and run it in a VM on my own bare metal server at home. The procedure is straight forward, because Netcup offers an option for this in the admin interface. A cool experience and I wanted to do the same thing with one of my VMs running in a Hetzner data center. Unfortunately, they have no such option. But then I remembered that a colleague told me a few months ago how that he downloaded the disk image of a running VM, so I gave the procedure a try myself. While the procedure requires a few steps, each one is pretty much straight forward to do:

A Word of Caution

One note before I start: It is also possible to perform the procedure that follows by shutting down the VM and launch it again in ‘rescue mode‘ This way, a rescue image is used to boot the VM and the real disk image just sits there as an unused block device (/dev/sda). But since the VM I wanted to download did not write anything to the disk apart from logs, I really wanted to try this procedure while the VM is running to check if the file system would be repaired to a sane state after downloading and running the disk image in a new VM on another server. If you have services with databases and other things running, you are probably better served by booting in ‘rescue mode‘.

That being said, here are the steps to download the disk image of a VM, running or not, and covert it for use with KVM/Qemu:

Step 1 – Download the Disk Image While the VM is Running

As disks are usually quite large, I chose a combination of ssh, dd and gzip to compress the disk image before transferring it over the Internet:

ssh root@myserver.com "dd if=/dev/sda bs=4M status=progress | gzip -c" | dd of=vm_disk_image.gz bs=4M

The virtual disk had a size of 40 GB and transferring it would take quite some time. Gzip compressed the data on the fly to a manageable 3 GB, mostly because the virtual disk was largely unused.

Step 2: Convert the Disk Image to QCOW2 Format

After I downloaded the disk image to my own bare metal KVM server, I unziped the file to its uncompressed version and converted it to the KVM/qemu qcow2 disk image format:

gunzip vm_disk_image.gz

qemu-img convert -p -c -O qcow2 vm_disk_image vm_disk_image.qcow2

At this stage, the raw image and the resulting qcow2 image have a size 40GB each again, i.e. the size of the original disk image.

Step 3: Set a New Root Password and Run the VM

Once in qcow2 format, the disk can be assigned to a new VM instance and the VM will boot up nicely. On first boot, it is detected that the disk was not shut down properly and a file system check and repair is performed. Exactly what I was looking for!

There are two problems however: First, the network configuration is not yet adapted to my local network and second, I don’t have a root password for the machine, because I only ever connected to it over ssh, which is not possible at this point because the network is not running.

The fix: Set the root password of the system running in the VM with KVM/qemu commands on the bare metal host. The commands temporarily mount the disk image and change the root password. A bit scary if you think about it. The disk image is not encrypted, however, so it’s only logical that the root password can be changed from the outside. This gave me a bit of an idea how secure/unsecure a VM is if the bare metal server on which it is run is not under your own control:

sudo virt-customize -a vm_disk_image.qcow2 --root-password password:super-secure-password

There’s also the option to specify –root-password random, but I didn’t use it because I had to type the password in by hand the first time. So I chose a password that I could type-in easily and changed it again once ssh access over the network was restored.

Step 4: Reconfigure the Network in the VM

Accessing the VM with the root password can be done in two ways: Either via the shell window in the Virtual Machine Manager (VMM) GUI or on the shell of the bare metal server with the following command

virsh console VM-NAME

The later has the advantage that copy and paste with keyboard shortcuts works, so I could copy a template network configuration into the VM without a lot of effort. The console command only works if the VM has the console activated on the serial interface. This is not the case for a default Ubuntu installation, but Hetzner configured it that way when the VM was first instantiated on their server, and I was glad for them giving me a hand this way.

And here is my network configuration with a fixed IP address that goes into the yaml file in

/etc/netplan

The two things I had to change in my template was the MAC address and of course the IP address the VM gets assigned in the local network

network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: 52:ab:cd:11:ca:fe
      set-name: eth0
      dhcp4: no
      addresses:
        - 192.168.11.242/24
      routes:
        - to: default
          via: 192.168.11.1
      nameservers:
        addresses: [8.8.8.8,8.8.4.4]

Important: To make sure the changed configuration is not overwritten during the next boot, network config on boot has to be disabled in:

/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

in my case the file was not present, so I created it and added the following line:

network: {config: disabled}

And that’s it! After a reboot, the network comes up and ssh access is possible again. At this point the only thing left to do is to change the root password again, even if password access over ssh is already disabled.

To make the cycle complete, the next step would now be to upload the ‘rescued’ VM to a Netcup bare metal server. More about that in the next post.