SSD, ZFS and VM performance – Workstation Power At Home – Part 12

Now that I have my ZFS filesystem spanning several drives in place on my Z440, I wanted to get an idea of how this setup is performing together with the underlying LUKS encryption. And while I am at it, the other questions I had in mind were: How does this compare to the ext4/LUKS combination I use on my notebook and how does the write performance look like from inside a virtual machine running on my Z440. A lot of obvious results but also quite a number of surprises.

To answer these questions, I wrote 10GB files to the different combinations of hardware, encryption and filesystems in two different ways as shown in the result graph at the top of the post: With random input data from /dev/random and with a text pattern. Here are the commands I used:

date; dd if=/dev/urandom of=10GB.bin bs=64M count=160 iflag=fullblock; date

date; dcfldd textpattern="Hello world" of=10GB.bin bs=64M count=160; date

Generating random data seems to be quite CPU intensive and as can be seen on the left side of the result graph, the Z440 can do about 200 MB/s when writing directly to an ext4 filesystem on one SSD and also to the ZFS filesystem that spans two SSDs and uses LUKS for encryption. The performance of the VM was around 8% lower. No surprises in these three numbers. My X250, however, seems to be a lot slower when it comes to random number generation and only delivers a fraction of the Z440’s values. That’s not a CPU limitation, however, as single core performance is around 70% of the CPU cores in the Z440. So this must be cause by something else.

Pure Write Performance – ZFS Shines!

As I didn’t really want to test the speed of the random number generator hardware and software but of the filesystem, LUKS encryption and the hardware, I ran the same tests again by writing a text pattern to get a 10 GB file. The results are shown on the right side of the graph. The performance of the X250 and the Z440 writing to a single drive, one with and the other without encryption are about the same and pretty much reach what the SSDs are capable of doing over the 6 Gbit/s SATA interface. When doing the same test on the ZFS filesystem that spans 2 SSDs on the Z440, the datarate almost doubles to 900 MB/s. That’s because ZFS writes the file to both SSDs simultaneously. This is why the yellow bar sticks out so significantly and also shows that the SATA ports on the Z440 are independent of each other.

LUKS CPU Requirements

The second screenshot on the left shows the difference in CPU usage when writing to a non-encrypted ext4 partition on my X250 notebook versus writing to a LUKS encrypted partition on the same SSD. In both cases, I achieved around 430 MB/s write performance. With LUKS encryption, the CPU is heavily involved (upper 4 graphs) while writing straight to the SSD requires almost no CPU power at all. The lowest graph in the screenshot is the throughput to the SSD which shows no difference between the encrypted and non-encrypted scenarios.

Puzzling VM Performance

I then tried the same in the virtual machine on the Z440 that also runs Ubuntu 20.04 and which wrote its data to a virtual ext4 filesystem in a file placed on the ZFS filesystem with 2 SSD drives and LUKS below. The big surprise here is that while ZFS again uses both drives to write the data to, the resulting speed is far below from the result of performing the same operation on the Z440 directly. I don’t think it’s CPU, because in the random generator test, the result was similar to the result achieved on the host itself. So I suspect the virtual disk driver as the most likely bottleneck.

Summary

So my takeaways from this exercise are the following:

• ZFS performs very well.
• LUKS encryption is quite CPU intensive but the Z440 can handle it well.
• For some reasons, the VM only writes at 50% of the speed that can be achieved on the host doing the same operation.