The 2 Hour Kernel Compile

Recently, Ubuntu 20.04 LTS bumped the Linux kernel version they use from 5.11 to 5.13. While that is generally welcome to support newer hardware, it unfortunately also broke the suspend/resume functionality of my notebook with an AMD Ryzen 7 4750U CPU. Bummer!

This seems to be a known problem in the 5.13 kernel line, and was subsequently fixed somewhere down the road. It’s definitely ok again in the current Linux 5.15 long term support kernel. More good news: Ubuntu 20.04 is likely to switch to 5.15 in a few months from now, and I could of course just stay with the last 5.11 kernel until that time. However, I’m not really happy with that, because security issues are no longer fixed in in 5.11, which potentially exposes me to security vulnerabilities in the next couple of months.

Long story short: To be on the save side, I started looking for ways to use a 5.15 kernel with Ubuntu 20.04 until Canonical moves to that kernel line on their own. Fortunately, there are options:

The Easy Way If You Don’t Have Self Compiled Kernel Modules

The most straight forward way is to use the 5.15 kernel line offered by Canonical themselves. There are tutorials out there that show how this can be done easily. The downside: Canonical’s 5.15 kernel line is compiled against library versions that are not included in Ubuntu 20.04. While the kernel doesn’t mind and works just fine, the header files can’t be inserted correctly, so programs that compile drivers against the currently running kernel with DKMS such as Virtualbox won’t run anymore. Therefore, that’s not an option for me.

The Latest Kernel With Correct Header Files – There’s a PPA For That!

I don’t seem to be the only one with that problem and Mark Boddington has come up with a nifty way to compile the latest kernels with Ubuntu 20.04’s library and header versions. His kernel compile script runs in a Docker container that is based on an Ubuntu 20.04 Docker image, and compiles any kernel version specified. His fully automated solution pushes the result into a PPA, which can be added to the list of a system’s apt update system. This way, one automatically gets the latest kernels (5.16.x at the time of writing) whenever a new bugfix or release becomes available. I gave this a try and it works like a charm.

Reducing the Chain of Trust and Compile Locally

Obviously, adding a PPA to a system increases the number of people and processes you have to trust, and I try to avoid this as much as possible. As Mark has put the script and the Dockerfile on Github, it’s easy to create the Docker image locally and then run it to compile the kernel locally as well. I’ve had a look at the Dockerfile and build.sh file that is executed in the container, and from a security point of view, they looked fine to me. The kernel source tree that has to be git cloned to a local directory is owned by the kernel team at Canonical (the company behind Ubuntu), so that looks fine as well.

When it comes to the Docker container image, one can either pull it directly from Docker Hub, or, which I did in my case, I created the image locally, so I don’t have to trust an external source about the contents of the image. The following Docker command takes the Dockerfile from Mark’s Github page as input and creates the image for the container locally:

docker build -t martin-build-environment:latest .

And here are the two commands to run the locally generated container instead of pulling it from Docker Hub to compile Linux kernel version 5.16.3:

# Create a folder for the source and the generated *.deb files
#
mkdir /mnt/extra-drive/kernel-build
cd /mnt/extra-drive/kernel-build
mkdir debs; mkdir source

# Let's make sure the Docker container has access to the folders
#
chmod -R 777 /mnt/extra-drive/kernel-build

# Now git clone the Kernel source to the local directory
#
cd source

git clone --depth=1 -b cod/mainline/v5.16.3 \
  git://git.launchpad.net/~ubuntu-kernel-test/ubuntu/+source/linux/+git/mainline-crack .

docker run -ti -e kver=v5.16.3 \
   -v /mnt/extra-drive/kernel-build/source:/home/source  \
   -v /mnt/extra-drive/kernel-build/debs:/home/debs \
   --rm martin-build-environment:latest \
   --update=no --flavour=generic --shell=yes

On my hexacore Xeon based workstation, compiling the kernel takes around 2 hours. Yes, it’s quite a bit of stuff that runs through the compiler and linker. While the output files are only a few megabytes in size, intermediate space requirements for object files is around 60 GB.

Once the .deb files are generated, they can be installed on an Ubuntu 20.04 system and removed again as described in this post.

Summary

Depending on your level of trust, installing the latest mainline kernel on an Ubuntu 20.04 system is either a super straight forward process by just adding Mark’s PPA, or a bit of additional work with a local Docker container, a compile session and manual kernel installation afterward. Canonical stresses that mainline kernels should not be used on production systems, which I think is good advice. However, a broken suspend/resume functionality and no security patches for a few months is also not something I can live with on a production system. So one has the choice.