Despite running a lot of services in my cloud at home and in a data center, I am very positively surprised that unless I want to actively add, remove or change anything, there is very little intervention required on a day to day basis. That’s because apart from very few exceptions, my security updates are automated or semi-automated with Ansible scripts. But I have noticed that over time, some technical debt has crept up that needs to be dealt with now.
That’s mostly because some of the services I host simply break when updating the underlying operating system from Ubuntu 18.04 to 20.04 (or now perhaps 22.04). Fortunately, all services in question run in virtual machines, so going back to a sane state after a broken update is easy by creating a VM snapshot before the update.
While I don’t mind running some of my services on an older but still supported Ubuntu variant, the lifecycle of Ubuntu 18.04 is coming to an end in 2023. So it’s time to think about what to do for those cases. While that’s of course a bit of a pain point, it also triggers the healthy process of deciding whether its worth to reinstall that service on a new OS version, retire it for good or to look for an alternative.
So here’s the services and my current line of thinking what I will likely do with them in the next couple of weeks or months:
OpenVPN: Currently the virtual machine I run my OpenVPN installation on is still running Ubuntu 18.04. When I do an OS upgrade, OpenVPN just stops working. I’m not sure at this point what goes wrong, so perhaps going trough the logs will reveal a trivial reason that can be fixed easily. For some time, I thought about sending OpenVPN to the junkyard for good and rather put a Wireguard server in its place. Unfortunately, Ubuntu 20.04 doesn’t have a Wireguard VPN GUI plug-in and it’s even still missing in Ubuntu 22.04. There are third party add-ons for this but I don’t want to go down that alley. So even though it will be a bit of a pain, this service needs to stay.
BBB: I use BigBlueButton for audio and video conferencing a lot and I currently run version 2.4.x on Ubuntu 18.04. This summer, version 2.5.x became available, which is based on Ubuntu 20.04. BBB recommends reinstalling on a few OS install instead of updating. Fortunately, I still remember from last time how to reinstall BBB without loosing my Greenlight frontend configuration and user data, so I can hopefully do this within an hour or two.
Fritzbox/Grafana: Two years ago, I built a monitoring system for my VDSL router at home (a Fritzbox), based on an Influx-DB, some glue software and a Grafana frontend. Unfortunately, the glue software has become unsupported in the meantime and updating the underlying OS from Ubuntu 18.04 to 20.04 breaks the setup. I like the Grafana frontend and analysis possibilities, but by now it’s mostly a nice to have service that I only use occasionally. Also, since the glue software that connects the Fritzbox API to the database is no longer maintained, I guess I have to look for something completely new. Or I keep running it and go for the Ubuntu 18.04 extended support line, which is free for non-commercial setups. But it would just delay the inevitable.
Mail server: For internal emails, I have my own mail server based on Dovecot and Postfix. It was a major pain to get it installed initially and updating from Ubuntu 18.04 to 20.04 just utterly breaks it. Quite frankly, I’m not even in the mood to start debugging this one, even though it’s a service that is essential to me. So perhaps I will look at an entirely different setup to replace it. Mailcow or Mailu that hide the complexity in containers might be the right thing for me.
Turn servers: I run two turn servers, one for Nextcloud Talk and one for my Prosody XMPP server for voice and video calls. Both also break when updating the OS. Fortunately, these are relatively easy to set-up with a few commands and a few changes to the default config files so I guess I will just re-install them.
Yes, so it looks like a bit of work is ahead for keeping my cloud in a good shape. For the re-installs, I will likely use Ubuntu 22.04, which will buy me 4 years until I have to repeat the procedure again.