In case you use Linux on the desktop, you might have been here before: When running out of memory, the system suddenly crawls to a halt and becomes totally unusable. In such cases it is often not even possible anymore to ssh into the system to reboot it. Sometimes, the system recovers after a very long time once the kernel finally gives up and invokes its out of memory procedures to terminate a process to free up memory. But that requires a lot of patience and the typical ‘quick’ solution is a power cycle. But recently I found a better way to deal with this: EarlyOOM.
Early Out of Memory (EarlyOOM) is a user space program that monitors RAM and swap. Once both fall below 10%, it selects a process that consumes a lot of memory and terminates it with a gentle SIGTERM. If the process is stubborn and remains running, SIGKILL will be sent once memory and swap fall below 5%. On the desktop, the usual culprits are Firefox, Chrome, Thunderbird or Liberoffice, as they just keep allocating more and more memory if not closed for a long time.
This works great in practice and I haven’t had a single ‘frozen’ system since I’ve started using it. Instead, I get the occasional “my Firefox has just crashed” user complaint that requires a bit of explanation that it hasn’t really crashed, but that’s much preferable from “my system has frozen again” complaint.
There are quite a number of configuration options to adapt the default options. In my case, I’ve put Thunderbird and Libreoffice (soffice) on the list of processes that should not be terminated. By default, the list already includes the init process, X, and the ssh daemon. To see how things are going, earlyoom makes a syslog entry once a minute, so the admin can observe how memory is slowly eaten bit by bit over time, and can even predict when earlyoom jumps into action to preserve sanity on the system.
It’s also good to know that EarlyOOM is not an ‘esoteric’ 3rd party program, as it’s included in many distributions. On Ubuntu, it’s only an ‘apt install earlyoom‘ away. On Ubuntu 20.04, it’s not quite the latest version, but still recent enough for me to avoid downloading and compiling it myself. More information on how to use it and the latest version can be found on Github.