In the past few years, the Linux community has polarized over the “systemd” management tools that have been created and slowly crept into all of the major distributions. Some love the tools and think they’re a big step forward in handling server-level events, and others abhor them and feel they’re a betrayal of the “Unix philosophy” of many small, simple tools instead of a single monolithic one. I’m in the latter camp. I can’t stand systemd, the arcane commands necessary to use and monitor it, and the way it basically takes over the system.
The latest fun happened this morning when I was upgrading servers to apply security patches. Apparently something had gone wrong with the systemd process on one server, and as a result it was not only impossible to upgrade the software, but many common system commands simply didn’t work. They’d hang for a minute or two and then report that systemd had timed out.
Finally I decided to reboot the server, thinking that it would reset systemd and get things back to normal. Fortunately it wasn’t a production machine, since that would’ve taken down client web sites or databases or whatever. Imagine my surprise when I couldn’t reboot!
Failed to start reboot.target: Connection timed out
See system logs and ‘systemctl status reboot.target’ for details.
Failed to open /dev/initctl: No such device or address
Failed to talk to init daemon.
Wow. Because systemd is so tightly integrated with low-level system processes, it controls the shutdown process… and since it was horked, I couldn’t run a simple reboot command. I was eventually able to work around it and get the server back online, but it reminded me of one of the many reasons I dislike systemd. Argh.