systemd, as a service manager, is not actually a bad piece of software by itself. The fact it can act as both a service manager and an inetd(8) replacement is really cool. The unit file format is very nice and expressive. Defining mechanism and leaving policy to the administrator is a good design.
Of course, nothing exists in a vacuum. I don’t like the encouragement to link daemons to libsystemd for better integration – all of the useful integrations can be done with more portable measures. And I really don’t like the fact they consider glibc to be “the Linux API” when musl, Bionic, and other libcs exist.
I’d like to dive into detail on the good and the bad of systemd, as seen through my eyes as all of: end user, administrator, and developer.
↫ awilfox
awilfox is a maintainer of Adélie Linux, which does not use systemd, but this blog post is one of the few reasonable, well-written, and substantiated critiques of systemd – as opposed to the usual mindless screeching you usually hear about systemd. A great read.
Thom Holwerda,
Many of us have been pointing out the exact same points the author makes right here on osnews. I generally agree with what the author is saying, it is a good review of the critiques. As someone who never liked sysvinit, I think systemd is a lot better, but then it stepped on many other projects and norms without any regard to the unix ecosystem and “KISS” principal. Systemd leaves us with less choice and more coupling to systemd specific tools (what the author calls “tentacles”), which aren’t necessarily better than the tools that systemd replaced and sometimes are worse.
I couldn’t have said it better.
You may say I’m “screeching” Thom, but I’m just pointing out that systemd isn’t right *for me* yet. And I do mean “yet”; like Wayland, it could one day become what it seems to want to be. But that time is not now, so for now I avoid it when possible, which is easy to do when the best OSes out there today (Void Linux, OpenBSD, and Haiku OS) don’t use it anyway.
Hey you might be confusing my Wayland position here. I have no particular feelings for or against systemd, as I barely have to interact with these things to begin with. The few times I had to – when messing around – I found systemd perfectly easy to use, but I’m no sysadmin or whatever so I really have no opinion on if it’s any good or not. Similarly, when I use Void, I’m just as happy with how easy runit is to use as a mere user.
Sorry, I’m so used to being called psychotic or brain-dead or a zealot (not by you, by others here and elsewhere in the tech world) because I don’t want to migrate to a distro that uses systemd. My apologies, your comment about screeching just rubbed me the wrong way. That’s a me problem, not a you problem: You’re right that there are some pretty loud and obnoxious voices in the anti-systemd crowd.
Anyway, my aversion to systemd is purely pragmatic: It doesn’t solve any problems I can’t solve with runit on Void or sysctl and rcctl on OpenBSD. I prefer simple and straightforward to overly complex and broadly scoped, and systemd as it stands now is a ton of stuff I don’t need to accomplish simple system management. But that’s just for me; I am aware that it has taken such a foothold in the Linux world not just through political maneuvering behind the scenes of the major distros, it does actually make sense in some scenarios, and people smarter than me like it, so I won’t say it’s bad and leave it at that. It’s just not for me.
And I’m glad to know you use and appreciate Void, I really wish its userbase would grow so they can get some more developers and package maintainers to work on it. I also wish I was skilled in those areas enough to contribute.
I see a difference between Wayland and Systemd. Wayland takes a lot of flack for not being exactly X11 that I do not think is fair as an exact clone of X11 is obviously not the goal. The project also takes a lot of heat for its elitism which is perhaps fair but, despite that criticism, the project seems to be headed in a direction of addressing real needs in a cooperative way. In fact, one of the criticisms of Wayland is that it relies too much on technologies that X11 fans do not see as making sense for a display server ( like dbus and pipewire ). Wayland is not trying to “be a platform” but it is trying to define a platform in terms of the cooperation between different services to provide the overall capability. Still, it does not mandate anything as low level as which C library you are going to use.
I feel that it is totally valid to say “Wayland does not meet my needs…yet” because it seems reasonable to assume that it will get there.
Systemd on the other-hand seems to be going the other direction. What we had before was a situation like Wayland is promoting where we had cooperation between different components. Systemd, like X11, is trying to be the complete platform it seems. It is at least modular but, at the same time, it seems to want to be responsible for everything. For the parts it does not implement itself, like the C library, it is very narrow in what it supports. When people complain about the viral nature of Systemd, they get called out as irrational opponents of progress. Even Thom’s “screeching” comments feel along these lines.
I am not sure that you can say that “Systemd does not meet my needs…yet” as it may be an explicit non-goal of the project to support you. Does Red Hat or Systemd see supporting musl as a good thing( or relibc, or LLVM libc, or BSD libc, or anything else )?
Both projects get accused of being too “Linux centric”. Again though, I think the case could be made that the kinds of things Wayland is relying on are just part of the evolution. Other platforms can either reject these and stay on X11 or they can evolve and adopt Wayland and other components it relies on. It is more of a rethink than a land-grab. Creating a hard dependency on something like glibc is a lot more of a land-grab though. The GNU/Linux crowd would obviously disagree with me but the C library is not what makes a Linux distribution a Linux distribution. Adelie Linux, Alpine Linux, Void Linux, and Chimera Linux are all definitely Linux distros but either explicitly or optionally non-glibc.
People that resist Systemd get called out as luddites and “boomers” ( regardless of age ) as they are seen as opponents of change. But you get distributions like Chimera that embrace solutions like Wayland and Pipewire explicitly to jetison “legacy” components and yet still reject Systemd. The Chimera Linux author by the way has also written in a very balanced and reasonable way about Systemd.
Systemd became popular due to influential sponsorship and how terrible the alternatives were at the time. Yet people continue to advocate for alternatives to it ( not just old ones but new ones as well ). I do not see that happening with Wayland. It may take a while to get everybody on the boat but there are no “Wayland” alternatives on the horizon. It will be more of a one way transition, regardless of of how long it takes. I am not convinced that everybody will use Systemd ( not just yet but ever ). Not everybody uses Wayland…yet.
tanishaj,
I hear this alot, but honestly I don’t see that, at least not here on osnews. We don’t expect wayland to be X11, but it’s slightly offensive when we keep being told that wayland is ready or that X is dead (as Thom’s been doing) when wayland functionality still remains broken. I think team wayland needs to focus less on declaring wayland ready and more on getting actually wayland features finished for everyone (not just gnome users).
Yes, I think it’s fair to say that, at least I hope it gets there.
You make an interesting point. Systemd works for me now and it technically meets my needs, but I do feel the coupling of daemons to systemd was unnecessary and keeps creating controversy by stepping on software developers and admins who ordinarily wouldn’t have strong objections to an init system. Ordinary init systems aren’t offensive the way systemd is.
There were viable alternatives, I even had my own. But history is written by the victors and redhat’s sponsorship was extremely influential. Redhat could have taken a simple init system like runnit or openrc, enhanced it with a few missing features like jails/cgroups, and boom 99% of users and admins would have loved this outcome. A strait forward init that works well and is true to unix philosophy with no tentacles – “Do one thing and do it well”. What’s not to love. The work could have even been appreciated by both linux and BSD users. I really like this idea of unification, but for whatever reason linux developers seem keen to move away from a common unix platform to an environment full of non-portable linuxisms. Oh well, I guess that’s life.
Alright. What if I consider glibc to be the Linux API? Me. What if it was me?
Libc incompatibilities isn’t why I run Linux. I tried running Alpine, and that’s when I realized, I might as well run a BSD. They’ve had decades of fixing libc incompatibilities, and Linux is merely a means to an end.
From the article…
There’s plenty of competition. Most haven’t gained any traction outside of the niche distro hosting them, or got replaced in the case of upstart.
OpenRC is the most well known. There is also runit (Void) and GNU shepherd (Guix). I think Alpine is starting a new Linux init project to replace OpenRC.
I happen to like journald. I don’t think it has reached it’s full potential, but it’s nicer then what was there before.
The problem is letting libc handle DNS look ups. A real DNS resolver at the system layer is awesome!
https://blogs.gnome.org/mcatanzaro/2020/12/17/understanding-systemd-resolved-split-dns-and-vpn-configuration/
Unbound, PowerDNS Recursor, DNSmasq, and Bind are too much. DNSmasq has way too much junk. Bind is a recursor and authoritative DNS service. Unbound and PowerDNS Recursor are enterprise level recursors focused on servers. As much as I like Unbound and PowerDNS recursor, I’d rather not run them on endpoints. A resolver which is simpler and more focused would be better.
FreeBSD ships Unbound as the local recursor, and it’s annoying and way to much. A simpler daemon would be better and give a better user experience.
OpenBSD used to ship Unbound in base, and they replaced it with the simpler unwind.
Anyway, I’m very happy resolved exists. Somethings are half-baked, but that’s kind of Linux.
It’s tool for systemd devs to test changes to systemd. That’s where it came from.
It also happens to be useful for simple system containers. I use them quite frequently on my home servers. The networking is simpler then application containers, like those created by podman or docker, since they can get an IP address from my DHCP pool.
They do have their limitations, but most of the time I want an isolated Linux install on my system to test something or run a service. LXC could possibly do it. That’s more services and daemons when I would like something simple, and mostly, out of the box.
Because tightly coupling the tools to the system gives better results, or at least allows devs to improve user land. A kernel is kind of useless without the tools, and GNU tools being general tools has been a problem in the past when devs have wanted to make changes to user land. This is why the iproute tools exist, and why ifconfig got deprecated.
NetworkManager would be the GUI. NM is an abstraction layer. It’s like how Firewalld manipulates nftables, and previously iptables, under the hood, or how PulseAudio or Pipewire are built on top of Alsa for audio.
Networking is pretty fundamental to computers these days, so it makes sense to push it down.
On a server, NetworkManager is a rather dumb idea, and not something I particularly like. Remember all the articles which said to remove NM from servers as one of the first things to do after an install? Yeah, they’re still valid. networkd is an upgrade, especially if a server is managed with config management. Just drop a file!
???? The real NTP client is timesyncd. timedatectl is just a tool to configure things like time, date, and timezone.
Importantly, timesyncd is only a client. Stripping out the server part simplifies the configuration and leans out the code on the endpoint.
systemd-boot only supports EFI, which makes it non-portable and inflexible. You won’t find EFI on Power or Z, and I have plenty of ARM boards that don’t support mainline U-Boot as well.
…
What is concerning is the fact that distros like Fedora are pivoting away from GRUB in favour of it, which means they are losing even more portability.
Configuring systemd-boot is much nicer then configuring Grub.
One, Fedora doesn’t support that many tier 1 ISAs. It’s not Debian or NetBSD, and it’s mainly focused on x86 and aarch64, mostly server stuff which uses UEFI to boot. If it works on the platforms they support, they should go for it. A Linux distro doesn’t have to be everything to everyone.
Two, it’s been proposed, and the proposal didn’t go anywhere. Mostly due to broken UEFI implementations, if I remember correctly. OpenSUSE might be closer to having systemd-boot as an option in the installer then Fedora. Anaconda being inflexible is a bigger problem then Fedora’s default bootloader.
Three, UEFI is being pushed as the universal way to boot a system. Good and bad, people want UEFI everywhere.
I’m having to install and configure many of these tentacles on Fedora (Fedora!), so they aren’t doing a very good job of being tentacles.
The tentacles are rather nice. Leaner, fitter, more productive servers make me happy. Yeah, some are half-baked, but that’s kind of Linux. There are many things which are half-baked.
Flatland_Spider,
My problem with most of Lennart Poettering’s work is his severe case of not-invented-here syndrome. There are FOSS projects lead by experienced developers with mature, secure, and capable work, etc. But instead of working with them he always reinvents the wheel with his own resulting in a less capable, less flexible, less thought out solutions that hold back standards like secure DNS on linux. Hopefully systemd catches up eventually, but it’s a case of “one step forward, one step backward” rather than “several steps forward” we should be striving for.
I faced similar issues with containers that were more capable prior to systemd. Of course eventually containers got rewritten around systemd specific interfaces – at the expense of other init systems. BTW. At least the dust has settled, but it does seem to be a recurring problem for FOSS. I wish we had a way of incentivizing more team players. It could just be wishful thinking on my part, but I feel that collectively we could accomplish so much more in the long run, not only for linux but BSDs and others as well.
I really don’t mind light weight daemons. The thing is nobody was asking for an init system with these tentacles, they didn’t need to be there at all. They were solutions in search of a problem. The method by which systemd resorts to tight coupling and displacing alternatives is arguably an anti-feature for FOSS. Not only is it harder to for owners switch, but it hurts compatibility with other software and unix systems. I concede that sometimes incompatibility is inevitable, but sometimes I worry that incompatibility may actually be a strategic goal rather than a mere side effect. We expect these kinds of monopoly strategies from commercial operating systems, but it feels wrong to me when FOSS developers do it.
I agree on the Glibc categorization. It is the standard, if you like it or not.
Try to run a JVM on a ARCH64 Void/Musl container and see what you get. It’s just big mess.
SystemD is a different thing though. It works fine on all my devices, but I never really liked it:
1) setting up a simple “TimeTask” is an endless trial and error
2) it is totally unclear, where the Units shall be stored. Some are under /usr/… some under /etc/…
3) the commands are terrible to memorize. Whenever I have to change services I end up in a long ChatGPT session and lots of copy’n paste
OpenRC just did everything as well as SytemD and somehow never stood in my way. As shame most distros dropped it.
Largely agreed. I think there is a lot of confusion on what is or is not a required part of systemd “tentacles” And to be fair its not really clear in some instances. The systemd tools in the repo are there just for a convince, not all of them are required to be used simply because systemd is the init system. Nothing forces you to use nspawn for containers. You’re free to use whatever. its not a monolith and it can be excluded from the system. Journald to my knowledge is not optional and needs to be there to handle logging correctly from the system and services.
Bill Shooter of Bul,
That’s not the full problem though. It’s not so much that we can’t build a distro with parts of systemd that we want, but rather that omitting components of systemd you don’t want (in favor of components with better feature set for example) breaks package dependencies and now you have to go about fixing hard coded dependencies (and maintaining) packages that don’t belong to you. In this sense it’s certainly much more monolithic than other init systems ever were. Systemd can also require processes to use it’s APIs for things like managing cgroups.
Unlike most other loosely coupled init systems, changing away from systemd isn’t trivial and running it as a subordinate init system is a no go. Even in instances where it would be genuinely useful to run systemd under itself, such as in a container can be problematic for users 🙁