In case you missed it, Red Hat announced they will no longer be providing the means for downstream clones to continue to be 1:1 binary copies of Red Hat Enterprise Linux (RHEL). Very quickly, both Jack and I shared some initial thoughts, but we intentionally took our time deciding the next right step for AlmaLinux OS. After much discussion, the AlmaLinux OS Foundation board today has decided to drop the aim to be 1:1 with RHEL. AlmaLinux OS will instead aim to be Application Binary Interface (ABI) compatible.
[…]For a typical user, this will mean very little change in your use of AlmaLinux. Red Hat-compatible applications will still be able to run on AlmaLinux OS, and your installs of AlmaLinux will continue to receive timely security updates. The most remarkable potential impact of the change is that we will no longer be held to the line of “bug-for-bug compatibility” with Red Hat, and that means that we can now accept bug fixes outside of Red Hat’s release cycle. While that means some AlmaLinux OS users may encounter bugs that are not in Red Hat, we may also accept patches for bugs that have not yet been accepted upstream, or shipped downstream.
I wonder just how much consumers care about the strict 1:1 with RHEL. With this change to AlmaLinux, we’re about to find out.
I like this move by Alma Linux.
This allows them to create a real distribution that is able to contribute, fix bugs, and differentiate itself where it makes sense. It will be a RHEL clone in that almost all RHEL documentation should apply to it and it will be ABI compatible but it is not trying to guarantee that it is bug-for-bug “identical”.
Pretty much everything Red Hat does will benefit Alma Linux as, in addition to documentation, Red Hat does contribute back essentially all their code to CentOS Stream. Hopefully this more clearly exposes what nonsense some of the more extreme rhetoric we have seen around Red Hat becoming “proprietary garbage” ( actual quote from a prominent Linux You Tuber ) as both Red Hat and Alma will now be downstream of exactly the same community effort and contributing together to the same code base.
In my view, this allows Alma to actually practice the “community values” that the other RHEL clones have been making noise about in pursuit of their commercial ambitions. This is what Red Hat wanted them to do.
Rocky has said that it will continue to release a bug-for-bug replica of RHEL by finding alternative ways to get the actual RHEL packages. Unlike Alma, this means that Rocky cannot even contribute a bug fix to “the community” without breaking their core value proposition. Oracle will be doing essentially the same although Oracle does differentiate from RHEL in the kernel I believe.
It is a bit unclear to me what SUSE is doing. They have said that they will “fork” RHEL to create “RHEL compatible” distro that they will then support via their Liberty Linux offering. Most people have interpreted that to mean they will do what Rocky and Oracle are doing ( though perhaps through more heavy lifting of their own ). It could be that SUSE is proposing something more similar to what Alma is now doing though. I hope they are.
tanishaj,
Distros that fork centos-stream are fine. But this particular argument seems bogus to me. Alma linux and rocky linux were never prevented from sending patches to centos-stream before. Naturally it takes patches longer to reach RHEL because it’s more stable and has to undergo additional QA before going downstream. This is also true of other unrelated distros. Consider the relationship between debian testing and debian stable. Are we be critical of debian stable forks of on account of patches getting into debian testing first? No we aren’t, that’s silly. This is just the inherent nature of stable distos.
I think the real crux of the issue, what’s getting people upset, is distros that are freeloading. These side arguments like RHEL forks not being able to contribute are rather weak and don’t hold much water.. If there were no issues with the fairness of copying RHEL, it would be perfectly sensible to fork RHEL *and* contribute to the upstream distro designated for testing & merging, which is exactly what centos-stream is.
I see your argument.
RHEL is not just a stable snapshot of CentOS Stream in the say that Debian Stable in its project though. The fact that you do not know exactly what point any given package in RHEL is from is why people are so hot and bothered about getting it from Red Hat direct instead of just CentOS Stream. The kernel in particular is a menagerie of old base code and newer patches. Other packages, maybe key packages like Glibc or GCC, may be similar. So distros like Rocky are only indirectly contributing to their own distro if they send a patch to CentOS Stream.
Also, Debian Stable can still get patches ( especially for security ). It is not like they have to wait years for stuff to come down from the more active branches. What you are describing relates more to functionality than to security and compatibility. Rocky cannot even patch its own distro to address a security problem. Debian Stable sure can.
Does Rocky Linux ( or Alma before ) ever actually contribute any code to CentOS Stream? It is my impression that they did not.
tanishaj,
I see your argument as well. On some level, 100% of linux distros are downstream of the linux kernel project, but upstream submissions can take a very long time to come percolating through the pipeline.
Debian stable doesn’t get updates in a timely manor, software can be a year or more behind. You’re right that Debian prioritizes security patches. I believe the policy is similar with ubuntu LTS – don’t expect many feature updates, only security patches.
I wasn’t trying to compare rocky to debian stable. that comparison seems weird to me. This is more along the lines I was thinking…
Debian-Stable is to Debian-Testing the way RHEL is to CentOS-Stream.
A more suitable comparison for Rocky would need to be something that forks Debian Stable rather than Debian Stable itself. LMDE comes to my mind. However unlike Rocky, LMDE is NOT a bug for bug clone of Debian Stable. They install their own features and different/newer versions of software like firefox.
So when you say that “Rocky cannot even patch its own distro to address a security problem”, I must disagree. Technically Rocky can do that just like LMDE does, but it goes against their mission, which is to continue the original centos mission and create a bug for bug clone of RHEL. To the extent that you consider it a shortcoming, that’s fine, but my point is that it’s not a shortcoming that arises from forking RHEL, rather it’s only a shortcoming that arises from the distro’s own goals.
I don’t know the answer to your question, but I think this is what people are upset over. Those who fork a stable distro and are not putting in their own work to improve the distro are freeloading. This raises an interesting question for me: what if there was something like a “linux mint redhat edition”. Like Rocky they would fork RHEL but unlike Rocky they would introduce their own features and changes. In terms of the GPL both are equally permissible, but would this draw the same level of wrath given that they would be putting in more work? What do you think?
I agree. This is a good move for Alma and the EL community. 🙂
Suse is funding Rocky Linux.
This is a brave move. Alma is a “free Redhat”. That is basically their “USP”. Without that, I don’t get compatibility guarantee with software vendors. I don’t get to take advantage of the massive support network. I don’t get the financial stability guarantee of RH without the cost. I have specific bugs and the hope they might fix them without breaking something else. Why would I choose it over any other distro at that point? For my (business) use case they have basically taken themselves completely off the table.
Good points. As you say, the value for Alma for many people has almost certainly been that it is RHEL for free ( as in beer, not freedom ). I guess they are hoping people will interpret things less extremely. Let me address some of your points:
“I don’t get compatibility guarantee with software vendors”
Alma is saying they will be “ABI compatible”. Does that mean that they will use the same compiler and library versions for a given release? If so, I would expect RHEL compatible software to run just fine on Alma unless it depends strongly somehow on some bug that Alma has fixed ( a bit unlikely ). Any bug that Alma has that RHEL does not would get fixed I presume ( but really, what are the chances that it would not have already been fixed in CentOS Stream? )
“I don’t get to take advantage of the massive support network.”
I am not sure what you meant by this. The documentation and training? I would expect that to still apply. People with RHEL certifications or skills? Again, I would think those could be used just as effectively on Alma. The differences will be lower level than that.
“I don’t get the financial stability guarantee of RH without the cost.”
Do you mean that you can be sure that RHEL will continue to exist but cannot say the same for Alma? CentOS Stream also benefits from Red Hat’s financial strength. So, you still benefit there. You are correct though that whatever Alma “exactly” becomes could go away if the Alma project becomes non-viable financially. CentOS Stream will still be there though as well any other RHEL-a-likes.
When it comes right down to it, it all depends on how completely you depend on the “bug-for-bug” identicality of the two. If that does not matter, it is essentially a complete clone and the ability to pivot to another essentially equivalent distro will always be there.
Does everybody using Alma today actually need that? Or do they need all the same features, the ability to apply the same documentation / training / expertise, and the ability to run compatible software? I bet a lot of the people using Alma do not actually need it to duplicate RHEL so exactly. Will the people that do not need the same bugs actually appreciate some of them being fixed? Will people appreciate any extensions that Alma is able to add?
It is an interesting experiment.
To try to expand my thinking as much as this format allows
No one chooses Redhat to be cutting edge. They choose it for compatibility and LTS. Can I rely on Alma supporting a 10 year support (or longer) cycle? Realistically, no. Obviously, I might look back in 20 years on this comment and laugh at it’s idiocy!!
ABI compatibility is fine, but it’s the kernals and drivers that are what make Redhat Redhat. The UBI source code is all still available via those images, it’s only the driver/kernal layers that’s being limited. Alma will be offering something similar to Oracle’s unbreakable Linux kernal. It’s great, but not Quite the same as Redhat. For many users these differences truly will not matter. And they’ll probably have a small market that choose them over Suse or Oracle.
But when you need to ensure your hardware works, you need to know it’s running the same drivers. The more Alma diverge, the less documentation is cross-compatible and the smaller the Alma community simply won’t be able to help in the same way as the more established distributions.
You are completely right, it’s an experiment. Alma have rolled the dice, that people dont Need RH kernals/drivers anymore in the cloud era, and UBI is enough. They might well be right! Time will tell.
Adurbe,
That’s the thing, redhat doesn’t have the latest cutting edge software. Corporations are generally looking for stability above all else. Not only aren’t cutting edge features interesting for them, these can actually be the source of major headaches and support costs. To these corporations, changes are bad and boring distros are good. RHEL is a boring distro, which is good. This isn’t everyone’s cup of tea obviously, but for the corporations that it is true for distros that are identical to RHEL hold value because they are identical rather than because they get more updates or features.
I concur, it will be interesting to see how the market reacts long term. My gut feeling is that the cent-os stream branch is less appealing for corporate use cases than RHEL. If this weren’t the case, corporate customers could have switched from RHEL to a centos-stream already. For these kinds of customers, long term stability is king.
It depends on why they picked Alma.
If the want free RHEL, yes.
If the they have some software package, probably commercial, which mandates RHEL, yes.
If they want something RHEL like with a slow release cycle and timely security updates, something close enough will work.
Mostly this. They don’t have to learn new tools and adjust current tooling to account for, say, Debian-isms. Then there is SOC stuff which requires a changes to be vetted and approved.
Alma is the upstream of CloudLinux, so bug-for-bug compatibility matters less to them. (https://www.cloudlinux.com/)
Binary compatibility matters more. Breaking WordPress or some other web app would be a problem for them. Having newer versions of ruby, python, or php matter more then bug-for-bug.
Hosting a web app on RHEL/CentOS gets tedious after awhile, and it turns into a game of working around old library versions via 3rd-party repos, custom compilation, or environment managers. There is a market for a RHEL-like which is tailored to web hosting, so this is a good move and liberating for them.
Considering their website renders so badly on my android mobile phone, I wish them the best at peeling away RH customers looking for webhosting…
Yes, this is exactly what happens in practice. I administered a large PHP website, and it was a nightmare to manage. As described above, like a cat-and-mouse game of the PHP project dropping support for the version that the webapp was designed for, checking if the Linux distro still offers security fixes for that PHP version, if not then migrate to another distro that does, then eventually the webapp stops security patches for the old version and requires updating to a major new version with newer PHP requirements, which might in turn necessitate upgrading the Linux distro to a newer version or even migrating to a different distro. As much as I hate the piecemeal and opaque scripting magic behind modern docker-compose workflows, it sure works wonders to satisfy all of the requirements of the webapp and avoid those “games”. With container workloads the underlying distro and the package versions that it provides doesn’t really matter as long as it supports Docker or Podman.
But then that goes back to the question of the value of an ultra-slow changing 1:1 copy of RHEL, if containers have taken over the dirty work of satisfying dependency version requirements then it doesn’t really matter in a practical sense if the distro is RHEL or a 1:1 copy or a RHEL upstream or a slight downstream RHEL or even a completely different distro lineage. The only remaining hard requirement that I can see for an exact copy of RHEL is for some kind of certification requirement or proprietary software that runs checksums against .so libraries and refuses to run if they don’t match (which I have seen in practice). So it seems to me that the main raison d’être of RHEL and its 1:1 clones is just to *be* 100% pure RHEL, and if that specific requirement doesn’t exist I’m not sure that a slightly different derivative like AlmaLinux is now proposing will offer any more practical value than even a completely different distro from another lineage.
rahim123,
Here here.
I think most of us have had similar experiences with PHP. PHP was developed without any formal CS training and somehow it ended up becoming the standard it is despite such amateur roots. Fortunately every new version fixes old mistakes and the language has gotten much better. The problem though is that many of these transitions would cause major breakages leaving admins like us in support hell. Linux distros that bundled PHP were in the same boat. Regardless of what version is installed, something would be broken because changes between versions were incompatible. I hope the majority of this is behind us now that the language is more mature.
It’s not PHP per say. It’s the distros bundling everything and the kitchen sink rather then having a core and extras, like the BSDs do.
Same problem with Python. I have a Django app which is maxed at a Django 4.1 (?) due to the CentOS 7 only supporting Python 3.6, and I need to migrate that app to something which supports python 3.7+.
Newer versions of RHEL provide modules to install specific versions of tooling, but that is a all or nothing approach rather then fine grained.
Flatland_Spider,
Yes I agree BSDs handle a base system better. It would also help if the distros supported additional versions being installed side by side.
Is that really a python language problem though? It seems like the issue here is that centos 7 doesn’t make the latest version available (BTW I’ve had the same issue with debian stable).
The reason I am critical of PHP specifically, and I believe rahim123 as well, is because so many PHP updates have broken existing sites & scripts. This means even if your distro has the latest version of PHP (good), updating to it can break PHP software (bad).. I don’t typically face this much backwards incompatibility after updating other software and languages. If you are hosting a traditional multiuser/multiapp environment, you inevitably face the wrath of users who’s software got broken after PHP updates. I give customers their own environment to run whatever version they need, sometimes we need to isolate software at the application level too.
This should be one of the pros of flatpak/snap/appimage, although so far I haven’t had much hands on experience with them beyond seeing how much snap distros pollute the mounting namespace, haha.
It is all about the ability to tap into support agreements. Plenty of software (or hardware) is being supported only on RHEL. Not because RHEL is especially good but because it is stable, common, and well-defined. If the problem arises, the first question from the support team is usually about checking if the customer runs the software on the supported version of the OS.
The RHEL solved the problem of fragmentation of Linux. That is the thing that IBM is trying to milk right now.
If somebody wants an alternative Linux OS, not being RHEL, there are already options: Ubuntu LTS, Suse.
Bogdanow,
I hadn’t looked at this before, but nvidia’s doesn’t appear to provide official driver support for centos stream. only centos 7. Meanwhile RHEL 7 8 9 are supported and so are rocky linux 8 and 9 (which makes sense).
https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=CentOS&target_version=7
I have found 3rd party procedures for installing nvidia drivers on centos stream, but rocky might be preferred due to first party support. Also when it comes to nvidia’s virtual GPU licensing feature that uses DRM, an official distro might be required and it’s not clear whether centos stream can be used unofficially.
That’s the point. NVIDIA tests the drivers against the RHEL. If there is an issue, the support team will recreate it on the given version. If the problem is deemed to be linked to the OS, the ticket to RH will be opened.
This is good. I like that we now have two approaches. And I like that it makes choosing between Rocky and AlmaLinux just that much easier for people. Hopefully it means that we can consolidate the resources that believe the traditional 1:1 bug-for-bug rebuild is valuable into a singly community via Rocky.
The reason a ‘free RHEL 1:1’ is important, is the support and bug tracker from Redhat. I have used it many times to figure out why certain things are behaving a certain way, and how to fix it. Without that, and without Alma having their own (yet) it becomes problematic to justify what to suggest to use.