What follows in the article is an analysis of the “dependency hell” problem in Linux and the issues surrounding software installation. I make a proposal of my own and critique the suggested “solutions” to these problems and point my finger at “the” (my perception) source of these problems.
I have used Linux for almost 8 years now and exclusively for the last three, so I have experienced what many refer to as “dependency hell”. Two years ago I switched to Gentoo which radically reduced my experiences with dependency hell. I will not say it eliminated it-for I love to play with the very newest and latest, but as regards those packages in portage I have had extremely few dependency problems -read perhaps 3 times.
In fact my experience with Gentoo has been so positive that I doubt I will ever use a binary distribution again-except the one I roll for myself. With this as a background you can probably imagine the immediate utility of “grand solution” to “dependency hell” for some one like me: ie. negligible.
Yet I wish that some of what I have grown accustomed to in Gentoo would be available to those who use the major distributions: for one it might stop the endless tirade of “Linux needs a good installer” posts and much more importantly prevent draconian changes to Linux in the name of a “grand solution” to dependency problems. This article, however, is not about Gentoo, nor about it’s virtues and vices. It is about possible approaches to this massive beast of a problem.
I do understand that the vast majority of Linux users do in fact use binary distributions and that there is a need for a viable solution to creating cross-distro binary packages. I also understand the plight of software developers in general and commercial developers in particular as regards offering their packages in an easy to install format, regardless of the distribution and version they are using.
I also understand the importance of things like LSB for the really large distros-those who are seeking certification by governmental bodies and standards committees and who are working closely with giants like Oracle. Yet after having read the article by Arvind Narayanan and other recent articles about the LSB that there is a) a lack of creative problem solution b) that the “solution” may in fact be worse than the “problem” itself and c) that beating this dead horse with a stick again isn’t going to make it go anywhere.
I have also closely followed the development of Mike Hearns’ Autopackage project and Thomas Leonards’ Zeroinstall project. These projects are attempts to solve some of the issues of “dependency hell” which think “outside of the box” of the conventions already established in the Linux community.
So without any further ado I wish to make some of the following observations and possible alternatives and solicit a response from you regarding your take on such and whether or not others in the community have been talking about such.
Lets start with the basics. Where does “dependency hell” come from. Dependency hell occurs when an application requires a multitude of tightly interdependent libraries and those versions expected by the application conflict with previously installed software on the system or when other applications require the same libraries but with different version numbers. This is how the “users” experience dependency hell. Being as it is that most free software makes liberal usage of existing libraries (code re-usage) the potential for “dependency hell” is given. However this situation is compounded again and again by issues which are external to any particular application itself-
Distribution specific collections of default installed libraries. SuSE, Redhat, Fedora, Mandrake etc. are markedly different in the specifics regarding which versions of which libraries are installed by default.
Distribution specific attempts at wider-scale integration of software which mandate exactly which libraries are compiled with which specific features and which specific patches.
Intra-Distribution version schemes and the correspondingly different constellation of installed default libraries. Major distributors pursue branding schemes which compounds such problems beyond the differing rates of development of the effected libraries.
These issues are homegrown issues brought on by the distributors themselves-they are responsible for this situation and they are obligated to finding ways to alleviate this situation. To the extent that they amongst themselves can work towards standardization via things like LSB they are working towards minimizing the negative costs of their market driven choices. Inter-distribution incompatibility is primarily a function of being market driven: built-in incompatibility is the key to all forms of customer lock-in, this is a trick they learned from their proprietary cousins. One does not have to be cynical to understand the market importance of branding and the most effective way to pursue such branding is to create distributions which differ sufficiently that users are bound to that distributor for the software of their choice.
Redhat and now Fedora have used this process to create their desktop solutions, ie. the fiasco about changes to KDE and Gnome to create the “unified desktop”-or SuSE and it’s tailoring of the KDE and GNOME environments. In both of these cases the software available to the distributors in source form was not the source of the problem, rather the deviation from the source is what caused such problems. GNOME and KDE and even X itself have also been part of this problem too: which is why it took a while for fontconfig and freetype to become THE anti-aliased font solution for Linux-X and KDE came with their own version of fontconfig compounding problems for users-luckily this situation has been resolved.
Now to the extent that a particular application is not overly dependent upon tight system integration I imagine that Autopackage can be a useful inter-distribution distribution mechanism. Yet it is exactly the lack of system-wide integration in Linux applications which people are constantly complaining about: and this integration is only currently possible via linking “policies” with other libraries and this requires specific compilation options and/or patches neither of which can be adequately addressed by Autopackage due to the differences in the distributions themselves. Steps are being take to abstract some of these issues to facilitate wider-scale system integration without resorting to specific compilation options and patches: dbus, hal, shared-mime-info etc. but they are still new developments. To the extent that small libraries and daemons can provide a richer context of system integration-ie. provide “hooks” for applications to make use of to effect integration the more useful something like Autopackage becomes.
The situation is improving all of the time but we still face issues concerning which local email system(sendmail, qmail ?) is installed by default, which sound system is being used by default(ALSA, OSS, arts, esd, jack ?), which printing system(CUPS, Omni, LPR ?) is installed by default etc. In fact a myriad of micro-libraries have been written to render each of the interfaces offered by such basic-system services compatible with one another-introducing another layer of dependency issues, eg. alsa-oss.
Recently I took a look at Zeroinstall and was quite impressed by what I saw. It seems to me that a combination of Autopackage and Zeroinstall might form important parts of a potential “solution” to dependency hell. Yet there seems to to still be a missing element: autobuilding of software form source.
IF the distributors and software developers could agree on package naming conventions and something like pkgconfig(or maybe pkgconfig itself) itself was universally used to index all available software already installed on the system, and distributors would not patch their sources in ways which break compatibility between libraries-and document those cases where they do in such a way that the system “knows” about such incompatibilities(ie. like a pkgconfig file which ascribes a version number to a package which precludes it from being used by applications expecting different functionality)-and this is a mighty big IF- one could then implement a background autobuilding environment(sandboxed of course) to quickly compile any specific libraries which are still outstanding.
Of course I won’t go into the details of how utterly horrid the build systems are in some of the best known distributions: but if their was a law against insane build environments which have been hacked together beyond mere mortal comprehension Redhat and SuSE would be punished for life in compilation purgatory ;). (Notice how I have not mentioned Debian here-although I do not use Debian I truly respect the quality build environment which Debian has created-which is probably why Debian is used as the basis for so many distributions.)
My idea (but perhaps not mine alone?) concerning a autobuilding environment looks like the following: a user wishes to install an application, this application is dependent upon a set of specific libraries further specified by unique version numbers. Of the list of dependencies only two are found that are not offered in that version by the distributor themselves. Because the operating system “knows” this “automagically” through rigorous adherence to effective “documentation” of installed libraries(ie. via pkgconfig) including “incompatible libraries” (ie. crippled and lamed libraries which only have a use in the specific context of that particular distribution and that particular distribution version number) and naming conventions (major.minor.bug.distrofix), the system could “automagically” initiate compilation of said libraries from source, silently, so as not to bother those frightened by gcc. The issue I see involved in such are:
How would the system “know” which compilation options to use(ie. which can it use safely and which may cause other dependency problems.)? If the distributors would encode their build environment specifics in some kind of machine readable configuration file which specifies which options were used when compiling each library and specific versions of specific patches used in the compilation-these factors could form the “knowledge” of the autobuild environment.
What kind of works needs to be done with ldconfig to enable “slots” and differentiated dynamic loading of specific libraries for specific applications: ie. the system should easily facilitate multiple concurrent version of installed libraries which are incompatible with each other and dynamically choose which it needs at run time? (Gentoo already has this).
If such things were attainable autobuild(TM 😉 ) could silently quickly “solve” most of the “dependency hell” issues-in most cases the compilation time would be negligible-anything other than large libraries is compiled in mere minutes on any system sold in the last 5 years.
The way I see it:
Autopackage attempts to define a distribution neutral alternative to software distribution-such along with an adhered to LSB could provide a way for third parties to distribute their software to all Linux distributions.
Zeroinstall allows users of Linux systems who do not have root priveledges (common in the “enterprise market” and all multi-user installations) to have access to applications transparently without any adverse effects on the system upon which it is running. Great for “total packaged” solutions.
“autobuild” would give the big binary distributors a tool with which to combat the incompatibilities borne of their market driven branding needs. If all of the major distributors had something like “autobuild” the draconian effects of trying to curtail code re-usage(read limit dependencies) and the problems with static packages could be avoided.
I applaud the work on the LSB and I hope that it becomes reality. I also am pleased to see that the partisan infighting which has accompanied all discussion as to what should be part of “Linux” is taking a backseat to the spirit of cooperation embodied in freedesktop.org. A lot of progress has been made and the signs look promising for the future but the problems are still here and have not been completely resolved: why is it that the GTK libraries end up in /usr/lib but QT/KDE does not ? why is it that dbus is dependent upon glib and how should KDE reconcile itself to such a dependency(Not Invented Here syndrome).
But whatever may become of the LSB if it’s success is dependent on “avoiding dependencies”, ie. less code re-usage, or statically and artificially ossifying the rapid development of Linux applications and there dependent libraries in order to achieve “solutions” for third party application developers the “solution” may in fact become worse than the “problem”. If Linux was still a “no man’s land void of applications” as it was many years ago and “winning over” new applications was the biggest task in Linux land I would be more prone to accepting such draconian measures. But this is not the case any more, of course the situation can be improved and I am not saying that there are not any problems, on the contrary the potential for problems is abundantly clear.
But the question is where and on whose shoulders the responsibility for such rests. The main culprits in this situation are the major distributions themselves and proprietary third-party software developers. For those of you who have not noticed NVIDIA includes an “autobuilding” tool in their proprietary binary packages-this should be commended and I wish that the major distributors would take it upon themselves to develop such technology so as to offset the incompatibilities created by their endorsement and sponsorship of and by proprietary software. Such a move would go much farther towards addressing issues of “dependency hell” than any LSB could ever attain. But then again the undertaking involved in (re-)creating sane build environments to facilitate such on demand, dynamic “autobuilding” of dependencies is an undertaking at least as grand in scale as adherence to some proposed LSB.
As a footnote, “autobuild” could completely solve the legal problems surrounding the issues of proprietary codec support and DVD playback in Linux. Instead of shipping crippled media players with Linux which are worse than useless a simple script could be written to invoke autobuild to automagically download the source and compile those libraries needed to provide such support under Linux-absolving the distributors of any legal liabilities because they would not be “distributing” the software. This issue is actually where most Linux users first encounter “dependency hell” and such a “solution” could be advantageous to all parties involved.
What do you think of such an idea ? Does such sound plausible to you ? Are there issues which I not taking into account which you have encountered ? Have you heard of any similar discussion within the free software community ?
About the author:
I am Karl Zollner. I have frequented the OSNEWS forums for the past two years and have been involved with computers for over 25 years. OS’s have come and gone along with the platforms upon which they were built-I finally found my home on Linux.
If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.