What follows in the article is an analysis of the “dependency hell” problem in Linux and the issues surrounding software installation. I make a proposal of my own and critique the suggested “solutions” to these problems and point my finger at “the” (my perception) source of these problems.
I have used Linux for almost 8 years now and exclusively for the last three, so I have experienced what many refer to as “dependency hell”. Two years ago I switched to Gentoo which radically reduced my experiences with dependency hell. I will not say it eliminated it-for I love to play with the very newest and latest, but as regards those packages in portage I have had extremely few dependency problems -read perhaps 3 times.
In fact my experience with Gentoo has been so positive that I doubt I will ever use a binary distribution again-except the one I roll for myself. With this as a background you can probably imagine the immediate utility of “grand solution” to “dependency hell” for some one like me: ie. negligible.
Yet I wish that some of what I have grown accustomed to in Gentoo would be available to those who use the major distributions: for one it might stop the endless tirade of “Linux needs a good installer” posts and much more importantly prevent draconian changes to Linux in the name of a “grand solution” to dependency problems. This article, however, is not about Gentoo, nor about it’s virtues and vices. It is about possible approaches to this massive beast of a problem.
I do understand that the vast majority of Linux users do in fact use binary distributions and that there is a need for a viable solution to creating cross-distro binary packages. I also understand the plight of software developers in general and commercial developers in particular as regards offering their packages in an easy to install format, regardless of the distribution and version they are using.
I also understand the importance of things like LSB for the really large distros-those who are seeking certification by governmental bodies and standards committees and who are working closely with giants like Oracle. Yet after having read the article by Arvind Narayanan and other recent articles about the LSB that there is a) a lack of creative problem solution b) that the “solution” may in fact be worse than the “problem” itself and c) that beating this dead horse with a stick again isn’t going to make it go anywhere.
I have also closely followed the development of Mike Hearns’ Autopackage project and Thomas Leonards’ Zeroinstall project. These projects are attempts to solve some of the issues of “dependency hell” which think “outside of the box” of the conventions already established in the Linux community.
So without any further ado I wish to make some of the following observations and possible alternatives and solicit a response from you regarding your take on such and whether or not others in the community have been talking about such.
Lets start with the basics. Where does “dependency hell” come from. Dependency hell occurs when an application requires a multitude of tightly interdependent libraries and those versions expected by the application conflict with previously installed software on the system or when other applications require the same libraries but with different version numbers. This is how the “users” experience dependency hell. Being as it is that most free software makes liberal usage of existing libraries (code re-usage) the potential for “dependency hell” is given. However this situation is compounded again and again by issues which are external to any particular application itself-
Distribution specific collections of default installed libraries. SuSE, Redhat, Fedora, Mandrake etc. are markedly different in the specifics regarding which versions of which libraries are installed by default.
Distribution specific attempts at wider-scale integration of software which mandate exactly which libraries are compiled with which specific features and which specific patches.
Intra-Distribution version schemes and the correspondingly different constellation of installed default libraries. Major distributors pursue branding schemes which compounds such problems beyond the differing rates of development of the effected libraries.
These issues are homegrown issues brought on by the distributors themselves-they are responsible for this situation and they are obligated to finding ways to alleviate this situation. To the extent that they amongst themselves can work towards standardization via things like LSB they are working towards minimizing the negative costs of their market driven choices. Inter-distribution incompatibility is primarily a function of being market driven: built-in incompatibility is the key to all forms of customer lock-in, this is a trick they learned from their proprietary cousins. One does not have to be cynical to understand the market importance of branding and the most effective way to pursue such branding is to create distributions which differ sufficiently that users are bound to that distributor for the software of their choice.
Redhat and now Fedora have used this process to create their desktop solutions, ie. the fiasco about changes to KDE and Gnome to create the “unified desktop”-or SuSE and it’s tailoring of the KDE and GNOME environments. In both of these cases the software available to the distributors in source form was not the source of the problem, rather the deviation from the source is what caused such problems. GNOME and KDE and even X itself have also been part of this problem too: which is why it took a while for fontconfig and freetype to become THE anti-aliased font solution for Linux-X and KDE came with their own version of fontconfig compounding problems for users-luckily this situation has been resolved.
Now to the extent that a particular application is not overly dependent upon tight system integration I imagine that Autopackage can be a useful inter-distribution distribution mechanism. Yet it is exactly the lack of system-wide integration in Linux applications which people are constantly complaining about: and this integration is only currently possible via linking “policies” with other libraries and this requires specific compilation options and/or patches neither of which can be adequately addressed by Autopackage due to the differences in the distributions themselves. Steps are being take to abstract some of these issues to facilitate wider-scale system integration without resorting to specific compilation options and patches: dbus, hal, shared-mime-info etc. but they are still new developments. To the extent that small libraries and daemons can provide a richer context of system integration-ie. provide “hooks” for applications to make use of to effect integration the more useful something like Autopackage becomes.
The situation is improving all of the time but we still face issues concerning which local email system(sendmail, qmail ?) is installed by default, which sound system is being used by default(ALSA, OSS, arts, esd, jack ?), which printing system(CUPS, Omni, LPR ?) is installed by default etc. In fact a myriad of micro-libraries have been written to render each of the interfaces offered by such basic-system services compatible with one another-introducing another layer of dependency issues, eg. alsa-oss.
Recently I took a look at Zeroinstall and was quite impressed by what I saw. It seems to me that a combination of Autopackage and Zeroinstall might form important parts of a potential “solution” to dependency hell. Yet there seems to to still be a missing element: autobuilding of software form source.
IF the distributors and software developers could agree on package naming conventions and something like pkgconfig(or maybe pkgconfig itself) itself was universally used to index all available software already installed on the system, and distributors would not patch their sources in ways which break compatibility between libraries-and document those cases where they do in such a way that the system “knows” about such incompatibilities(ie. like a pkgconfig file which ascribes a version number to a package which precludes it from being used by applications expecting different functionality)-and this is a mighty big IF- one could then implement a background autobuilding environment(sandboxed of course) to quickly compile any specific libraries which are still outstanding.
Of course I won’t go into the details of how utterly horrid the build systems are in some of the best known distributions: but if their was a law against insane build environments which have been hacked together beyond mere mortal comprehension Redhat and SuSE would be punished for life in compilation purgatory ;). (Notice how I have not mentioned Debian here-although I do not use Debian I truly respect the quality build environment which Debian has created-which is probably why Debian is used as the basis for so many distributions.)
My idea (but perhaps not mine alone?) concerning a autobuilding environment looks like the following: a user wishes to install an application, this application is dependent upon a set of specific libraries further specified by unique version numbers. Of the list of dependencies only two are found that are not offered in that version by the distributor themselves. Because the operating system “knows” this “automagically” through rigorous adherence to effective “documentation” of installed libraries(ie. via pkgconfig) including “incompatible libraries” (ie. crippled and lamed libraries which only have a use in the specific context of that particular distribution and that particular distribution version number) and naming conventions (major.minor.bug.distrofix), the system could “automagically” initiate compilation of said libraries from source, silently, so as not to bother those frightened by gcc. The issue I see involved in such are:
How would the system “know” which compilation options to use(ie. which can it use safely and which may cause other dependency problems.)? If the distributors would encode their build environment specifics in some kind of machine readable configuration file which specifies which options were used when compiling each library and specific versions of specific patches used in the compilation-these factors could form the “knowledge” of the autobuild environment.
What kind of works needs to be done with ldconfig to enable “slots” and differentiated dynamic loading of specific libraries for specific applications: ie. the system should easily facilitate multiple concurrent version of installed libraries which are incompatible with each other and dynamically choose which it needs at run time? (Gentoo already has this).
If such things were attainable autobuild(TM 😉 ) could silently quickly “solve” most of the “dependency hell” issues-in most cases the compilation time would be negligible-anything other than large libraries is compiled in mere minutes on any system sold in the last 5 years.
The way I see it:
Autopackage attempts to define a distribution neutral alternative to software distribution-such along with an adhered to LSB could provide a way for third parties to distribute their software to all Linux distributions.
Zeroinstall allows users of Linux systems who do not have root priveledges (common in the “enterprise market” and all multi-user installations) to have access to applications transparently without any adverse effects on the system upon which it is running. Great for “total packaged” solutions.
“autobuild” would give the big binary distributors a tool with which to combat the incompatibilities borne of their market driven branding needs. If all of the major distributors had something like “autobuild” the draconian effects of trying to curtail code re-usage(read limit dependencies) and the problems with static packages could be avoided.
I applaud the work on the LSB and I hope that it becomes reality. I also am pleased to see that the partisan infighting which has accompanied all discussion as to what should be part of “Linux” is taking a backseat to the spirit of cooperation embodied in freedesktop.org. A lot of progress has been made and the signs look promising for the future but the problems are still here and have not been completely resolved: why is it that the GTK libraries end up in /usr/lib but QT/KDE does not ? why is it that dbus is dependent upon glib and how should KDE reconcile itself to such a dependency(Not Invented Here syndrome).
But whatever may become of the LSB if it’s success is dependent on “avoiding dependencies”, ie. less code re-usage, or statically and artificially ossifying the rapid development of Linux applications and there dependent libraries in order to achieve “solutions” for third party application developers the “solution” may in fact become worse than the “problem”. If Linux was still a “no man’s land void of applications” as it was many years ago and “winning over” new applications was the biggest task in Linux land I would be more prone to accepting such draconian measures. But this is not the case any more, of course the situation can be improved and I am not saying that there are not any problems, on the contrary the potential for problems is abundantly clear.
But the question is where and on whose shoulders the responsibility for such rests. The main culprits in this situation are the major distributions themselves and proprietary third-party software developers. For those of you who have not noticed NVIDIA includes an “autobuilding” tool in their proprietary binary packages-this should be commended and I wish that the major distributors would take it upon themselves to develop such technology so as to offset the incompatibilities created by their endorsement and sponsorship of and by proprietary software. Such a move would go much farther towards addressing issues of “dependency hell” than any LSB could ever attain. But then again the undertaking involved in (re-)creating sane build environments to facilitate such on demand, dynamic “autobuilding” of dependencies is an undertaking at least as grand in scale as adherence to some proposed LSB.
As a footnote, “autobuild” could completely solve the legal problems surrounding the issues of proprietary codec support and DVD playback in Linux. Instead of shipping crippled media players with Linux which are worse than useless a simple script could be written to invoke autobuild to automagically download the source and compile those libraries needed to provide such support under Linux-absolving the distributors of any legal liabilities because they would not be “distributing” the software. This issue is actually where most Linux users first encounter “dependency hell” and such a “solution” could be advantageous to all parties involved.
What do you think of such an idea ? Does such sound plausible to you ? Are there issues which I not taking into account which you have encountered ? Have you heard of any similar discussion within the free software community ?
About the author:
I am Karl Zollner. I have frequented the OSNEWS forums for the past two years and have been involved with computers for over 25 years. OS’s have come and gone along with the platforms upon which they were built-I finally found my home on Linux.
If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.
When I read most of the first page and the author hadn’t yet made his point, I couldn’t be bothered to read the rest.
Anyway, IMHO … the problem isn’t dependency hell, as it seems that both apt, portage, and probably other package managers are able to approriately deal with those.
The problem as I see it is that most distros tend to want to do their own thing with package managers, and so when you pick a distro, you’re automagically limited to whatever packages the distro and/or the distro’s community provide you, unless you want to do configure, make, make install and hope that works.
I have said for a long time that what is needed it to define a standard for how packages should be laid out (ok, two standards – one for source and one for binary distros), and let the package mangers each do it their own way. Just like HTML doesn’t care about the web browser, neither should the packages care about the package manager either.
> as it seems that both apt, portage, and probably other
>package managers are able to approriately deal with those.
I don’t believe they do. There are always cases where not-so-intelligent packagers screw up the packages and deps don’t resolve correctly, or they end up overwrite versions of libraries you really need for other apps. Fink (apt-based) was the latest problem I encountered the other day (I wrote about it on my blog). It sucked. I wish things were statically linked like apps are on Win/BeOS/OSX. Things just work (TM).
I still don’t see a perfect solution for package management. we have apt, yum, opencarpet, etc
but i would like to see the distros duke it out a bit more and let the market decide
the last thing i want is the LSB to pick a winner. the market is better at picking winners than a committee.
I am fine with this. As long as installation of the app is as easy as it is with statically linked apps, I am willing to have a few extra KBs of filesize (which actually makes the app load faster because it doesn’t try to go find all these libs one by one — a reason for gnome apps loading not as fast as they should) and to download the full app when a new version is out (I am on a fast 512 KB/sec cable line here).
>And should be recompiled for every update. …
From the moment that the “statically link” route is taken, that would be the job of the developer, not mine, so the “pain” wouldn’t be mine, as a user.
Remember, I am not talking about statically link everything, but surely everything that is not standard on distros, e.g. libsigc++ or gtkmm. But certainly not things like libpng or libfreetype.
Gotta agree with Gabriel here. I am often forced to compile some of my code statically, esp. in re Intel’s MKL or other numeric libraries. And the problem gets really annoying. Keeping around a few versions of 50+ MB executables starts getting to me.
And, has anyone ever tried to figure out what static version of popular software would bloat to? Some of the GNOME and KDE programs are already pretty big using dynamic linking.
> the last thing i want is the LSB to pick a winner.
Almost nobody cares about LSB anymore (except maybe commercial distros).
Neither debian nor gentoo are LSB-compliant. They don’t even use rpm currently, so IMO there’s no way they’d ever adopt a LSB-glorified package manager (unless it’s apt, of course).
>Keeping around a few versions of 50+ MB executables starts getting to me.
If gcc was more clever, it should not be 50MBs of executables. I am talking about a few more KBs more for each binary package! That’s negligible!
SGI’s compilers KNOW how to correctly statically link only the needed symbols from libs, it’s time GCC learns how to do that too instead of statically compile all the cr*p inside and bloat things.
@Eugenia
Uh, at last check GCC does only link in needed Code. I’ve done quite a bit of static / dynamic mixed linking with executables I’ve built and all of the documentation clearly states that only needed code is linked in. Perhaps you should further research your claim.
> From the moment that the “statically link” route is taken, that would be the job of the developer, not mine, so the “pain” wouldn’t be mine, as a user.
s/developer/maintainer/
I’m sure no developer with limited resources would take that route, many are already only releasing source and that number would only increase with your proposal.
>I’m sure no developer with limited resources would take that route,
That’s their problem. If they want their app to be used further, they should provide binaries that really work.
>Uh, at last check GCC does only link in needed Code.
Last I checked, it didn’t.
” why is it that the GTK libraries end up in /usr/lib but QT/KDE does not ? why is it that dbus is dependent upon glib and how should KDE reconcile itself to such a dependency(Not Invented Here syndrome).” NOTE TO AUTHOR – read this quote from their DBUS website:- “D-BUS has no required dependencies, but includes GLib, Qt, Python and .NET APIs to the D-BUS system, if you want to build those.”
So are you just trolling regarding the KDE reconcile comment?
GTK libraries in /usr/lib – good question why is it there? Does it comply with the LSB in being there?
The author seems to forget that Debian solved dependency hell about 7 years ago with the introduction of apt, and have been laughing at the increasingly fractured and broken RPM distros ever since. I’m surprised the editors of OSNews would allow such a glaring oversight get published.
> I am talking about a few more KBs more for each binary package! That’s negligible!
A few KBs? Are you kidding (from the GTK tutorial; GTK version 1.2):
gebner@schnecke2 ~/tmp/scribble-simple [0] $ gcc -static scribble-simple.c `gtk-config –libs –cflags` -lpthread -o scribble-simple.static
gebner@schnecke2 ~/tmp/scribble-simple [0] $ gcc scribble-simple.c `gtk-config –libs –cflags` -lpthread -o scribble-simple.dynamic
gebner@schnecke2 ~/tmp/scribble-simple [0] $ ls -lh scribble-simple.*
-rw-r–r– 1 gebner gebner 4.6K Jun 17 1999 scribble-simple.c
-rwxr-xr-x 1 gebner gebner 17K Aug 4 21:17 scribble-simple.dynamic
-rwxr-xr-x 1 gebner gebner 2.7M Aug 4 21:17 scribble-simple.static
gebner@schnecke2 ~/tmp/scribble-simple [0] $
Not trying to flame or any thing…
but why not copy the ports/package system from FreeBSD or the like?
It is OSS, so there is nothing stopping it from being ported and it handles dependency problems wonder fully. Except where stuff is mutually exclusive, but that is a given…
>A few KBs? Are you kidding (from the GTK tutorial; GTK version 1.2):
Gabriel, READ again the above comment about GCC not knowing how to ONLY statically link the symbols it really needs! What you linked there was the FULL gtk, NOT the symbols that are really needed!
>The author seems to forget that Debian solved dependency hell about 7 years ago
Read my comment above about Fink (apt-based OSX system).
@Eugenia
Last I checked, it didn’t.
Perhaps you should check again then. GCC only links in all symbols of a particular object when linking if it was directly referenced on the command line as “foo.o”, and then only on certain platforms. When linking to library code GCC only links in symbols (pointers to code / data basically) that have been referenced by the program and global symbols in the library. It does NOT just link in the entire library. Read the GCC documentation. Executables can be further shrunk after compilation by using the “strip” program supplied with their compiler toolkit.
In my experiments even the Intel Compiler kit for Linux doesn’t produce smaller executables, in fact it generally produces larger ones.
Besides, we are not talking about statically link libs that are standard. GTK+ is standard. Most other third party libs that are not standard are usually much smaller.
BTW, you didn’t strip your binaries.
strip –strip-all scribble-simple.static
ls -l scribble-simple.static
I wish things were statically linked like apps are on Win/BeOS/OSX. Things just work (TM).
I wonder why nobody (no distro that i know) never tried to provide statically linked versions of Gnome/KDE/etc? Is it hard to do?
Victor.
> Gabriel, READ again the above comment about GCC not knowing how to ONLY statically link the symbols it really needs! What you linked there was the FULL gtk, NOT the symbols that are really needed!
-rw-r–r– 1 root root 3.9M Jul 30 15:03 /usr/lib/libc.a
My static libc is bigger than the static executable I presented, so gcc/ld has to strip at least something.
If you had bothered to read the article you would see that the problem I was writing about effects all Linux distributions with no exceptions. Debians apt-get is only really a solution for Debian packages-whereas what I was trying to point out is what would be necessary for third-party developers to distribute applications and libraires for any and or all distros. Although I do not use Debian I do like the package management system used by Debian(I have used it before).
This is why I specifically did not address the differing package management systems which are currently in use by different Distros- this article is not about RPM vs. APT vs. vs Portage etc. No package management system from any distro can address the needs of all distros-and when it comes to distributing and installing software for the “Linux platform” what is needed is alternatives which can dynamically adapt to the particular distro on which the application is to be installed. LSB represents one approach to this problem, for those distros which conform to the LSB- many wish to radically reduce the number of dynamic dependencies- either by statically linking everything- which is simply not viable for most software- or by trying to outright reduce the number of libraries used in software development- which means less code reusage, which is the native advantage of opens source software.
My text is an attempt to explore an alternative-autobuild environments which can compile only those specific libraries which are needed for a particular app, dynamically. No packaged binary solution can have absoulute all dependencies covered when one takes into account all of the different Linux distros. And no package management system covers absolutely all needed libraries.
So I did not overlook Debian….
LSB does not require you to statically link the libc.
> BTW, you didn’t strip your binaries.
The difference is tiny:
-rwxr-xr-x 1 gebner gebner 11K Aug 4 21:31 scribble-simple.dynamic
-rwxr-xr-x 1 gebner gebner 2.4M Aug 4 21:31 scribble-simple.static
compared to:
-rwxr-xr-x 1 gebner gebner 17K Aug 4 21:17 scribble-simple.dynamic
-rwxr-xr-x 1 gebner gebner 2.7M Aug 4 21:17 scribble-simple.static
As I told you, your sample is not representative. GTK is a big library and it is standard. Try to compile a real app that has 2-3 deps (or more) to non-standard third party libs (which are usually small libs). Even if the binary ends up be big, it well worth the pain of installing these additional libs by hand or by a package management system that is prone to screw up things regularly.
> or by a package management system that is prone to screw up things regularly.
If you use such a package management system, that’s your fault. I’ve never had any problems with apt/debian with regard to this.
“Perhaps you should check again then. GCC only links in all symbols of a particular object when linking if it was directly referenced on the command line as “foo.o”, and then only on certain platforms. When linking to library code GCC only links in symbols (pointers to code / data basically) that have been referenced by the program and global symbols in the library. It does NOT just link in the entire library. Read the GCC documentation. Executables can be further shrunk after compilation by using the “strip” program supplied with their compiler toolkit. ”
1. GCC is not a linker.
2. Do you have any idea how linkers work?
3. Care to explain how a linker could do dependency analysis, figure out what pieces of data a piece of code needs, as well as what other procedures it might call, and then only extracts the relevant code from the library?
Gabriel, I have used pacman, apt-get, yum, Fink, urpmi, BSD ports and Gentoo among others. I am not a newbie around here. I have found problems with ALL of these package manangements that they simply try to resolve the impossible and they try to solve the WRONG problem.
I am against dependency-resolution package managers. It is a solution for the wrong problem.
Now, you can either respect my opinion as I do yours and give an end to this discussion, or try to be sarcastic and caustic as you were earlier and that won’t help you around here.
Unfortunately sometimes static linking isn’t an option, either due to licensing issues (LGPL requires a dynamic executable) or platform problems.
For example, I’ve been maintaining the Linux port of the ags (Adventure Game Studio) runtime engine for a while now. I originally distributed binaries that were statically linked to Allegro, and Allegro in turn would dynamically load a module for arts support on the system. However, when it does this on newer Linux distributions it seg faults. When I filed a big with one Linux distribution vendor I was told in no uncertain terms that statically linking to a library that dynamically loaded libraries was unsupported and strongly discouraged.
> 3. Care to explain how a linker could do dependency analysis, figure out what pieces of data a piece of code needs, as well as what other procedures it might call, and then only extracts the relevant code from the library?
1. Figure out the entry points of the code.
2. Traverse all references from the entry points.
3. Only link those functions for which references were found.
Am I too naive?
@tore
“1. GCC is not a linker.”
I know that, but it does manage the linking process.
“2. Do you have any idea how linkers work?”
Enough to use it and work with its idiosynchrasies (at least as far as the GNU Toolkit is concerned).
“3. Care to explain how a linker could do dependency analysis, figure out what pieces of data a piece of code needs, as well as what other procedures it might call, and then only extracts the relevant code from the library? “”
Care to explain why the GNU documentation explicitly states that when liking to library code only referenced symbols from that library are linked in?
I don’t pretend to understand every single step it performs, but I know enough from reading the documentation and having done several different types of linking and maintaining a Linux port of a game engine to know that it doesn’t link an entire library in.
@Gabriel
Thank you, thank you, thank you
And yes, BTW that’s how it works from what I’ve read…
The FHS, which is a Linux standard for directory layout,
indicates that /usr/bin is where an executable goes unless
one of:
a. it is only run by root (use /usr/sbin then)
b. it is required at boot (use /bin then)
c. both “a” and “b” (use /sbin then)
d. it comes with XFree86 (may use path with “X11R6” in it)
e. it doesn’t come with the system — see below
You use $HOME/bin for personal stuff, /usr/local/bin for
stuff the admin compiled up for all users, and a place
under /opt for add-on packages (sold separately).
By this, we conclude that KDE is either a distinct add-on
package that doesn’t ship with the OS (it is usually not)
or that KDE is getting installed in the wrong place.
When I tried to install a program (rpm based) and I found out I had to download other little pieces of software on the internet, I gave it up.
Then I tried my first debian based distribution.
I honestly think the debian “way” of installing programs is the solution.
This basically what gentoo does with portage/emerge. But this won’t work because of the same reasons that the different BSD’s port systems are not 100% compatible. Not being up to speed on this I believe the Open and Net ones are compatible. As the BSD’s are similar in distro’s that they patch and edit al included software to a degree to cope with for example kernel and device system differences. The problem would also occur if you downloaded a random (Ports is source based? isn’t it?) that is binary for say open BSD and insert it in a net BSD. A few behavioural differences between the two where they store files would give the same troubles.
But the reason this dependency hell exists is partly also due that packages depend on other packages and not the files that those packages contain. This is the central flaw of the package system and is also why the auto package system works so well. This means that an RPM package named AAA says I need package AAB and does not say I need file libAAB.so. This is the major shortcoming of the package system and also the reason why this problem occurs as different distro’s put files in different package. So in SuSE the package is named AAS.rpm and in Red Hat AAR.rpm a distro’s alien package wants the package AAA.rpm as this is the one it expects will provide libAAA.so and does not know that both the SuSE and Red Hat Package also provide the library libAAA.so.
You must not forget that a package is merely some binary program (often) with a description of what that program requires to work and where it should put it self.
And that is why even as Ports has been ported to gentoo it won’t solve dependancy hell as it is as the major commercial distro’s are linked very much to their rpm/deb based way of doing things and they don’t yet have the time or commercial luxury to change things. As doe the ISV who dislike any changes at all. I hope you find my points interesting. Feel free to reply or feel free to assign this to the garbage.
Jerven
@velox:
but why not copy the ports/package system from FreeBSD or the like?
I thought Gentoo was special in the Linux world, because it uses a BSD like ports system.
And I am such used to building everything on my FreeBSD system from the latest source tree (kernel, usertools, apps) that I seldom trust one of my generated binaries to run on another system, because I fear it is not in sync with the libs there. 🙂
Regards,
Marc
…is to decide on one packaging format (.dpkg or .rpm or a new one perhaps) but there’s no guaranteeing that all the distros will support it.
I think software makers should just make their own packages, like an installer. People like Real and NVidia do this already. Should it really be the distro’s responsibility to make sure that the installation of 3rd party software packages run smoothly?
I installed freeBSD on my laptop recently (dual boot with Mandrake Linux), and suffered through the interminable cvsup routine, updating the OS, downloading and building the ports. Due to dependency problems in the ports collection, I had to repeat that part of the process a couple times (something to do with updating perl to 5.8.5, etc). The bottom line is my poor laptop sat spinning furiously for the better part of 2 weeks. All in all, I bet it compiled code for 40-60 hours or more. The poor thing was hotter than a fire-cracker, and I bet I took a good 6 months off it’s life. Yeah, I want to go through that again! I’m thinking the laptop HP now sells pre-loaded with Linux sound better every day.
Well I was just checking out Autopackage and they just added(dont know when) support for sealed installers!!!
So now Autopackage can work like the old loki installers with no need for network dependency resolution. Very nice.
I prefer to build from source. Though Slackware has a pretty nice package tool, I normally will download the source tarball and go from there.
./configure
make
make install
That easy. And if you are missing a dependency, the configure script tells you.
Uhm, dude, that’s the whole point… That dependency then needs another dependencies, which, in turn, have dependencies of their own and so on and so forth.
./configure-make-make install is not the solution to this problem.
Although I have to admit that ‘back in the day’ when I used Linux, I also tended to use compile-from-source.
Eugenia is right, GCC is very lazy when it comes to staticly linking. GCC doesn’t look further then the object file (there are multiple object files in a library) to check what has to be included. The result is very very _very_ large staticly linked binaries.
But this lazyness of GCC can be worked around like the guy of dietlibc did by making every single function its own object file (plus other size saving hacks). He has some links to applications (grep, ldap server, etc) that use his library staticly linked that are of sizes you can’t get a staticly linked glibc hello world binary.
… but I wouldn’t wish to see that. I for one would absolutely not be happy if some commercial rpm based distromaker would make the rest stick with them.
I for one enjoy (how peculiar that may seem to many of you) the diversity of package management systems. It just is part of what the open development process means. Or should I say “this is who we are”.
I have to tell you, I always thought that package dependency hell is only a problem of the lazy (some of you will take this offensive, hey, I can live with that). There is no problem in linux what a caring mind cannot solve (yeah, you might also argue on this as long as the Nile goes).
I have to tell I am a debian man in the first place (though not exclusively). I have never ever encountered such a hell that couldn’t be changed into joy over some nights and by some beers (okay okay, I know how that sounds
). I’m not telling I always like the solving process, but the joy when being done has always beeing worth it.
So for me, I don’t care if someone will introduce some standard on a rainy afternoon someday, I just wish I will still have the good old opportunity to use what I want.
<grin> How about using a tool that does not need to be told three times where the dynamic libraries are supposed to be, after which it still doesn’t feel like doing the right thing</grin>. Seriously, even though this would be a major architectural change, why not use, reimplement, or copy the dynamic linking features of Darwin?
Autobuild sounds like a great idea. I think I’d like [binary + source] packages that can be _rebuilt_ on the fly, or whenever the admin so whishes. If the binaries work, if you’re not paranoid, or you’re low on CPU – no need to compile. If they don’t work, or you can afford it – have the system rebuild the package(s) seamlessly.
Source trees are great for developers, but I’d rather not have hundreds of thousands of files consuming filesystem time or space. Keep the user happy and the admin sane by keeping it simple.
Another idea might be to separate metadata from the common binary packages, and have an Internet-service that pushes updates to package metadata – dependencies, conflicts, etc, as well as storing all ever published packages so you’ll never ever be missing one because it’s not available anymore.
If this can be tied to some kind of failsafe install/remove mechanism where packages aren’t really uncompressed in common locations, but kept apart from each other, in the form they came, more or less, I’d be thrilled. Perhaps something like what QNX has. (Until recently I think. Did they remove it?).
DragonFly BSD has plans for a new package system:
http://www.livebsd.com/cgi-bin/trac.cgi/wiki/dflypkg
I am against dependency-resolution package managers. It is a solution for the wrong problem.
I’d be very curious if you would explain what you think is the problem.
Resolving installation dependancies and application library dependancies are completely separate problems. Static binaries are a possible solution for small programs, but not for programs that take tens of MB of space. Its also not a particularly nice solution if a major flaw is found in a critical library (e.g., the libz flaw a couple years ago, and I’m pretty sure glibc has had such flaws, openSSL has recently had such problems).
The big problem is that the major libraries: libz, glibc, etc. are different on major distros. Sometimes these differences are incompatible, even between distro minor versions. When you have the source to a program, this isn’t really too much of an issue: you simply (may not so simply depending on the distro) recompile.
But, when you have a binary only program (i.e., the Intel Fortran Compiler), there is no way to do such. Now, “yous takes your chances” is the relevant theme, but many people desire such programs to work in the Linux world. Ideology aside, its a “good thing” to help them.
The question of how is very, very hard on an operating system where most applications are assumed to be source compatible, not binary compatible. Face it, there is no quick-and-dirty solution. The only real solution for these vendors is if their primary platform targets cooperate and agree to make the job easier. I don’t how they can do this and expect things to work in Gentoo/Slack/Jo-Blow Linux as well. Its probably and intractable problem.
Face it, GNU operating systems were designed to be source compatible, not binary compatible. Them’s the breaks. It opens up a whole world of possiblities, but there will probably never be a way for any random binary application to work perfectly across all Linux distros. There’s too much variety, but at least a subset will work. The LSB will help that subset identify a target for vendors, and it may help out-of-set distros, but it can’t be responsible for everyone.
Hmm, now that I think of it, one possiblity is that the LSB should not insist on particular library versions. Instead, the LSB should insist on certain implimentation functionality. Maybe specifying such functionality as “version X” of a library.
Now, when a distro includes incompatible versions of a library, they must include an LSB library (libLSB) that provides the specified functionality. For instance, if glibc changes to provide a newer version of “foo”, then libLSB should contain the older “foo”, which will be linked against.
I’ll admit to not thinking about this for more than 5 minutes, so any objective criticism is welcome. This also means that all LSB-compliant distros (something done by certification) will end up supplying their own versions of libLSB. That seems like a reasonable compromise to me, as it allows for strict compliance with several LSB versions as well as letting source-compatible applications follow the moving targets.
At first, I was inclined to agree with your opinion of Eugenia, but then I did some testing.
1. The gnu manuals do indeed say that only the necessary symbols are loaded for static linking.
2. The gnu manuals lie.
—a.h—
#include “b.h”
int foo(int x);
int bar(int x);
int baz(int x);
—a.c—
int foo(int x) {return x + 42;}
int bar(int x) {return x * foo(x);}
int baz(int x) {return bar(x) – wombat(x);}
—b.h—
int wombat(int x);
—b.c—
int wombat(int x) {return x*x;}
—main.c—
#include “a.h”
int main() {return bar(57);}
gcc -c a.c
gcc -c b.c
ar rc libFoobar.a a.o b.o
ranlib libFoobar.a
gcc -c main.c
gcc -o main-static main.o -L`pwd` -lFoobar
My gcc-3.2.3 puts all 4 functions (foo, bar, baz, wombat) into the executable, even though only 2 are actually used.
You are correct I stand corrected. I found this information out myself shortly after having written the article. This was a thorny debate issue a while back-and furthermore it serves as an example of such issues. It was not my intent to troll about KDE or GNOME- I am glad that DBUS is being written in such a way as to have no dependencies and allowing glib and KDE to provide wrappers. But this was not the first such issue and probably won’t be the last.
My example was slightly stupid, because main.c #included all the 4 functions. Should have done 1 more minute of thinking.
Anyway, it seems that gcc includes all the symbols that it finds the program accessing, even if it’s just through an #include and the corresponding function never gets used. So, for example, if you changed my files so that a.h does not include b.h, you will find that wombat doesn’t show up in the executable, but baz (which never gets called) does show up.
Exactly.
This is why I said that GCC doesn’t do it right, no matter what the manuals say. SGI’s compilers, I know, they do it more correctly.
It maybe that GNU systems wered designed to be source compatible-but if you look at the build environments of the most popular distros all you find is complete and utter chaos(Redhat, Fedora, Mandrake, SuSE).
There are many issues responsible for this: a) maintaing build environments which build a humongous library of software on multliple different architectures and most importantly b) the “branding” needs of distributors in the form of specific “features” brought on through patching and specific compile time options.
The build environments present in the major distributions make the possibility of an “autobuild” solution quite difficult- the build environment needs to be relatively clean and coherent. But if all of the mature distributors could provide clean and coherent build environments they could standardize on a kind of “autobuild” system which would allow third-party application developers to let such a system dynamically resolve dependencies not managed by the package management system itself.
Their is probably a direct relationship between how clean and coherent the build environment of a distribution is and how much it would profit from such a “autobuild” system. Debian and Gentoo would only profit slightly from such because their package management systems have solved most of the problems with dependencies- whereas Redhat, SuSE, Fedora, Mandrake, Lycoris,Linspire,Xandros etc. could profit immensely from such. But the point of such a system would to be to facilitate work for third-party developers.
At any rate a key to solving these problems is rigorous adherance to library naming conventions: we need:
liblib-Major.Minor.Bugfix.PackageSystemFix
The libraries as released from the authors can only include:
liblib-Major.Minor.Bugfix
But given that most binary distributors also change their versions of the third-party libraries over time to enchance integration and ease of management-a system needs to be built which documents which patches and compile options were used by the distributor so that third-party developers can write their code independent of the particular hacking of particular distributors….
A “major” release number indicates changes in the API which may break backwards compatibility. “Minor” indicates incremental changes in a given API which expose new functionality not included in the older API. “Bugfixes” explains itself- ie. has this or that exploit been fixed?. And finally “PackageSystemFix” should describe the changes made in the .deb/.ebuild/.rpm spec file which allow for integration of the library into a package manangement system.
liblib-Major.Minor.Bugfix.PackageSystemFix
I personally would prefer the Gentoo naming convention; it is often convenient to explicitly state whether a given package/library is beta or alpha or release candidate or whatever.
http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&c…
The dependency problem is _solved_ lots of times ago. apt exist. statically linked packages exist (see the opera deb package in opera webpage).
Dude, dependecy problems are _already_ solved.
The author suggest yet another source-based package system. As far as I know, source-based packages systems are _dead_. From the controversy generated in the latest year, one can very well conclude that _nobody_ is happy with either binary packages or source-base compilations. Everyone wants both. And since binary packages must have _some_ way of being compiled from source, they’re the ones who really satisfice all the users.
The “dynamic dependency” crap (aka USE flag) has no sense. Good software allows you to configure at *runtime* its settings not at compìle time. If I’ve to choose at compile time if X program is going to use OSS or alsa, well, that software is _broken_, fix it. Decent software uses OSS or alsa or esd or arts or…., depending on what is available, not having to choose what one you’ll use at runtime in a fucking environment variable.
i think the solution partly lies in packages not only asking for packages but for specific files so that if the package is not around the file can be looked for by the installer.
then there is the “problem” of multiple library versions. is there a way to make it smarter so that you can have multiple versions of the same library installed and the app can tell what lib version it wants to the system? sure it will lead to install bloat in that you have multiple versions of the same library installed but that is unavoidable, either you build all yourself or you risk a bit of install bloat. sure this is similar to static linking but it will lead to smaller installs overall (compared to pure static links) as when you staticly link you get a new version of the lib for every app you install. with the system above you only get multiple versions of you have a app that must have the new version but allso have a app that need the older version.
and what if the installer is a bit smarter so that when you uninstall/upgrade something it allso parses the dependency tree for what your uninstalling/upgradeing and cleans up the tree for packages that are no longer in use (go down one level, do a scan for installed packages that depend on this package, if none then remove/upgrade it and check the dependencys of that package in the same way until you hit rock bottom). sure if you mix it with source installed stuff this may break but thats allso why i try to avoid source installs as mutch as i can.
as for distros useing modified libs and compiles well that is i fear unavoidable as the text points out that while its new school in the way the source is handled it still old school business style when it comes to user lock-in, this will allways be the case i fear. but then if apps could request diffrent lib versions and the system could install it without breaking stuff allready in the system the power of lock-in would be gone. just install a set of “standard” libs on top and be happy
if a lib or app was changed compared to the default one then they should carry a tag saying so in the package name. like say rh have a modified version of lib foo. then it would be named libfoo.x.x.x-rh.xxx and you would know that this is a home grown version. if a package then requested libfoo.x.x.x.xxx it would be installed and the user told the reason is that the -rh one is a rh specific variation and the app he is installing is requesting the clean one.
“what one you’ll use at runtime in a fucking environment variable.” -> compiletime, obviously
BTW, I very much agree with other people here.
“Sorry, I’ve debian stable, I try to use gnome 2.7.53-alpha3.2 packages from GodKnowsWhere and I’ve dependendy problems”. Of course you’ve dependendy problems. The thing, is, a normal user wouldnt want to do stupid things and he wouldn’t install those packages.
Its the fact that “backward compatible” isn’t in the average OSS developers vocabulary. Why does every version of a library break applications written for a previous one? Newer should always be compatible, but improved.
It is the same problem with the kernel. Modules won’t migrate from one kernel to the next. Not even through a simple recompile in almost all cases.
The other thing (that autopackage works around) is the way libraries are linked. Unless you use a special linking procedure, just having a compatible library on the system still may not work.
Packaging is the LEAST of our worries.
(Sorry if this is a rehash, I wasn’t interested in reading 50 of the same post)
You are kidding? right?
Typical example for you: lots of simple qt applications can be trivially made to look and behave like kde applications by replacing QApplication, QFileDialog, etc. with KApplication, KFileDialog, etc., since the function parameters match. Typically you do it like this:
#ifdef USE_PRETTY_KDE
KFileDialog dialog(dirname);
#else /* use ugly qt */
QFileDialog dialog(dirname);
#endif
Now, are you suggesting that instead of this elegant mechanism (which is directly exploitable by Gentoo USE flags), an application should always compile against kde (and which version of kde? there are odd incompatibilities) and using some complex fragile system-dependent magic degrade to qt if kde is not found?
A pity you can’t read and/or comprehend . If you would have bothered to read and try to understand what I wrote you would see that I am not talking about a new source based package management system. For your info source based systems are actually quite new in the Linux World, with the exception of LFS. And they are far, far from dead. But *my point* is that the binary distributions could benefit from using some of the technology found in source based distributions.
Firstly source based distribtuions provide clean and coherent build environments-they must or there is no software to run. Secondly source based distributions like Gentoo already know how to deal with multiple versions of the same library. Now if binary distributions could provide a kind of “autobuild” environment-standard on all distributions- third party developers could write their application and let “autobuild” only compile those libraries not already provided for in their own package management systems.
“Dependency Hell” may be too strong a word-but most downloadable binary applications only run on a handful of Linux distributions without problem. 90% of the time they require that certain libraries which may be present on the system were compiled with specific options which may or may not present. This is “dependency hell” if someone needs this application. Further I am not talking about USE flags- although I think it is important that the system have a way of documenting which compilation options were used during compilation-and which patches were used. Lastly I am not talking about forcing people to do ./configure, make and make install.
What I am talking about is distributors commiting themselves to providing clean and coherent build environments and to implement some kind of “autobuild” system, which could be some kind of new standard, which would allow for dynamic compilation of libraries which export the exact symbols needed by a binary pacakage and which would not conflict with other already installed libraries.
Certain major applications have provided their own build environments for years-X, Mozilla and OpenOffice to name a few. These build environments are massive undertakings for third-party developers and few rarely compile such large systems on their own machines. But until very recently if one wanted a version of Mozilla which actually used the gtk libraries and the local fonts one had to actually compile the program on their own machine unless and/or until their distributor offered a binary package for that specific release of that particular version of their Linux distribution.
What I am arguing for is that the distributors themselves take on a larger part of the burden for insuring that open source software is trully portable across distribution lines.
Please, for the love of god would people stop saying “everyone drop what your doing and switch to this brand new package manager that I’ve just invented but haven’t implemented and which won’t work unless everyone uses it.”
Red Hat won’t drop RPM, Debian won’t drop DEBs and Slack won’t drop TGZs. To fix “dependency hell” in an inter-distro fashion, create a machine readable way of describing dependencies that package authors can use to complement their packages and distro authors can use to make their hacks work. Then write some software to use that info – not too hard.
Mostly these problems are solvable on a distro-by-distro basis and it is up to software authors to ensure their target market can use their software. This is something commercial software vendors do well.
There is no reason why people who are scared of make should be compiling from source – it gives very little beneift in terms of speed (I use LFS). I wish software authors would provide FHS compliant, distro-independent binaries but rpms can usually be made to work.
More libraries should be modules and loaded as needed if present not this –with-png –with-gtk. If the application is compiled with ‘hooks’ to allow for loading module libraries than we can cut down on dependency hell significantly. Though, we might run into module version ABI/API incompatabilities…
but it gets us a little further ahead.
Howdy
Has anyone actually filed a bug report telling the GCC folks that it is not working as expected? .. just a thought.
Static linking if done correctly would aleviate some of the more funky one-off libs that you need and maybe just leave larger ones to be dynamic (KDE and Gnome etc etc).
I have just installed Gentoo 2004.1 and was compiling KDE-3.2.3 the other day and the little dependencies it wanted were incredible!
I have compiled the same KDE using Konstruct on Redhat 8.0 and most deps didn`t exist! (and no they weren`t previously installed) so i`m not sure what was happening there but maybe the Gentoo ebuild specified different params to the make file???
SuSE/Redhat use RPM, and without apt-rpm, dependency hell is almost guaranteed, and apt-rpm still needs to be installed.
Debian uses DEB, and with apt-get and use of only the stable
repositories, it is possible to avoid dependency hell, but the stable branch is way too old. I have hosed more than one machine in short order trying to use the unstable branch, simply by doing an ‘apt-get upgrade’.
Gentoo, on the other hand, has in ‘portage’ taken the BSD ports metaphor and taken it to another level of ease of use and configurability. Gentoo supports both source and binary packages and has been nearly bulletproof regarding package updates in my experience. Most packages are nearly bleeding edge, yet ebuilds currently marked stable are 99.99% of the time exactly that – stable and “just work”. It is trivial to update/upgrade/uninstall packages. The only cost is CPU time to compile the source, and it’s almost impossible to irreparably break your install just by upgrading, which is something I can’t say for RPM or even DEB based distros.
We don’t have to re-invent the wheel – we simply need to take a look at what Gentoo has done correctly and learn.
Kudos to Daniel Robbins and the whole Gentoo team!
Check out:
http://www.dragonflybsd.org/goals/packages.cgi
With all the ranting that is going on about gcc makes me think that we should have at least two quality opensource c/c++ compilers. Competition between the two should produce better compilers much like competition between kde and gnome pushed both projects farther.
With all the ranting that is going on about gcc makes me think that we should have at least two quality opensource c/c++ compilers. Competition between the two should produce better compilers much like competition between kde and gnome pushed both projects farther.
Actually, there is a second compiler under development…TenDRA…it’s BSD Licensed too for those who swing that way. (Like myself :p)
http://www.tendra.org/
I haven’t personally tried it, apparently ANSI C works pretty well, and C++ support is partially there.
It would be interesting to see an article comparing GCC, TenDRA, and Intel’s C/C++ Compiler. *hint*
To be on topic…perhaps I’m the exception rather than the rule, but since using APT on Fedora Core, I have had almost no dependency problems. Perhaps you guys just go off the beaten path and work around the packaging system more often than I do? Anyway…with GCC in its current state, static linking every application would be insane.
Just my thoughts.
look..this is a moot point among ‘nix users…..If you are a gentoo user you use portage..or compile from source.. if you use slack.. you use .tgz..or compile from source.. If you use debian youuse ports and .debs..if you use redhat.. you use redhat rpms or suffer.. same with fedora and mandrake.. the problem doesnt seem to be with ‘nix users.. it seems to be with the press who make this into more of a problem than it really is…
Ive been running the same slack box for over 2 yrs.. and i have software on it that frankly ive forgottne about.. and have never really experienced problems with mainstream applications..such as oOo.org the gimp..mozilla.. or the like. Sure..there will always be some wild little proggie that need sspazlib or some other esoteric library to run.. but its sort of the same on winboxes.. they may need some proprietary or little used .dll to run….Really.. its time to get off this.. when an enterprise decides to go with a particular distro.. they will know which packages to use.. and there will be no prob.. nuff said.
I’ve heard of and toyed with tendra. Anything that has partial c++ support is hardly production quality. TenDRA has a lot of potential but it isn’t there yet. I seem to recall it was hanging in limbo for a while. Has work on the project picked up?
I wish things were statically linked like apps are on Win/BeOS/OSX.
Um, no they’re not (well, they might have been on BeOS – I doubt it though – but they certainly aren’t on Windows and OS X). Windows and OS X apps certainly *can* be statically linked, but as a rule, they are not.
You are, perhaps, thinking of the way Windows and OS X apps are often *distributed* with any non-standard libraries and store them “locally” in the application directory, but that is *very* different to static linking.
It does not matter how, as long it works. The problem with unix is that if these apps place their libs on /usr/lib or /usr/local/lib they will overwrite existing same libs (of different version), and that’s not what you want. So, there are two things to do:
1. Either statically link
2. Don’t use /usr/l*
In any case, the core of the matter is that the user should not see dependency problems whatsoever.
When I read most of the first page and the author hadn’t yet made his point, I couldn’t be bothered to read the rest.
Well, you should have. This is one of the most intelligent proposals I’ve seen so far about dependency resolution and vendor-neutral software distribution.
Of course, what he proposes his quite a task, but it’s definitely doable. The question is, would the various distros get with it, or would they fight it…
Really, you should have RTF[riendly]A. Even though the style may be a bit dry at times, there is truth in those words.
1. the (gnu)linker’s working unit is the .o file not C functions. b.o was linked in because it’s refered to by a.o ‘s baz().
2. it’s unlikely any other linker would do better with the example (a.c b.c and main.c) above unless that linker can do code-elimination at the function level (which is arguably NOT the linker’s job. Plus, in the case of asm code, how do you determine where functions start and end).
3. In the above example, just put one function in one file to get the desired effect. Or make baz() static and gcc-3.4.x will remove it (then the linker won’t link in b.o).
4. In the old days, I was taught to put each C function in its own file. But now a day: disks are cheap, ADSL is cheap and OSes do demand-paging. All become a NON-issue.
5. Most ppls on OSNEWS are more qualified for debating GNOME-vs-KDE than saying coherent things about linker/compiler.
I am against dependency-resolution package managers. It is a solution for the wrong problem.
While I personally haven’t had any real problems with urpmi, I can definitely understand the arguments for a standardization of software management on Linux.
So, it’s clear you don’t like dependency-resolution package managers, but what I’d be curious to know is what you think of Karl’s proposal. It strikes me as a very interesting approach, as it would – in my view – promote standardization while allowing for a transition for distros that use lots of patched and quirky libraries. It’s not as much a revolution as an evolution – which means it also fits the open nature of Linux.
Anyway, I was just curious to heard what you thought of it.
Dude, dependecy problems are _already_ solved.
The author suggest yet another source-based package system.
Dude, RTFA. It’s clear from your comments that you didn’t.
Dependency problems in Linux are relatively minor, especially for an OS that has such a complex developer ecosystem. They are certainly not crippling problems, and if you stay within the confines of the packages available for your distro, then dependency issues are pretty much non-existant (due to the dependency-resolving package managers Eugenia dislikes).
But start mixing and matching distro rpms with that of other distros, or deb packages, and source packages…dependency problems do occur. What the author suggest is an interesting way to reduce these problems even further.
Newer should always be compatible, but improved.
At some point, compatibility means bloat, stagnation or instability. Look at what MS has had to do to make sure that backward compatibility was maintained, such as coding in dozens exceptions for specific third-party software products. The Windows codebase has become monstrous, I can understand why they consider dropping some old cruft in the next version.
The point is, in order to remain compatible, you have to allow for exceptions for older apps, but at the same time you don’t want to penalize people who don’t use these apps. As you can see, the situation isn’t quite as clear-cut as you indicate.
So, there are two things to do:
There are other options:
A 3rd option that some of us are going towards: Including precompiled versions of any “obscure” or “critical” libraries we use and installing them into the application’s directory. Most Linux distributions honour LD_LIBRARY_PATH so an app can simply install libs into it’s own directory and use those instead of the ones located in the system.
Bioware does this with neverwinter nights, they have a local library directory that’s used for the Miles Sound Design libraries and a precompiled version of SDL that they link against. This eliminates the dependency problem for the user, but bloats the executable, and sometimes the library won’t work on the system.
One problem with Eugenia’s response about how the developer is responsible for depedencies is that graphics library (especially those like SDL, Allegro, etc.) and audio libraries cannot be compiled on one distribution and be guaranteed to work on other distributions.
For example, the version of SDL included with Bioware’s NWN works fine with RedHat, and Fedora but flakes out and has other problems on SuSE 9.1. Additionally, there are licensing considerations and other technical limitations that often prevent the developer from being able to statically link against code or include a precompiled version. So, not even the 3rd option is a complete solution either.
A 4th option that developers have is to limit their official support to one or two Linux distributions (much like Oracle does).
A 5th option is to go through the hellish experience of distributing a different binary for each Linux distribution they want to support.
With a commercial package an autobuild system is not a viable solution.
“Dependency hell occurs when an application requires a multitude of tightly interdependent libraries and those versions expected by the application conflict with previously installed software on the system or when other applications require the same libraries but with different version numbers.”
I think the “two” scenario’s described are actually the same
A needs xyzlib v1
B needs xyzlib v2
xyzlib v1 and v2 may pose a conflict – meaning one will override/erase the other on installation.
But would this still be a problem if v2 is backward compatible with v1 e.g. if v2 includes v1 unmodified – or if the libraries are named differently and only known as such throughout the system e.g. no symlinking using a common name?
Personally, it has not been a big issue. I’ve seen conflicts but not many.
I think another thing is that the packagers sometimes require higher versions than what’s actually necessary. I think this happens because the dependency requirements are derived from whatever libs happen to be on the system on which the packaging is done. And more often than not these libs are the very latest.
Dude, you are aware there is absolutely no OS called BSD? And that Net/Free/OpenBSD, Darwin, and ect are not the same OS?
Yeah, the ports system is source based, but the package system is not, but works with it.
So what is wrong with something being mutually exclusive? It happens on every OS, including windows. Don’t see any reason to whine pointless over that.
BTW the FreeBSD port/package system was NOT ported to gentoo. It is similar, but a entirely different beast. Of which does not have a problem with comercial software installs requiring stuff. It just requires a port for that software to install it and ect. There are several in the FreeBSD ports tree like that and require a disk to install it.
Then you really did not do something right… try reading the docs and ect next time…
I’m glad to see discussion about problems surrounding linux installation and package management because I’ve been complaining and suggesting change for a long time.
I’m to the point that I’ve given up pushing Linux and I’m just waiting for something better to come along which deserves my attention and support effort.
I really wish all Linux supporters luck as I believe in the ideals, I just don’t like the flavor.
Windows solved the problem by simply installing different software (including libs) to different independent folders. System is looking at program’s folder first when looking for dynamic lib (so called “dll”). And that’s it. Program can be uninstalled simply by removing program’s folder. Now, I know some people there came with all sorts of stupid ideas like OCX (like dlls, but depends on registry), dlls in system folders, using registry etc. But we can ignore it. I think Gobo Linux learned that lesson right.
Statically linking libs is something that should have died 20 years ago. In fact, the only proponents and users of static linking are commercial vendors. It doesn’t make sense to use in free and open source software where sharing and reuse are standard and second nature.
While stactically linked packages would make installation easier, I doubt it is a solution to the “supposed dependency hell” people are experiencing. I haven’t experienced one of those myself in years.
From a technical standpoint, the benefits of dynamically shared and linked libraries far outweigh that of their static counterpart.
In the end, it all falls back to package managers and an intelligent package managing system. I doubt any free/open source developers in their right mind will provide statically linked packages, well, except again commercial vendors.
Just give us the source, and let the package manager take care of the rest. In my opinion, dependency hell has already been solved by two great existing package managers, it’s just not universal yet, because Red Hat wouldn’t just let RPM die!
Gentoo for instance has a resaonably intelligent manager that is capable of installing different versions of libraries and packages side by side on the system, don’t know of any other manager that can do that. Unfortunately, Portage isn’t a universal system, so people still experience this dependency hell.
If I had my way Apt will be the universal manager for binary based distros, and Portage will the universal manager for source based distro. But man can dream. Anything but statically linked packages, please!
>I wish things were statically linked like apps are on Win/BeOS/OSX. Things just work (TM).
Sorry if someone has said this already.
Not all MacOS X apps are statically linked. In fact if it’s a Objective-C app then it has a relatively small amount of statically linked code. Cocoa Frameworks are not linked statically and the code which goes into you executables is dynamically linked which allows from some message passing magic.
OS X apps are bundles which means that it is possible to put libraries within them as a separate files/frameworks.
An idea which is kind of the way OS X does it, I think:
Although it would add blought and a substantial change in the run time it could be used to resolve conflicts. If the application cannot find the library it needs or the library doesn’t have the correct function then it search’s the application bundle for an appropriate library.
During installation if an application needs a specific library version which would conflict with the currently installed then it is installed into the bundle. I can see a number of issues with this but I’m sure they could be ironed out.
Another thing which I think would be useful is compatibility information between library versions. Even if it was just to the extent that the package manager new which functions a library provided and what the application needed from it. Some information which said whether and function from library version 1 was compatible with version 2’s function would go a long way.
Maybe libraries should be stored in a database arrangement on the system giving access to all this versioning information.
Obviously this would add work to the library writer but surely couldn’t some of this be automated.
Well I better get some work done and chatting rubbish. Please don’t flame me they are just ideas.
Portage does exactly what the author describes as the solution.
Eugenia mentioned in a comment that, “I don’t believe they [, including portage, handle dependency resolution appropriately]. There are always cases where not-so-intelligent packagers screw up the packages and deps don’t resolve correctly, or they end up overwrite versions of libraries you really need for other apps.”
Well, in 2 years of using Gentoo, I’ve never had such a problem. Gentoo supports slots, you can have multiple versions of any library installed. Portage will not overwrite versions of libraries that other apps need. And you can literally use portage on other distributions (there’s a how-to in the Gentoo forums) so it’s not even distribution specific.
I appreciate the feedback guys(and gals). I know the article was somewhat longwinded and indeed it was written very “dry”. This was the first article which I have submitted for publication. I have read all of the posts in this thread and I will be trying to further refine the ideas presented here based on the criticisms and the issues which have been brought up here.
I hope that there is sufficient interest here in these forums for aticles which *attempt* to dig a little deeper into complex issues-I have no interest in writing articles about “I installed distro x and spent 5 minutes and this is why is 5ux0rz or r00lz”. Also I am not so much interested in pitting Linux vs other OS’s or distro x vs. distro y. I have used many different distros and OS`s over the years-of course I have my preferences- but I would like to think that it is possible to address larger issues which go beyond distro infighting.
My own coding skills are rather lacking- I have not actively programmed in many, many years-but I have invested a lot of time in trying to grok the issues which hackers/developers face and I have been rolling my own distro’s for a couple of years now-so I have some understanding of almost all facets of distribution-enough to feel competent speaking about such- yet eager to learn more and be corrected. Although I could not write the “autobuild” system I suggested in this article I do know what would need to be done to make such a potential project doable-ie. which issues must be resolved and ways in which such issues can be programatically resolved.
I hope in the future to further contribute such articles and serve as a bridge between the excellent coders who may lack a broader-range view of the complex distro issues and the advanced users and administrators of systems who may lack the more specific knowledge involved in distro building.
Several of you have commented that this or that distro solves theses problems. Some have pointed to Debian and others have pointed to Gentoo. I myself am an avid Gentoo user for the last 3 years. But my interests always tends to focus on issues which go beyond any single distro-I have had very little in the way of dependency problems with Gentoo-but Gentoo will always be a niche distro and Gentoo is not immune to these problems because it exists in a much larger world of Linux distros and software.
Finally I did not envision this “autobuild” idea as a be-all-end-all solution to such problems. Rather I see it as a very useful tool for dealing with these issues. I also believe that tools like autopackage and zeroinstall could be parts of a broader solution. I do not believe that statically linking libraries solves these problems-in fact statically compiled apps ussually contribute to such problems. I also do not think that altering the file-layout of Linux will really resolve such issues-if for no other reason than the lack of standard file locations amongst the differing distributions. I also do no think that reducing dependencies can be a viable approach to solving these issues-dependencies mean code reuse and this is the greatest thing about Linux in the first place and open source in general.
Again thanks for the feedback and a big thank you to those who took the time to read the article and actually weigh the issues I was addressing.
Portage gets inspiration from *BSD ports, that are known to work very well without problems, the only problem maybe is the time needed, for example, to compile a big app, like KDE or OpenOffice, so if it could be able to handle binary pkgs (I know that by now is possible, but it’s of little use in gentoo) there could be a common repository of sources for all distros, pkgs could be still mantained by distros, but sources could be exchangeable (maybe with a set of patches specific for the distribution where you install the package) in this way if a package isn’t avaiable for your distribution you could simply build it automagically like in gentoo, you could then make pkgs for later reuse.
This also could automatize the creation of binaries by the distros, all this using a single tool…..am i dreaming? 😛
I’m not quite sure what this autobuild does that isn’t already covered by zero install. There, you link your application against a particular build of a library (eg, “the build of GTK from gtk.org”).
If you need something that version doesn’t have, you link against a build that *does* have it. If two apps link against the same build, they’re shared automatically.
As for building from source, this nearly works automatically. If you click on, eg, ROX-Filer and no binary is available, it will try to compile it there. You need to specify a place to put the result (currently using 0divert, though it could be made automatic).
You can also drag a particular version of ROX-Filer to your home directory, edit the source, and recompile it there. No problem. Of course, this could be made completely foolproof by putting gcc in Zero Install too. Then, a C compiler is always available.
http://0install.net
Are you perchance Thomas Leonard ?
Bogolisk: Thanks for clarifying what I had said. I bet most people would understand your explanation better than mine.
To others: If gcc is linking in every function listed in all headers included, that’s a big gcc bug, because it shouldn’t be including functions not referenced. And from my experience, it does behave as it should. If you want to test, use dynamic linking, or maybe even static linking, and run ‘nm’ on the generated binary. It’ll list ONLY those symbols explicitly referenced in your source code.
I see people left and right comparing random RPMs which are downloaded off some website with installing the same application through Debian’s APT system (assumingly using the vastly superior, official package repositories). That’s comparing apples to oranges. The reason why Debian works that way is because there’s rigorous quality control on the packages. The whole debian system (maintainers, users and (very important!) policies) ensures that by the time a release is made which is usable by Joe Sixpack dependency bugs have been ironed out. That’s why ‘stable’ is called ‘stable’. Not because it doesn’t crash your computer, but because bugs like this have been ironed out. When there’s an update to ‘stable’, it’s almost always a security bugfix, which drops right in an existing system. Painlessly. If not, that’s considered a grave bug, and the package maintainer is receiving some serious flak. You can rest assured it won’t happen again. Breaking ‘stable’ is simply Not Done.
Where things go awry is when people:
1) Start installing their own created packages (I do 😉
2) Download some random packages off the ‘net and force install those (apt-get.org anyone?)
3) Mix and match releases (stable + unstable, for instance)
All three above (not exhaustive) would cause confusion in the package manager. It’s all fixable, but annoying nontheless. Also keeping up with the bleeding edge (running ‘unstable’) sometimes introduces little problems, which are usually fixed quickly. But that’s the user’s own choice. Running unstable isn’t for Joe Sixpack, and definately not for people who have nothing better to do than run distributions into the ground by writing “reviews” and “opinion articles” on websites all day long.
So what’s the difference between “apt-get” and installing a random RPM? The source, and the quality of the package. Not to mention the habit people have to force install a package after clear messages there are problems with it.
The problem isn’t “dependency hell”, the problem is with people’s expectations, flawed logic and sheer stupidity when dealing with installing software.
Oh yeah, and standards are good. But instead of agreeing on some magical package format, we should agree in parts of the LSB which make sense, and lift Debian’s packaging policies (and QA tools) to industry standard. Resolving the problem of installing software is just a waste of a lot of energy, IMO.
> then there is the “problem” of multiple library versions. is there a way to make it smarter so that you can have multiple versions of the same library installed and the app can tell what lib version it wants to the system?
Yes, and it has existed for ages. Libraries have a minor and a major version number. When only the minor version differs, the libraries are supposed to be binary compatible. When a new version introduces binary incompatibility, the major version number should increase.
Applications link against a major version number, and the link loader normally loads the most recent minor version with that major version number.
This system breaks down when library and application are e.g. compiled with different versions of the compiler that introduce binary incompatibility trough no fault of the library.
But I have the impression that the system doesn’t work as well as it’s supposed to, even with the same compiler version. I have no idea why that is.
the (gnu)linker’s working unit is the .o file not C functions. b.o was linked in because it’s refered to by a.o ‘s baz().
If gcc is linking in every function listed in all headers included, that’s a big gcc bug, because it shouldn’t be including functions not referenced.
I bow to your greater knowledge of linking.
If I understood you two right, static linking an executable will
1. split the .a static lib into its component .o object files;
2. statically link in those .o files that contain a symbol used in the program or parts of the static lib that were already linked in.
In this case, I have a couple of questions:
1. Are (or can) the .o files internally be structured to show block boundaries of the original C code?
2. How does the number and size of .o files in a single library or executable affect performance? (you imply it will decrease memory usage)
I’ve made a number of RPM pacakages and would concur that urpmi and yum replicate the Debian apt system. So long as you stick within your distribution, there is little dependency issues. Problem do arise when you attempt to mix-and-match things from various distributions.
For commercial entities, the short term and simple solution is simply to provide the dependencies for the taret distributions or a platform neutral incarnation of them. For example, you have an app dependent on libpng, so provide a libpng RPM that has “Provides” notes that apply to the naming conventions of other distributions.
It would be easier if distributions (and developers) settled on a standard naming convention. Mandrake and Gentoo have similar conventions now, but Debian and Fedora are typically on their own.
Ideally package managers (RPM, APT, etc.) would support the notion of “bundles” like PERL packages from CPAN. This way you can explicitly bundle dependencies and have them optionally installed (rather that requiring a package manager like URPMI or what have you to be setup to obtain a package from an external source or some local medium).
Debians apt-get is only really a solution for Debian packages-whereas what I was trying to point out is what would be necessary for third-party developers to distribute applications and libraires for any and or all distros.
<p>
Who needs any other package system? I’m saying RPM is obsolete.
Al I said is that you can’t use a binary port file created for let’s say DragonFlyBSD with lets say openBSD(in the last post I wrote this as open BSD, small difference but it does not mean i do not know that there is no BSD os as such, but that there is a family of BSD o.s.’s).
Why people whine is that they in at least the Linux world have to download files published for their own distro’s as you in the BSD world you need ports for you individual bsd os. So if a third party developer creates a binary program and you want to download this from the net and use it. You need to check back (quote from freeBSD manual)
“If you do not know the name of the application you want, try using a site like FreshMeat (http://www.freshmeat.net/) to find an application, then check back at the FreeBSD site to see if the application has been ported yet”
So the advice of the freeBSd manual would partly translate to the current Linux world as, check if you distro’s packages this program so that you don’t have installation problems. (The program might just not work on freeBSD yet as it is not designed for an UNIX system for example.)
You are correct in saying that gentoo’s portage is not a port of port(Al this port makes me want to drink a glass of it) I erred on this in the last paragraph
As is said before i’m not up to speed on this as I am not a user of the BSD’s other than darwin.
The dependency hell is an unnecessary hell. It occurs on windows and is then called the dll hell. But this a problem that can be designed against.
The static library solution is not a solution as this would decrease performance if you use one library a lot among different programs. As near worst case scenario in the case of kde if you compile all programs in kdebase statically. you have a big size bloat both on disk and in ram. The individual installation program works on windows as this is only a single distro’s with multiple versions. And the layout of files is different per version in single way there is no difference between windows XP home and windows XP home where they store things.
the make configure install routine does not solve this problem as to my current knowledge they are relatively difficult to deinstall from a computer.
The Mac OS X way of providing bundles is also depended on the assumption that all Mac OS X computers are different out in a specific way.
The Linux world is a lovely chaotic world where everyone put everything where they want to. The installation methods used should take account of this and there should not be a reason to implement a ridged file system hierarchy to solve this problem. The package handling program should deal with this. The developer shouldn’t need to know anything about the users system.
Just my 2 eurocents.
I think the problem of dependencies and configuration hell is quite real for linux. Despite what many of the previous posters have asserted this has *not* been solved many times. I will use my experience with debian as an example.
True apt-get and so forth worked quite well so long as I was using a stable release and only interested in debian software. However, like many people who use linux I would often want to install cutting edge packages or compile other programs not supported by debian’s package system. For instance when I commited the cardinal sin of installing binary only commercial software. Often newer version of the libraries would be required and I had to choose to either wait for debian to catch up, make my entier system unstable by upgrading to the unstable version of debian (just for a couple of libraries) or compile the library from source. Unfortunatly, mixing a binary distribution with newer libraries you compile for source is not a good mix (often you break other programs). I have also experience the same thing in redhat.
I have some sympathy for the people who insist that GNU/linux is a source based system and woe be to those who insist on using binaries. However, this simply isn’t practicable to continually update manually via source all the various libraries and so forth. I think the deciscion to use dynamic rather than static libraries is a good one but as a result you can’t require manual source update (automatic source update has exactly the same issues as does manual update plus more).
So what does a solution require? For one it clearly requires good data about version numbering of libraries and requirements of various binaries. While we might be able to do this by a good major.minor.bugfix sort of system ultimately we need more. The user should be able to specify and change what libraries/programs a given binary might require. Of course this needs to be combined with a system which allows multiple differnt versions of the same library to be kept on the system (I guess gentoo portage my already do this but it needs to be made universal).
So instead of being really pedantic with library naming I suggest a modification to the library/executable format. In addition to the normal data include an XML section for metadata. This could be used to explain what dependencies the program requires (library > version blah..but also what I don’t think is supported now library > blah1 but not blah2…avoid specific versions with security holes etc..) but it’s xml charachter would allow for extension when new types of dependencies emerge (for instance only works on systems with this certain type of hardware…or perhaps info on how to download new versions letting the user merge in new packages to his automatic update). If the binary loader and dynamic linker were aware of this information it could automatically load the appropriate library version or offer the user the option to download a new version.
Some of you might ask what this offers over apt-get or similar. The difference is small but crucial, it provides a standard way for applications to indicate their dependencies. This lets the user add new source only programs and modify the required dependencies (for instance might want to not let it run with libraries known to have security issues). In short we need to move all this information out of distribution specific databases and into the executable and library files themselves (or at least into an invisible .file saved with them).
In my opinion the real problem is not the way programs are built (as they require specific versions for some shared objects) or the package manager (be it rpm/deb/portage/etc.). If you get more dependency problems it means in most cases that the spec file (taking the rpm case) is written in enought details which is good. The problem comes from another spot: the package manager front-end. The front-end (yum/apt/emerge/etc.) should sugest the user how to solve those problems or solve them by it-self (with user notification). It should have access to decent repositories with all the required packages. Here debian and gentoo are leaders and Fedora is slowly catching up (fedora.us/rpm.livna.org/atrpms/freshrpms/dag/etc.). If there was a sole decent repo for these distros I’m pretty sure that the dependency problems could be solved by the front-end (yum/apt/rcd).
Having well written deps in a package is a blessing (no runtime errors), if you can get past the rpm errors then everything is ok.
1. Are (or can) the .o files internally be structured to show block boundaries of the original C code?
ELF already supports this. The problem is what if your source code is asm (which has just labels and not any kind of block structure.) How about language that support local functions? That’s why the linker sticks to .o files as linking unit.
2. How does the number and size of .o files in a single library or executable affect performance? (you imply it will decrease memory usage)
Nothing, with a demand-paging OS like Linux. The kernel only maps in what you use (read/write/execute). The pages containing only unused code/data will stay on disk and never make it to ram/cache. Diskspace is so cheap (unless you’re mirror the entire Debian archive) therefore the GCC’s bug ppl are discussing is kinda funny.
“By this, we conclude that KDE is either a distinct add-on
package that doesn’t ship with the OS (it is usually not)
or that KDE is getting installed in the wrong place.”
Are you comparing a toolkit (gtk) with a “package” KDE? “qt” (the toolkit KDE uses) is in “/usr” , the same as “gtk”
On SUSE, “gnome” is in “/opt”, the same as KDE.
Both “gnome” and “kde” are add-on packages.
so I don’t have that problem.
it is true i also switched to gentoo and forgot about all depencities hell.. a clear and most notable example is “mplayer”.
Under mandrake I had to hunt the internet for the right .rpm’s etc..now I just do :
emerge mplayer
and i’m ready.
>>During installation if an application needs a specific library version which would conflict with the currently installed then it is installed into the bundle. I can see a number of issues with this but I’m sure they could be ironed out.
Say, my python2.2 or perl5 or Apache came with its own libssl.so.
From now on, after I upgrade my system wide libSSL with security patch, I wouldn’t know if I’m really safe. That’s a another hell.
And you tell me how could it be ironed out?