Issues regarding UNIX software package managment have brought us many usable, yet complicated package management tools, such as those seen in RedHat Linux, Solaris, BSD systems and Mac OS X. While being very useful at their tasks, they bring unnecessary complexity of the management software used, as well as dependency on central package database.A larger list of problems with the current package management schemes can, probably, be found elsewhere. Here we will concentrate on few ones, as the main task of this paper is to propose potentially more elegant and, first of all, usable solution
to package management problems.
First problem is package installation, or, at least, gaining access to package contents. Many modern systems use either variations of well-established formats, thus producing new format that is similar, yet can not be dealt with using conventional tools (RedHat RPM), or using unaltered format whose nature is somewhat concealed from the user (Debian DEB/cpio). While there is, actually,
nothing wrong with Debian approach, RedHat approach leaves us another “support entity”.
Second problem is package removal. As long as package contents is merged into the existing filesystem structure, a database of package contents is necessary for a safe and full deinstallation. Whether package contents is stored in the central database, or in database unique to each installed package, this accounts for another “support entity”.
Third problem is the coexistence of various versions of the same package. While idealogically unnecessary, sometimes this is the only way to do “smooth upgrade” on a living system. And we know that sometimes system uptime and availability is much more precious than idealogy.
At last, complexity of support software brings us another “support entity”, thus driving the final nail in the coffin of easy portability, and, which is much more important, in the coffin of the KISS principle. Rarely we can perform abovementioned operations easily without resorting to the third-party tools.
The package system proposed in this paper is supposed to solve at least four problems mentioned above, while make it easy to implement mode advanced features such as dependency tracking. These features will be, with some degree of probability, addressed in future revisions of this document.
Cornerstone problem, that of semi-portable package formats and support tools, is solved by using standard tar format (whether with gzip/bzip2 compression or not) for package files, and simple shell scripts as a management tools. While management tools are used, operations they perform can be easily done with standard UNIX tools.
Deinstallation problem is solved by using installed package hierarchy somewhat similar to the one used in Solaris and the one proposed in Linux FHS. Every package is installed inside its own subdirectory inside, say, /opt or /usr/opt filesystem. Thus, package install prefix will be something like /opt/name-version. This brings us the following problem: what to do with numerous search paths for binaries and libraries?
We can solve this problem by using the feature readily available in BSD systems and, as far as it known, in Linux since 2.6 kernel version. This feature is called “union mounts” in BSD, and, probably, something else in Linux.
In order to avoid unnecessary long search paths, we simply remount each package directory under, say, /usr/local, while making sure /usr/local stays at the top of the union, so any additions to it, whether intended or unintended, not end in some unfortunate package”s own directory.
Using discussed /opt scheme we can easily install two versions of the same package, and have at least two options for smooth transition: whether simply not mount the old directory, or mount the new one over the old. The latter option also allows us to do somewhat “incremental” upgrades. Furthermore, if two versions are so distant that they do not collide much, we may leave them both to support sometimes weird software dependencies.
Package deinstallation is as easy as “rm -rf”, assuming we do not do extra things like dependency checking, or do them by hand. Installation, on the other hand, is as easy as “tar xv[z][j]f”. Rolling your own package is similarly easy, you just make a tarball of already installed package”s directory under /opt filesystem.
The catch is that, your system has to support union mounts. As of BSD systems, they have it from long ago. Recent Linux kernel 2.6 is also promised to have this feature.
Please feel free to propose any additions to this document, or comment on possible stupidity of the overall idea, if such stupidity actually has place.
If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.
The following is what I did. I used a shell script so no special OS features are necessary.
– Create /Applications
– In that, create .bin, .etc, .man and so on
– Create a shell script (Update) that populates these directories with symlinks
– add /Applications/.bin to $PATH and /Applications/.lib to /etc/ld.so.conf
And voila, unzip an application to /Applications, run the Update script and everything is ready to use. Delete the application with rm -rf, run Update and the removal is completed.
It does not do dependency checking; versioning also isn’t implemented, but for its task it actually works quite well.
The described idea is pretty much package management at it’s most simple. (Except for the union mounts bit)
I can pretty much do this already manually from source.
I must say that I’m more in favour of the database style option. However the database would have to be better organised than it currently is.
I suspect that part of the problem lies in the convoluted directory structure that Unix uses. Binaries, Libraries amd other stuff are jumbled together in discrete sensible locations for universal access, rather than grouped based on application.
Perhaps what we need is a better way of locating and linking to/from these fragments of application software. This will sound very bad, but a register of installed libraries/versions/locations could be the way to go. The application on installation or compilation could check for installed versions of the libraries it wants/needs. I believe this is already possible, however the libraries have to be installed in the right location. Applications and libraries should know how to find nearly everything else they need themselves.
As for locating binaries. That’s trickier. As the user may not know what version they want. Multiple versions might not be ideal from a usability perspective anyways. Even with merged bin folders you have to deal with name conflicts etc.
I guess I don’t have any answers after all.
http://www.gobolinux.org/
http://zero-install.sourceforge.net/
I think the problem is non existant for most linux users. Most distros can now use synaptic, Mandrake has had urpmi for quite a while and a cosmetic improvement that could be brought to the distros I know is something like click-and-run for the web and the file system.
And of course, source based distros have their own elegant, if slower solution.
That being said, for desktop applications, I think autopackage is the way forward, especially when they’ll have achieved interoperability with the 2 main package management formats.
There is also InstallShield and InstallAnywhere (used by Limewire) along the same lines.
There is another interesting project that lets you run apps from a web repository (http://zero-install.sourceforge.net/index.html). That’s a fine mechanism for internal apps in a corporate environment for example.
So the solutions are there, some better than others but honestly, there is always a simple solution and if your distro doesn’t have one, then there is distrowatch.
How fast would it be to have 2000 union mounts on /usr/local ? One could just symlink everything required instead of union mounts, and have an uninstall script clean up those links.
Dependency checking would be kind of slow without a database. On installation one would have to check for the existance of various dependent libraries by searching the library path. But what about de-installation? How does one know that removing Mozilla will break galeon, epiphany, and anything else that relies on the Mozilla libraries.
So I don’t think we can get rid of the database. Though we could make it a bunch of xml/text files so that we don’t have to worry about a large binary file being corrupted and hosing the system.
Perhaps adapting autopackage to have a form that can manage an entire distribution would be good.
You described almost exactly the slackware package system, and you added the option to install each app in its own directory. The problem of union mount is easily solved by adding symlinks from the apps to /usr/bin or similar.
Each app gots it dir, most apps uses fonts, lib, and so on dirs in that one, if you want to have it that way, fine, if you want to share your libs or whatever, copy them to fonts: or libs: (which lives in /fonts and /libs) and remove them from the specific app.
Each app just looks for the files in it’s own fonts-dir or the system wide one.
Only problem is dependencies, in the amiga world that’s not a problem either since you just replace your older lib of whatever with the newer version, in the unix world it might be it because people tend to change apis, plain simply stupid.
This very much resembles the package system I developed, ezpkg. It used simple tarballs for packages, which would be extracted in a directory specific to that package and version. Symlinks (specified in a file in the package) would then be created in the system directories (backing up any existing files with the same name).
ezpkg is simple, portable, gracefully deals with multiple versions of the same package, and checks dependencies (also taking into account software installed outside ezpkg). However, it doesn’t do anything intelligent about configuration files. I think any system that cannot perform necessary edits to configuration files is for that reason not ready for release. I am not developing my system for now, as .deb works fine for me and I see no need to replace it. A .deb is an ar archive with three special files in it – that’s about as simple as it gets.
Some apps also looks for say gimp: (if that would be on the amiga) which means the disk called gimp, however, you can easily assign any directory to the name gimp: so basically you could have:
/apps/gimp-2.0
/apps/gimp-2.0/gimp
/apps/gimp-2.0/plugins
/apps/gimp-2.0/plugins/… (loads of files)
/fonts/gimp.font
/libs/gimp.library
and gimp: pointing at /apps/gimp-2.0 (in this case say /apps/default/gimp symlink to /apps/gimp-2.0).
So if gimp needs it’s fonts, it looks in /fonts or gimp:fonts (/apps/default/gimp/fonts) which resolves to /apps/gimp-2.0/fonts. If you don’t want to spread your apps around, they all live nice in /apps, if you want to share the fonts or say qt-40.library just copy them over to /fonts and /libs.
AmiBSD.. Aminux.. ahh, the possibilities!
Instead of using union mounts, take a look at GNU Stow. It should be a lot faster.
GNU Stow helps the system administrator organise files under /usr/local/ by allowing each piece of software to be installed in its own tree under /usr/local/stow/, and then using symlinks to create the illusion that all the software is installed in the same place.
Unfortunately the proposed “solution” does nothing to tackle the problems of upgrading between software versions (eg, automatically running scripts to rewrite configuration files, run ldconfig etc), or resolve package interdependencies. Just use Debian and “apt-get” — I switched five years ago, and I’ve never looked back. Stop trying to re-invent the wheel.
DarwinPorts and pkgsrc use this approach, though – at least DarwinPorts – hardlinks the files on ‘activation’ to your choosen prefix (say /opt/local); the ports themselves are put aside – each in its own subdir, incl. version and variants (like ssl, x11, etc.).
This way you even can have multiple versions of the same port installed and can activate the one you just need.
I agree – lets stop reinventing the wheel. there wouldn’t be much trouble if everyone used the same package management. and I guess .deb would be the best choice. not only because it is already used the most, and has the biggest amount of already packed, high quality software, but also because its quite flexible. although I think some changes could be made.
lets say
1= make apt able to handle source code better (a la gentoo, BSD ports)
2= possibility to install software as user, with apt (in home dir)
3= apt should be able to handle several versions of a package at a time
4= apt should be able to take hand-installed (configure/make/make install) software in account when tracking dependencies (autopackage)
5= maybe add some zeroconf capabilities
Is it just me, or are many of the comment posters who say “XYZ packing already does this” missing the article author’s goal?
If I am using XYZ format, I have to:
a) Wait for the official XYZ repository to package the latest version of app A
b) find some unofficial XYZ package of A
c) download A’s source, and use the XYZ package builder utils (which I have found painful at times)
Although I think 2000 Union mounts would be horribly inefficient (although I really don’t know for sure), the principal idea *I* gleaned from the original article was that I should be able to grab a raw source-ball, using standard configure + make (or other build process defined by source author, NOT by XYZ author), and have the latest/greatest installed in a way that I can easily upgrade, uninstall, etc.
Or, am I missing the point?
The author of the article misses out on a few other methods which would be simpler than union mounts..
1. use an automounter. have the automounter key point to whichever version of the application you’re currently supporting, but the key is always the same. so application foo can always be found by users in /opt/foo even though the application physically resides in /path/foo_version. An upgrade of application foo just requires an update of your automounter map.
2. use symlinks to accomplish the same role as an automounter (mentioned above).
When I compile a package from source, I configure with –PREFIX=/usr/local/packages/PACKAGE_NAME_VERSION so a ‘make install’ builds its tree in its own self-contained space.
The only tool is a script that creates relative symlinks into the /usr/local tree, checking for filename conflicts on the way, releasing the package into the system.
What’s easy is you can maintain more than one version of a software package in the packages directory, and can backout a new version and re-release an older one if something doesn’t work out.
Further, rather than a database, the tree itself describes the file structure the package, and ‘ls -ld /usr/local/bin/foo’ reveals what package and version foo belongs to, e.g. /usr/local/packages/FOO-PACKAGE-1.01/bin/foo.
Also, since the symlinks are relative to /usr/local, a computer on a network can simply mount /usr/local by NFS and to take advantage of the (remotely administered) software library.
Works for me better than scratching my head over RPM’s dependency hell.
Peter, thanks for pointing that out, very astute.
1. symlinks wont work as some programs (e. g. python) track the symlinks (python cant find its modules when symlinked)
2. I still cant have
-library v1 + program A linked against lib-v1
AND
-library v2 + program B linked against lib-v2
simultaneously
As said by amiga fans – app directories. If applications are in their own directories – a simple directory delete is all that is needed.
By extension use the framework feature in MacOS X, the code is already in gcc.
GnuSTEP already uses both of these concepts.
Frameworks can be versioned as well. App bundles can be easily portable by having multiple binaries under the proper system type.
You’ll run into problems with many apps that don’t have proper –prefix behavior. For example, KDE expects all it’s stuff to live in the same prefix, because of it’s configuration file hierarchy. On top of that, many packages just plain don’t like being installed in places other than /usr or /usr/local.
Don’t Mac OSen, including OS X, have the app directory thing (app installs into own directory, to delete just delete directory – in OS X, that’s drag ‘n’ drop to Trash) already?
What’s needed for any universal package management scheme to be more useful than all those that already exist is a willingness by developers of OSs and apps to standardize filesystem/dependency/install/uninstall behavior across the universe of OSs/distros and applications. Since we’re talking about folks who tend to be independent-minded, I don’t see this happening real soon.
A lot of the issues that come with having each program in a separate directory have been overcome in GoboLinux. All programs are created in the /Programs/Version/ directory and SymLinked to /System/
I’ll admit that there are still some hickups and issues that you run into, but the system works great and the filesystem is beautiful to look at.
/Programs
/System
/Users
From what I imagine, a simple dependency “control” system could use some standard format file (xml based?) to have the dependency information in the folders that come with the application.
The main issue would be to have a standard defined for the “control” files, and additional “configuration” files that the OS and other components could “add together” during their initialization phase, so that all configuration files could be handled in an by application independant way.
so add a new application tree and it starts working, remove the directories, and the application is gone.
Use the “dependecy” application to check hoe your system is, download missing components, remove old superfluous stuff…and so on.
No Database, just the application files.
Hope my explanation is good enough, have to write some better paper to expand the concept.
Alexandre.
“So I don’t think we can get rid of the database.”
One could run a daemon (FAM / IMon-alike or using such application) which detects the removal of a file which is part of the package. If that happens, actions are taken: the user is asked wether he/she is sure the package should be removed, the other files which are part of the packages are shown, etc but i’m not sure wether such interactive way is a good thing but it could be a good thing to prevent the breaking of the package database.
Why do people want to get rid of that package database anyway? I seriously don’t get it.
Regarding symlinks: one has to be careful to stay “NFS-friendly”. Frankly, i don’t find /Applications, /Libs, etc NFS-friendly.
“1= make apt able to handle source code better (a la gentoo, BSD ports)”
You’ll have to describe what’s wrong with APT in this aspect as it exists right now in order to say it has to become better and specify what “a la Gentoo, BSD ports” means. The authors most likely value constructive feedback, too.
PS: I’d also like to see more information on Union Mounts. The author could have provided a link to a description and/or implementation.
Maybe I’m missing the point myself, but it seems like this article is talking about a really interesting idea. I’ve wanted this for a while: the ability to mount two things to one “logical” location and have the files from both of them visible there. One should then be able to set it so that files written to that location are then divided up among the actual directories mounted there in a rules-based manner.
I think the author is hinting at this:
/config/app1 + /bin/app1 -> /usr/app1. If you change/add a config file in /usr/app1, that file should actually be written to /config/app1. This way all the files for a program could appear in one place without sacrificing the beneficial seperation between configuration and binaries.
dpkg + debconf + apt + apt-listbugs + debfoster = perfection
If you’re a little slower then most… aptitude/synaptics/kpackage will suite you…
Saying package management is bad on Linux is kinda ignorant. If all you use is the mainstream package manager, RPM, yes its cumbersome. Even with YUM, URPMI, you don’t have complete control.
Debian does one thing, and does it well, it manages packages.
This may or may not get moderated, but I really wish these articles wouldn’t be so ignorant to “The Debian Way”… For the most part, its very well thought out, and is the most logical method.
There is even a project called Kalyxo intergrating KDebConf into KDE amoungst other things. So those saying you can’t use Debian because there are no good GUI’s, there are, you just are too lazy to look. (granted the example in this paragraph may not be a final release yet, and doesn’t really help gnome users in this area).
Package management is why I go back to Debian every time. Nothing else compares, but no articles such as this even try to compare it.
</rant>
If you HAVE to have an up-to-date system… Debian provides SID… packages are new… but may have bugs etc, but then, if you are upgrading all the time, you are used to bugs. apt-listbug helps in this area in that there is a good chance someone found problems you may encounter before you even install it, then you can hold the package untill you know its fixed…
This even works with Experimental, although that I really don’t recommend. If you MUST be bleeding edge though, bugs.debian.org + apt-listbugs + knowing how to hold a package = fairly safe.
At this point, maybe zealot comes to mind? Perhaps… I have just used a lot of distro’s, none make things easier then Debian. You say SID isn’t supported? Well, neither is stable… its a not-for-profit organization, if you are using Debian, you likely don’t need support….
Worst comes to worst… it takes me about 10 mins to install a base system using the new installer… thats not bad if you ask me…
If you don’t currently run Debian however… wait until after Sept. 12th
No one that downloads a free iso is getting support, no matter what the developers etc say. You can’t call them and get help, you either get by with no help, or you live with it.
Most Linux users fit into this category, and thus do not get support either way. Saying “but sid isn’t supported” is not valid.
I think I have covered most angles to ensure I don’t get flamed 8)
For the record… I have contributed money to the Debian project… I purchased a book I didn’t need simply because it ensured me 40% of proceeds would go directly to the project.
Debian is the only project I have invested money into however.
(it was a potato book, I started early woody)
From the mount_unionfs(8) man page on FreeBSD 5.2.1:
BUGS
THIS FILE SYSTEM TYPE IS NOT YET FULLY SUPPORTED (READ: IT DOESN’T WORK)
AND USING IT MAY, IN FACT, DESTROY DATA ON YOUR SYSTEM. USE AT YOUR OWN
RISK. BEWARE OF DOG. SLIPPERY WHEN WET.
This code also needs an owner in order to be less dangerous – serious
hackers can apply by sending mail to <[email protected]> and announcing
their intent to take it over.
Without whiteout support from the file system backing the upper layer,
there is no way that delete and rename operations on lower layer objects
can be done. EROFS is returned for this kind of operations along with
any others which would make modifications to the lower layer, such as
chmod(1).
Now, that aside, I love the idea of a union mount. On FreeBSD w/ jails and some thought-out security considerations in mind, one _could_ expose a union fs to numerous jails. There’s a performance gain from this, since all the nasty disk IO is performed once; once the page is in memory it’s exposed to the cache. That’s the security issue, too. So, just make the files that are known not needing to be changed by the jails mounted RO, and now one can manage jails in more efficiently. If the jail wishes to change its version, only that jail will see the change. All other jails see the underlying files.
Works for me better than scratching my head over RPM’s dependency hell.
The same dependency hell .deb’s have with dpkg. The same dependency hell almost any other binary package management system has. One thing that alleviates that ‘hell’ for both RPM based systems and DEB based systems are frontend management systems like apt, urpmi, etc.
I’m tired of people acting like only RPM has dependency problems. DEB packages have the exact same issue. Most people don’t realize that dpkg (the Debian package management prgoram) and rpm both have the same issues. The only thing that gets rid of these issues is a higher level program like apt or rpm.
/applications/mozilla
/applications/mozilla/bin
in the mozilla folder,
/applications/mozilla-1.7/install.sh
install.sh contains
ln -s /applications/mozilla-1.7/bin/run-mozilla.sh /applications/bin/mozilla
and all relevant binaries
then add PATH=/applications/bin;/applications/sbin
so no symbolic links are in /usr/bin or anything.
use FAM or something similar to watch for changes in the file structure, and when changes are seen, run a script that deletes any broken links
no problem with uninstalls, because the links will be deleted automatically.
also, dependancies. these are simple if you use a new filesystem like reiser4(i think). one file can exsist in multiple locations(NOT LINKS), and if you delete one out of a folder, it is not really deleted on the storage device unless it was the last copy.
so if galeon needs mozilla files, it would just register that it wanted those files in the install.sh script, then if you deleted mozilla, the necessary files would remain but the directory would be gone.
of course you would have
/applications
/libraries
and other needed directories.
config files would need to be linked to etc, OR be multipoint files(for lack of a better name) that will be left intact as long as any program claims them.
the install.sh script would also need to scan the /libraries and /applications etc etc folders for dependancies like “mozilla-1.>7” or something, then register that it wants those files to the filesystem.
—
of course its not perfect, as i’m just one guy, and it has the limitation of requiring the use of a filesystem that supports specific features.
any thought on this?
The reason FHS is layed out the way it is, isn’t just to have $PATH convinience and know where to find files. Much of the layout is based on the selection of media to store files on.
ex.
/tmp as ramdisk
/var/tmp as a fast solidstate disk
/home on the large raid
/usr ro nfs mounted from the network storage server
/srv/www shared between the load balancing servers
… and so on.
With that said, even if we assume an single desktop system, with a single harddrive, the biggest problem with package management isn’t with keeping track of files, most of them does this easily. The big problem is maintaining a package namespace, version tracking, dependency resolving and system integration modifications.
First we need a package namespace, so that we can refer to the files. I suggest we just use URIs.
Then we need the files installed. I suggest install(1).
The package integrated intoo the system… I have no idea… a distro specific db is probably needed.
I have been playing with the though about system integration hints that the distro integration manager would parse.
We could call the tool deploy(1)
Upgrades, rdf feed to keep track, deply(1) would call diff3(1) to modify altered files.
unistallation, install(1) should have recored all files in a db.
Dependency resolving, a rdf/owl parsing webservice, or local service.
What have I missed?
also, dependancies. these are simple if you use a new filesystem like reiser4(i think). one file can exsist in multiple locations(NOT LINKS), and if you delete one out of a folder, it is not really deleted on the storage device unless it was the last copy.
Actually this is a feature most filesystems support, it is called a hardlink (man 1 ln).
There is another bug that was not mentioned. “Running find(1) over a union tree has the side-effect of creating a tree
of shadow directories in the upper layer.” I see that this is mentioned in both netbsd and freebsd man pages. For what this article seeks to accomplish, this would create a huge mess on the top layer of the 2000 odd union mounts.
That aside, I have had no issues with using mount_union on NetBSD.(The man page is not so nervous about letting the user use it either). In fact I have used it before with other tools such as pkg_comp from pkgsrc to build chrooted environments under which I can compile packages for slower netbsd boxen.
Please stop saying “Debian solved this.” Because all it means is: 1) you don’t understand the problem; 2) you don’t understand dpkg – because dpkg does nothing to solve this problem. You also don’t understand BSD Unix. Please inform yourselves by reading Martin Cracauer’s FreeBSD page ( http://www.cons.org/cracauer/freebsd.html ): “Perhaps the most impressive feature is that the FreeBSD “ports” system is built to handle applications that require incompatible versions of C libraries.”
Another approach is the one taken by the aforementioned GoboLinux, IMHO a greatly underestimated Linux distro. It could be regarded as cutting-edge and original, even. It doesn’t pay to be original sometimes, does it?
Debian with their volunteer-build database approach failed miserably in their original goal of having an up-to-date distro – proof of this is that every advanced Debian user and/or developer does _not_ run stable – and thereby thwarting their own policy. It’s gotten to be so chronic that Debian releases are farther and farther behind schedules, and their project leaders are elected on basis of an “I will do it, I’ll get everyone to get work done” platform.
The [semi-]succesful approaches (BSDs and GoboLinux) rely on the information that is _already_ on the build process itself. It doesn’t add another layer of complexity.
I guess we should take database approaches with a grain of salt from now on. More automation (BSD) approach, less humans (Debian). Humans are hard to manage, and lazy (Debian).
In fact, this problem of software dependency with a database approach might well be a _very_ hard mathematical (algebraic) problem. deb and rpm come nowhere near solving this in their actual state.
The database approach would facilitate management but in reality it severily limited the reality of management of the database facility approach.
I don’t see anyone mentioning Gentoo (www.gentoo.org)..?