Linked by Thom Holwerda on Wed 9th Nov 2011 21:26 UTC, submitted by edwin
General Unix Way back in 2002, MIT decided it needed to start teaching a course in operating system engineering. As part of this course, students would write an exokernel on x86, using Sixth Edition Unix (V6) and John Lions' commentary as course material. This, however, posed problems.
Thread beginning with comment 496893
To view parent comment, click here.
To read all comments associated with this story, please click here.
RE: binary for windows....
by bogomipz on Thu 10th Nov 2011 22:35 UTC in reply to "binary for windows.... "
bogomipz
Member since:
2005-07-11

I guess all that is missing is a functional libc, and shared memory, shared libraries...

What makes you so sure shared libraries is important or even desirable?

http://9fans.net/archive/2008/11/142

Reply Parent Score: 2

RE[2]: binary for windows....
by Alfman on Fri 11th Nov 2011 04:04 in reply to "RE: binary for windows.... "
Alfman Member since:
2011-01-28

bogomipz,

Regarding shared libraries, I've often wondered this myself. Shared libraries are the cause of all dependency problems, but is there really that much of a net benefit?

I think maybe when RAM was extremely tight, the answer may have been yes. But these days, they may be of less value, we really ought to test our hypothesis.

Consider, that shared libraries can't be optimized over API calls. It might take 30 bytes to call a libc function, which shifts the bytes around again to do a syscall. In a static application, it could theoretically optimize away all the glue code to do a syscall directly while saving space.

Obviously we have to look at bigger libraries too, like libjpg, but even there I wonder how much space would be wasted if it were statically compiled.

This isn't to preclude to the use of shared libraries for applications which are genuinely related and deployed together. But I do see an awful lot of application specific libraries under /lib which have to be managed in lockstep with their associated application, why should these be shared libraries at all?

Reply Parent Score: 3

Vanders Member since:
2005-07-06

Shared libraries are the cause of all dependency problems


Library versioning and symbol versioning are a solved problem. It's only when developers do not follow the standards that they introduce problems. In a properly managed library dependency issues are non-existent. Glibc is the obvious example here: since Glibc2 (libc.so.6) was introduced on Linux, the library has remained both forwards and backwards compatible.

Consider, that shared libraries can't be optimized over API calls. It might take 30 bytes to call a libc function, which shifts the bytes around again to do a syscall. In a static application, it could theoretically optimize away all the glue code to do a syscall directly while saving space.


The glue code is either needed, or it is not needed. If it's needed, then you can't simply "optimize it away". If it's not needed, then you can just remove it from the syscall shim.

Remember that in many cases the syscall shim will also attempt to avoid a syscall. It's much, much better to do the sanity checking and return early if the arguments are bad before making the syscall. It may require a few more instructions before the syscall happens, but that's still far less expensive than making the syscall only for it to return immediately because the arguments are wrong.

In some cases it's also entirely possible for the syscall shim to satisfy the caller entirely from user space I.e. it doesn't even need to call into the kernel.

Obviously we have to look at bigger libraries too, like libjpg, but even there I wonder how much space would be wasted if it were statically compiled.

This isn't to preclude to the use of shared libraries for applications which are genuinely related and deployed together. But I do see an awful lot of application specific libraries under /lib which have to be managed in lockstep with their associated application, why should these be shared libraries at all?


Now this is where I do a 180 and agree with you! Shared libraries are overused and in a large percentage of cases are used inappropriately. As a simple rule of thumb I'd prefer that system libraries are shared, and any third party libraries required by an application should be static. That's not fool proof, but it's a starting point.

Reply Parent Score: 3

christian Member since:
2005-07-06

"I guess all that is missing is a functional libc, and shared memory, shared libraries...

What makes you so sure shared libraries is important or even desirable?

http://9fans.net/archive/2008/11/142
"

All well and good until you find a critical bug in that DNS client library that every network capable program you have installed uses, and you now have to recompile (or relink at least) every single one of them.

Shared libraries may give memory usage benefits or not, may cause DLL hell in some cases, but from a modularity and management point of view, they're a god send.

Performance poor? That's an implementation detail of using lookup tables. There's nothing stopping the system implementing a full run time direct link, at the expense of memory and start up performance.

Reply Parent Score: 2

bogomipz Member since:
2005-07-11

The argument that updating a library will fix the bug in all programs that dynamically link said library goes both ways; breaking the library also beaks all programs at the same time.

And if security is a high priority, you should be aware that dynamic linking has some potential risks on its own. LD_LIBRARY_PATH is a rather dangerous thing, especially when combined with a suid root binary.

Reply Parent Score: 2

Alfman Member since:
2011-01-28

christian,

"All well and good until you find a critical bug in that DNS client library that every network capable program you have installed uses, and you now have to recompile (or relink at least) every single one of them."

Your point is well received.

However this person has a slightly different suggestion:

http://www.geek-central.gen.nz/peeves/shared_libs_harmful.html

He thinks applications shouldn't use shared libraries for anything which isn't part of the OS. This would largely mitigate DLL hell for unmanaged programs.

I realize this answer is gray and therefor unsatisfactory.


A better solution would be to have a standardized RPC mechanism to provide functionality for things like DNS. The glue code would be small, and could always be linked statically. This RPC would be kernel/user space agnostic, and could be repaired while remaining compatible. I think the shift from shared libraries to more explicit RPC interfaces would be beneficial, but it'd basically need a new OS designed to use it from the ground up - now that linux hosts tons of stable code, it's unlikely to happen.

Reply Parent Score: 2

RE[2]: binary for windows....
by jabjoe on Fri 11th Nov 2011 11:32 in reply to "RE: binary for windows.... "
jabjoe Member since:
2009-05-06

Shared objects aren't really about saving space any more (much of Window's bloat is having a massive sea of common DLLs that might be needed, and multiple versions of them, for both x86 and AMD64). It's about abstraction and updates. You get the benefits of shared code from static libs, but to take advantage of new abstractions or updates, with static libs, requires rebuilding. That's a lot of rebuilding. Check out the dependency graph of some apps you use some time. They are often massive. To keep those apps up to date would require constant rebuilding. Then the update system would have to work in deltas on binaries else you would be pulling down much much more with updates. With shared objects you get updated share code with nothing but the shared object being rebuilt. Easy deltas for free. Having to rebuild everything will have a massive impact on security. On a closed platform this even worse because the vendor of each package has to decide it's worth them updating. Often it's even worse because each vendor has their own update system that may or may not be working. Worse, on closed platforms, you already end up with things built against many versions of a lib, often needing separate shared object files (which defeats part of the purpose of shared objects. Manifest is crazy with it's "exact" version scheme.) Static libs would make this worse. With shared objects not only do you get simple updates but abstraction. Completely different implementations can be swapped in. Plugins are often a system of exactly that. Same interface to the plugin shared objects, but each adds new behaviour. Also put something in a shared object with a standard C interface, and many languages can use it.

With an open platform and a single update system, shared objects rock. You can build everything to a single version of each shared object. You update that single version and everything is updated (fixed/secured). You can sensibly manage the dependencies. You removed shared objects if nothing is using them. You only add shared objects something requires. This can and is, all automated. This does save space, and I would be surprised if that if you build everything statically the install wasn't quite a lot bigger, unless you have some magic compressing filesystem witch sees the duplicate code/data and stores only one version anyway. But space saving isn't the main reason to do it.

Any platform that moves more to static libs is going in the wrong direction. For Windows, it may well save space to move to having static libs for everything because of the mess of having so many DLLs not actually required. But it will make the reliability and security of the platform worse (though not if it already has an exact version system, then it's already as bad as it can be).

In short, you can take shared objects only from my cold dead hands!

Reply Parent Score: 3

bogomipz Member since:
2005-07-11

In short, you can take shared objects only from my cold dead hands!

Haha, nice ;)

I agree with those that say the technical problems introduced with shared libraries and their versioning have been solved by now. And I agree that the modularity is nice. Still, the complexity introduced by this is far from trivial.

What if the same benefits could have been achieved without adding dynamic linking? Imagine a package manager that downloads a program along with any libraries it requires, in static form, then runs the linker to produce a runnable binary. When installing an update of the static library, it will run the linker again for all programs depending on the library. This process is similar to what dynamic linking does every time you run the program. Wouldn't this have worked too, and isn't this the natural solution if the challenge was defined as "how to avoid manually rebuilding every program when updating a library"?

Reply Parent Score: 2