There are many intertwined reasons for this phenomenon, and a good place to start looking at them is the duality of opposites between operating system standards - POSIX most prominent among them - and innovative, revolutionary ideas. One the one hand, implementing a standard such as POSIX gives a new operating system a leg up and a library of working or partially working software, always a boon to an OS looking to build a userbase. On the other hand, the real gain of many revolutionary ideas like orthogonal persistence (definition: making RAM persistent by writing it to disk at checkpoints rather than using a filesystem to permanently store data) can only come from breaking compatibility with these standards and with mainstream computer use, isolating the project in question! If we ever hope to go beyond Unix and Windows a solution must be found so that new systems can do their own thing without cutting themselves off from the rest of the world.
The Unununium project found an interesting way to handle this that actually helps solve a couple of other issues, too. Rather than require that all programs for their operating system be written in a compiled language like C and link to their system calls to start with, they're building a Python interpreter that can talk to the OS. This will render any standard Python code which doesn't link to special operating system features or other software (a sizable minority) nearly instantly portable, irrespective of the fact that they implement that orthogonal persistence I was talking about above.
The next largest problem for new operating systems is, of course, hardware driver support. Thanks to fragmentation and disagreement in the OS community every new system must over years of work laboriously build its very own unique library of drivers to drive the very same hardware everyone's already using on more mainstream operating systems! Some people think Linux is slow to gain hardware support, but at least it eventually does. Most hobby and academic projects can expect to be abandoned before they gain enough driver support to even match Torvalds's kernel.
Here, at least, there is an apparent solution: a standard interface for device drivers to talk to kernels with. The problem is that it was tried and Project UDI's efforts at a Unified Driver Interface failed to be adopted by anyone with enough political clout or driver support to count. Even the companies that created, funded and poured effort into UDI haven't actually adapted their operating systems to use it. The F/OSS software community doesn't support it either, as when the debate about a driver API/ABI for Linux came up on Slashdot a month or two ago, the overwhelming opinion was that not forcing drivers to link to the kernel sources would lead to too many proprietary drivers! This sort of attitude might work for a system with Linux's following, but no new OS is going to gain any support by forcing manufacturers or even other hobbyists and researchers to rearchitect their hardware drivers for each new, quirky kernel they want to run them on. UDI may not "benefit the Free Software movement", but for those of us who hope to someday see our kernels running on real, everyday hardware it is a godsend. And just imagine if a major kernel like Linux or a BSD fork did support it, rendering many of its drivers usable under hobby system XYZ! UDI may be sponsored by proprietary software makers, even the latest "Great Satan" SCO, but it has reasonable technical merit and does the job it's intended to do. More hobby and academic kernels should be UDI-compliant.
Yet what peril even awaits those who can write or obtain device drivers? The lack of runnable software. Every operating system that is developed must always have every single program of new software ported to it, even its own development toolchain. Again, while standards like POSIX and other similarities to mainstream operating systems can make this work easier, it is still a major piece of work. The integration of interpreted and high-level languages into the userland of the OS also helps in this regard, but more needs to be done. Every operating system ports certain important pieces of software like its toolchain and usually some user stuff like a text editor as well, and we have tutorial websites like BonaFide OS Development and The OSFAQ Wiki. Put them together! When you port a piece of extremely useful or common software you think everyone else will support, write about how you did it so everyone can learn from your experience! I know I will, and this collaboration should help with the effort of getting a userland set up.
Of course these few ideas won't make the world an OSdevers paradise, but I humbly think they will help. Standardized device drivers will aid new OSes in picking up the hardware support they often desperately need, while high-level and interpreted languages at the basic levels of systems will provide a small but substantial software base to systems getting onto their feet. Finally, I'd love to see an effort to document how common, everyday programs can be ported to new operating systems, because we all know mucking about in the GCC source tree without a map is nobody's idea of fun.
About the author:
Eli Gottlieb is an operating system hobbyist whose pet kernel lives at Glider. He hopes he's helpful, or at least sparks off good dialectics in the comments.
If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.