You can spend no more than an hour researching online, and you will find literally hundreds of open-source operating system projects. Deep down, the developers of these projects hope to become the next Linus. Their dream is to complete a working system, and be swamped in e-mail from all over the world with stories of how reliable and friendly their nanokernel-based system is.
All of us are happy if others find the creation of our own design useful. To paraphrase Frederick P. Brooks Jr., author of "The Mythical Man-Month", desire for praise starts early, with children making pencil holders “for daddy's office".
Going back to the online projects, most are unfortunately stuck in the dream stage or boot loader stage. Grandiose plans are expressed with a boot sector and sometimes a "hello world" in C, last modified two years before.
There are many reasons for operating system projects to end that early in the cycle. In a previous article, I shared my experience of developing a closed source project to completion and listed generic points. They do not guarantee success, but in my experience they are important to help focus on the end goal. Weed the distractions out. Get the job done.
You have decided on your camp. You know your audience. You have decided to get real about what you can do. You have agreed to be the benevolent dictator of your project. Well, you still have to write your operating system, now don't you?
Cover your bases
Systems programming is the hardest form of programming. To paraphrase the good Mr. Brooks Jr. again, compilers are three times more complex than regular applications, and operating systems are three times more complex than compilers. That was his rule of thumb in 1975, after managing the IBM OS/360 project to completion. If we consider how much is expected from operating systems today, we can chance that the chasm is even wider. Ironically, phenomenal progress has been made in easing application development, while you still need to write kernel code with your bare hands.
Let it sink in. You want to climb Mount Everest. It is fun and rewarding to climb Mount Everest, but you don't want to do that with a small plastic hammer, wearing shorts and a t-shirt, at minus fifty degrees.
As your first operating system project is by definition the first one, you want to give yourself some credit and just dive in, after you have all the basics covered. If operating system programming is at least nine times harder than regular applications programming, logic would dictate that you better be a good applications programmer. You better know how to manipulate all basic data structures. Lists, arrays, hash tables, arrays of function vectors, bit arrays will all become useful as you develop your kernel. Any error in manipulating them will make you a new member of the triple-fault club. Worse, subtle non-systematic bugs will annoy you for days and take the fun out.
Good software engineering dictates modularity. This is not just for teaching in computer science classes. If you're a good programmer, you're lazy. You don't want to spend hours isolating a bug to finally locate it in your umpteenth list insertion inline code. Make one set of dependable primitives, and use them. The net effect in terms of bugs and frustration saved is tremendous.
A fellow systems programmer, who responded to a previous piece, mentioned he saw the same attitude while a mentor for aspiring game writers. People who had never written a working Tetris game contacted him and asked "How do I write Quake?". Get real and make sure you stand on solid ground before progressing higher on the mountain.
Any good practice of software engineering applies to operating system development, and is at least nine times as important. I suggest you draft at least some sort of specifications and design document, keep things modular, write test suites, use version control, a bug database, and all sorts of good practice you may have encountered while writing other kinds of software.
As a systems programmer, you know you don't know what you are doing. So do it carefully.