The article which you're currently reading aims at helping you answering a rather simple, but very important question: are you really ready to get into OS development? Few hobbies are as demanding, time-consuming and late-rewarding at the same time. Most hobby OS projects end up reaching a dead-end and slowly being abandoned by their creators because they didn't understand well enough what they were getting into and wrote insufficiently maintainable and ill-designed code or were overwhelmed and depressed by the amount of time it takes to get a mere
Some bad reasons for trying hobby OS development
"I'll bring the next revolution of computing": Simply put: no. Long answer: think of yourself as a small research team. You can make something very neat, sure, but only at a small scale. Without massive backing, your research won't make it to mass distribution. And no, you can't emulate an existing OS' software in order to appeal to a wider audience.
"I want to become proficient with a programming language, and need some challenging exercises": Some people try to learn a new programming language through OS development, thinking that they need something difficult to do in order to really understand the language. That's not a good idea either.
First, because it makes things even harder than they already are for the rest of the OS-deving world, and you shouldn't overestimate your ability to withstand frustration and strain.
Second, because being new to the language, you'll make more mistakes and those mistakes are much harder to debug when done at a low level than they are at the user application level where you have neat tools like GDB and Valgrind to help you.
Third, because you won't actually learn the language itself, but only a subset of it. The whole standard library will be unavailable and except for a few languages designed for the job like C, most programming languages have features which require a runtime support you won't be able to provide for some time. A few examples: C++ exceptions and vtables, C# and Java's garbage collectors and real-time pointer checks, Pascal Object's dynamic arrays and nice strings... More details here and there.
In short, you'll have a hard time coding things. When you do, you'll write poor code and you won't actually learn the language. Don't do it.
"Implementing a new kernel is horribly complicated, but there's a much easier path! I'll just take the Linux sources and tweak them till I get an operating system which fits my needs, it shouldn't be so hard": May I be the first to introduce you to what Linux's sources actually look like by showing you its main.c file. Your first mission will be to locate where the "actual" main function is. Your second mission will be to understand where each mentioned function is located and what it's supposed to do. Your third mission will be to estimate the time it'll take you to understand all of that.
As you should have understood by now, modifying an existing, old codebase without a good knowledge of what you're getting into is really, really hard. Code reuse is a very noble goal and it has many advantages, but you must understand now that it won't make your life simpler but rather make it much worse because you'll not only have to learn kernel development as a whole but also all the specifics of the codebase you're looking at.
"I don't like detail X in Linux/Windows/Mac OS X, these should totally be re-done": All modern operating systems have their quirks, especially the desktop ones, no question about that. On the other hand, if your gripes against them remain relatively minor, you really should consider trying to contribute patches which fix them to the relevant projects and only rewriting the incriminated component(s) if this approach fails.
"I'm looking for an exciting job": cf. "I'll bring the next revolution of computing". Except if you're in a research lab which works on operating systems, you'll hardly get any money from your OS as long as it has no practical uses (and it can remain the case for a very long time).
Some good reasons for trying hobby OS development
"I want to experiment with new OS design": You have your own idea of what an operating system should be like and you can't find it in the existing OS market ? Hobby OS development can be a way to try designing and implementing what you've envisioned and see if it actually works in practice.
"I'm now quite experienced with a language and looking for a programming challenge, or want to know how my computer works at the lowest levels": OS development certainly provides both, and to a great extent. You'll learn how to debug software with hardly anything more than text output facilities, how to do without a standard library or any sort, why so many people complain about this x86 architecture which works so well on your computers, what bad documentation from the vendor truly is...
"I've been working on an operating system for some time, know its codebase well, and have many gripes with it": If addressing these gripes would imply major changes, the kind of which makes software incompatible, you'll probably no choice but to fork, which while not described in this article is a totally valid path for OS development (and one which is certainly more rewarding than writing a new OS from scratch).
"The set of changes I want to bring to an OS is too big for simply patching or forking part of it": There sure is a point where starting a new project becomes the best option. If you think you have reached it, then welcome to the club!
"I'm looking for an exciting hobby": Well, depends a lot on what your definition of "exciting" is, of course, but OS development can be one of the most exciting hobbies on Earth if you're sufficiently prepared. It's somewhat akin to raising a child or growing plants: sure, it's slow, and sometimes it gets on your nerves and feels highly unrewarding. But this just makes success feel even better. Plus it's incredibly fun to be in full control of what's happening and have absolute power on your hardware.
So, are you ready?
Aside from the questioning phase, here are a few things which you must know or learn quickly in order to take this road successfully:
General computer science: You must know well and be able to fluently manipulate binary and hexadecimal numbers, boolean logic, data structures (arrays, linked links, hash tables, and friends), and sorting algorithms.
Computer architecture: You should know the global internal organisation of a desktop computer. The words and expressions ALU, interrupts, memory and PCI buses, and DMA should not sound mysterious to you. Knowledge of more automated mechanisms like the TLB, multilevel caching, or branch prediction, would be a very valuable asset in the long run, once you try to optimize code for performance.
Compiler and executable files internals: You must roughly know what happens during the preprocessing, compilation, and linking steps of binary generation, and be able to find everything you need to control your toolchain in its manual. You should also learn about linker scripting and the internal structure of executable files.
Assembly: You must know what Assembly is and how it works, and you must be able to code simple Assembly programs (playing with the stack and the registers, adding and subtracting numbers, implementing if-like branching...). You can write most of your operating system in other programming languages, but at some point you will need assembly snippets to do some tasks.
C or C++: Even if you plan to use another programming language to write your OS in, you'll need to know C or C++ in order to understand the code you'll find on the web when looking at other operating systems' source or other OS-deving tutorials.
Your system programming language: No matter whether you plan to write your OS in C#, Java, Assembly, BASIC, C, C++, Pascal, Erlang or REBOL, you should be very familiar with it. In fact, you should even have access to a description of its internals. As an example, you should know how to call a function written in your language of choice from Assembly code, what kind of runtime support your language requires and how it can be implemented.
(List based on this page of the OSdev wiki and tweaked based on personal experience.)
This ends the "Are you ready ?" part of this OS-deving tutorial. If you have reached this point, congratulations ! You should have the right mindset and sufficient knowledge to begin the hobby OS development adventure. The next article will explain to you how to set goals and expectations for your OS project, a truly vital step if you actually want it to get somewhere in the end.