Sharpen your tools
It is crucial you prepare a reliable set of tools before you get started. You will find this kind of advice in any book on software development so let me be more specific. You will encounter really weird problems, like hard reboots, or registers being thrashed in your thread for no logical reason. Your scheduler may pick the wrong task, or execute code in the wrong location. Even with the best programming abilities, miscommunication between programmers, or your brain spacing out for two minutes will do that, and when it happens, you need to be certain your code did it.
I tried to be smart once, and started using gcc 3.0 for development, instead of the 2.95 series. I was happy to find the new compiler reported more useful warnings and was a lot cleverer about subtle potential problems in the code, so I started adopting it for daily work. All was good, until I tried using it on strongarm. The scheduler would insist on running the idle task, which strongly reduced the usefulness of our system. The bug occurred in portable code, but the strongarm code base was new at the time, and there is no end to the side effects it could introduce. In my mind, a compiler couldn’t generate subtly wrong code, so I never questioned it. Out of things to try, I reverted to gcc 2.95. Sure enough, it made the code work, and disassembling shown that gcc 3 was optimizing so much, that it removed a variable related to the priority level, which forced it to be the idle one. Now gcc 3 made it fast! And useless.
Continuing in the open source world, the same goes for binutils. A lot of times, new versions have really interesting bugs. Symbols that are offset by 4 bytes in the internal representation of COFF, compensated by a hack that says /* I don't know why I need to do this */ somewhere else in the code. If you hack binutils and bfd to produce your own format, be aware that this is major quicksand. Use ELF and a reliable, official release of binutils unless you have a truly relevant reason for doing otherwise.
The moral of the story is to select reliable, proven tools. Save yourself frustration and time by being unable to accuse the toolset when something goes wrong.
Use a high-level language
This is not as controversial as it sounds. If you insist on using assembler even for kernel development, I assume you don't need advice and you are making good progress. Some projects can pull it off. I have written real-time kernels and even applications fully in assembler before. I was probably more stubborn than you can be about the virtues of assembler, but I would never go back to it now.
I won't even discuss the portability advantages of a high-level language. If you think C is not much more than a high-level assembler, you need to use it more. This said, you might not even have portability as one of your goals and don't necessarily care.
Using C, C++ or a high level language of your choice shields you from a whole class of problems: using the wrong register, swapping operands by mistake, using the wrong opcode, or miscalculating how many bytes you need for something. GNU tools don't even make it remotely easy. Going to the dentist is a nicer experience than writing assembler with gas or inline with gcc. You can always use nasm on x86, but it doesn't cover any other cpu. If you want to be portable, you are stuck with gas syntax.
If you're comfortable with keeping track of what is in every register and certain you will not pass an integer to a function expecting a pointer to your threads hash table, fine. If not, high-level languages provide you with very powerful tools to diagnose a problem when your brain spaced out. Types and prototypes will generate warnings or errors when you mismatch things by mistake.
No early optimizing
This can be seen as controversial advice again, but if you decide to use a high-level language, you should try to write as much as possible in that language, even if it is tempting to have an optimized version of some part of memory management or semaphores in assembler. It is highly probable your internal design will have changed a lot by the time you can run applications. By definition, if you optimize a critical piece, it is critical. It will be called a lot. If you introduce a bug in it, you will wreck all kinds of things.
At first you want to make it work. Then you will have a good view of how to make it work fast.
Don't try to code smart
In the same way, any attempt at smart coding should be avoided like the plague or going to the doctor. Don't save sixteen bytes by writing cryptic, bug-prone code. You will thank yourself a year later. Or you won't, a few days later, when you finally locate this really strange bug.
Use an instrumented environment
If possible, and if you're going for a bootable, standalone environment, the icing on the cake is to use an instrumented target machine.
Without one, you will have absolutely no help if your system doesn't boot as expected. Being able to execute your code step-by-step, set breakpoints, and watch memory or registers as your code executes, is invaluable. Later, when your operating system boots, the instrumentation can be used to monitor memory access patterns and execution time of your crucial code.
A lot of embedded boards come with JTAG ports, In-Circuit Emulators, and software to do exactly that. If you develop for fun at home, you probably do not want to afford them, but a reasonable starting point is then an emulator such as Bochs. It obviously executes much slower than a real machine, but this is not a problem when you start writing the system. You can step your code, in a virtual machine, without the need for another PC. A hack I contributed to Bochs lets you dump traces to an I/O port and read them on your console.
If nothing else, pepper your code with optionally compiled traces, coupled with a reliable method for displaying them. If your code does not work as expected, you then have the option to enable the traces and see what got printed last.
Small is beautiful
Once you have bootstrapped your effort, you probably want to have the smallest interface to your kernel as possible. Logic dictates that the less entry points you have from the outside world, the less you have to code and document. It is important to keep a small and consistent design in the external view you will give of your kernel, as much if not more than the internal view.
- "Introduction, Cover your bases"
- "Advices Part I"
- "Advices Part II and Conclusion"
