Now that you have an idea of where your OS project is heading as a whole, it's time to go into specifics. The first component of your OS which you'll have to design, if you're building it from the ground up, is its kernel, so this article aims at being a quick guide to kernel design, describing the major areas which you'll have to think about and guiding you to places where you can find more information on the subject.
Post a Comment
Member since:
2010-03-08
Member since:2006-02-15
Member since:2008-11-27
Member since:2010-03-08
AFAIK, the PM feature of OSnews is discontinued (look at http://www.osnews.com/messages?op=compose&uid=5 ).
However, as an editor, I've got access to the mail address you used for subscribing, so I can contact you this way if you're okay with that 
Member since:2008-11-27
Member since:
2009-04-23
While you might be able to argue that a VM kernel could be more reliable with GC and array bounds etc.... arguing that it is more secure is a losing battle. VMs can and have been broken out of in the past.... I would actually think they would increase the amount of code were an exploit could occur.
Edited 2011-02-20 15:11 UTC
Member since:2010-03-08
Member since:2009-04-23
While any vailidation of your efforts on my part was wholly unintentional... you're welcome just the same :-)
I wish there were more information on exokernels myself.... the idea seems to have stalled back in 2000. XOK doesn't even compile with a modern compiler. Myself I am looking at going through the MIT OS developement course on my own... through the xv6 code and development of JOS which is an exokernel http://pdos.csail.mit.edu/6.828
EDIT: I found an old copy of the code apparently a newer copy is here http://pdos.csail.mit.edu/6.828/xv6/xv6-rev5.tar.gz
It doesn't require any hacking up to compile it... just make qemu or make bochs
Edited 2011-02-21 01:05 UTC
Member since:
2006-02-03
Member since:2010-03-08
Indeed. I'll correct this along with some English mistakes once I get access to some real internet connection (EDGE with locked ports is the new 56K).
When looking for examples, I've been highly displeased to discover that most operating systems which were said to be VM-based (like Inferno or JavaOS) were in fact nothing but a java virtual machine running on top of a regular C/C++ kernel, and have given up a bit too quickly apparently.
SharpOS was also an interesting project, but apparently development has ended.
Edited 2011-02-20 18:45 UTC
Member since:2005-07-08
That is because in the area VM operating systems the author is either not old enough, or did not research enough the subject.
Pascal MicroEngine in 1976, which used to process P-Code as instruction set.
The Lisp machines the early 80s.
The original Smalltalk environment at Xerox in
Forth is a VM, compiler and operating system, all in one, in the early 70s.
Modula-2 based system for the Lilith architecture.
Some versions of Oberon operating system, use the modules in bytecode form and compile them on load.
Granted this systems still do use some assembly at the core of their implementations, but so do C based OSs.
Member since:2010-03-08
What was the point of coding a kernel in an interpreted language without even having the benefits of modern interpreted languages like type safety and pointer/array bounds checking, just for the sake of having the code being interpreted ? Apart for proving "yeah, it's possible", sounds like a waste to me...
Member since:2005-07-08
Member since:2010-03-08
So, you are telling me, without a smile, that Forth manages to be type-safe without even having a type system ?
Indeed, especially when trying to understand computer history... These benefits go in both directions of the "age" axis though. I've had the chance to know the time where not every game was an FPS and to learn to program with Delphi without having COBOL and FORTRAN giving me nightmares during my childhood, so I'd say I came in this world just at the right time as far as PC evolution is concerned ^^
Edited 2011-02-21 08:22 UTC
Member since:2005-07-08
Member since:
2005-07-06
How do loadable kernel modules enter into the micro v. monolithic kernel debate? Is it possible that a microkernel could minimize the penalty of IPC by adding these services as modules? i.e. does "micro" mean merely that the kernel remains small during runtime, or that the kernel's core codebase is small, but by loading modules it could grow?
Member since:2010-03-08
Afaik, microkernels are defined by the way they put some of the services they offer in isolated processes. Microkernels are intrinsically modular, but monolithic kernels *can* also be modular without becoming microkernels.
Whether monolithic kernels are modular or not, most of their main advantages and drawbacks remain, since most code runs in the same address space, in kernel mode, with full access to hardware. Modularity acts independently from that.
Edited 2011-02-20 20:54 UTC
Member since:2005-07-06
Okay. So you could have a monolithic kernel that is actually quite small, and loads modules dynamically into memory as needed. The drawbacks of security would remain, and might even be worsened if the administrator was dumb enough to load an unsecure module to extend the OS' capabilities.
The reason I ask is that modularity could decrease code size significantly. You mention that it is common for desktop monolithic kernels to include facilities for rendering GPU-accelerated graphics and managing every single filesystem in existence, which, okay, could be bad if all these filesystems are in memory all the time, but isn't so bad if they are loaded only when needed. It likewise could provide a defense against what you cite as unmaintainable, non-portable, non modular code.
It makes me wonder whether most monolithic kernels do this in practice.
I guess I should say that this line of questioning is inspired by my acquaintance of Microware's OS-9, which I once heard described as neither monolithic nor micro but modular. But I don't know more than that, and even that was a distant memory.
Member since:2010-03-08
Indeed, more modularity can be used as a way to reduce the disadvantages of the monolithic approach on "large" kernels, without going as far as the microkernel approach. It's an interesting in-between solution.
On the other hand, there must be a policy somewhere that forces kernel devs to put new features in separate modules whenever possible. Otherwise, you get something like Linux : the kernel is modular, but outside of the realm of hardware drivers its modularity capabilities are heavily under-used (and since there's no standard, stable module interface, third-party module development does not work well... But that's another story)
Member since:
2006-05-09
Beautiful article!!!
Some years ago I was fascinated reading about L4 and its several implementations. What I find great on microkernels is the way of implementing everything as servers (isolated processes) or implementing several "personalities" running on top of the microkernel; so you could, in theory, having your microkernel with several "virtual machines" running on top of it. Could you think on Xen or VMware ESX Server as having some microkernel-like design?
Member since:
2007-08-20
Windows is more hybrid in recent years since the introduction of User-mode Driver Framework: http://en.wikipedia.org/wiki/User-Mode_Driver_Framework
IIRC, Aero graphics drivers run in usermode.
Another VM-based OS that's had some coverage on OSNews is Phantom OS: http://dz.ru/en/solutions/phantom/
Their goal is full persistence of all processes, data etc across shutdowns. Last blog post was October 2010 so perhaps it's not dead yet.
Member since:2010-03-08
IIRC, Aero graphics drivers run in usermode.
Indeed. That thing crashes so often on my machine that if it was in kernel mode, I'd spend more time rebooting than doing something useful when I'm on Windows, like in the Win9x days...
Don't know which part exactly of Windows' graphics stack is in user mode, though. I find it hard to believe that they could just fully move graphics driver in user space. That would break driver compatibility, which is definitely not Microsoft's thing.
Their goal is full persistence of all processes, data etc across shutdowns. Last blog post was October 2010 so perhaps it's not dead yet.
Added, along with some links.
Edited 2011-02-21 08:28 UTC
Member since:2005-07-06
Don't know which part exactly of Windows' graphics stack is in user mode, though. I find it hard to believe that they could just fully move graphics driver in user space. That would break driver compatibility, which is definitely not Microsoft's thing.
Microsoft did break compatibility with Windows Vista by pushing more responsibility off onto the hardware vendors hence WDDM (Windows Display Driver Model) required virtual re-write of drivers hence the reason why Windows Vista launch was so problematic. Windows 7 has WDDM 1.1 which has bought back hardware acceleration for some GDI functions given in 1.0 GDI was totally software driven.
As for Windows crashes; when you have multiple points of failure; crappy hardware, crappy drivers and a difficult to understand driver API then things will never work out as planned. With that being said, however, if your hardware vendor produces quality hardware and uses the latest Windows Driver Kit and takes advantage of the 'tried and tested' templates that exist then many of the issues shouldn't arise (at least in theory).
Member since:2010-03-08
Well, although there are some random Aero crashes, Aero mainly crashes when running old games and other things running in full screen which do not use DirectX (or at least not the latest releases), so I bet I'm just not in the official test cases
Also, I have an exotic GPU setup (Optimus), which further increases the likelihood of something breaking.
I've reported that on Microsoft's bug reporting tool, but they were not very helpful in terms of telling me who else I should contact if it's not their fault.
Anyway, I've switched back to using Linux as my main OS some months ago, so I don't care so much about that anymore. And as said before, I must admit that it breaks quite nicely, silently falling back to CPU rendering without a glitch.
Edited 2011-02-21 11:40 UTC
Member since:
2011-02-21
Member since:2006-03-03
Member since:2011-02-21
I'd rather take back my comment, actually, as it didn't really add much to the conversation. I had poked around the FAQ and forum a bit and was dismayed by how discouraging people were sometimes being to newbies.
...but if I had to point something out, there were a lot of articles on the initial boot-up of systems which talked as if the details were universal, when they were really all x86-specific.
Again, though, I wish I could retract my previous comment. The site *does* have a lot of no-doubt useful information.
:)
Member since:2005-11-16
Hi,
This isn't intentional. Anyone who has information on other platforms is encouraged to add to wiki.osdev.org.
The reality is, most hobbyist OS developers (just like normal users) have easy access to one or more "PC compatible" computers (required for testing, etc), and don't have easy access to anything else.
Note 1: "testing" means being able to find bugs that exist but only show symptoms in some situations, by being able to test on a (hopefully large) number of different systems.
Note 2: "easy access" means you (and other people that volunteer to test your OS) can mess with system software without special equipment (JTAG cables, flash/EPROM burners, etc) and without worrying about bricking the system.
Most other platforms are either rare (e.g. SPARC), obsolete (e.g. Alpha), or too expensive (e.g. Itanium). This even includes other platforms that use x86 CPUs (UEFI based systems), which are still far less widespread than "PC compatibles".
Then there's embedded CPUs (MIPS, ARM) which can't really be considered a platform because there's no standardisation for much more than the CPU's instruction set. For these, you can write an OS for one system and it won't really work on other systems that have the same CPU type. For example, you can write an OS for one of the ARM development boards, but you'd need to port the OS to any other ARM systems you want to support, which makes it hard to test on a wide variety of systems (especially during the early stages of development).
Anway, the end result of "all the above" is that most hobbyist OS developers want information for "PC compatible" computers, and eventually become people who are able to add information about "PC compatible" computers to wiki.osdev.org.
I'd also point out that some of the information on wiki.osdev.org applies to any platform. This includes the "theory" sections, some hardware information (PCI and USB), etc.
- Brendan
Member since:2011-02-21
Member since:
2005-11-22
AFAIK because NT kernel does indeed include some features in kernel space where as pure microkernels separate all of those features into user space, but it still loads other features as services into user space, Therefore NT kernel is called a hybrid kernel. it is not actually monolithic and different from Linux's module architecture. (BeOS is also similarly a adopted a hybrid kernel architecture.) So it is not true to call it monolithic kernel architecture because everything is separated.
Member since:2005-11-22
Member since:
2011-01-28
I am absolutely delighted that my suggestion to use a type safe language to provide isolation got attention in the "VM Based Kernels" section.
"The kernel must include a full featured interpreter for the relevant language, which means that the codebase will be huge, hard to design, and that very nasty VM bugs are to be expected during implementation."
Yes, a type safe VM language will require more work than simply using a traditional compiler.
As for devel effort, with any luck we wouldn't have to completely re-invent the wheel and could use existing VM implementations like Java or Mono. Writing a VM from scratch would be a ton of work - even if it has merit.
I think the term "interpreter" mis-characterizes the approach. A type safe language capable of isolation is not dependant on being interpreted, it can use JIT and pre-compilation too.
"Making a VM fast enough that it is suitable for running kernel-level code full of hardware access and pointer manipulation is another big programming challenge."
As discussed in the earlier comments, I don't think the type safe language would imply any overhead overhead over a correctly implemented unsafe-language version. All we require are safe language constructs which map efficiently over top of the underlying hardware like (memory mapped devices, port IO, DMA).
"A VM running on top of bare hardware will be harder to write, and thus more buggy and exploitable, than a VM running in the user space of an operating system."
This depends on how deeply integrated Java/Mono are with external dependencies (such as pthreads, or libc, or syscalls). Since I don't know the answer, I'll let the criticism stand.
"Currently, the Java Virtual Machine is one of the biggest sources of vulnerabilities in existence on desktop PCs."
Citation?
Also, remember that, unlike a web browser sandbox, the kernel/VM isn't required to protect from maliciously altered kernel modules (untrusted code). It only needs to ensure that modules written in type safe code remain isolated.
As long as the compiler only produces in-spec modules, then we can reasonably get away with a VM which produces undefined results for out-of-spec modules.
Features I thought about for my OS many years ago:
It would be very nice to have support for clusters within the OS, such that you could take any running application and migrate it to another node while continuing to run. Similar to what VMware/KVM/Xen do, but would work with arbitrary applications.
Every single kernel interface should have the ability to be virtualized such that all interfaces on PC-A could be transparently redirected/aliased on PC-B without explicit support for this within the drivers.
PC-A: soundblaster, webcam
PC-B: remote alias to peripheral devices from PC-A
This would go hand in hand to make the application migration feature seamless.
This also means that all local applications could bind to remote devices transparently without being explicitly written to do so.
Other thoughts:
I hope you're not planning on copying the *nix signal model, it's pretty bad for modern requirements.
I also hope you opt for an asynchronous IO design within the kernel over a threaded IO design. This has been one of the weaknesses plaguing linux for years.
While Posix plays an important role in compatibility and standardization, it's a major impediment to revolutionary designs. Therefore I think strict Posix compliance should be an explicit non-goal, particularly with regards to fs permissions and some of the braindead APIs.
Now if only someone would employ me to work on these things... Is anyone else here woefully under employed?
Member since:2011-02-24
JNode and SqeakNOS are practical examples of VM based kernels - esp. check out Squeak and it's associated OS branch squeakNOS. It looks like development has stalled again, so it's missing COG integration (JIT compiler) and the latest I/O libraries, and the optimisation is pretty non-existant at this stage. It does run reasonably quickly however.
As per the argument put forward by the Phantom OS team, you should be able to (long term) optimise this sort of OS better than a monolithic kernel thanks to zero context switching.
In addition, the VM would take away a whole class of bugs in user-level code eg. buffer overrun bugs in flash plugins. OS-wide I think you'd have a net gain and any VM bugs would show themselves quickly (and get fixed for an open source project anyway).
Checkout the Spoon image - it can do this and much more (versioning is handled well), major update due in March!
Member since:
2011-02-23