Linked by Hadrien Grasland on Sun 20th Feb 2011 13:20 UTC
OSNews, Generic OSes Now that you have an idea of where your OS project is heading as a whole, it's time to go into specifics. The first component of your OS which you'll have to design, if you're building it from the ground up, is its kernel, so this article aims at being a quick guide to kernel design, describing the major areas which you'll have to think about and guiding you to places where you can find more information on the subject.
Order by: Score:
Phew, finally !
by Neolander on Sun 20th Feb 2011 14:26 UTC
Neolander
Member since:
2010-03-08

And now, let's go back to my own OS and those damn accomodation issues in Uppsala ^^ I didn't expect this article to take so long, but you should probably not expect the next one to come next week either ;)

Edited 2011-02-20 14:28 UTC

Reply Score: 1

RE: Phew, finally !
by WereCatf on Sun 20th Feb 2011 14:29 UTC in reply to "Phew, finally !"
WereCatf Member since:
2006-02-15

Thanks for yet another great article, I'm sure there's plenty of people here who have found these a great read ;)

Reply Score: 4

RE: Phew, finally !
by abstraction on Tue 22nd Feb 2011 08:49 UTC in reply to "Phew, finally !"
abstraction Member since:
2008-11-27

Great choice of University. You won't be disappointed =) If you have any questions about Uppsala or the uni don't hesitate to ask me. Drop a PM.

Reply Score: 1

RE[2]: Phew, finally !
by Neolander on Tue 22nd Feb 2011 15:14 UTC in reply to "RE: Phew, finally !"
Neolander Member since:
2010-03-08

AFAIK, the PM feature of OSnews is discontinued (look at http://www.osnews.com/messages?op=compose&uid=5 ).

However, as an editor, I've got access to the mail address you used for subscribing, so I can contact you this way if you're okay with that ;)

Reply Score: 1

RE[3]: Phew, finally !
by abstraction on Tue 22nd Feb 2011 22:15 UTC in reply to "RE[2]: Phew, finally !"
abstraction Member since:
2008-11-27

No problems!

Reply Score: 1

Comment by cb88
by cb88 on Sun 20th Feb 2011 15:10 UTC
cb88
Member since:
2009-04-23

While you might be able to argue that a VM kernel could be more reliable with GC and array bounds etc.... arguing that it is more secure is a losing battle. VMs can and have been broken out of in the past.... I would actually think they would increase the amount of code were an exploit could occur.

Edited 2011-02-20 15:11 UTC

Reply Score: 1

RE: Comment by cb88
by Neolander on Sun 20th Feb 2011 18:24 UTC in reply to "Comment by cb88"
Neolander Member since:
2010-03-08

Well, I thought that this article displayed too much my preference towards the microkernel approach ;) Thanks for showing me that I did a good job balancing it after all.

Reply Score: 1

RE[2]: Comment by cb88
by cb88 on Mon 21st Feb 2011 00:59 UTC in reply to "RE: Comment by cb88"
cb88 Member since:
2009-04-23

While any vailidation of your efforts on my part was wholly unintentional... you're welcome just the same :-)

I wish there were more information on exokernels myself.... the idea seems to have stalled back in 2000. XOK doesn't even compile with a modern compiler. Myself I am looking at going through the MIT OS developement course on my own... through the xv6 code and development of JOS which is an exokernel http://pdos.csail.mit.edu/6.828

EDIT: I found an old copy of the code apparently a newer copy is here http://pdos.csail.mit.edu/6.828/xv6/xv6-rev5.tar.gz

It doesn't require any hacking up to compile it... just make qemu or make bochs

Edited 2011-02-21 01:05 UTC

Reply Score: 1

jnode
by antonone on Sun 20th Feb 2011 17:56 UTC
antonone
Member since:
2006-02-03

Thanks for this article, but it appears that the JNode system got missed from the sample OS'es based on VM approach. It even has a GUI so it might be interesting thing to watch, if anyone's interested.

Reply Score: 1

RE: jnode
by Neolander on Sun 20th Feb 2011 18:36 UTC in reply to "jnode"
Neolander Member since:
2010-03-08

Indeed. I'll correct this along with some English mistakes once I get access to some real internet connection (EDGE with locked ports is the new 56K).

When looking for examples, I've been highly displeased to discover that most operating systems which were said to be VM-based (like Inferno or JavaOS) were in fact nothing but a java virtual machine running on top of a regular C/C++ kernel, and have given up a bit too quickly apparently.

SharpOS was also an interesting project, but apparently development has ended.

Edited 2011-02-20 18:45 UTC

Reply Score: 1

RE[2]: jnode
by moondevil on Sun 20th Feb 2011 22:33 UTC in reply to "RE: jnode"
moondevil Member since:
2005-07-08

That is because in the area VM operating systems the author is either not old enough, or did not research enough the subject.

Pascal MicroEngine in 1976, which used to process P-Code as instruction set.

The Lisp machines the early 80s.

The original Smalltalk environment at Xerox in

Forth is a VM, compiler and operating system, all in one, in the early 70s.

Modula-2 based system for the Lilith architecture.

Some versions of Oberon operating system, use the modules in bytecode form and compile them on load.

Granted this systems still do use some assembly at the core of their implementations, but so do C based OSs.

Reply Score: 3

RE[3]: jnode
by Neolander on Mon 21st Feb 2011 07:32 UTC in reply to "RE[2]: jnode"
Neolander Member since:
2010-03-08

What was the point of coding a kernel in an interpreted language without even having the benefits of modern interpreted languages like type safety and pointer/array bounds checking, just for the sake of having the code being interpreted ? Apart for proving "yeah, it's possible", sounds like a waste to me...

Reply Score: 1

RE[4]: jnode
by moondevil on Mon 21st Feb 2011 07:55 UTC in reply to "RE[3]: jnode"
moondevil Member since:
2005-07-08

Funny that all the languages I mention have those benefits.

And most of the examples given by me have JIT implementations.

Do you know that Sun's Java Hotspot was actually developed for Smaltalk (Self)?

Age has some benefits... ;)

Reply Score: 2

RE[5]: jnode
by Neolander on Mon 21st Feb 2011 08:02 UTC in reply to "RE[4]: jnode"
Neolander Member since:
2010-03-08

So, you are telling me, without a smile, that Forth manages to be type-safe without even having a type system ? ;)

Age has some benefits... ;)


Indeed, especially when trying to understand computer history... These benefits go in both directions of the "age" axis though. I've had the chance to know the time where not every game was an FPS and to learn to program with Delphi without having COBOL and FORTRAN giving me nightmares during my childhood, so I'd say I came in this world just at the right time as far as PC evolution is concerned ^^

Edited 2011-02-21 08:22 UTC

Reply Score: 1

RE[6]: jnode
by moondevil on Mon 21st Feb 2011 11:53 UTC in reply to "RE[5]: jnode"
moondevil Member since:
2005-07-08

Ok, in what concerns Forth you are right.

Reply Score: 2

kernel modules
by jack_perry on Sun 20th Feb 2011 20:31 UTC
jack_perry
Member since:
2005-07-06

How do loadable kernel modules enter into the micro v. monolithic kernel debate? Is it possible that a microkernel could minimize the penalty of IPC by adding these services as modules? i.e. does "micro" mean merely that the kernel remains small during runtime, or that the kernel's core codebase is small, but by loading modules it could grow?

Reply Score: 2

RE: kernel modules
by Neolander on Sun 20th Feb 2011 20:47 UTC in reply to "kernel modules"
Neolander Member since:
2010-03-08

Afaik, microkernels are defined by the way they put some of the services they offer in isolated processes. Microkernels are intrinsically modular, but monolithic kernels *can* also be modular without becoming microkernels.

Whether monolithic kernels are modular or not, most of their main advantages and drawbacks remain, since most code runs in the same address space, in kernel mode, with full access to hardware. Modularity acts independently from that.

Edited 2011-02-20 20:54 UTC

Reply Score: 1

RE[2]: kernel modules
by jack_perry on Sun 20th Feb 2011 21:45 UTC in reply to "RE: kernel modules"
jack_perry Member since:
2005-07-06

Okay. So you could have a monolithic kernel that is actually quite small, and loads modules dynamically into memory as needed. The drawbacks of security would remain, and might even be worsened if the administrator was dumb enough to load an unsecure module to extend the OS' capabilities.

The reason I ask is that modularity could decrease code size significantly. You mention that it is common for desktop monolithic kernels to include facilities for rendering GPU-accelerated graphics and managing every single filesystem in existence, which, okay, could be bad if all these filesystems are in memory all the time, but isn't so bad if they are loaded only when needed. It likewise could provide a defense against what you cite as unmaintainable, non-portable, non modular code.

It makes me wonder whether most monolithic kernels do this in practice.

I guess I should say that this line of questioning is inspired by my acquaintance of Microware's OS-9, which I once heard described as neither monolithic nor micro but modular. But I don't know more than that, and even that was a distant memory.

Reply Score: 2

RE[3]: kernel modules
by Neolander on Sun 20th Feb 2011 21:57 UTC in reply to "RE[2]: kernel modules"
Neolander Member since:
2010-03-08

Indeed, more modularity can be used as a way to reduce the disadvantages of the monolithic approach on "large" kernels, without going as far as the microkernel approach. It's an interesting in-between solution.

On the other hand, there must be a policy somewhere that forces kernel devs to put new features in separate modules whenever possible. Otherwise, you get something like Linux : the kernel is modular, but outside of the realm of hardware drivers its modularity capabilities are heavily under-used (and since there's no standard, stable module interface, third-party module development does not work well... But that's another story)

Reply Score: 1

Thanks!
by ebasconp on Mon 21st Feb 2011 03:12 UTC
ebasconp
Member since:
2006-05-09

Beautiful article!!!

Some years ago I was fascinated reading about L4 and its several implementations. What I find great on microkernels is the way of implementing everything as servers (isolated processes) or implementing several "personalities" running on top of the microkernel; so you could, in theory, having your microkernel with several "virtual machines" running on top of it. Could you think on Xen or VMware ESX Server as having some microkernel-like design?

Reply Score: 2

Hybrid windows
by Morph on Mon 21st Feb 2011 07:21 UTC
Morph
Member since:
2007-08-20

Windows is more hybrid in recent years since the introduction of User-mode Driver Framework: http://en.wikipedia.org/wiki/User-Mode_Driver_Framework
IIRC, Aero graphics drivers run in usermode.

Another VM-based OS that's had some coverage on OSNews is Phantom OS: http://dz.ru/en/solutions/phantom/
Their goal is full persistence of all processes, data etc across shutdowns. Last blog post was October 2010 so perhaps it's not dead yet.

Reply Score: 1

RE: Hybrid windows
by Neolander on Mon 21st Feb 2011 08:07 UTC in reply to "Hybrid windows"
Neolander Member since:
2010-03-08

Windows is more hybrid in recent years since the introduction of User-mode Driver Framework: http://en.wikipedia.org/wiki/User-Mode_Driver_Framework
IIRC, Aero graphics drivers run in usermode.

Indeed. That thing crashes so often on my machine that if it was in kernel mode, I'd spend more time rebooting than doing something useful when I'm on Windows, like in the Win9x days...

Don't know which part exactly of Windows' graphics stack is in user mode, though. I find it hard to believe that they could just fully move graphics driver in user space. That would break driver compatibility, which is definitely not Microsoft's thing.

Another VM-based OS that's had some coverage on OSNews is Phantom OS: http://dz.ru/en/solutions/phantom/
Their goal is full persistence of all processes, data etc across shutdowns. Last blog post was October 2010 so perhaps it's not dead yet.

Added, along with some links.

Edited 2011-02-21 08:28 UTC

Reply Score: 1

RE[2]: Hybrid windows
by kaiwai on Mon 21st Feb 2011 09:17 UTC in reply to "RE: Hybrid windows"
kaiwai Member since:
2005-07-06

Indeed. That thing crashes so often on my machine that if it was in kernel mode, I'd spend more time rebooting than doing something useful when I'm on Windows, like in the Win9x days...

Don't know which part exactly of Windows' graphics stack is in user mode, though. I find it hard to believe that they could just fully move graphics driver in user space. That would break driver compatibility, which is definitely not Microsoft's thing.


Microsoft did break compatibility with Windows Vista by pushing more responsibility off onto the hardware vendors hence WDDM (Windows Display Driver Model) required virtual re-write of drivers hence the reason why Windows Vista launch was so problematic. Windows 7 has WDDM 1.1 which has bought back hardware acceleration for some GDI functions given in 1.0 GDI was totally software driven.

As for Windows crashes; when you have multiple points of failure; crappy hardware, crappy drivers and a difficult to understand driver API then things will never work out as planned. With that being said, however, if your hardware vendor produces quality hardware and uses the latest Windows Driver Kit and takes advantage of the 'tried and tested' templates that exist then many of the issues shouldn't arise (at least in theory).

Reply Score: 2

RE[3]: Hybrid windows
by Neolander on Mon 21st Feb 2011 11:38 UTC in reply to "RE[2]: Hybrid windows"
Neolander Member since:
2010-03-08

Well, although there are some random Aero crashes, Aero mainly crashes when running old games and other things running in full screen which do not use DirectX (or at least not the latest releases), so I bet I'm just not in the official test cases ;)

Also, I have an exotic GPU setup (Optimus), which further increases the likelihood of something breaking.

I've reported that on Microsoft's bug reporting tool, but they were not very helpful in terms of telling me who else I should contact if it's not their fault.

Anyway, I've switched back to using Linux as my main OS some months ago, so I don't care so much about that anymore. And as said before, I must admit that it breaks quite nicely, silently falling back to CPU rendering without a glitch.

Edited 2011-02-21 11:40 UTC

Reply Score: 1

Glass houses
by AFreeQuark on Mon 21st Feb 2011 18:03 UTC
AFreeQuark
Member since:
2011-02-21

The people at wiki.osdev.org/Main_Page sure are elitist considering how much of their information assumes the silly behavior of x86.

I think being an obnoxious snob carries certain responsibilities, such as knowing how large a world one lives in.

Reply Score: 1

RE: Glass houses
by Kochise on Mon 21st Feb 2011 19:57 UTC in reply to "Glass houses"
Kochise Member since:
2006-03-03

Could you... elaborate ?

Kochise

Reply Score: 2

RE[2]: Glass houses
by AFreeQuark on Tue 22nd Feb 2011 02:00 UTC in reply to "RE: Glass houses"
AFreeQuark Member since:
2011-02-21

I'd rather take back my comment, actually, as it didn't really add much to the conversation. I had poked around the FAQ and forum a bit and was dismayed by how discouraging people were sometimes being to newbies.

...but if I had to point something out, there were a lot of articles on the initial boot-up of systems which talked as if the details were universal, when they were really all x86-specific.

Again, though, I wish I could retract my previous comment. The site *does* have a lot of no-doubt useful information.

:)

Reply Score: 1

RE: Glass houses
by Brendan on Mon 21st Feb 2011 23:45 UTC in reply to "Glass houses"
Brendan Member since:
2005-11-16

Hi,

The people at wiki.osdev.org/Main_Page sure are elitist considering how much of their information assumes the silly behavior of x86.


This isn't intentional. Anyone who has information on other platforms is encouraged to add to wiki.osdev.org.

The reality is, most hobbyist OS developers (just like normal users) have easy access to one or more "PC compatible" computers (required for testing, etc), and don't have easy access to anything else.

Note 1: "testing" means being able to find bugs that exist but only show symptoms in some situations, by being able to test on a (hopefully large) number of different systems.

Note 2: "easy access" means you (and other people that volunteer to test your OS) can mess with system software without special equipment (JTAG cables, flash/EPROM burners, etc) and without worrying about bricking the system.


Most other platforms are either rare (e.g. SPARC), obsolete (e.g. Alpha), or too expensive (e.g. Itanium). This even includes other platforms that use x86 CPUs (UEFI based systems), which are still far less widespread than "PC compatibles".

Then there's embedded CPUs (MIPS, ARM) which can't really be considered a platform because there's no standardisation for much more than the CPU's instruction set. For these, you can write an OS for one system and it won't really work on other systems that have the same CPU type. For example, you can write an OS for one of the ARM development boards, but you'd need to port the OS to any other ARM systems you want to support, which makes it hard to test on a wide variety of systems (especially during the early stages of development).

Anway, the end result of "all the above" is that most hobbyist OS developers want information for "PC compatible" computers, and eventually become people who are able to add information about "PC compatible" computers to wiki.osdev.org.

I'd also point out that some of the information on wiki.osdev.org applies to any platform. This includes the "theory" sections, some hardware information (PCI and USB), etc.

- Brendan

Reply Score: 3

RE[2]: Glass houses
by AFreeQuark on Tue 22nd Feb 2011 02:02 UTC in reply to "RE: Glass houses"
AFreeQuark Member since:
2011-02-21

Good points, all.

Hopefully I will have things to add to the wiki in the near future. ;)

Reply Score: 1

Comment by t3RRa
by t3RRa on Tue 22nd Feb 2011 08:30 UTC
t3RRa
Member since:
2005-11-22

AFAIK because NT kernel does indeed include some features in kernel space where as pure microkernels separate all of those features into user space, but it still loads other features as services into user space, Therefore NT kernel is called a hybrid kernel. it is not actually monolithic and different from Linux's module architecture. (BeOS is also similarly a adopted a hybrid kernel architecture.) So it is not true to call it monolithic kernel architecture because everything is separated.

Reply Score: 2

RE: Comment by t3RRa
by t3RRa on Tue 22nd Feb 2011 22:50 UTC in reply to "Comment by t3RRa"
t3RRa Member since:
2005-11-22

So it is not true to call it monolithic kernel architecture because everything is separated.

I meant "everything is not separated". common mistake while changing and moving around..

Reply Score: 2

VM Based Kernel
by Alfman on Wed 23rd Feb 2011 08:12 UTC
Alfman
Member since:
2011-01-28

I am absolutely delighted that my suggestion to use a type safe language to provide isolation got attention in the "VM Based Kernels" section.


"The kernel must include a full featured interpreter for the relevant language, which means that the codebase will be huge, hard to design, and that very nasty VM bugs are to be expected during implementation."

Yes, a type safe VM language will require more work than simply using a traditional compiler.

As for devel effort, with any luck we wouldn't have to completely re-invent the wheel and could use existing VM implementations like Java or Mono. Writing a VM from scratch would be a ton of work - even if it has merit.

I think the term "interpreter" mis-characterizes the approach. A type safe language capable of isolation is not dependant on being interpreted, it can use JIT and pre-compilation too.


"Making a VM fast enough that it is suitable for running kernel-level code full of hardware access and pointer manipulation is another big programming challenge."

As discussed in the earlier comments, I don't think the type safe language would imply any overhead overhead over a correctly implemented unsafe-language version. All we require are safe language constructs which map efficiently over top of the underlying hardware like (memory mapped devices, port IO, DMA).


"A VM running on top of bare hardware will be harder to write, and thus more buggy and exploitable, than a VM running in the user space of an operating system."

This depends on how deeply integrated Java/Mono are with external dependencies (such as pthreads, or libc, or syscalls). Since I don't know the answer, I'll let the criticism stand.


"Currently, the Java Virtual Machine is one of the biggest sources of vulnerabilities in existence on desktop PCs."

Citation?

Also, remember that, unlike a web browser sandbox, the kernel/VM isn't required to protect from maliciously altered kernel modules (untrusted code). It only needs to ensure that modules written in type safe code remain isolated.

As long as the compiler only produces in-spec modules, then we can reasonably get away with a VM which produces undefined results for out-of-spec modules.


Features I thought about for my OS many years ago:

It would be very nice to have support for clusters within the OS, such that you could take any running application and migrate it to another node while continuing to run. Similar to what VMware/KVM/Xen do, but would work with arbitrary applications.


Every single kernel interface should have the ability to be virtualized such that all interfaces on PC-A could be transparently redirected/aliased on PC-B without explicit support for this within the drivers.

PC-A: soundblaster, webcam
PC-B: remote alias to peripheral devices from PC-A

This would go hand in hand to make the application migration feature seamless.

This also means that all local applications could bind to remote devices transparently without being explicitly written to do so.


Other thoughts:

I hope you're not planning on copying the *nix signal model, it's pretty bad for modern requirements.

I also hope you opt for an asynchronous IO design within the kernel over a threaded IO design. This has been one of the weaknesses plaguing linux for years.

While Posix plays an important role in compatibility and standardization, it's a major impediment to revolutionary designs. Therefore I think strict Posix compliance should be an explicit non-goal, particularly with regards to fs permissions and some of the braindead APIs.


Now if only someone would employ me to work on these things... Is anyone else here woefully under employed?

Reply Score: 1

RE: VM Based Kernel
by alexisread on Thu 24th Feb 2011 11:01 UTC in reply to "VM Based Kernel"
alexisread Member since:
2011-02-24

I don't think the type safe language would imply any overhead overhead over a correctly implemented unsafe-language version. All we require are safe language constructs which map efficiently over top of the underlying hardware like (memory mapped devices, port IO, DMA).


JNode and SqeakNOS are practical examples of VM based kernels - esp. check out Squeak and it's associated OS branch squeakNOS. It looks like development has stalled again, so it's missing COG integration (JIT compiler) and the latest I/O libraries, and the optimisation is pretty non-existant at this stage. It does run reasonably quickly however.

As per the argument put forward by the Phantom OS team, you should be able to (long term) optimise this sort of OS better than a monolithic kernel thanks to zero context switching.

"A VM running on top of bare hardware will be harder to write, and thus more buggy and exploitable, than a VM running in the user space of an operating system." This depends on how deeply integrated Java/Mono are with external dependencies (such as pthreads, or libc, or syscalls). Since I don't know the answer, I'll let the criticism stand. "Currently, the Java Virtual Machine is one of the biggest sources of vulnerabilities in existence on desktop PCs." Citation? Also, remember that, unlike a web browser sandbox, the kernel/VM isn't required to protect from maliciously altered kernel modules (untrusted code). It only needs to ensure that modules written in type safe code remain isolated. As long as the compiler only produces in-spec modules, then we can reasonably get away with a VM which produces undefined results for out-of-spec modules.


In addition, the VM would take away a whole class of bugs in user-level code eg. buffer overrun bugs in flash plugins. OS-wide I think you'd have a net gain and any VM bugs would show themselves quickly (and get fixed for an open source project anyway).

Features I thought about for my OS many years ago: It would be very nice to have support for clusters within the OS, such that you could take any running application and migrate it to another node while continuing to run. Similar to what VMware/KVM/Xen do, but would work with arbitrary applications. Every single kernel interface should have the ability to be virtualized such that all interfaces on PC-A could be transparently redirected/aliased on PC-B without explicit support for this within the drivers. PC-A: soundblaster, webcam PC-B: remote alias to peripheral devices from PC-A This would go hand in hand to make the application migration feature seamless. This also means that all local applications could bind to remote devices transparently without being explicitly written to do so. Other thoughts: I hope you're not planning on copying the *nix signal model, it's pretty bad for modern requirements. I also hope you opt for an asynchronous IO design within the kernel over a threaded IO design. This has been one of the weaknesses plaguing linux for years.


Checkout the Spoon image - it can do this and much more (versioning is handled well), major update due in March!

Reply Score: 1

SymbianOS missing
by maze on Wed 23rd Feb 2011 18:38 UTC
maze
Member since:
2011-02-23

I think you forgot the SymbianOS. It might be worth to mention it since it has some interesting concepts, even though it is a bit unclear for how long it will exist (and for how long you can look at the source code to learn from it)

Reply Score: 1