OSNews: http://www.osnews.com/story/24405/Hobby_OS-deving_3_Designing_a_Kernel Exploring the Future of Computing en-us Copyright 2001-2013, David Adams adam+nospam@osnews.com Wed, 22 May 2013 22:14:06 GMT http://www.osnews.com/images/osnews.gif OSNews.com http://www.osnews.com Phew, finally ! http://www.osnews.com/thread?463345 http://www.osnews.com/thread?463345 And now, let's go back to my own OS and those damn accomodation issues in Uppsala ^^ I didn't expect this article to take so long, but you should probably not expect the next one to come next week either ;) Edited 2011-02-20 14:28 UTC Sun, 20 Feb 2011 14:26:00 GMT donotreply@osnews.com (Neolander) Comments RE: Phew, finally ! http://www.osnews.com/thread?463346 http://www.osnews.com/thread?463346 Thanks for yet another great article, I'm sure there's plenty of people here who have found these a great read ;) Sun, 20 Feb 2011 14:29:00 GMT donotreply@osnews.com (WereCatf) Comments Comment by cb88 http://www.osnews.com/thread?463361 http://www.osnews.com/thread?463361 While you might be able to argue that a VM kernel could be more reliable with GC and array bounds etc.... arguing that it is more secure is a losing battle. VMs can and have been broken out of in the past.... I would actually think they would increase the amount of code were an exploit could occur.Edited 2011-02-20 15:11 UTC Sun, 20 Feb 2011 15:10:00 GMT donotreply@osnews.com (cb88) Comments jnode http://www.osnews.com/thread?463377 http://www.osnews.com/thread?463377 Thanks for this article, but it appears that the JNode system got missed from the sample OS'es based on VM approach. It even has a GUI so it might be interesting thing to watch, if anyone's interested. Sun, 20 Feb 2011 17:56:00 GMT donotreply@osnews.com (antonone) Comments RE: Comment by cb88 http://www.osnews.com/thread?463380 http://www.osnews.com/thread?463380 Well, I thought that this article displayed too much my preference towards the microkernel approach ;) Thanks for showing me that I did a good job balancing it after all. Sun, 20 Feb 2011 18:24:00 GMT donotreply@osnews.com (Neolander) Comments RE: jnode http://www.osnews.com/thread?463381 http://www.osnews.com/thread?463381 Indeed. I'll correct this along with some English mistakes once I get access to some real internet connection (EDGE with locked ports is the new 56K).

When looking for examples, I've been highly displeased to discover that most operating systems which were said to be VM-based (like Inferno or JavaOS) were in fact nothing but a java virtual machine running on top of a regular C/C++ kernel, and have given up a bit too quickly apparently.

SharpOS was also an interesting project, but apparently development has ended.Edited 2011-02-20 18:45 UTC Sun, 20 Feb 2011 18:36:00 GMT donotreply@osnews.com (Neolander) Comments kernel modules http://www.osnews.com/thread?463392 http://www.osnews.com/thread?463392 How do loadable kernel modules enter into the micro v. monolithic kernel debate? Is it possible that a microkernel could minimize the penalty of IPC by adding these services as modules? i.e. does "micro" mean merely that the kernel remains small during runtime, or that the kernel's core codebase is small, but by loading modules it could grow? Sun, 20 Feb 2011 20:31:00 GMT donotreply@osnews.com (jack_perry) Comments RE: kernel modules http://www.osnews.com/thread?463395 http://www.osnews.com/thread?463395 Afaik, microkernels are defined by the way they put some of the services they offer in isolated processes. Microkernels are intrinsically modular, but monolithic kernels *can* also be modular without becoming microkernels.

Whether monolithic kernels are modular or not, most of their main advantages and drawbacks remain, since most code runs in the same address space, in kernel mode, with full access to hardware. Modularity acts independently from that.Edited 2011-02-20 20:54 UTC Sun, 20 Feb 2011 20:47:00 GMT donotreply@osnews.com (Neolander) Comments RE[2]: kernel modules http://www.osnews.com/thread?463407 http://www.osnews.com/thread?463407 Okay. So you could have a monolithic kernel that is actually quite small, and loads modules dynamically into memory as needed. The drawbacks of security would remain, and might even be worsened if the administrator was dumb enough to load an unsecure module to extend the OS' capabilities.

The reason I ask is that modularity could decrease code size significantly. You mention that it is common for desktop monolithic kernels to include facilities for rendering GPU-accelerated graphics and managing every single filesystem in existence, which, okay, could be bad if all these filesystems are in memory all the time, but isn't so bad if they are loaded only when needed. It likewise could provide a defense against what you cite as unmaintainable, non-portable, non modular code.

It makes me wonder whether most monolithic kernels do this in practice.

I guess I should say that this line of questioning is inspired by my acquaintance of Microware's OS-9, which I once heard described as neither monolithic nor micro but modular. But I don't know more than that, and even that was a distant memory. Sun, 20 Feb 2011 21:45:00 GMT donotreply@osnews.com (jack_perry) Comments RE[3]: kernel modules http://www.osnews.com/thread?463410 http://www.osnews.com/thread?463410 Indeed, more modularity can be used as a way to reduce the disadvantages of the monolithic approach on "large" kernels, without going as far as the microkernel approach. It's an interesting in-between solution.

On the other hand, there must be a policy somewhere that forces kernel devs to put new features in separate modules whenever possible. Otherwise, you get something like Linux : the kernel is modular, but outside of the realm of hardware drivers its modularity capabilities are heavily under-used (and since there's no standard, stable module interface, third-party module development does not work well... But that's another story) Sun, 20 Feb 2011 21:57:00 GMT donotreply@osnews.com (Neolander) Comments RE[2]: jnode http://www.osnews.com/thread?463417 http://www.osnews.com/thread?463417 That is because in the area VM operating systems the author is either not old enough, or did not research enough the subject.

Pascal MicroEngine in 1976, which used to process P-Code as instruction set.

The Lisp machines the early 80s.

The original Smalltalk environment at Xerox in

Forth is a VM, compiler and operating system, all in one, in the early 70s.

Modula-2 based system for the Lilith architecture.

Some versions of Oberon operating system, use the modules in bytecode form and compile them on load.

Granted this systems still do use some assembly at the core of their implementations, but so do C based OSs. Sun, 20 Feb 2011 22:33:00 GMT donotreply@osnews.com (moondevil) Comments RE[2]: Comment by cb88 http://www.osnews.com/thread?463433 http://www.osnews.com/thread?463433 While any vailidation of your efforts on my part was wholly unintentional... you're welcome just the same :-)

I wish there were more information on exokernels myself.... the idea seems to have stalled back in 2000. XOK doesn't even compile with a modern compiler. Myself I am looking at going through the MIT OS developement course on my own... through the xv6 code and development of JOS which is an exokernel http://pdos.csail.mit.edu/6.828

EDIT: I found an old copy of the code apparently a newer copy is here http://pdos.csail.mit.edu/6.828/xv6/xv6-rev5.tar.gz

It doesn't require any hacking up to compile it... just make qemu or make bochsEdited 2011-02-21 01:05 UTC Mon, 21 Feb 2011 00:59:00 GMT donotreply@osnews.com (cb88) Comments Thanks! http://www.osnews.com/thread?463442 http://www.osnews.com/thread?463442 Beautiful article!!!

Some years ago I was fascinated reading about L4 and its several implementations. What I find great on microkernels is the way of implementing everything as servers (isolated processes) or implementing several "personalities" running on top of the microkernel; so you could, in theory, having your microkernel with several "virtual machines" running on top of it. Could you think on Xen or VMware ESX Server as having some microkernel-like design? Mon, 21 Feb 2011 03:12:00 GMT donotreply@osnews.com (ebasconp) Comments Hybrid windows http://www.osnews.com/thread?463455 http://www.osnews.com/thread?463455 Windows is more hybrid in recent years since the introduction of User-mode Driver Framework: http://en.wikipedia.org/wiki/User-Mode_Driver_Framework
IIRC, Aero graphics drivers run in usermode.

Another VM-based OS that's had some coverage on OSNews is Phantom OS: http://dz.ru/en/solutions/phantom/
Their goal is full persistence of all processes, data etc across shutdowns. Last blog post was October 2010 so perhaps it's not dead yet. Mon, 21 Feb 2011 07:21:00 GMT donotreply@osnews.com (Morph) Comments RE[3]: jnode http://www.osnews.com/thread?463456 http://www.osnews.com/thread?463456 What was the point of coding a kernel in an interpreted language without even having the benefits of modern interpreted languages like type safety and pointer/array bounds checking, just for the sake of having the code being interpreted ? Apart for proving "yeah, it's possible", sounds like a waste to me... Mon, 21 Feb 2011 07:32:00 GMT donotreply@osnews.com (Neolander) Comments RE[4]: jnode http://www.osnews.com/thread?463461 http://www.osnews.com/thread?463461 Funny that all the languages I mention have those benefits.

And most of the examples given by me have JIT implementations.

Do you know that Sun's Java Hotspot was actually developed for Smaltalk (Self)?

Age has some benefits... ;) Mon, 21 Feb 2011 07:55:00 GMT donotreply@osnews.com (moondevil) Comments RE[5]: jnode http://www.osnews.com/thread?463462 http://www.osnews.com/thread?463462 So, you are telling me, without a smile, that Forth manages to be type-safe without even having a type system ? ;)

Age has some benefits... ;)


Indeed, especially when trying to understand computer history... These benefits go in both directions of the "age" axis though. I've had the chance to know the time where not every game was an FPS and to learn to program with Delphi without having COBOL and FORTRAN giving me nightmares during my childhood, so I'd say I came in this world just at the right time as far as PC evolution is concerned ^^Edited 2011-02-21 08:22 UTC Mon, 21 Feb 2011 08:02:00 GMT donotreply@osnews.com (Neolander) Comments RE: Hybrid windows http://www.osnews.com/thread?463463 http://www.osnews.com/thread?463463
Windows is more hybrid in recent years since the introduction of User-mode Driver Framework: http://en.wikipedia.org/wiki/User-Mode_Driver_Framework
IIRC, Aero graphics drivers run in usermode.

Indeed. That thing crashes so often on my machine that if it was in kernel mode, I'd spend more time rebooting than doing something useful when I'm on Windows, like in the Win9x days...

Don't know which part exactly of Windows' graphics stack is in user mode, though. I find it hard to believe that they could just fully move graphics driver in user space. That would break driver compatibility, which is definitely not Microsoft's thing.

Another VM-based OS that's had some coverage on OSNews is Phantom OS: http://dz.ru/en/solutions/phantom/
Their goal is full persistence of all processes, data etc across shutdowns. Last blog post was October 2010 so perhaps it's not dead yet.

Added, along with some links.Edited 2011-02-21 08:28 UTC Mon, 21 Feb 2011 08:07:00 GMT donotreply@osnews.com (Neolander) Comments RE[2]: Hybrid windows http://www.osnews.com/thread?463468 http://www.osnews.com/thread?463468
Indeed. That thing crashes so often on my machine that if it was in kernel mode, I'd spend more time rebooting than doing something useful when I'm on Windows, like in the Win9x days...

Don't know which part exactly of Windows' graphics stack is in user mode, though. I find it hard to believe that they could just fully move graphics driver in user space. That would break driver compatibility, which is definitely not Microsoft's thing.


Microsoft did break compatibility with Windows Vista by pushing more responsibility off onto the hardware vendors hence WDDM (Windows Display Driver Model) required virtual re-write of drivers hence the reason why Windows Vista launch was so problematic. Windows 7 has WDDM 1.1 which has bought back hardware acceleration for some GDI functions given in 1.0 GDI was totally software driven.

As for Windows crashes; when you have multiple points of failure; crappy hardware, crappy drivers and a difficult to understand driver API then things will never work out as planned. With that being said, however, if your hardware vendor produces quality hardware and uses the latest Windows Driver Kit and takes advantage of the 'tried and tested' templates that exist then many of the issues shouldn't arise (at least in theory). Mon, 21 Feb 2011 09:17:00 GMT donotreply@osnews.com (kaiwai) Comments RE[3]: Hybrid windows http://www.osnews.com/thread?463481 http://www.osnews.com/thread?463481 Well, although there are some random Aero crashes, Aero mainly crashes when running old games and other things running in full screen which do not use DirectX (or at least not the latest releases), so I bet I'm just not in the official test cases ;)

Also, I have an exotic GPU setup (Optimus), which further increases the likelihood of something breaking.

I've reported that on Microsoft's bug reporting tool, but they were not very helpful in terms of telling me who else I should contact if it's not their fault.

Anyway, I've switched back to using Linux as my main OS some months ago, so I don't care so much about that anymore. And as said before, I must admit that it breaks quite nicely, silently falling back to CPU rendering without a glitch.Edited 2011-02-21 11:40 UTC Mon, 21 Feb 2011 11:38:00 GMT donotreply@osnews.com (Neolander) Comments RE[6]: jnode http://www.osnews.com/thread?463484 http://www.osnews.com/thread?463484 Ok, in what concerns Forth you are right. Mon, 21 Feb 2011 11:53:00 GMT donotreply@osnews.com (moondevil) Comments Glass houses http://www.osnews.com/thread?463512 http://www.osnews.com/thread?463512 The people at wiki.osdev.org/Main_Page sure are elitist considering how much of their information assumes the silly behavior of x86.

I think being an obnoxious snob carries certain responsibilities, such as knowing how large a world one lives in. Mon, 21 Feb 2011 18:03:00 GMT donotreply@osnews.com (AFreeQuark) Comments RE: Glass houses http://www.osnews.com/thread?463513 http://www.osnews.com/thread?463513 Could you... elaborate ?

Kochise Mon, 21 Feb 2011 19:57:00 GMT donotreply@osnews.com (Kochise) Comments RE: Glass houses http://www.osnews.com/thread?463514 http://www.osnews.com/thread?463514 Hi,

The people at wiki.osdev.org/Main_Page sure are elitist considering how much of their information assumes the silly behavior of x86.


This isn't intentional. Anyone who has information on other platforms is encouraged to add to wiki.osdev.org.

The reality is, most hobbyist OS developers (just like normal users) have easy access to one or more "PC compatible" computers (required for testing, etc), and don't have easy access to anything else.

Note 1: "testing" means being able to find bugs that exist but only show symptoms in some situations, by being able to test on a (hopefully large) number of different systems.

Note 2: "easy access" means you (and other people that volunteer to test your OS) can mess with system software without special equipment (JTAG cables, flash/EPROM burners, etc) and without worrying about bricking the system.

Most other platforms are either rare (e.g. SPARC), obsolete (e.g. Alpha), or too expensive (e.g. Itanium). This even includes other platforms that use x86 CPUs (UEFI based systems), which are still far less widespread than "PC compatibles".

Then there's embedded CPUs (MIPS, ARM) which can't really be considered a platform because there's no standardisation for much more than the CPU's instruction set. For these, you can write an OS for one system and it won't really work on other systems that have the same CPU type. For example, you can write an OS for one of the ARM development boards, but you'd need to port the OS to any other ARM systems you want to support, which makes it hard to test on a wide variety of systems (especially during the early stages of development).

Anway, the end result of "all the above" is that most hobbyist OS developers want information for "PC compatible" computers, and eventually become people who are able to add information about "PC compatible" computers to wiki.osdev.org.

I'd also point out that some of the information on wiki.osdev.org applies to any platform. This includes the "theory" sections, some hardware information (PCI and USB), etc.

- Brendan Mon, 21 Feb 2011 23:45:00 GMT donotreply@osnews.com (Brendan) Comments RE[2]: Glass houses http://www.osnews.com/thread?463529 http://www.osnews.com/thread?463529 I'd rather take back my comment, actually, as it didn't really add much to the conversation. I had poked around the FAQ and forum a bit and was dismayed by how discouraging people were sometimes being to newbies.

...but if I had to point something out, there were a lot of articles on the initial boot-up of systems which talked as if the details were universal, when they were really all x86-specific.

Again, though, I wish I could retract my previous comment. The site *does* have a lot of no-doubt useful information.

:) Tue, 22 Feb 2011 02:00:00 GMT donotreply@osnews.com (AFreeQuark) Comments RE[2]: Glass houses http://www.osnews.com/thread?463531 http://www.osnews.com/thread?463531 Good points, all.

Hopefully I will have things to add to the wiki in the near future. ;) Tue, 22 Feb 2011 02:02:00 GMT donotreply@osnews.com (AFreeQuark) Comments Comment by t3RRa http://www.osnews.com/thread?463556 http://www.osnews.com/thread?463556 AFAIK because NT kernel does indeed include some features in kernel space where as pure microkernels separate all of those features into user space, but it still loads other features as services into user space, Therefore NT kernel is called a hybrid kernel. it is not actually monolithic and different from Linux's module architecture. (BeOS is also similarly a adopted a hybrid kernel architecture.) So it is not true to call it monolithic kernel architecture because everything is separated. Tue, 22 Feb 2011 08:30:00 GMT donotreply@osnews.com (t3RRa) Comments RE: Phew, finally ! http://www.osnews.com/thread?463560 http://www.osnews.com/thread?463560 Great choice of University. You won't be disappointed =) If you have any questions about Uppsala or the uni don't hesitate to ask me. Drop a PM. Tue, 22 Feb 2011 08:49:00 GMT donotreply@osnews.com (abstraction) Comments RE[2]: Phew, finally ! http://www.osnews.com/thread?463586 http://www.osnews.com/thread?463586 AFAIK, the PM feature of OSnews is discontinued (look at http://www.osnews.com/messages?op=compose&uid=5 ).

However, as an editor, I've got access to the mail address you used for subscribing, so I can contact you this way if you're okay with that ;) Tue, 22 Feb 2011 15:14:00 GMT donotreply@osnews.com (Neolander) Comments RE[3]: Phew, finally ! http://www.osnews.com/thread?463602 http://www.osnews.com/thread?463602 No problems! Tue, 22 Feb 2011 22:15:00 GMT donotreply@osnews.com (abstraction) Comments RE: Comment by t3RRa http://www.osnews.com/thread?463606 http://www.osnews.com/thread?463606 So it is not true to call it monolithic kernel architecture because everything is separated.

I meant "everything is not separated". common mistake while changing and moving around.. Tue, 22 Feb 2011 22:50:00 GMT donotreply@osnews.com (t3RRa) Comments VM Based Kernel http://www.osnews.com/thread?463660 http://www.osnews.com/thread?463660 I am absolutely delighted that my suggestion to use a type safe language to provide isolation got attention in the "VM Based Kernels" section.


"The kernel must include a full featured interpreter for the relevant language, which means that the codebase will be huge, hard to design, and that very nasty VM bugs are to be expected during implementation."

Yes, a type safe VM language will require more work than simply using a traditional compiler.

As for devel effort, with any luck we wouldn't have to completely re-invent the wheel and could use existing VM implementations like Java or Mono. Writing a VM from scratch would be a ton of work - even if it has merit.

I think the term "interpreter" mis-characterizes the approach. A type safe language capable of isolation is not dependant on being interpreted, it can use JIT and pre-compilation too.


"Making a VM fast enough that it is suitable for running kernel-level code full of hardware access and pointer manipulation is another big programming challenge."

As discussed in the earlier comments, I don't think the type safe language would imply any overhead overhead over a correctly implemented unsafe-language version. All we require are safe language constructs which map efficiently over top of the underlying hardware like (memory mapped devices, port IO, DMA).


"A VM running on top of bare hardware will be harder to write, and thus more buggy and exploitable, than a VM running in the user space of an operating system."

This depends on how deeply integrated Java/Mono are with external dependencies (such as pthreads, or libc, or syscalls). Since I don't know the answer, I'll let the criticism stand.


"Currently, the Java Virtual Machine is one of the biggest sources of vulnerabilities in existence on desktop PCs."

Citation?

Also, remember that, unlike a web browser sandbox, the kernel/VM isn't required to protect from maliciously altered kernel modules (untrusted code). It only needs to ensure that modules written in type safe code remain isolated.

As long as the compiler only produces in-spec modules, then we can reasonably get away with a VM which produces undefined results for out-of-spec modules.


Features I thought about for my OS many years ago:

It would be very nice to have support for clusters within the OS, such that you could take any running application and migrate it to another node while continuing to run. Similar to what VMware/KVM/Xen do, but would work with arbitrary applications.


Every single kernel interface should have the ability to be virtualized such that all interfaces on PC-A could be transparently redirected/aliased on PC-B without explicit support for this within the drivers.

PC-A: soundblaster, webcam
PC-B: remote alias to peripheral devices from PC-A

This would go hand in hand to make the application migration feature seamless.

This also means that all local applications could bind to remote devices transparently without being explicitly written to do so.


Other thoughts:

I hope you're not planning on copying the *nix signal model, it's pretty bad for modern requirements.

I also hope you opt for an asynchronous IO design within the kernel over a threaded IO design. This has been one of the weaknesses plaguing linux for years.

While Posix plays an important role in compatibility and standardization, it's a major impediment to revolutionary designs. Therefore I think strict Posix compliance should be an explicit non-goal, particularly with regards to fs permissions and some of the braindead APIs.


Now if only someone would employ me to work on these things... Is anyone else here woefully under employed? Wed, 23 Feb 2011 08:12:00 GMT donotreply@osnews.com (Alfman) Comments SymbianOS missing http://www.osnews.com/thread?463731 http://www.osnews.com/thread?463731 I think you forgot the SymbianOS. It might be worth to mention it since it has some interesting concepts, even though it is a bit unclear for how long it will exist (and for how long you can look at the source code to learn from it) Wed, 23 Feb 2011 18:38:00 GMT donotreply@osnews.com (maze) Comments RE: VM Based Kernel http://www.osnews.com/thread?463811 http://www.osnews.com/thread?463811
I don't think the type safe language would imply any overhead overhead over a correctly implemented unsafe-language version. All we require are safe language constructs which map efficiently over top of the underlying hardware like (memory mapped devices, port IO, DMA).


JNode and SqeakNOS are practical examples of VM based kernels - esp. check out Squeak and it's associated OS branch squeakNOS. It looks like development has stalled again, so it's missing COG integration (JIT compiler) and the latest I/O libraries, and the optimisation is pretty non-existant at this stage. It does run reasonably quickly however.

As per the argument put forward by the Phantom OS team, you should be able to (long term) optimise this sort of OS better than a monolithic kernel thanks to zero context switching.

"A VM running on top of bare hardware will be harder to write, and thus more buggy and exploitable, than a VM running in the user space of an operating system." This depends on how deeply integrated Java/Mono are with external dependencies (such as pthreads, or libc, or syscalls). Since I don't know the answer, I'll let the criticism stand. "Currently, the Java Virtual Machine is one of the biggest sources of vulnerabilities in existence on desktop PCs." Citation? Also, remember that, unlike a web browser sandbox, the kernel/VM isn't required to protect from maliciously altered kernel modules (untrusted code). It only needs to ensure that modules written in type safe code remain isolated. As long as the compiler only produces in-spec modules, then we can reasonably get away with a VM which produces undefined results for out-of-spec modules.


In addition, the VM would take away a whole class of bugs in user-level code eg. buffer overrun bugs in flash plugins. OS-wide I think you'd have a net gain and any VM bugs would show themselves quickly (and get fixed for an open source project anyway).

Features I thought about for my OS many years ago: It would be very nice to have support for clusters within the OS, such that you could take any running application and migrate it to another node while continuing to run. Similar to what VMware/KVM/Xen do, but would work with arbitrary applications. Every single kernel interface should have the ability to be virtualized such that all interfaces on PC-A could be transparently redirected/aliased on PC-B without explicit support for this within the drivers. PC-A: soundblaster, webcam PC-B: remote alias to peripheral devices from PC-A This would go hand in hand to make the application migration feature seamless. This also means that all local applications could bind to remote devices transparently without being explicitly written to do so. Other thoughts: I hope you're not planning on copying the *nix signal model, it's pretty bad for modern requirements. I also hope you opt for an asynchronous IO design within the kernel over a threaded IO design. This has been one of the weaknesses plaguing linux for years.


Checkout the Spoon image - it can do this and much more (versioning is handled well), major update due in March! Thu, 24 Feb 2011 11:01:00 GMT donotreply@osnews.com (alexisread) Comments