General Development Archive

Operating system: from 0 to 1

Probably you asked yourself at least once, how an Operating System (OS) was written from the ground up? You probably have spent years programming, but still understand operating system as a collection of abstract concepts, not how to implement an operating system in actual code. In your mind, somehow the operating system can magically control the underlying hardware and do what you want through the higher level API of your favorite programming language. You wish to understand the details, but for some reason, it seems too difficult because regardless how much you learn, it is never enough. You may feel that you are missing an important piece of the puzzle, and get stuck. However, deep inside you still want to write an operating system without a crystal clear understanding. After all, you are a software engineer, and an operating system is a software. You should know your software better than anyone else!

If that is the case, this book is for you. By going through this book, you will be able to find the missing piece that is essential and enable you to implement your operating system, from scratch!

A free detailed book about writing your first operating system.

POSIX has become outdated

The POSIX standard for APIs was developed over 25 years ago. We explored how applications in Android, OS X, and Ubuntu Linux use these interfaces today and found that weaknesses or deficiencies in POSIX have led to divergence in how modern applications use the POSIX APIs. In this article, we present our analysis of over a million applications and show how developers have created workarounds to shortcut POSIX and implement functionality missing from POSIX.

Wine 2.0 released

This release represents over a year of development effort and around 6,600 individual changes. The main highlights are the support for Microsoft Office 2013, and the 64-bit support on macOS.

It also contains a lot of improvements across the board, as well as support for many new applications and games. See the release notes below for a summary of the major changes.

As awesome of a project Wine is, I wonder how many people actually use it on a daily basis. Maybe I'm wildly off base here (honestly, I probably am), but it seems like if you're running Linux, there's really nothing Windows applications offer that Linux can't.

Making a game in PICO-8

I'm going to use PICO-8, which its creator, Joseph "Zep" White, calls a 'fantasy console', but really it's like an indie-fied emulator of the computers I grew up with, like the BBC B. When you start it, you're presented with a 128 by 128 pixel display glitching into life, this little do-do-do-do! jingle, and a command prompt.

Everything you need to make games is right there: a mini Lua code editor, sprite and map editors, and sound and music editors. It's reactive, instant to test to see if things work, and generally delightful. And the stuff people have made in it is extraordinary. Little short-form games: colourful, fun, immediate, varied. Type SPLORE into the command prompt and this little browser for games posted to the PICO-8 forum comes up. Since no game, including its graphics, is bigger than a 65K text file, you're playing them pretty much instantly. It's lovely.

This is just the first article in a series.

Brainfuck: code that was designed to hurt

Software is the umbrella term for computer programs and libraries, the coded logic that makes our machines tick. At the root of all software is the code, the instructions that enable a human to tell a machine what to do. This code is written in one of the hundreds of different programming languages - such as C, Java, or Python - each of which has its own eccentricities and context-dependent advantages.

Yet regardless of the programming language being used, the functionality, logic, and efficiency of the language are always paramount - unless, of course, you're talking about Brainfuck, an esoteric programming language that champions purposefully overcomplicated code.

I had no idea esoteric programming languages were a thing.

The little book about OS development

This text is a practical guide to writing your own x86 operating system. It is designed to give enough help with the technical details while at the same time not reveal too much with samples and code excerpts. We've tried to collect parts of the vast (and often excellent) expanse of material and tutorials available, on the web and otherwise, and add our own insights into the problems we encountered and struggled with.

CHIP-8 in Common Lisp: the CPU

A while back I decided to try to write a Game Boy emulator in Common Lisp based on this series of articles. I made some good progress but eventually got bogged down because I was trying to learn a bunch of complex new things at once.

Instead of dragging on, I decided to take a break and try something simpler: a CHIP-8 emulator/interpreter. The CHIP-8 is much simpler than the Game Boy, which made it easier to experiment with the rest of the infrastructure.

In this post and a couple of future ones I'll walk through all of my CHIP-8 emulator implementation.

The BASIC issue with retro computers

When you have built your retro computer the chances are you’ll turn it on and be faced with a BASIC interpreter prompt. This was the standard interface for home computers of the 8-bit era, one from which very few products deviated. If you were a teenager plugging your family's first ever computer into the living-room TV then your first port of call after getting bored with the cassette of free educational games that came with it would have been to open the manual and immerse yourself in programming.

The trouble is, in the several decades since, 8-bit BASIC skills have waned a little. Most people under 40 will have rarely if ever encountered it, and the generation who were there on the living room carpet with their Commodore 64s (or whatever) would probably not care to admit that this is the sum total of their remembered BASIC knowledge.

10 PRINT "Hello World"
20 GOTO 10

If you have built a retro-computer then clearly this is a listing whose appeal will quickly wane, so where can you brush up your 8-bit BASIC skills several decades after the demise of 8-bit home computers?

When I was very, very young - I'm from 1984 - I did some very basic BASIC, mostly on an MSX, but I remember very little of it. BASIC programming didn't grab me as a kid, and as such, I never went down the programmer's path. Today, with an adult life with adult responsibilities, learning to program seems like such a daunting undertaking, for which I simply don't have the time.

GPLGPU walkthrough

The goal was to publish source code to a GPU that is register compatible with the late 90's era Number Nine "Ticket To Ride IV" GPU. Although the project didn't meet its funding goal, the person behind it later published the code on github.

Despite the fact that this is an older design, it has lots of stuff that is worth studying. It's interesting to compare this design to the VideoCore GPU that I walked through in a previous post. While there are some fundamental differences, there are surprising number of functions that are similar, which shows how modern GPUs evolved from earlier ones.

A walkthrough of the GPLGPU as well as some history and backstory of the Number Nine "Ticket To Ride IV" GPU.

Preserving Mesa, the Xerox Star system programming language

The goal of this project is to preserve and present primary and secondary source materials (including specifications, source code, manuals, and papers discussing design and implementation) from Mesa, the system programming language designed at Xerox PARC in the 1970s and used to implement the Xerox Star office automation system and its follow-ons. The editor greatly appreciates comments, suggestions, and donations of additional materials.

Wikipedia has a short overview of Mesa, and here's the 1979 Mesa Language Manual, which is obviously a lot more in-depth.

How a robot coded ‘Super Mario Maker’ onto an SNES, live

By now, simply taking over a game and replacing it with a brand new app was beginning to feel a little predictable. So this year, TASBot decided to show off a new skill. At the AGDQ marathon, the bot set out to edit new features onto a game that's still running in active memory. TASBot wanted to be magnanimous with its new capabilities, too, allowing human players (and livestream viewers) the opportunity to edit the game on the fly.

But just how did TASBot - and the team of coders behind it - intend to turn an old game of Super Mario World, running on a standard SNES, into a heavily editable game of Super Mario Maker? Luckily, we had a behind-the-scenes invite to the event and the opportunity to find out.

I spent most of last week watching AGDQ (and donating, of course), and this particular segment blew my mind.

Beating Castlevania: Symphony of the Night with assembly code

So, Castlevania: Symphony of the Night is one of my favourite games of all time, and it's also generally considered to be one of the best games ever made. And, as with all games, people 'speedrun' this game, which means trying to beat the game as fast as is humanly possible. There are several categories, each with certain rules and things that are and are not allowed.

This particular speedrun of Symphony Of The Night by Cosmo takes this concept to a whole new level. The end time of 7 minutes and 52 seconds is mind-blowing enough, but how he actually gets there is just utterly insane. Basically, he procures a very specific set of items in his inventory, and then proceeds to manipulate the items in his inventory in a extremely specific way, within very specific fractions of seconds of game-time, to use the sorting mechanism of the inventory to manipulate the assembly code in memory to make the game finish itself. All this, on the actual console itself, without tools, without additional software, without emulators, without anything.

The actual science or coding behind this technique was discovered and developed by a person named Sockfolder, and he put up a 40-minute stream to explain in detail what's going on, with the contents of memory on the side of the screen so you can see exactly what's happening. It's mesmerising (even though I don't fully understand what's going on).

While the actual coding part of it can be discovered and explored in relative comfort of an emulator and other tools, actually pulling this off 'live', with just the tools at the disposal of any regular player, is absolutely amazing. This kind of stuff sits at the very fringes of programming, and I find it incredibly impressive.

Getting started with Project Islandwood

If you are an iOS developer, the Windows ecosystem can appear a strange and frightening place. Writing an app for Windows requires an investment in all kinds of new things: new tools (Visual Studio), new languages (C#), new APIs and Controls (Win32, XAML), new graphics engines (DirectX) and before you know it, life seems too short and wouldn't another Flappy Birds clone be more fun anyway?

Fear not, brave adventurer, for Project Islandwood is here.

LLVM to get FORTRAN compiler

Today, the US Department of Energy announced that it had established a partnership with NVIDIA that would be enhancing the LLVM compiler collection. The goal will be to port an existing FORTRAN compiler that targets massively parallel GPUs. The results are expected to be released as open source in late 2016.

Cutting-edge research still universally involves Fortran; a trio of challengers wants in. While FORTRAN isn't a mainstream language, it's still heavily used in scientific computing, and there's lots of legacy code that relies on it. A lot of that code is maintained by people at the US National Labs, and the new project is being organized by staff at Lawrence Livermore, Sandia, and Los Alamos.

League of Legends’ chat service architecture

League of Legends players collectively send millions of messages every day. They're asking friends to duo-queue, suggesting a team comp on the champ select screen, and thanking opponents for a good game. On July 21st of this year (I picked a day at random), players forged 1.7 million new friendships in the game - that's a lot of love! And each time players send a message they trigger a number of operations on the back-end technology that powers Riot chat.

In the previous episode of this series on chat, I discussed the protocol we chose to communicate between client and server: XMPP (Extensible Messaging and Presence Protocol). Today I'll dive into the mechanisms in place on the server-side and the architecture of the infrastructure, and I’ll discuss the work we’ve done to ensure that our servers are scalable and robust. Like the last article, I hope it’ll be interesting to anyone building out chat features to a distributed client base.

The limits of language

The best class I took in college was on the philosopher Ludwig Wittgenstein. Until that point, I had avoided philosophy of language as simply being too esoteric and hermetic to be of use. David Pears, a prodigious yet modest and approachable figure visiting from Oxford, changed my mind. In large part because of Pears' instruction, Wittgenstein's philosophy has been directly relevant to my thinking about computer science, artificial intelligence, and cognitive science. When other scholars were thinking that language and thought could be reduced to a universal, logical language, Wittgenstein turned the matter to practical questions and raised incredibly inconvenient questions that gained traction in artificial intelligence in the 1970s, 40 years after he was working on them.

Great article. I found this paragraph especially interesting:

Here's one example. The French equivalents for here and there are ici and là respectively. But if I point to a pen and say, "The pen is here," the French equivalent is not "Le stylo est ici," but "Le stylo est là." In French, là is always used to refer to a specific place or position, while in English here or there can both work. This rule is so obscure I never learned it in French classes, but obviously all native speakers learn it because no one ever uses it differently. It could just as easily be the other way round, but it's not. The situation is not arbitrary, but the way in which language carves up the interaction between mind and world varies in such a way that French speakers recognize certain practices as right or wrong in a different way than English speakers do. This may seem a trivial point, until you have to program a computer to translate "I pointed to Paris on the map and said, 'She is here.' " into French - at which point it becomes a nightmare. (If you are a translator, on the other hand, this is great news.)

Aside from the obvious fact that I can relate to the remark about translators, the author touches upon something that I benefit from every day. I always feel that being multilingual (just Dutch, English, German, some French, and a basic grasp of ancient Greek and Latin - relatively limited when compared to true multilinguals) makes it easier for me to express myself. Being able to use words, concepts, ideas, structures, and conventions from foreign languages and incorporate them into my Dutch - even if only in my inner monologue - allows me to describe objects, concepts, and situations in a more fine-grained, and therefore, more accurate manner (accurate to my perception, which does not mean "more correct" in more absolute terms).

I appreciate how ridiculously pretentious this sounds, but I do firmly believe this is true: being able to understand, read, write, and speak multiple language makes me better at language.

I'm no programmer - something I like to repeat as often as I can to make sure everyone knows where I'm coming from on the subject of programming - but I get the idea that programming is not very different in that regard. That is, being able to program in multiple programming languages will make you better at programming, and not just in the sense that you will be useful in more situations (you can find a job both as a Java and an Objective-C programmer, for instance), but also in the sense that knowledge and experience in programming language Abc will give you new and different insights into programming language Xyz, allowing you to use a certain language in more unconventional ways that people with knowledge of fewer languages might not.

As much as language is an expression of culture, a programming language is an expression of how a computer works. Both contain within them invaluable knowledge that cannot be easily expressed in other languages - and as such, they are invaluable in preserving knowledge, both culturally and digitally.

Go 1.5 released

This release includes significant changes to the implementation. The compiler tool chain was translated from C to Go, removing the last vestiges of C code from the Go code base. The garbage collector was completely redesigned, yielding a dramatic reduction in garbage collection pause times. Related improvements to the scheduler allowed us to change the default GOMAXPROCS value (the number of concurrently executing goroutines) from 1 to the number of logical CPUs. Changes to the linker enable distributing Go packages as shared libraries to link into Go programs, and building Go packages into archives or shared libraries that may be linked into or loaded by C programs (design doc).