This article offers feature suggestions to budding OS developers looking for that neat edge. This osnews mini-series is a series to give people developing hobby operating systems – for fun – ideas that take their OS from a kernel to a ‘system’.In previous articles we have described using URIs [1] for file paths and the idea that streams of various types can be transparently converted between by the operating system [2]. This article describes how these might be unified into a single, consistent user interface.
UNIX has the design principle that ‘everything is a file’ [3]. A file is, more properly, a stream. I have introduced, in the previous article, the idea that a stream, if we know its type, might be presented transparently to a program as another, compatible type. This means that conceptually a file is in fact many streams, with each stream marked by type [4].
Streams don’t just refer to physical files on your computer or remote computers. Streams can be generated by other things such as applications and device drivers. In UNIX, named pipes and UNIX-domain sockets provide this kind of facility [5].
Many UNIX programs are an excellent example of applications that provide streams programmatically. The ‘sort’ command takes one stream as input and outputs to another, as does the ‘grep’ command. In fact, all command line programs do this. As our hobby operating system is marking streams by type, you could expect all these command line applications to be reading and writing text/plain streams. If you want to ‘grep’ on a rich document, for example, the operating system would be quite capable of converting that to text/plain – if the virtual file system has the right plug-in – and ‘grep’ need not know [6].
We’ll make the jump that your hobby OS includes a GUI of sorts! How might applications provide streams for the user to interact with?
GUI widgets (windows, even individual controls such as text boxes) might report to the system that they provide/consume streams of certain mime-types. This would allow the window manager to provide drag-n-drop stream connections and work as a versatile clip-board. This is not so strange – this is pretty much what the windows clipboard is doing anyway!
So think of ‘open’ and ‘save’ as reading and writing streams, rather than files (a slightly thin distinction to make, I know). Applications conventionally use standard system dialogs for allowing the user to select files to open and names of files to save-as. This allows the hobby operating system author to unify application streams, remote streams and file streams and present them to the user in a sensible manner.
You’ll see from my dialog mockup that I allow the user to select from a recent streams history, from a location (much like a conventional file-open dialog) and by type. This later category is something akin to a search. Popular consensus in the human-computer-interaction camp at the moment seems to think that users don’t really remember where they put files or what they are called. Modern operating systems such as BeOS, Windows and the like are moving in the direction of letting the user search for files instead. This is the whole big idea behind the ‘file system is a database’ hype at the moment!
So the user doesn’t care where the stream comes from, they care only that they find their stream. The ‘by type’ presents all files of the appropriate type, regardless of location. In the example dialog an application-provided stream is shown highlighted, mixed in with streams provided by conventional files from the user’s home directory.
A ‘by type’ query that is constantly updating live is termed an ‘active query’ and is one of the many big features in BeOS. The idea is that, while you have a search window open, files that are added, modified or removed that fit the terms of the query are automatically updated [7].
The Save-As dialog might also present application streams as valid destinations. A user might ‘save’ a bitmap to their presentation software which is running at the moment, and that presentation software might insert that received image file into the actively-being-edited application.
Opening a stream is assisted by search agents. The idea is that various mime-types might register search agents that take words, phrases or even complex query expressions and filter a list based upon user-entered criteria. You might want to only documents that contain a reference to a certain activity, for example. Various search agents might use meta-data, a contextual search or an index to find matches; it depends upon the mime-type and the agent’s needs.
It could be taken further and integrate search agents that represent remote locations e.g. coalescing a Google search into a local files search.
[1] “Network Transparency and the Multi-Root File System“
[2] “Files and Programmatic Streams“
[3] Google results
[4] Some file systems allow a single file to consist of more than one actual physical stream. On the Mac, a file consisted of exactly two streams, one containing the ‘meta data’ (as more recent file systems placed in peer attributes). On Windows NTFS, files can contain many streams (although this is hardly a much-used feature) and so on. When your hobby operating system has to mount such file systems, this leads to stream ambiguity. One good sounding approach is to give each native stream a name (most such file systems expect this) and treat them as individual files with the actual multi-stream node being presented as a directory.
[5] This provides a good overview to UNIX inter-process-communication (IPC), which is about files, pipes and sockets.
[6] As an aside, it probably would make sense for a hobby version of ‘grep’ to work from a dedicated application/x-grep stream that might contain meta information such as logical locations; this would allow a viewer of the actual native stream type to understand the stream positions of the matching terms and highlight it appropriately. Such a system would be very useful for doing regular expression searches across a wide range of actual native stream types that contain text elements.
[7] One approach to adding active queries to a hobby operating system is described here.
If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.
… This can all be done with userland file servers. You really need to stop writing articles and actually see what the operating system world beyond Linux has to offer.
Great article — but best of all, Bach gets a mention! Huzzah for the ultimate master of counterpoint. Mmmm, baroque ‘n roll…
> This can all be done with userland file servers.
What exactly do you mean by “user land file servers”? Im not familiar with the term. Also, why is it a better method?
The article starts with stating that there is a *nix design philosophy that states that “everything is a file”. This is a widely spread idea, but it is also an idea that is not very true. When you think about it, how much of a modern *nix is actually a file? It turns out that there are a lot of fundamental concepts that are not files:
* Processes – A process cannot be manipulated like a file. There are filesystem extentions that allow you to inspect the state of a process like it was a collection of files, but it is not possible to e.g. create a process like you would create a file.
* Network sockets – There is no way to create or manipulate network sockets like they were files. There is an old proposal called “portals” for BSD that actually makes network sockets appear in the file system, but this has not been adopted.
* Network interfaces – It is not possible to configure a network interface by using the file system – we are forced to use ifconfig (or the underlying kernel communication magic).
* Users – A user does not have any representation as a file. Sure, all users are stored in the passwd file, but that does not make a user a file.
* Un-named pipes (the ones that appear if you type the | character between commands) – they may be implemented using files, but that does not make them files
* Signals – You cannot send or receive signals using the file system.
Maybe it is time to finally drop the idea that “everything is a file” and acknowledge the fact that most things in modern *nixes are *not* files.
You should also read this enlightening works:
http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm
http://www.namesys.com/whitepaper.html
Instead of having a naspace identifying collection of bits in the computer memory the namespace should identify the conceptual data, the system should just handle representations of the concept.
I’m pretty sure the BeOS find dialog is not find-as-you-type.
See “Plan 9” for a more extreme implementation of “everything is a file”.
* Un-named pipes
Even if the pipes isn’t represented as files, I do belive that you could find a stdin/stdout/stderr for each process in the pipeline at /proc/[0-9]+/fd/*. Am I wrong here?
in case anyone has a morbid interest in these sorts of things in our beloved linux kernel, you might like to have a quick glance at this very heated debate that’s been raging on the linux kernel mailing list for some time.
google groups says the thread has reached over 480 posts in just a few days!
http://groups.google.co.uk/groups?hl=en&lr=&ie=UTF-8&threadm=fa.g6o…
a quick executive summary is that reiser4 allows some neat things with its idea of plugins (streams and meta-data are the focus of the discussion but the sky is the limit). unfortunately this is not compatible with POSIX, SVR4 or current linux VFS semantics. cue a major philosophical flame-fest.
Maybe it is time to finally drop the idea that “everything is a file” and acknowledge the fact that most things in modern *nixes are *not* files.
Yes. I don’t like the way drivers are done on Unixes– device files, with horrible hacks like ioctl to get around the fact that the file is not a suitable interface for most devices, and now all this UDev and DevFS palaver. Why not just expose a nice set of interface functions for the device class (with the option of hardware-specific functions). Ioctl is just a silly way of doing exactly this, but incurs more overhead and is harder to use.
The file system does provide one useful feature for device drivers, that is a way of accessing and listing by name. But you can do this some other way.
It would appear that the historical significance of “everything is a file” has been lost. When this philosophy was coined everything had a different interface. So when you wanted to write to a device you had to learn a brand new interface. What’s more you had to learn how to refer to that interface. After looking at a lot of these interfaces it was found that most of them have a lot in common. It was decided that there needed to be a standard way of talking to stuff and the “file level” was the lowest common demoninator. When we say “everything is a file” we’re saying “you can use the file interface to talk to everything”. If you want to do something special with the device then you can extend the file interface (with ioctl). Userland libraries can provide more “pretty” interfaces if you really need, but by making everything conform to this standard interface it makes it easier to join programs together. The pipe is simply two such interfaces with a buffer in-between, and by using a lot of programs and devices which all share the same interface we can get a very powerful system from a small number of components. Remember, as system’s power is measured by the number of components raised to the power of the number of possible interconnections between those components.
You are right that everything is not a file, but IMHO an operating system which gives uniform interface to access device is always going to be easier and liked by developers. How exactly do you think people writing code at high level manage difference between those devices?
Today i can use same WriteFile functions on a network device as well as a file system and it is much better than having 100s of different functions specific to each device and making it unusable when the device change.
May be what you want to propose is a more powerful interface than file but not non-uniform interface based on devices.
The purpose of having everything, from a user process standpoint, appear as a file descriptor, is that it gives you a simple and powerful way of cleaning up when the process exits.
ioctl() is indeed not a very elegant way of supporting interfaces, even though you could publish interfaces in the C++/component sense by marshalling/demarshalling and using
the file descriptor as a ‘wire’.
However, it simplifies the OS and the applications a lot that everything supports open() and close().
See what happens when a process crashes in Syllable and it uses the “non file compliant” message ports and windows for example: they don’t go away.
> However, it simplifies the OS and the applications
> a lot that everything supports open() and close().
Last time I looked into /dev and /proc, I wasn’t terribly impressed by the “simplicity” of it, and I can’t say I remember ever open()ing or close()ing anything in there.
I think while the “everything is a file” metaphor makes things easier for (a certain brand of) developers, it’s more than a bit confusing for the rest, including end users.
Unfortunately, *nix isn’t exactly about end users, anyway.
Well, in Linux (and I guess that in all other unixes) when I want to see if my usb mouse has been recognised I do “cat /dev/input/mice”, and I bet that many, many other people do similar things.
When you open a terminal, the terminal open(s) /dev/console, /dev/ttyS0, etc. and this is passed over to child processes.
/dev is a way for user processes to open() stuff that isn’t a physical file. In itself it’s not a bad idea; what is a bad idea is that it’s exposed to the end user (you and me); whether it’s physical inodes or a filesystem like devfs doesn’t really make any difference.
The Windows NT kernel works the same way (I think the hierarchy is called \device.. NT gurus please correct me 🙂 but you don’t see it exposed directly to apps.
So, again, there is nothing wrong (and a lot of good) with wrapping the ugly semantics of the “everything looks like a file” using nice C++ or component-based interfaces.
By simplifying life for applications, I mean that you don’t have to explicitly free and close everything when your application exits, and the system (or yourself) can safely slay a process without leaving things hanging. If you ever programmed for AmigaOS, you know what I mean.
That was my point 🙂
One interface to rule them all, one place to find them, one call to open them all and in the darkness bind them.
If it scares you when you ls /dev then don’t.
Or find another distro that doesn’t make a scarey /dev.
I think everything should be a file. But I can deal with the fact that some things, like users, won’t become files in the near future. Though I might treat them like files if all they do is complain.