File systems need to change. Current file systems are horribly out-of-touch with the realities of what users need to effectively find, organize, and modify their vast quantities of files. Unfortunately, no major consumer OS vendor (Microsoft, Apple, various Linux distos, etc.) has had the foresight, the will, and most of all, the cajones to implement anything more elaborate than a small departure from the standard hierarchical name-space which we all grew up on and should rightfully deplore. Worst of all, the best suggestions for changing the current entrenched standard are incredibly toothless, incredibly feeble.
Not a single proposal I’ve read has ever started by considering the most important motivator of good file system design: how will the user interact with it? The tacit assumption that everyone seems to be making is that by making it easy for programmers to code for, the program can then present whatever “user-friendly” interface it wants to the hapless user. This is a damaging notion, and is born of the misplaced belief that any user-centered file system must necessarily be difficult to interface with for the programmer. False! Programmers must translate user intent into machine code. When the file system is created to support a user’s intent, the programmer’s job becomes trivial. This, then, is our task: design a file system sympathetic to the user’s weaknesses which nonetheless fits all the needs of programmers. For this I propose the Brendan File System, or BFS.
The Lost Art of Collating
The traditional hierarchical structure was introduced to create structure where there was none, in order to aid both the user and the programmer in categorizing information. Unfortunately, it became the only way to organize structure on-disk, and so became much more restrictive than the flat file system’s disorganized chaos. We’ve come to the point in OS design where over-simplification causes nothing but headaches and provides no measurable performance advantage. BFS throws hierarchy out the window. Instead, folders become “groups”, and files can meander about and show up wherever they’re needed, even in multiple places at once. Groups function much like folders, but a file is never tied to a particular group; files have an independent existence and can belong to any number of groups (or none!).
Let’s say a user is starting a new programming project. She first creates the group for the new project, optionally names it (who said a group had to have a name?), and starts adding some of the relevant documentation to the group. Need that primer on C++ templates? Add it to the group, and never mind how many other “copies” you have floating around in other groups of yours. When you’re done, you can just remove it from the group and you needn’t worry about it disappearing or being deleted before you need it again. There is, of course, a separate delete command available if you never want to see the file again. Certainly, all this can be done with folders and symlinks and the like, but it’s just so much cleaner this way.
Finding Simplicity in Complexity
Well, if you don’t have any directory structure, how on earth do you find things? You can’t use file names because there’s no guarantee you’ll remember the name (besides, who said your file had to have a name?), and you can’t use the file’s “location” because it doesn’t really have one, so what to use? The best answer is a good search mechanism. The Be file system basically nailed this with extendable attributes and live queries, and the BFS embraces this system whole-heartedly. Issue a query for all mail messages from Bob Smith and you get a list of all of them, updated in real time as new ones come in. But where the Be file system made these queries persist at the application level, the BFS lets users and programmers create and extend groups by adding a search pattern to their metadata. The user could create a group for storing all her HTML documents which would automatically be updated as new ones are created; the user need never physically add an HTML file to the group. Of course, you can make groups which are mixtures of a live query and user-placed files. In the HTML group, you could also stick your favorite browser for easy previewing. The possibilities are limited only by the complexity of the search and the imagination of the user. This has tremendous potential for the programmer as well, and will be further elaborated on in the section on the System group.
Metadata: Man’s Best Friend
Metadata is worth its weight in gold. There is absolutely nothing as important to a modern file system as metadata. Without it, you simply cannot find, catalog, or classify the vast amount of data present on even the lowliest consumer PC. To that end, there is absolutely no reason why file systems should in any way limit the type or quantity of metadata associated with a file. Once again, the Be file system shines in this regard. Storing small, keyed attributes, such as the file’s type (probably as a MIME type), the default application for the file, the various dates associated with it, and the other, custom, types is the only sane way to organize and classify data in a modern filesystem. To this end, everything about the file, including it’s name (and any localized names), should be stored in the auxiliary metadata file associated with every regular file on the system.
Of utmost importance is that there be no limit on the size of type of the metadata present. The metadata file is designed to be the way to identify a file. Keywords, document titles, authors, etc. should all be stored in the file’s metadata, allowing the user to issue searches for “all documents with author ‘Brendan’ and contain the word ‘money'”. Nothing less will suffice as the gateway to the vast quantity of files a use must deal with. Metadata searching is the biggest aid to productivity since the whip, and BFS will have it.
Some Groups are More Equal than Others
Okay, while it is all fine and dandy to have a system that is extremely easy to navigate for the user and which does away with many of the annoying constraints of modern systems, there is a vast swath of code out there written for the usual, hierarchical structure and which is not going to change over night. Very well, since programmers demand a highly structured disk layout, we shall give it to them in the form of a System group. Like the “blessed” folder on the Mac OS and the root directory on Unix, this is where the off-limits system files and usual hierarchical structure reside. If the program gets a directory listing of “/usr/bin”, the file system will return all the files present in the group System -> user -> bin. If a program deletes a file, it disappears from all the groups in which it currently resides. Do note that, although the System group looks just like “/” to the programs, the hierarchical structure is only skin deep. The actual layout on disk bears no resemblance to the hierarchical structure present in System.
Beyond just providing a compatibility layer for older programs, the System group can play a greater role in aiding applications in finding the files/information they need. For instance, even if the user doesn’t have all their applications in a single group named “Applications”, the System group could contain a group called “Applications” which would have a live query associated with it that found all application files on the system. Hence, no more expensive searching to find the location of a certain application and no more popping up a dialog asking the user to locate one. The same goes with other things of interest to programs such as: all removable media, all USB devices, all connected monitors, etc. Unix started the trend of making everything available through the file system, why not finish it?
So far we’ve divorced files from their locations, from their names, and from their metadata constraints. Why not divorce them from their file type too? Why not add a plugin architecture (technically not part of the file system, but it’s related) which could allow applications to install various filters for certain types of files. Why have to save a file once as a PNG and once as a GIF to have access to both types for different applications? BFS and any related software above it will let applications install a PNG -> GIF converter to allow any application which understands GIF to open and PNG document. A program need not look only for the files it can understand, but simply tell the OS which file types it can accept and let the OS decide whether a given file can be converted to the appropriate type.
Beyond even this, BFS will allow users to create “doppelgangers” of various files whereby any modification to the original file will automatically trigger an update to the doppelganger. Imagine having an important group full of billing information for your company. Create a doppelganger of that group which gzip’s the group’s contents into a backup archive. As changes are made to the original files, the doppelganger is replaced with the compressed, updated files. The link between the original file and its doppelganger can be broken at any time, thus making it easy to create stationary snapshots.
The user interface for all this should probably be much like the spacial Finder of Mac OS 9. However, the multitude of windows spawned should be cut down to one. A zooming interface wherein groups and their contents zoom out to let you inspect them is probably better suited to navigation. The desktop metaphor works very well with the group metaphor in that the desktop is always your temporary dock where you can toss stuff onto and then wipe clean again. Note that tossing files on to the desktop does not remove them from their previous groups. You need to issue a “remove” command to make that happen. Groups should probably be displayed as a clump of the files it contains. This allows the user to quickly glance and see which groups are fuller than others.
Files, without a location, without a specific group to belong to, without even a name, should be identified at all times by a unique number, probably 64 or 128 bits in length. This number should be unique to the volume that they are on and should be intelligently used by the system to find a given file and to index information about a file. For instance, there might be various B+trees scattered about the disk, all of them containing a portion of inode lookup tables (sort of like the directory structure of today). A file with a number ending in 16 one bits would be found in the B+tree associated with 16 final one bits and so on. The file number should be persistent so that applications can store the number in a file and then find the file again after a restart. Note that the file number does not depend at all upon which group(s) the file resides in.
Metadata is likely to take the form of a small file and can probably be efficiently packed into blocks with other small metadata files. Various techniques like placing the first couple of bytes directly after the file inode should also help. Metadata journaling should be present as well.
Finally, the elimination of a hierarchical structure should free the implementor from having to worry about unbalanced directories, strange create/delete patterns that strain block allocation policies, etc. Further, no centralized directory need exist; all information for a section of the disk could be stored at the head of the section like ext2 and could be easily recovered and indexed at boot time.
The Final Retort
As I have said previously, the current hierarchical structure of file systems is unnecessarily constraining. Creating a more free-form storage allows the user or the System group to create hierarchy when necessary and refrain from doing so when unnecessary. The end result is a system more suited to a user’s organizational skills and less dependent on the fragility of keeping a location and name for every file on the system. Addition of superior metadata handling is the coupe de grace which creates order out of chaos and fully exploits the human ability to describe rather than identify.
I Want to Know More
Raskin, Jef The Humane Interface Addison-Wesley, 2000
An excellent introduction and forward-looking view of human-interface design. A must for any serious programmer dealing with GUIs. Check out his website http://humane.sourceforge.net/home/index.html.
Giampaolo, Dominic Practical File System Design Morgan Kaufman Publishers, Inc., 1999
The creator of the Be file system tells you everything you ever wanted to know about creating a file system. Very interesting read. Buy it at Amazon.
Reiser, Hans Name Spaces as Tools http://www.namesys.com/whitepaper.html
The creator of the ReiserFS talks about his vision for the future of file systems. Rambling, at times completely incoherent, but holds a couple of jewels of insight. Be sure to check out the section: “Storage Layers Above the FS: A Sure Sign the Developer Has Failed”.
About the Author:
“I am currently a junior at the University of Wisconsin – Madison studying Mathematics. In my spare time I program for fun and profit for a small company doing graphical logging of roadways. I also manage to inflict on the world my limited theatrical talents by performing in plays, musicals, and the like.”