Though great advances in Linux Desktop usability have been made in recent years, one rather large area of difficulty still exists, posing some serious issues on the ease-of-use front. There is, however, a little-known solution to these problems…
Introduction
Many Linux users spend quite some time at a GNOME/KDE desktop. These
modern graphical environments have improved the standing of Linux as a
desktop OS remarkably over the past few years. They are able to make the
system easy to use for newbies and can also allow experts to simply get
on with their work without the DE getting in the way.
One of the most notable features of these modern desktop environments is
their ability to connect to remote computers for the purposes of
accessing files. These files could be on Windows shares, FTP servers or
SSH accounts, for example. The desktop environments provide a way to
access them just as one would access the files on our local machines…
or do they?
While it’s true to a large extent that these subsystems – know as
Virtual Filesystems (VFS for short) – provide some pretty nifty
features, my experience of them has been made fairly miserable for a
number of reasons, which I hope to describe to you now. My experience
lies mostly with GNOME but I have had similar issues with KDE when I
last used it (which was some time ago; so please forgive me for any
errors). As it happens, the points I’m about to make are fundamental
design issues relating to VFS layers in general and are not aimed at any
particular desktop environment.
What’s Wrong with the popular VFS implementations, then?
You would be right to ask this question, given that I’ve just explained
how useful a VFS can be. The answer comes down to how the VFS subsystems
that are in use have been implemented, rather than the principles behind
them.
Have you ever accessed an FTP site via Nautilus (your home directory on
a server, for example) and wondered why you can’t move files you own to
the Wastebasket (instead you have to delete them directly), or see the
same types of information in the properties dialogue box as you do for
files that live on your desktop machine? How about being told that
OpenOffice.org can’t open a file on an SSH share you mounted, despite
GNOME/Nautilus being able to see it? Why can Totem play files from a
remote music library but XMMS can’t? Some of the answers to these
questions are “because the underlying file service can’t support X, Y or
Z”, however, that’s not always the case. Sometimes, users are left
wondering why they can’t open a file that, to them, clearly exists right
in front of them on their screen. This is how the problems of modern-day
VFS are made visible. The power users amongst us may know what the
problems are caused by and how to fix them, but that doesn’t help
someone who’s accustomed to easier-to-grasp OSes from being totally
confused and maybe even unable to do their work effectively because they
don’t know a way around the problem.
The main issue is that the code which runs the VFS is implemented at the
wrong layer in the system. By this, I mean that the VFS is usually
implemented too far away from the kernel of the operating system,
alongside the high-level desktop infrastructure. This means that they
can’t provide facilities to programs that run “lower down” in the system
– and that’s where most of the problems are caused. This will,
hopefully, be made clear very soon through some examples.
*Side note:* One of the reasons for their implementation at such a high
level is that the desktop environments they’re part of, being portable,
need to rely on common underlying software to function. Different OSes
provide different ways of accessing remote filesystems and that can
cause trouble for porting the software. However, there is a solution
available for Linux which could be cloned in other OSes and, hopefully
in your opinion after reading this, should be used where possible.
Anyway, back to the problems caused by the VFS being implemented at the
wrong level…
The Problems Caused
Here I’ll explain a few situations in which you may have experienced
problems with a VFS and explain why this is the case.
*Totem can play remote music but XMMS says “file not found”:* This is a
common situation and occurs with more programs than just music players.
The problem is that Totem, being part of GNOME, has been linked to the
GNOME-VFS library, which provides remote file operations for GNOME apps.
XMMS, however, has not – so even though Nautilus can see a file on your
other box, XMMS can’t because it doesn’t know how to talk GNOME-VFS.
I’ve also had this type of error whilst trying to open an OpenOffice.org
document over a SSH connection to a remote home directory – though I
believe the OOo team is working on GNOME-VFS support for the next
release, it begs the question “Surely not every app has to provide
specific support for every VFS (i.e. separate support for GNOME, KDE,…)?”
*Can’t perform some file operations or see some properties information
for remote file-shares:* This problem arises sometimes when the protocol
used to share/access the files is incapable of performing some
functions, but in honesty it shouldn’t happen much. The protocols in use
today are all capable of doing pretty much every common file operation –
it seems to be limitations in VFS implementations which don’t fully
support some protocols that is causing some of these issues.
*Can’t use cp, rm, mv on remote files:* Because they’re not really
“mounted” in the UNIX sense of the word – the kernel has no idea that
the remote system is being accessed and can’t present it in the
filesystem like any other directory. This is a major annoyance for many
people as they’d like to access remote files easily with tools like the
above and any other console programs such as Vim.
*Desktop interoperation is discouraged:* I’m sure many of you are aware
of the amazing work that Freedesktop.org have done in partnership with
GNOME and KDE (and many other DEs) to give users much more choice and
flexibility in which environment they run and that environment’s ability
to talk to programs written for the others. VFS layers pose one of the
last few major hurdles that face interoperability amongst DEs – as
you’ve seen, they are coded to be DE-specific and all the applications
in a given environment are expected to support that environment’s own
VFS. For KWrite to work seamlessly with files mounted by Nautilus would
require the KDE apps to be given support for GNOME-VFS – not something
that would be very popular (either way ’round).
A Solution to All of these Problems
There exists an Open Source project that provides a solution to all of
the issues discussed above – LUFS [http://lufs.sourceforge.net/]. The
initials stand for “Linux Userland FileSystem”. LUFS provides a way for
code in userspace to provide support for various filesystems. These are
mostly remote ones like ftpfs and sshfs, but could be local ones too.
LUFS contains a very small kernel module that links to a userspace
daemon. This daemon provides the basic infrastructure for
filesystem-specific mounting programs to run on top of. The magic thing
about LUFS is that it allows you to access the remote files, no matter
how you mounted them, as if they were local to your system.
The result of this is that *all* of your programs, from rm to vim and
GEdit to JuK, can see and work with the files automatically, with no
extra coding effort required. Interoperation is guaranteed not only
amongst desktop environments but console apps too! Some examples of use
are that you can open your remote music files in any player, regardless
of which desktop environment you run and/or which media player you use.
You can even stream video over FTP as you watch it! Many options can be
passed to the filesystems you mount to control a lot of their workings
and get things set up to your taste.
And if that’s not enough…
LUFS provides the ability to look at file access in a novel way. Because
anyone can develop a program to mount any filesystem that they like,
some very interesting new uses can result. One example is the Gnutella
filesystem, which allows you to echo into a certain file what you’d like
to search for and marvel as the results gradually appear as files in a
directory below this. You can stream the files from the peer that owns
them without even copying them to your hard disk! Though this may be
impractical for some people, it just goes to show what can be done.
Watching films over FTP or streaming music seamlessly is very fun and
not something that you’d want to give up when you’ve tried it. The LUFS
team say they’re working on such things as httpfs, which could provide
an interesting way to navigate the web…
Conclusion
I hope I’ve explained some of the issues involved in the VFS layer as we
know it. Even if you have not had the problems that I have, I hope I’ve
shown how powerful doing things at the right layer can be and how this
could benefit the whole desktop environment scene by promoting
interoperability and getting rid of a load of repeated code that each DE
currently has to maintain. I see no reason why LUFS shouldn’t be part of
Linux (the kernel) already. If it was, adopting solution like LUFS is
something that Freedesktop.org could heavily promote in the future;
after all it will cut down on a great deal of duplication of code (and
bugs, no doubt!) and should also help everyone use the programs they
like regardless of which desktop environment they use. I feel that
freedom of choice is one of the main things that Free Software should
promote – LUFS seems like a prime example of how to do this effectively.
About the author:
Matthew Atkinson is a Computer Science finalist at Loughborough University in the UK. He is co-founder of AGRIP [http://www.agrip.org.uk/] which is an Open Source project that aims to make mainstream games accessible to blind players. Currently Quake has been made accessible. He also recently set up AGDev [http://www.agdev.org/] which is a community site for accessible game developers. He’s used Linux since August 2002, and has been happily reading OSNews since just before then.
/*Author’s note:* Please be aware that I have no relation to the LUFS project; I simply think it rocks and deserves some publicity :-)./
If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.
Both Gnome and KDE aren’t Linux-only, but support Unix. Relying on a Linux-only kernel service to implement an important part of their DE isn’t something that can be done in reality.
I’m not sure, but to do it correctly (having password dialogs come up at the right time, display progress) would seem to be a little difficult. It would be easier to fix the problems you mention.
Yes, lufs is fantastic. I was having some irritating problems mounting my media directory from my HTPC on my other machines via nfs and samba last week, and someone suggested I just use lufs instead. Result? urpme samba-server , disable nfs and portmap services, urpmi lufs on the client machines, write one line into fstab and I’ve got a perfectly transparently ‘mounted’ filesystem over ssh. Hassle-free remote mounting and a drastic cut in the servers running on my network? I’m all in favour.
Altough it’s not the same code base I thin, FUSE (another userspace filesystem alternative) has been included in the -mm tree and will be probably included in future releases of linux!
Thank you for a well-written and reasoned article. We don’t get many of those around here.
The problem with LUFS as you stated is portability. KDE and Gnome have refused to adopt it for this reason because Solaris and xBSDs will not be able to use these features.
I find this a little silly. Even though there are some desktop users in the xBSD family, most of the people using BSD, use it as a server.
At least, on the desktop where VFS filesystems are most useful,I think it is time for the KDE and Gnome people to realize that most of their users are on Linux and that holding back the platform in the name of neutrality is actually hurting their respective platforms.
One last thing, KIO slaves have worked beautifully for the past 2 years for me, but I would also like to have a desktop independent solution that was pervasive across the whole Linux stack.
LUFS is good, FUSE is good too. They are about the same thing and progress is being made by the FUSE developer to include it in the kernel. Check out http://fuse.sourceforge.net/ and the get latest -mm kernel and test! Unfortunately linus is hesitating a bit about this.
I am new to LUFS. Could you post your /etc/fstab line to check out the syntax for LUFS?
Thank you.
Both Gnome and KDE aren’t Linux-only, but support Unix. Relying on a Linux-only kernel service to implement an important part of their DE isn’t something that can be done in reality.
Then tell me why HAL is supported in both Gnome and KDE ? HAL only exists on Linux, not on Solaris/xBSD.
There goes your argument 🙂
fuse is a filesystem that is designed to export filesystem functionality to userland tools. lufs is similar but not designed as cleanly. fuse support has continued development today and should prove more useful/functional in the long term. If you must have lufs, look for the lufs module for fuse. fuse supports lufs as well as its own native modules.
http://fuse.sf.net
not right now, I’m at my non-MDK job, windows only here! if you google for lufs you get the project home page which has some example fstab lines, that’s what I based mine off. If you don’t figure it out by the time I get home, I’ll try and remember to post mine for ya. sorry!
Then tell me why HAL is supported in both Gnome and KDE ? HAL only exists on Linux, not on Solaris/xBSD.
There goes your argument 🙂
I think that’s fairly obvious. HAL is a nicety. It’s not required for basic expected functionality of the desktop. The underlying OS in many cases already provides this functionality in another form. For example, Solaris 10 automatically handles mounting of CDs, USB keys, and other media perfectly from what I can tell.
But, when you try to do something like this to a VFS layer, which is a fundamental part of the desktop environment, suddenly you’ve left a lot of users out in the cold.
1. The author doesn’t mention FUSE at all. Since it’s going to be included in linux LUFS will become rather irrelevant.
2. “The problem is solved too far away from the kernel”. This is simply not true. You could move e.g. gnome-vfs to the kernel, it wouldn’t change a thing, you still wouldn’t be able to use mv,rm,OpenOffice etc with remote filesystems. Similarly, you could implement a POSIX-style API in userspace and make it work with mv,rm and OpenOffice. It’s a question of API, not user-level vs kernel-level code.
3. Frameworks like KIO and gnome-vfs handle a lot of things that POSIX-style API:s such as LUFS or FUSE don’t. Things like progress bars, key management, password dialogs. They are also much better suited for non-filesystemlike API:s like HTTP for instance. This is the major reason GNOME and KDE developers are not interested in LUFS or FUSE. It’s not the first time this suggestion have been made. Yet this article doesn’t adress these points at all.
Um, LUFS is pretty much dead. Some posters already mentioned FUSE, and that’s for a good reason as FUSE is actively worked on (I don’t have this impression on LUFS).
Pardon me, maybe I’m ignorant, but I don’t understand why someone who discovers a pretty old and not very well written project one day (a lot of people in here have been using it two or three years ago) gets an article on osnews. Write about LUFS, instead. Well, right, maybe I should simply start writing an article instead of blaming osnews.
Didn’t I have to install LUFS to use the captive ntfs drivers? I’m still not sure what userland utilities do, but if I have LUFS can I just mount a Windows partition to read/write or do I need to continue using the captive drivers for that?
You can use a kio-slave in every shell command via kfm-exec, e.g
kfm-exec cat fish://[email protected]/etc/hosts
Bye
Thorsten
Its appreciable that GNOME-VFS and KIO-Slaves do some different things that FUSE/LUFS cannot do, its also true that there are some common tasks. This just cries out for a common backend to unite the technologies.
Does anyone know if the freedesktop people are trying to unite GNOME-VFS and KIO-Slaves into a shared desktop VFS layer? I would then make sense to impliment a backend driver for that layer to use FUSE on Linux systems when its available, and to use the desktop-VFS layer backend for other platforms and things not handled by FUSE.
File systems in userspace is very similar to the ideas of translators available in the Hurd.
http://www.gnu.org/software/hurd/whatis/translator.html
Here’s a list of things which could be made possible by these ideas.
http://hurd.gnufans.org/bin/view/Hurd/TranslatorWishList
Thanks, all, for your posts and constructive criticism of the article. Also, cheers for pointing out about FUSE — I was not aware of it. As has been mentioned, it is in the same vein as LUFS so it’s good to know that those ideas are being continued. I discovered LUFS almost a couple of years ago and have been meaning to write to OSNews about it for nearly that long :-). I’ve been on the lookout for other such projects and things like this getting into the kernel but FUSE seems to have passed me and my LUFS-using friends by. Mostly, I was trying to explain the principles that LUFS stood for and I’m pleased that something like it will probably now get into Linux. Hopefully we’ll be able to benefit from it soon.
I thought the points made about the cross-platform (or not) nature of some of the other technologies the DEs depend on was quite good. HAL will be more and more important in the future. It would be nice to see it ported to the other *NIXes (or is that *NIXen? hehe), but using them where we can does seem to make sense at the moment, especially given the competition which is represented by that other, quite easy-to-use, OS…
I’ve been interested in trying the HURD for some time and probably will do now the LiveCD is out. I look forward to using translators first-hand!
best regards,
Matthew
There is a freedesktop.org project to unify the VFS implementations. But it’s not mature or accepted enough by GNOME or KDE teams AFAIK
http://freedesktop.org/wiki/Software_2fdvfs
v9fs – A /Sane/ Answer to the Problems of Desktop Virtual Filesystems. (and much more)
v9fs.sf.net
The problem with filesystems is, that you can only access them trough unix file operations. But you cannot wrap asyncron file operations in unix file operations.
To cite Havoc Penning:
“An async/gnome-vfs/http type API can wrap POSIX file APIs, but the reverse isn’t true without breaking POSIX semantics in major ways (and keeping you from implementing things you need, like progress bars). The desktop file API can really be extremely simple: get/put user-visible documents/data. For config files and caches and such, using the vfs isn’t even right; you want a guaranteed local/fast filesystem or an API like gconf. The way to think about a proper gnome-vfs would be a document load/save API more than as a streams API. This is ignoring things like Storage.”
AGRIP [http://www.agrip.org.uk/] which is an Open Source project that aims to make mainstream games accessible to blind players. Currently Quake has been made accessible.
Dear god. Is it textmode quake translated to brail?
I posted something like this to a KDE mailing list, as you’ve got wonderful technology in io-slaves but you simply cannot open a file across them without it downloading the whole damn file to your hard drive. This creates usability problems for users, as they expect the file they open to be located wherever they opened it.
some people ask KDE to use fuse.
yeah. ever heard about KIO-fuse? you can mount any KIO slave as a folder. so every app can use KDE’s IO slaves. problem solved?
Ummm…. In order a view a file, the contents do have to be downloaded. Why would you want to have KIO Slaves act in any other manner than they currently do?
In general, almost all programs act upon complete files, not chunks or streams. You could rewrite them, but that is somewhat perverse for a word processor/image editor, etc.
KIO Slaves download, allow editing and implictly re-upload the results (with rw-capable protocols, at least). To do this, I can just do fish://user:passs@myserver in Konqi and middle-click or open/edit as I need to, or I can just have that in the file selector. When I click ‘Save’ toolbar icons or menu items, the file is resaved to the SSH server through SCP automatically. As a user, I don’t need to know about any of that. That’s network transparency as it is meant to be.
Downloading a complete piece of data prior to editing, for media that need it is essential, as semi-streaming everything produces massive problems with network disconnects, coherency and versioning. You’d *really* not want to do it.
In order a view a file, the contents do have to be downloaded.
Nope, the whole of a file should not need to be downloaded to work on it. It is over a network after all.
Why would you want to have KIO Slaves act in any other manner than they currently do?
Errrrrrrr, so I don’t have every MP3 downloaded to my local machine when I play it, or I don’t have a several hundred meg video file downloaded needlessly? Just a thought
.
Downloading a complete piece of data prior to editing, for media that need it is essential, as semi-streaming everything produces massive problems with network disconnects, coherency and versioning.
We are actually in the 21st century you know? The vast majority of networks, wireless as well, have handled this fine for a very long time.
You’d *really* not want to do it.
Works fine on a Windows Network. Don’t see why it shouldn’t elsewhere.
Does anyone know if the freedesktop people are trying to unite GNOME-VFS and KIO-Slaves into a shared desktop VFS layer?
There is a proposal on the Freedesktop.org wiki to build a common VFS (ala KIO) on top of D-BUS: http://www.freedesktop.org/wiki/Software_2fdvfs
It’s not a unification of KIO and GNOME-VFS, but rather a reimplementation of both.
The last time i tried it, you can not open/edit/save
text files with Lufs. Your files
will get corrupt!
Works fine on a Windows Network. Don’t see why it shouldn’t elsewhere.
It works on windows shares. Windows has extremely poor support for non-local files otherwise. KDE (and perhaps Gnome, I don’t really know) has far superior VFS support than any other environment.
I think the problem with it downloading the entire file is limited to the smb:/ io-slave in KDE. You can play a video over http:/ just fine. It’s just over a windows share it downloads it first. No idea why though. I wish they would fix that. Time to check bugs.kde.org.
HAL support is not neccesary for KDE. I think the media:/ ioslave will function just fine without it.
HAL in its current state is highly overrated anyway. I don’t see why I should run some daemon 24/7 sucking up system resources just so I can save 3 minutes every 6 months when I plug in some new hardware.
The problem with filesystems is, that you can only access them trough unix file operations. But you cannot wrap asyncron file operations in unix file operations.
I’m not sure why that is supposed to be so, Posix does support non-blocking file operations after all.
But in any case, if there are problems, surely the solution should be to extend Posix (and its implementation in the kernel and glibc), so that virtual file systems become transparently and consistently available to every program, including the command line.
KIOSlaves and Gnome_vfs were great to get things moving in the right direction, but they really need to be replaced at a lower level.
In Konqueror, When clicking on a zipped file on a local drive, it is opened right there using the zip kioslave, which I find very useful.
However, when clicking on a zipped file on a remote drive, e.g. sftp or smb, it is opened using “ark”, which isn’t half as good as the kioslave.
That’s confusing and inconsistent.
The problem is that kioslaves can’t be nested. When clicking on a local zip file, the resulting URL is something like:
zip:/dir/file.zip
This means that the ‘zip’ slave directly accesses local files, which duplicates functionality of the ‘file’ slave and stops it from working on remote files (unless they’re mounted locally of course).
Really, kioslaves like zip should be wrappers around other slaves, thus allowing for remote access, e.g.:
zip:(file:/dir/file.zip)
zip:(sftp://user@server/file.zip)
The parentheses would be necessary to disambiguate directories when browsing into archives, e.g.:
zip:(sftp://user@server/file.zip)/dir
Nested archives would be possible too, e.g.:
zip:(zip:(file:/dir/file.zip)/foo.zip)
Do LUFS or FUSE support that kind of thing? Are there any plans for KDE 4.0 to include nested ioslaves?
I recall vaguely something like this mentioned on the dot, but I don’t know for sure. but I think you are right, there should be more flexibility here. some applications don’t support it correctly, too (can open remote files from the filemanager, but not save them – while it works with the file-open dialogue) – I filed some bug reports, as its very easy to fix (they just have to change 1 or 2 characters in the application’s .desktop file).
check bugs.kde.org, and/or file a wish, as this sounds very cool.
“Pardon me, maybe I’m ignorant, but I don’t understand why someone who discovers a pretty old and not very well written project one day (a lot of people in here have been using it two or three years ago) gets an article on osnews. Write about LUFS, instead. Well, right, maybe I should simply start writing an article instead of blaming osnews.”
But remember google is so large…
I personaly have found lots of old projects (dead) but full of information.
We would all enjoy more articles joe (IP: —.stusta.mhn.de)
The problem with LUFS as you stated is portability. KDE and Gnome have refused to adopt it for this reason because Solaris and xBSDs will not be able to use these features.
what people use gnome/kde on solaris/bsd ? all these people run blackbox or enlightnement any way and prefer screen sessions to a desktop environment. i hardly doubt people there use nautilus at all.
“you want a guaranteed local/fast filesystem or an API like gconf”
The problem with LUFS as you stated is portability. KDE and Gnome have refused to adopt it for this reason because Solaris and xBSDs will not be able to use these features.
what people use gnome/kde on solaris/bsd ? all these people run blackbox or enlightnement any way and prefer screen sessions to a desktop environment. i hardly doubt people there use nautilus at all.
Just because you don’t use KDE/GNOME on Solaris or BSD doesn’t mean there aren’t a lot of us out there. There are. Considering how often the KDE/GNOME ports are updated on FreeBSD, I’d say there are a large number of us. Sure, maybe there are more KDE/GNOME users running on some version of an OS using some version of the Linux kernel. But that’s no reason to alienate an entire section of your userbase.
KDE and GNOME are cross-platform and portable by design. Let’s keep it that way. Create the VFS layer and make it portable. Underneath the VFS, let it use whichever technologies are available on that OS (LUFS, FUSE, SSH, NFS, RSH, whatever). Make the user-visible layers work the same, and make use of the best techonologies available in the OS on the backend.
For the record.
The article speaks of VFS as the Desktop VFS. However right at the heart of the linux kernel there is a subsystem called VFS.
Those are two different things implemented at two oppsite levels. Distinction should be clearly made between them, even they somehow provide same functionality.
It is unfortunate that there is a second VFS at the desktop level.
–sadyc
If the browser would do what he suggest, when the user removes a files from a FTP server, the browser would *download* the file into the wastebasket to be ‘coherent’ with local file deletion..
How nice! Users will love downloading files that they want to remove.. I think not!!
Actually this means that the browser cannot let a VFS layer handles totally the files: it must handle differently ‘local files’ and remotes files.
That said, a desktop agnostic VFS is a good idea of course.