In a multi-user, multi-process operating system, files are continually being created, modified, and deleted, often by apparently unrelated processes. This means that any software that needs to keep aware of what is happening in a filesystem needs to employ a file monitoring technique. Monitoring, in this sense, means keeping a watch over a set of files, waiting for any of them to change. Read the article at DevChannel.
File Alteration Monitoring Techniques under Linux
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
Good solid article. I happen to need to know how to do file monitoring, glad there’s a somewhat portable way of doing it — didn’t know about it before the article and now I intend to put it to use.
Erik
I tought: (F)ile (A)lteration (M)onitoring? use FAM.
then I read the article, and it goes three pages long to say: use FAM.
Five minutes of my life I won’t get back. 🙁
on my archlinux box it seems as if fam is polling the filesystem — i get an annoying crunching sound from the harddrive every 5 seconds or so… maybe have to modify and rebuild that package myself. what a pain life can be
There is currently no good mechanism to provide file alteration monitoring in the linux kernel. FAM itself uses the dnotify mechanism, but that is very limited. For example it does not work recursively, so in order to watch for changes in a large tree, you have to watch each directory.
I wrote an enhanced dnotify mechanism for the linux kernel. It can be found here:
<http://www.lambda-computing.com/~rudi/dnotify/>
There has already been some discussion on the lkml.
The correct url of the enhanced dnotify page is:
http://www.lambda-computing.com/~rudi/dnotify/
Should have used the preview…
If your using resierfs it writes its journal to disk every few seconds, but not via the filesystem layer so it will always write, even if you try to spindown your disk or use the no-flush daemon.
interesting for sure – and tuttle’s patch would make the UI more user-friendly.
putting noatime in the fstab entry is recomendable, also
Is it just my browser of are the header file names missing from the sample code?
Wow, just makes me apprecciate Be’s NodeMonitors that much more!
watch_node(my_node_ref, B_WATCH_ALL, be_app);
void
MyBeApp :: MessageReceived(BMessage*msg)
{
// DO WHAT I NEED TO DO WITH MESSAGES.. 🙂
}
If you want a high level object oriented api for file change notification, there is the KDE class KDirWatch. It uses FAM, which then uses dnotify if available or polling if not.
http://developer.kde.org/documentation/library/cvs-api/kio/html/cla…
I needed something ~like~ a file alternation monitor sometime ago. However, what I needed was a system which could give me notifications about any access made to a file whether or not the access was actually granted. FAM/dnotify etc are limited in scope that they notify only when a file is altered. They don’t have notifications for access to files which have been denied. I needed a cross platform solution which worked on NFS, AFS and non-distributed file systems. Found none! I am currently using an ugly glibc interpose hack to get these events from the system!
I just upgraded my kernel from 2.24.26 to kernel 2.6.6 and it seems a lot slower, especially with regard to file I/O. I am running a P2-450Mhz with 384MB of RAM and a 20GB ATA100 drive.
Why is the stock 2.6.6 kernel so much slower?
This project wants to be a platform independent file access notification/control mechanism. Currently it works for Linux 2.2-2.6 and BSD, but they plan to support Windows and OSX in the future. Maybe you should take a look at this:
http://www.dazuko.org/
It is not nearly as efficient as my patch, but it has a different goal: it allows you to intercept file system calls and implement additional security features etc.
“I just upgraded my kernel from 2.24.26 to kernel 2.6.6 and it seems a lot slower, especially with regard to file I/O”
i get the feeling that x sometimes isnt 100% responsive during heavy file i/o. i cannot recall experiencing that with the 2.4 series
while (1>0); do ls -l $file; done
haha
or try my ‘search engine’ –
find . -type d | xargs grep ‘foobar’
PLEASE MOD DOWN! CONTAINS COPYRIGHTED SOURCE OWNED BY YAHOO!
This kind of news makes me glad that I don’t run Linux. What is with this trend of putting everything “in-kernel”? What’s next, a snack-bar, a weenie roaster? How about a web server? (har har)
Not to mention we’ll now see “expert programmers since yesterday” using this crap and then getting all huffy when people tell them their code sucks because it’s not at all portable.
As others mentioned, there’s this:
http://oss.sgi.com/projects/fam/
You don’t know what your talking about.
FAM is just a daemon. FAM itself uses dnotify if available. If nothing like dnotify is available, FAM will use polling.
It is completely reasonable to put some kind of file change notification mechanism in the kernel, unless you like polling.
Nobody uses dnotify directly. People use FAM or the even more high level objects such as KDirWatch in KDE or FileSystemWatcher in .NET. But for these high level objects to work there needs to be some low level infrastructure. dnotify is just that.
What if your program called rsync or used librsync in dry-run mode to see if a file has changed?
Any thoughts?
-John
That’s alot of code to monitor the file system!
I use FileSystemWatcher.
Re: John
creating hashes is quite computationally expensive. That would be a very expensive way to look for changes.
Re: Shapeshifter V.90
FileSystemWatcher under mono uses FAM or dnotify. So you would benefit from a better mechanism as well.
Back when I was new to NT4 programming (having had a unix background) I was very impressed by the Win32 api WaitForMultpleObjects. You can use WaitForSingleObject too. The greate thing about WaitForMultpleObjects if that the array of objects can include files, folders, semaphors, mutexes, threads or timers. This makes it real easy to implement timeouts etc. Also waiting in a folder means it’s whole contents recursively (not sure about mount points, they can in Win2k).
When I learnt this it was the first time I though ‘wow NT does have significant advantages over unix’. (Let’s ignore the window per process and other nastiness for now).
I don’t know if WaitForMultipleObjects first came about in NT, or started in some earlier system, but I wonder how hard it would be to implement in a unix kernal (linux, *bsd, darwin)? What are the main hurles to implmenting in unix when it’s considered bread and butter in NT?
For those that like reading APIs: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dl…
I agree that WaitForMultipleObjects is a very impressive API. The mono people are trying to emulate it under unix systems. If you use mono you have WaitForMultipleObjects available regardless of wether you are using UNIX or Windows.
“I just upgraded my kernel from 2.24.26 to kernel 2.6.6 and it seems a lot slower, especially with regard to file I/O”
i get the feeling that x sometimes isnt 100% responsive during heavy file i/o. i cannot recall experiencing that with the 2.4 series
Make sure that your X isn’t niced at -10. That used to be a trick to make X feel faster, but it doesn’t work with the 2.6 kernels. Renice X to 0 if that’s the case, it should help.
There is something similar on BSD like systems (such as OS X) named kqueue (kernel event notification mechanism). But there is a great limitation (for me) : because a file descriptor is opened for each file to monitor its not possible to watch lots of files at once.
What are the limitations for FAM ?
BeOS offered a very very good “file alteration monitor” called LiveQuery. It was great and *all* files on the hard drive could be watched.
> What are the limitations for FAM ?
>
You can watch for as many directories as you want. But since dnotify uses a file descriptor per directory, this can get quite expensive. And if the number of watched directories exceeds some number (I think 255), FAM resorts to polling. That sucks.
> BeOS offered a very very good “file alteration monitor”
> called LiveQuery. It was great and *all* files on the
> hard drive could be watched.
>
I loved BeOS. One of my main motivations to write the enhanced dnotify patch was to add the nessecary infrastructure to do live queries on linux.
With my patch, you can watch for all files on a partition by just watching the mount point recursively.
> The system facilities that provide file monitoring are
> now widespread and mature
That is such a lie. Perhaps widespread, but certainly not mature. I don’t think I could even imagine anything worse than dnotify (except for polling, of course). AFAIK the best thing out there for linux is tuttle’s dnotify replacement, but there are still some ugly problems with it, such as not seeing notifications of changes made to a file from a hardlink outside your “monitor zone”, even though the changed file is also inside that zone. I guess the cause of the problem is the stupidity of the person (Linus?) that didn’t include a “primary key” system in the FS API of linux.
> That is such a lie. Perhaps widespread, but certainly not
> mature. I don’t think I could even imagine anything worse
> than dnotify (except for polling, of course).
>
LOL. You are right, and most people on the lkml agree. The original dnotify is very limited and also a quite strange as an API. I enhanced it instead of completely replacing it to increase the chance of getting it included, not because I like it.
> I guess the cause of the problem is the stupidity of the
> person (Linus?) that didn’t include a “primary key” system
> in the FS API of linux.
>
That is not linus fault. Many file systems such as ext2/ext3 have the inode number as a primary key. The problem is that the inode number as defined in POSIX is only 32bit and therefore too small for the primary keys of many modern file systems. Some fs use as much as 128bit. That is the reason for all these kludges.
If linux were to support only one or two file systems like Windows NT, it would be much less difficult. But supporting many different fs is one of linux greatest strengths.
> > I guess the cause of the problem is the stupidity of the
> > person (Linus?) that didn’t include a “primary key” system
> > in the FS API of linux.
>
> That is not linus fault. Many file systems such as ext2/ext3
> have the inode number as a primary key. The problem is
> that the inode number as defined in POSIX is only 32bit
> and therefore too small for the primary keys of many modern
> file systems.
Yes, and that’s why there would need to be a “primary key” system so that each FS “module” can define its own primary key, instead of the kernel relying on e.g. inodes being unique.