Novell’s Nat Friedman put up a bunch of demos demonstrating Beagle on Linux. Beagle is an indexing & search utility similar to Apple’s upcoming Spotlight (both similar to Be’s BeFS regarding “live queries” and metadata, but Be’s solution could not search inside documents as both Beagle and Spotlight can).
Beagle Flash Demos Available
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
86 Comments
2005-01-24 5:54 pm
2005-01-24 6:04 pm
Anonymous
3) It would be wonderful if patches could be made to treat all spaces in filenames as underscores. GNOME devolopers have started using spaces in file and folder names-STUPID! This just creates havoc for those of us who administer systems. IF A FILENAME OR FOLDERNAME CANNOT BE PARSED BY THE COMMAND LINE TOOLS DUE TO EMPTY SPACES OR PARENTHESIS IT IS A BUG AND IF YOU IMPLEMENT SUCH DUE TO “APPEARANCES” YOU ARE CONTRIBUTING TO BREAKING LINUX.
Except almost every command line tool that I know of can parse file names with all kinds of characters including spaces in them. If they cannot, they are broken and need to be fixed. Lastly, please do not type in all caps, use bold instead. No one likes being yelled at 
2005-01-24 6:13 pm
2005-01-24 6:37 pm
2005-01-24 6:38 pm
2005-01-24 6:40 pm
2005-01-24 7:13 pm
2005-01-24 7:23 pm
2005-01-24 7:38 pm
2005-01-24 7:39 pm
2005-01-24 7:40 pm
2005-01-24 7:58 pm
2005-01-24 9:59 pm
2005-01-24 10:35 pm
2005-01-24 10:44 pm
2005-01-24 11:01 pm
2005-01-24 11:05 pm
2005-01-24 11:16 pm
2005-01-25 12:26 am
2005-01-25 12:57 am
2005-01-25 2:43 am
2005-01-25 2:49 am
2005-01-25 8:35 am
Anonymous
Beagle uses d-bus, an new ipc system developed by Robert Love, iNotify is the way that kernel is able to send messages over d-bus, so fam or things like that aren’t needed anymore.
The polling age is over
2005-01-25 9:25 am
2005-01-27 11:24 pm
2005-01-28 6:09 pm
And I have to say, the live query demo (second one down) is especially fun.
Beagle is going to be a nice addition to the Linux desktop. I also like that the Beagle developers have an open interface/API and even nice command-line tools, so that one can even write smart shell scripts and small programs using Beagle as a backend.
I wonder if the “tiles” interface that the Beagle developers came up with will fly back into standard GTK as a widget for widespread consumption… they are quite nice, and probably have a lot of other applications.
I just had a look at the demo and I must say it looks verly good function-wise. GUI Search tool’s have been quite a weak spot in Gnome (I won’t comment on KDE as I don’t use it).
Thanks alot Nat & Novell.
How is Beagle really different froma traditdonal search program like Kfind?
It searches file contents, mails, web pages you’ve seen, IM log conversations, stores metadata, and is really fast.
You should watch the demos.
View the demos more carefully. You obviously didn’t even watch the demos, but still, you felt like commenting.
apparently Google puts a lot of man hours into open source…im sure this would be a great area for them to help out the os community.
*hint*
Wondering if anythose are similliar to skyos’s find as you type thing? seems like they may be a little more advanced though
Is Beagle going to be an official part of GNOME 2.10?
No, because it is using C#, and Mono/C# is not part of Gnome, neither it will be any time soon, according to Red Hat’s Gnome engineers.
Hey all, I’m planning on starting a series of projects (sexy giFT frontend, Nero-like CD burning app, and a bittorrent client) and am tempted to use Mono/GTK#.
Can someone answer some questions.
Are there C# Gstreamer bindings(for embedded media previews)?
Good documentation?
Is performance decent?
Anyone using Mono now, any comments (good/bad)?
So it won’t be in Fedora 4 as a package then? Curses!
<<neither it will be any time soon, according to Red Hat’s Gnome engineers.>>
Its planned to be in Gnome3 apparently
….
http://live.gnome.org/ThreePointZero
Bottom, “Technology Integration”
When is GNOME 3? Are they planning 2.12 and 2.14 first or will it be straight after 2.10? APIs are being changed so it will be longer to build than the normal six month release schedule, won’t it?
It’s a lot more user friendly than Kfind, and the live queries as well as fid as you type features are nice.
However, isn’t it a bit too limited. I love the concept, but I want it to work for all file types, not just a few source code, IM, mail etc. I want a full and integrated solution. (btw whon opening things, does it respect mimetype settings and defaults?) Is it possible for Beagle to do mhat I’ve seen in the demos for all file types, maybe in the future even for audio (recognize speech). That would really be something
Basically, is there a clean C++ API for developers to add new functionality to Beagle, such as allowing it to search in new files?
Finally, from the demo it looks like the core functionality does not depend on GTK. Is this true? Does that mean that in the future I might see a Beagle povered Kfind? :p
>Its planned to be in Gnome3 apparently
….
No, it is not. That page doesn’t say anything about it, and even if it did, the point of the matter is that many Gnome devs are against it for patent reasons.
No ones knows
I’m guessing 2.12 is the last release in the 2.X series. That would put Gnome3…*does math*….2007/08
I do not know C# and I am not at all interested in the language.
>Is it possible for Beagle to do that I’ve seen in the demos for all file types,
More file types might be implemented later, for now, these will do. Apple’s Spotlight searches among 19 file types atm.
>maybe in the future even for audio (recognize speech). That would really be something
Yeah, in 10-20 years maybe. When CPU power will be a lot, and when storage will be even more. For now, you will need special servers to do something like that.
The least thing a user wants right now is to have a search daemon make the machine slow just because the user has 2,000 mp3s on his machine… Think about it, what you are proposing is science fiction (for now) and not really useful.
<<
No, it is not. That page doesn’t say anything about it, and even if it did,>>
It’s OK that you can’t read. I’ll quote for you.
<QUOTE>
We can more deeply integrate new technology that has appeared since the beginning of the 2.x series.
[…]
Beagle
</QUOTE>
http://live.gnome.org/ThreePointZero
Reading is fun!
<<the point of the matter is that many Gnome devs are against it for patent reasons.>>
UH-OH no Java then.
http://www.sun.com/legal/patents/
What makes you think there will be a GNOME 2.12 and they won’t just go into developing 3.0?
Also, what patent reasons would prevent Beagle from being part of GNOME 3.0? And no, I do not like C# all that much, in fact as a language I think Java and C++ (with the appropriate toolkit) are much nicer. Python is really good too
<<What makes you think there will be a GNOME 2.12 and they won’t just go into developing 3.0?>>
Read the mailinglists. They’re pushing back technologies/features and make specific reference to Gnome 2.12
<<Python is really good too
>>
Agreed!
“Yeah, in 10-20 years maybe. When CPU power will be a lot, and when storage will be even more. For now, you will need special servers to do something like that.
The least thing a user wants right now is to have a search daemon make the machine slow just because the user has 2,000 mp3s on his machine… Think about it, what you are proposing is science fiction (for now) and not really useful.”
Have you read up on Cell? This will probably be possible by 2010 and I think it will be REALLY useful. It’s so hard to find what you’re looking for in movies or audio files, that would be incredibly useful.
I’m more interested in how the demo works using ordinary web technology than I am in the actual content of the demo. It is very cool to have a movie-like demo that doesn’t require any type of plugin such as quicktime or real player. Beagle looks cool too, but I have no plans to use it anytime soon since I got a Mac.
Isn’t it Flash?
And Flash isn’t in every OS by default.
it does require flash….
Oh, never mind
.
As far as integration into gnome, don’t expect it soon (like 2.10). There certainly is a split over how much people in the gnome communinity want .net tied in any way to gnome. However I think it’s only a matter of time. Some of the most innovative and interesting developments in the gnome world are being done on mono so I think eventually it simply won’t be an option to not include it.
As for what distro’s will include it. At this point I would guess Novell, possibly Ubuntu (there are many devs that seem to like mono so at least available in some way), debian in some fashion. While Redhat won’t include or support it in Fedora there is no doubt the community will provide it.
As for the demo’s, I must say I’m impressed. It really looks like it’s come along and looks to be a very useful tool. I would simply love something like this for myself, can’t wait to use it (I use Ubuntu so here’s hoping Hoary has it).
Lastly I wouldn’t doubt to see a QT version sooner than later. Especially since I see this as something Novell will want to push as a solution for KDE and Gnome.
It is very easy to add new filetypes to beagle. All you have to do is write a filter (in C#) for the corresponding files. There is a nice public API which is easy to use. There are allready filters for .doc .pdf besides all those in the demo, plus lots more, it is really easy to extend.
well.. do not fail to notice that this softwares heart is the C# COPY of Lucene.. which is a long time java project. sigh.
That live query demonstration was amazing. That’s the way search should be done.
When I checked the info on the beagle website, and specifically, the filetypes which Beagle can handle, I saw that beagle can’t search for movies (avi, mpg, mpeg2, asf, wmv, etc.). Will these extensions be integrated in beagle soon?
If it has no support for, say, .avi, does it still show up “foobar.avi” if I do a search for “foorbar”?
Great search application btw, it looks better than my Google Desktop on Windows XP.
Beagle can’t search metadata. Beagle doesn’t support natural language queries. It’s at version 0.0.4. Also, it doesn’t integrate at a filesystem level like Spotlight.
Spotlight is the brainchild of Dominique Giampaolo, creator of BeOS LiveQueries, of which Spotlight and Beagle are obvious descendants. Spotlight will support any format for which commercial developers choose to release plugins for. I just think how great it will be when I can search for “guitar patches for the nnxt” and all my Reason Soundbanks and ReFills will be searched. Meanwhile Beagle will depend largely on reverse engineering to search file contents, and doesn’t search metadata so the query I just gave as an example would be impossible.
It’s another example of alpha quality open source being pitched against a mature counterpart professionally developed by people who know what they’re doing (you’ll be hard pressed to find someone more knowledgable about filesystems than Dominique Giampaolo). Spotlight provides a cohesive package from the raw filesystem to the UI level whereas Beagle is cobbled together and doesn’t integrate at the filesystem level (it uses extended attributes of modern filesystems but operates in userspace).
So, the open source world can say “Spotlight? Yeah, we have something like that” but it is certainly nothing but a laughable immitation.
After playing around a bit with beagle I have to say I’m quite impressed. This will really be a killer application. My only problem with it right now is that it uses a hell of a lot of memory, but I hope this will get better in the future.
Btw., any news about a new release?
@Bascule:
Just one little word of advice, try to at least glance over the article before you start your trolling the next time around. It isn’t that hard, it’s only 3 lines, but as you seem to have missed it:
“Novell’s Nat Friedman put up a bunch of demos demonstrating Beagle on Linux. Beagle is an indexing & search utility similar to Apple’s upcoming Spotlight (both similar to Be’s BeFS regarding “live queries” and metadata, but Be’s solution could not search inside documents as both Beagle and Spotlight can).”
Not that I’m agreeing with Bascule’s comments, but the quote you provided as a rebuttal doesn’t address his points.
The magic word here is metadata.
Directly or indirectly Spotlight technology will change the way we all perceive our local data.
Once again Apple drives the computing industry forward
I hope they’ll include it in SuSe Pro as soon as it’s mature enough.
Live queries are great but how much do they comsume, do you need a big cpu machine filled with memory to run it fast and without to many load?
I esp. enjoyed the glade / sharp programming demo, i think i am going to
dig further into it.
Big applause to the Gnome Community, Ximian and Novell for making all this possible.
ps. Is there going to come/be a Ximian Desktop 3 for Redhat or Mandrake with Beagle etc. included?
Beagle isn’t as tightly integrated into the FS as spotlight is. With spotlight, once a file is updated the changes are noted and the index is updated. Beagle requires some sort of process to periodically update (once a day?) the metadata. Correct me if I’m wrong.
I really hope Beagle get’s released soon and is comparable to Spotlight. Apple’s Tiger looks to be a very promising upgrade but I’m quite settled into using Linux on my Powerbook and don’t really feel like upgrading if I can get spotlight features on Linux.
But it all looks promising at the moment, so kudos to the GNOME and Ximian guys.
Beagle isn’t as tightly integrated into the FS as spotlight is.
Why does it have to be?
Beagle requires some sort of process to periodically update (once a day?) the metadata. Correct me if I’m wrong.
You’re wrong.
I really hope Beagle get’s released soon and is comparable to Spotlight.
Are you kidding me? Beagle has been released for well over months now!
Apple’s Tiger looks to be a very promising upgrade but I’m quite settled into using Linux on my Powerbook and don’t really feel like upgrading if I can get spotlight features on Linux.
Huh? Upgrade what? Just install beagle if you have the right packages and kernel. You don’t need to upgrade shit. If you don’t have a shitty distro, your repository should have it somewhere in their unstable branch.
<P>Sorry but you are wrong here- and if you had watched the flash demos you would see for yourself….Beagle uses either fam or gam to monitor files dynamically in real time….In the demo he does the following:</P>
<P> echo superblizzard >> foobar.txt </P>
<P>and less than a second later Beagle lists foobar.txt in the list of files found containing “superblizzard”</P>
sorry about the paragraph markers…..duh!
Beagle uses iNotify to notice real-time changes.
<cite>As far as metadata goes-until the use of extended attributes on the Linux filesystems becomes the defacto standard, and these extended attributes are the same for all of the various file systems there will be no unified metadata system in Linux land.</cite>
Metadata exists in a lot of places other than EAs, and has so for quite some time. Lots and lots and lots of file formats provide a way to embed metadata directly into a header section in the file.
And this is a good thing. Any data that doesn’t transfer with my file over the wire might as well not exist. The only intelligent way to do metadata is to put it in the file. Of course, this is all easier if we have a smallish number of common, open container formats.
<cite>DO NOT RELY ON FILENAME EXTENSIONS FOR DETERMINING DATA TYPES.</cite>
Gnome (and KDE) uses both filename globs (not just extensions), and file(1)-esque magic strings. In a lot of cases, the filename is the only way of identifying the type of the file correctly. It’s not a perfect system, but we can’t fix the world.
<cite>GNOME devolopers have started using spaces in file and folder names-STUPID! This just creates havoc for those of us who administer systems. IF A FILENAME OR FOLDERNAME CANNOT BE PARSED BY THE COMMAND LINE TOOLS DUE TO EMPTY SPACES OR PARENTHESIS IT IS A BUG</cite>
Yup, a bug indeed. A bug in the command line tools. Look, it isn’t at all hard to create scripts that can deal with non-sucky filenames. We have this wonderfully intelligent system where basically any character (except trusty old /) is allowed in a filename. Why can’t we use it?
My Neuros uses a FAT32 filesystem. It annoys me to no end that I have to mangle my filenames when I transfer music files to it. Oh, sorry, that character isn’t allowed. Oh, sorry, you can’t end a filename with a dot. Why?
Users shouldn’t have to figure out weird abbreviations or read underscores as if they were spaces. Does it make the life of a programmer more difficult? Maybe a little. But users always trump developers.
Hum, right, OSNews doesn’t allow the cite element. For the record, I did preview that comment, and it looked right.
Eugenia, I think the preview function doesn’t strip markup the same way as submitting does.
Underneath the comment box:
The only HTML/UBB tags allowed are for bold and italics.
They certainly could have picked a better name –
“Hey – let’s ,ake a search tool and cal it sobig.f or perhaps sasser!”
“Nah, how about klez, nimda or Beagle?”
“Beagle it is!”
Does anyone have any experience running Beagle on anything other than linux? I looked at their wiki but didn’t see any mention.
It seems like all of the dependencies exist on other platforms, with the one tricky one being the last one – the kernel dependency.
Specifically, could I convince Beagle to run on Panther-fink-gnome-mono? I’d like to play with something like this before Tiger.
My my, where to begin:
1) Beagle is actually here, you can install it and play with it. So the fact that it’s 0.0.4 is irrelevant: Beagle is here and Spotlight is not.
2) Beagle does index metadata, just watch the demos. What is the file name if not “metadata”?
3) I see no reason why commercial plugins would not be allowed, but even if they aren’t so what? Most of the filetypes I actually have on my system are either open (mp3, rtf, txt, jpg/png etc) or well known proprietary (MS Office formats etc). You’re really taking a huge leap of faith that those audio programs you named will *have* Spotlight plugins, whereas Beagle can index music, documentation, MS Office files, web pages, IM conversations etc today.
4) Search tools operating in userspace is good and the right way to design this stuff. I have every confidence that Spotlight will work in the same way – if it does not I would be most disappointed. With technologies like inotify there are no compelling technical reasons to provide explicit kernel support for such things, so why do it?
5) What makes you think the people writing Beagle don’t know what they’re doing? They certainly appear to know what they’re doing to me!
The rest of your comment is so trollish I won’t reply to it, it’s certainly a pity as many of your previous comments here have been excellent. You should be ashamed.
Just remembered something – Beagle is based on Lucene (shame they forked it to rewrite in C#), which is written by Doug Cutting: a guy who has written search engines for Excite and Apple, amongst other people. He has been writing search engines for a loooong time and definitely knows what he’s doing.
Meanwhile, it seems that Novell have stolen a lot of Apples thunder here: Beagle integrates deeply with the desktop and kernel to provide good performance, able to index even things like blogs and calendars, while providing a clean and beautiful UI on top. Where are the Flash demos of Spotlight?
While beagle looks really nice, I have to voice my concerns over the use of C# as their language of choice for the program.
It’s not that it’s horrible, just why not Python or any number of other languages? For a distro to include beagle now, they will be risking infringement upon MS’ patent/copyright/whatever it is. I won’t be using Beagle.
Well, functionally speaking this is quite useful. I don’t think it’s going to match something like Spotlight, simply because to do real fuzzy searching you need real metadata stored at a low level (but how do you generate the metadata in the first place!). Hans Reiser is certainly right about this, and is probably a large reason why Microsoft has delayed WinFS, along with the integration issues.
All Beagle currently is is a search tool integrating a standard search of your hard drive, IM and e-mail. What it would really need to do is to be able to search different kinds of documents that perhaps don’t store raw data as plain text, Open Office documents etc. Rather than have plugins for every file type on the planet you would need a standard metadata system and native filesystem support that could do this for everything. However, it’s more than most desktops have (KFind anyone?) so it’s encouraging.
Whether it will be part of Gnome I don’t know, but if people want it it will probably come. As someone has pointed out, I cerainly don’t see people like Red Hat coming up with any alternatives as much as they feel uncomfortable (probably quite rightly) about Mono, although it will still be as slow and consume as much resources written in Java.
Meanwhile, it seems that Novell have stolen a lot of Apples thunder here: Beagle integrates deeply with the desktop and kernel to provide good performance, able to index even things like blogs and calendars, while providing a clean and beautiful UI on top. Where are the Flash demos of Spotlight?
http://www.apple.com/macosx/tiger/spotlight.html for screenshots and technical data on Spotlight. Watch the Steve Jobs keynote for demonstrations of Spotlight in action.
Beagle isn’t available to the general public today. It won’t be until the inotify thingy is accepted into the plain vanilla kernel or distros start patching the kernels to contain inotify. Sure, you can download the alphas and play with them, but the same can be said about spotlight (if you’re an apple developer).
Still, I’m waiting for Beagle since I don’t want to upgrade to Tiger just for spotlight.
Afaik this inotify thingy is already in the vanilla kernel.
“And this is a good thing. Any data that doesn’t transfer with my file over the wire might as well not exist. The only intelligent way to do metadata is to put it in the file. Of course, this is all easier if we have a smallish number of common, open container formats.”
In theory I agree with you-placing metadata in the file is logically the best approach-unfortunately we in the FOSS world have little to no say over the vast majority of file formats with which we have to work. From my point of view the vast majority of data with which we work is not stored in “open container formats”- and as such this makes things quite intractable. If EA’S were in wide spread usage and our command line tools(tar, zip etc.) took this additional info into account there would be no fundamental problem with having such metadata extraneous to the file itself- then the “file” would simply become a unit composed of parts. I, however, am no expert on these issues, and am open to being corrected…
“Gnome (and KDE) uses both filename globs (not just extensions), and file(1)-esque magic strings. In a lot of cases, the filename is the only way of identifying the type of the file correctly. It’s not a perfect system, but we can’t fix the world.”
I am well aware of this. I am also aware of the fact that parsing documents according to extension names is far, far faster than actually extracting the magic-type information. And no we can’t change the world. But I do not want to see our software becoming dependent upon extension names. Perhaps it is a lesser of two evils situations-but named extensions offer zero guarantee that the file is actually of the type specified by the extension name-and although this is not particularly important for desktop home users it is crucial to no exactly what you are dealing with when administering systems with hundreds of users…I wish I knew the ultimate answer here-I don’t and perhaps the current compromise is best….
“Yup, a bug indeed. A bug in the command line tools. Look, it isn’t at all hard to create scripts that can deal with non-sucky filenames. We have this wonderfully intelligent system where basically any character (except trusty old /) is allowed in a filename. Why can’t we use it?”
Because if we do not delimit the valid charachters to be used in file and folder naming there is no way to parse it from the command line. Command line tools must have some form of syntax for dealing with patterns-and if there is no syntactical limit to the naming conventions used in our files there is no way, at least that I am aware of, that one can parse it from the command line. Desktop users should not have to be worried about such but administrators must be able to parse any and or all documents found on the system. At present this means learning an arcane system of escaping and meta-charachter sequences-but if there are not combinations of charachters which are not valid in file and folder names then there is no way to differentiate on the command line between those commands specific to the tool in use and those charachters which occur in names.
Again there maybe someway to deal with such-but I am unaware of how such is possible. The fact that command line tools have been using escape and meta-charachter sequences for as long as command line tools have existed seems to speak for some deeper rational about such issues. Although this is sacriledge- I would have no problem having a fundamental distinction between the file and folder names which the users see on their desktop and the actual identity, from the administration point of view. If a “file” was actually a unit consisting of parts-one of these parts could define how the name appears to desktop applications….The user friendly name would simply be a “view” of the file…
Users may trump developers in some sense which presupposes a fundamental dichotomy between the two-but system administrators trump both
inotify isn’t in the standard kernels. You need to patch the kernel. More info can be found on http://www.beaglewiki.org/index.php/inotify-Enabled%20Linux~*~@…
I might try this later tonight when I have time.
How does this relate to Gnome Storage?
http://www.gnome.org/~seth/storage/
“Storage is an exciting project to replace the traditional filesystem with a new document store. Storage is part of a larger design for a new desktop environment, more details on that to come. The current implementation offers natural language access, network transparency, and a number of other features….”
And I believe this is the best strategy-the are not talking about what they are going to do-they are simply doing it-and Redhat has no exclusive claim to Linux, SuSE has adopted it, as well as Ubuntu(repositories with packages exist for it) and Gentoo.
Suse has certainly not adopted Mono. They’ve included it in their packages here and there to keep certain people happy, but that’s a long way from actually adopting it. Certainly, no one is being recommended to develop ASP.NET applications with Mono. The official way of developing ‘Java-like’ web applications with SLES is through using JBoss, and they certainly haven’t adopted it on the desktop side. The only work being done is apparently via the Novell Desktop Unit.
The only two distributions that have really adopted Mono with any intent are the NLD and Ubuntu, but the level of desktop integration with development even there is few and far between.
UH-OH no Java then.