Linked by Thom Holwerda on Mon 30th Jan 2012 20:39 UTC
General Unix Finally something really interesting to talk about. If you've used UNIX or any of its derivatives, you've probably wondered why there's /bin, /sbin, /usr/bin, /usr/sbin in the file system. You may even have a rationalisation for the existence of each and every one of these directories. The thing is, though - all these rationalisations were thought up after these directories were created. As it turns out, the real reasoning is pretty damn straightforward.
Thread beginning with comment 505639
To view parent comment, click here.
To read all comments associated with this story, please click here.
RE[4]: We are stuck in the past.
by axilmar on Thu 2nd Feb 2012 12:16 UTC in reply to "RE[3]: We are stuck in the past."
axilmar
Member since:
2006-03-20

What info exactly do you expect to be able to query from a jpeg or mp3?


Size. Author. Title. Date. Compression rate. Encoding rate. Decoding rate. Etc. There are many other attributes to query.

and yes, that sort of data, makes sense in a database and often is put there.


But it is put there by specialized software. Querying for metadata is not a standard feature of most filesystems, as is, let's say, the POSIX file interface.

You could view the filesystem as a database where the path system is the primary index.


But it is not relational.

Find and Grep can be used, with others, to query.


These tools fail to return structured data, especially from non-text formats.

The kernel need only provide the basics, the primary key and the data.


I never said anything about kernels.

It's the wrong tool for the job.


Nope, it's the right tool for the job. The various development problems we are having today are due to the lack of databases in a large degree.

Many database just aren't design for storing GBs in a column entry for a row. It's just not what they are for.


These GBs that you speak of would be broken down to their individual parts, if stored in a database, and they will be indexable, and queriable, discoverable by any program, they would support transactions, and they would allow programs to be notified of changes in the data store. All these capabilities are absent, more or less, from today's data storage systems.

Reply Parent Score: 2

jabjoe Member since:
2009-05-06

Size. Author. Title. Date. Compression rate. Encoding rate. Decoding rate. Etc. There are many other attributes to query.

As I said before, metadata. Or maybe media information from something like 'mediainfo'.

But it is put there by specialized software. Querying for metadata is not a standard feature of most filesystems, as is, let's say, the POSIX file interface.


Nor should it be. Wrong level to have it.

But it is not relational.


Not quite, but it's not really hierarchical either, not with links, it's more of a network. Still easy to argue is a database of a form.

These tools fail to return structured data, especially from non-text formats.


The case in question is meta data, the tool 'extract' sticks the meta data in a set structure out to stdout. With that, find and grep, you could write a "query" that find all jpegs that where taken with a certain camera. Of course this is a sequential search and a indexed one would be better. Plenty of programs that build custom indexs/database of meta data for this kind of purpose, but you don't want to index everything in every way, just in case.

Nope, it's the right tool for the job. The various development problems we are having today are due to the lack of databases in a large degree.


Example please. Windows WMI certainly hasn't convinced me.

These GBs that you speak of would be broken down to their individual parts, if stored in a database, and they will be indexable, and queriable, discoverable by any program


What parts exactly? Not all fileformats are like that, many are just blobs. What about a video, once you dismiss the metadata stuff, how are you going to store the stream? Binary blob block/chunks, just like a filesystem? On their own one is likely meaningless, and won't have anything in it you can search. Then why bother? Creates lots of needless big vacuuming for no gain.

they would support transactions, and they would allow programs to be notified of changes in the data store. All these capabilities are absent, more or less, from today's data storage systems.


Many filesystem do offer transactions, and even with those that don't, you can still do it. The SQLite site has some a doc on how they do it generically: http://www.sqlite.org/atomiccommit.html

Pretty much every OS has a notification system for file or folder changes.
Linux's is inotify, or even just the "select" call.

Found this, sounds like what you want:
http://en.wikipedia.org/wiki/Pick_operating_system

Reply Parent Score: 2

axilmar Member since:
2006-03-20

As I said before, metadata. Or maybe media information from something like 'mediainfo'.


It's not metadata. It's data. And it's not only media that have useful data.

Nor should it be. Wrong level to have it.


Yes it should. There should be a standard for it, to allow applications to interoperate at data level.

Not quite, but it's not really hierarchical either, not with links, it's more of a network. Still easy to argue is a database of a form.


You can't run queries on a filesystem regarding the data inside the files, and therefore it is not a database.

The case in question is meta data


Nope. The discussion is about information management, not metadata only.

the tool 'extract' sticks the meta data in a set structure out to stdout.


The schema of the information output cannot be queried at runtime.

With that, find and grep, you could write a "query" that find all jpegs that where taken with a certain camera.


But not all jpegs taken on a certain afternoon, or within a specific time period, or with multiple cameras or users, or many other things.

Plenty of programs that build custom indexs/database of meta data for this kind of purpose


If this functionality was supported out of the box, these programs would be redundant.

Example please. Windows WMI certainly hasn't convinced me.


Almost every application has a layer of data input/output from/to files. This layer would be redundant if databases were supported out of the box.

What parts exactly? Not all fileformats are like that, many are just blobs


Nope. All file formats have an internal structure, otherwise they could be read after they were written.

What about a video, once you dismiss the metadata stuff, how are you going to store the stream?


As an array of frames, and each frame being an array of pixels.

On their own one is likely meaningless, and won't have anything in it you can search.


Nope. One can search for a particular scene, employing pattern matching with another picture, or even a hand-drawn one, for example.

Many filesystem do offer transactions


Many, but none of the major ones, as far as I know.

The SQLite site has some a doc on how they do it generically


SqlLite achieves transactions through various mechanisms, as described in the paper. These mechanisms are based on the functionality provided by filesystems, but the filesystems themselves do not have the concept of transaction.

In order to enable transactions in another application, one has to rewrite the same SqlLite mechanism.

Pretty much every OS has a notification system for file or folder changes.


But these notification mechanisms are not compatible with each other, many O/Ses don't even have such notification mechanisms, these notification mechanisms don't work over the internet, and applications cannot be notified about what exactly changed inside a file.

Found this, sounds like what you want:
http://en.wikipedia.org/wiki/Pick_operating_system


Not bad. It's a step in the right direction.

Reply Parent Score: 2