Linked by John Finigan on Mon 21st Apr 2008 19:00 UTC
Oracle and SUN When it comes to dealing with storage, Solaris 10 provides admins with more choices than any other operating system. Right out of the box, it offers two filesystems, two volume managers, an iscsi target and initiator, and, naturally, an NFS server. Add a couple of Sun packages and you have volume replication, a cluster filesystem, and a hierarchical storage manager. Trust your data to the still-in-development features found in OpenSolaris, and you can have a fibre channel target and an in-kernel CIFS server, among other things. True, some of these features can be found in any enterprise-ready UNIX OS. But Solaris 10 integrates all of them into one well-tested package. Editor's note: This is the first of our published submissions for the 2008 Article Contest.
Thread beginning with comment 310890
To view parent comment, click here.
To read all comments associated with this story, please click here.
RE[2]: Comment by agrouf
by jwwf on Tue 22nd Apr 2008 16:31 UTC in reply to "RE: Comment by agrouf"
jwwf
Member since:
2006-01-19

It's not just that. It's maintainability. When features get added to the wrong layer, it means code redundancy, wasted developer effort, wasted memory, messy interfaces, and bugs that get fixed in one filesystem, but remain in the others.

It does make a difference just how many filesystems you care about supporting. The Linux philosophy is to have one that is considered standard, but to support many. If Sun is planning for ZFS to be the "be all and end all" filesystem for *Solaris, it is easy to see them coming to a different determination regarding proper layering. Neither determination is wrong. They just have different consequences.

Perhaps btrfs will someday implement all of ZFS's goodness in the Linux Way. I confess to being a bit impatient with the state of Linux filesystems today. But not enough to switch to Solaris. I guess one can't expect to have everything.


This is a good, balanced explanation. I think the question is whether the features provided by ZFS are best implemented in a rethought storage stack. In my opinion, the naming of ZFS is a marketing weakness. I would prefer to see something like "ZSM", expanding to "meaningless letter storage manager". Calling it a FS makes it easy for people to understand, but usually to understand incorrectly.

I see ZFS as a third generation storage manager, following partitioned disks and regular LVMs. Now, if the ZFS feature set can be implemented on a second generation stack, I say, more power to the implementors. But the burden of proof is on them, and so far it has not happened.

I too am impatient with the state of Linux storage management. For better or worse, I just don't think it is a priority for the mainline kernel development crew, or Red Hat, which, like it or not, is all that matters in the commercial space. I think ext3 is a stable, well-tuned filesystem, but I find LVM and MD to be clumsy and fragile. Once ext4 is decently stable, I would love to see work on a Real Volume Manager (tm).

Reply Parent Score: 4

RE[3]: Comment by agrouf
by DirtyHarry on Tue 22nd Apr 2008 19:27 in reply to "RE[2]: Comment by agrouf"
DirtyHarry Member since:
2006-01-31

[q]I too am impatient with the state of Linux storage management. For better or worse, I just don't think it is a priority for the mainline kernel development crew, or Red Hat, which, like it or not, is all that matters in the commercial space. I think ext3 is a stable, well-tuned filesystem, but I find LVM and MD to be clumsy and fragile. Once ext4 is decently stable, I would love to see work on a Real Volume Manager (tm).


I think you wrong here. I do think the Linux kernel community is aware of the desperate need of a 'kick-ass-enterprise-ready-filesystem-like-ZFS'. A lot of people where waiting for the arrival of Reiser4, but we all know how that ended :-)

Ext4 is just a 'let's be pragmatic' solution: we need something better than Ext3.

ZFS for Linux is (besides license issues) a 'no-go' because of the (VFS) layering stuff.

But I think that there's hope: BTRFS. It doesn't sound as sexy as ZFS, but it has a lot to offer when it becomes stable and available. I'm following the development closely, and I get the idea that Chris Mason makes sure to not fall into the 'reiser trap' by communicating in a constructive matter with the rest of the kernel community.

Although not ready in the near future (read 2008), I personally have high expectations of BTRFS. And I believe it will become the default filesystem for many distributions when it arrives.

Regards Harry

Reply Parent Score: 1

RE[4]: Comment by agrouf
by Weeman on Tue 22nd Apr 2008 19:49 in reply to "RE[3]: Comment by agrouf"
Weeman Member since:
2006-03-20

The problem with BTRFS is that a whole team worked for several years on ZFS to get it to the current point, while BTRFS still has only one guy behind it (as far as I know). While there may be first workable prototypes pretty soon, the fleshing out of details (usually things work nice on paper, but practically prove to be crappy and need to be redesigned) and fine-tuning is going to need LOTS of time.

Reply Parent Score: 1

RE[4]: Comment by agrouf
by jwwf on Tue 22nd Apr 2008 20:26 in reply to "RE[3]: Comment by agrouf"
jwwf Member since:
2006-01-19

I think you wrong here. I do think the Linux kernel community is aware of the desperate need of a 'kick-ass-enterprise-ready-filesystem-like-ZFS'. A lot of people where waiting for the arrival of Reiser4, but we all know how that ended :-)

Ext4 is just a 'let's be pragmatic' solution: we need something better than Ext3.


I hope you are right. To be clear, I think ext4 could turn out to be an excellent incremental improvement.

But I think that there's hope: BTRFS. It doesn't sound as sexy as ZFS, but it has a lot to offer when it becomes stable and available. I'm following the development closely, and I get the idea that Chris Mason makes sure to not fall into the 'reiser trap' by communicating in a constructive matter with the rest of the kernel community.


Yes, I hope this turns out as well, since it seems to bypass some of the MD and LVM cruft. What I find interesting about BTRFS is that it is an Oracle project, and yet better storage for unstructured data is not really in Oracle's interest--what they would like is to get your unstructured data into their db.

Note: Not a conspiracy theory! I just think it is unusual.

Also, what weeman said ;)

Reply Parent Score: 2

RE[3]: Comment by agrouf
by segedunum on Tue 22nd Apr 2008 23:05 in reply to "RE[2]: Comment by agrouf"
segedunum Member since:
2005-07-06

I too am impatient with the state of Linux storage management.

I don't see many people sharing your impatience in all honesty. The software RAID subsystem within Linux is pretty good, and has been tested exceptionally well over and over again over a number of years. You need to have spent a reasonable amount of money on a pretty decent hardware RAID set up if you really want something better. That extends to ZFS as well, as software RAID by itself can only take you so far.

The only perceived problem is that you don't get volume management, RAID and other storage management features for free in one codebase. If distributions started partitioning by using LVM by default, created userspace tools that had a consistent command line interface, as well as GUI tools that made LVM and software RAID much more visible and usable, then you'd see them more widely used on a wider variety of systems.

...but I find LVM and MD to be clumsy and fragile.

You're going to have to qualify that one, because LVM and MD software RAID were stable and being used before ZFS was even a glint in Bonwick's eye. Indeed, ZFS has yet to be proven in the same way.

I would love to see work on a Real Volume Manager (tm).

You'll probably see one come about when it becomes a requirement to do storage management on running desktop systems and other appliances. Until that day arrives those who require the features of ZFS are already using hardware RAID and some form of volume management, and beyond that, something like the Veritas Storage System.

That's part of the fallacy I see in some of the hype surrounding ZFS. It's useful and a nice thing to have, especially when compared with what you got in Solaris before, but its significance is overplayed. Most of what's in there is only really useful to people with some pretty large storage arrays, and if you have something that matters to you then you'll already be running hardware RAID of some description and volume management, and if you have money riding on it then you'll use Veritas as they have the tools that make the thing really useful, and it's very well proven having been around for the best part of two decades.

When we do get to a stage where desktop systems and various appliances need these storage capabilities (real home entertainment systems, whenever they actually happen) we'll have far better and affordable solid state or holographic storage, and the vast majority of what ZFS provides that is useful will have been transparently rolled into the hardware and not even software and kernel developers will need to worry about it.

To conclude, ultimately, when it comes to storage management today, we are limited by the hardware and the storage mediums that we have. You can't polish a turd by slapping ZFS on it, no matter how much self-healing goes on.

Reply Parent Score: 2

RE[4]: Comment by agrouf
by jwwf on Wed 23rd Apr 2008 02:21 in reply to "RE[3]: Comment by agrouf"
jwwf Member since:
2006-01-19

"I too am impatient with the state of Linux storage management.

I don't see many people sharing your impatience in all honesty. The software RAID subsystem within Linux is pretty good, and has been tested exceptionally well over and over again over a number of years. You need to have spent a reasonable amount of money on a pretty decent hardware RAID set up if you really want something better. That extends to ZFS as well, as software RAID by itself can only take you so far.

The only perceived problem is that you don't get volume management, RAID and other storage management features for free in one codebase. If distributions started partitioning by using LVM by default, created userspace tools that had a consistent command line interface, as well as GUI tools that made LVM and software RAID much more visible and usable, then you'd see them more widely used on a wider variety of systems.

...but I find LVM and MD to be clumsy and fragile.

You're going to have to qualify that one, because LVM and MD software RAID were stable and being used before ZFS was even a glint in Bonwick's eye. Indeed, ZFS has yet to be proven in the same way.
"

Clumsy: See your comments about consistent userland tools. Actually I think LVM is ok, but I am not a fan of MD's userland tools, and I am not convinced that the separation of the two subsystems is optimal. I believe this is for historical reasons, anyway. Regardless, this is a matter of taste.

Fragile: I can crash Ubuntu 6.06 pretty easily like this:

1. LVM snap a volume
2. Mount the snap read only
3. Read the snap, write to the parent, let a couple of hours pass.
4. Unmount the snap.
5. Sleep a little while.
6. Destroy the snap.
7. Panic.

The solution? Sync between each step, 4 to 6. This is not the only weirdness I have seen, but it stands out. Humorously, I was trying to use LVM snapshots in this case to reduce backup-related downtime.

The problem with hardware RAID is that most of them aren't very good, and the ones that are are really expensive. To approximate ZFS features like clones and checksumming, not to mention well integrated snapshots, you really need a NetApp or, less elegantly, block storage from EMC. The price per gig of these is about 10 times that of the dumb storage you would use with a software solution. I find it really ironic to hear that one establishment Linux perspective is "just use expensive, closed, proprietary hardware".

And before anyone says that if you have the needs, you'll have the budget, remember that overpriced storage is a problem with any budget, since it causes more apps to be crammed onto a smaller number of spindles with correspondingly worse performance. Plus, what is good about the little guy being stuck with a mediocre solution?

Reply Parent Score: 2

RE[4]: Comment by agrouf
by phoenix on Thu 24th Apr 2008 04:45 in reply to "RE[3]: Comment by agrouf"
phoenix Member since:
2005-07-11

The only perceived problem is that you don't get volume management, RAID and other storage management features for free in one codebase. If distributions started partitioning by using LVM by default, created userspace tools that had a consistent command line interface, as well as GUI tools that made LVM and software RAID much more visible and usable, then you'd see them more widely used on a wider variety of systems.


The biggest issue with all Linux tools are that they are not designed to work together, in that they don't harmonise the commandline options (or even the command names).

For instance, why is that you can't just use:
"mount -t <fs> -o <options> <device> <mount point>

to mount any filesystem? Some can be, if they have a mount.fs command installed, while others require running fsmnt commands.

Or, why can't one just use:
"mkfs -t <fs> <device>"

to format partitions with the filesystem of choice? Some have mkfs.fs command that work that way. Others don't and require you to run the separate command.

Or, why can't one use:
"fsck -t <fs> <device>

to check filesystems for errors.

Or, why can't one use:
"ifconfig <interface> <options>

to configure wired NICs, wireless NICs, vlans, bridges, and tunnels?

There are too many separate projects doing each little piece of the pie in their own corner of the world with very little regard for what the others are doing. This wouldn't be so bad ... if the layers between them were clearly defined, and the interfaces clearly defined, and didn't change at the drop of a hat.

"...but I find LVM and MD to be clumsy and fragile.

You're going to have to qualify that one, because LVM and MD software RAID were stable and being used before ZFS was even a glint in Bonwick's eye. Indeed, ZFS has yet to be proven in the same way.
"

LVM, DM, and MD are crap when you are working with large filesystems. On one 64-bit Debian Lenny box, I can't create multiple LVM volume if their total size goes over 1.2 TB. On a separate 64-bit Debian Etch box, I can create a single LVM volume that spans 7 TB, but creating a second one of 10 GB (yes GB) on the same box fails. On a third 64-bit Debian Etch box, I had to manually stitch together a bunch of 1.9 TB physical volumes into one large 10 TB volume group, but I can only partition it into smaller logical volumes if the total size of the volumes is less than 2 TB.

So much for trying to simplify things by using volume management for storage of backups and as storage for virtual machines.

"I would love to see work on a Real Volume Manager (tm).

You'll probably see one come about when it becomes a requirement to do storage management on running desktop systems and other appliances.
"

I'd much rather see something useful in the server room. LVM, MD, and DM have a long way to go before that happens.

Most of what's in there is only really useful to people with some pretty large storage arrays, and if you have something that matters to you then you'll already be running hardware RAID of some description and volume management


We're using hardware RAID, working with multiple TB storage arrays, and struggling with volume management, as the current crop of Linux tools really, really, really suck. I'm very tempted to turn our VM boxes into nothing more than NFS servers so that I can run FreeBSD or OpenSolaris on them and get some useful volume/fs management tools.

Reply Parent Score: 2