Linked by John Finigan on Mon 21st Apr 2008 19:00 UTC
Oracle and SUN When it comes to dealing with storage, Solaris 10 provides admins with more choices than any other operating system. Right out of the box, it offers two filesystems, two volume managers, an iscsi target and initiator, and, naturally, an NFS server. Add a couple of Sun packages and you have volume replication, a cluster filesystem, and a hierarchical storage manager. Trust your data to the still-in-development features found in OpenSolaris, and you can have a fibre channel target and an in-kernel CIFS server, among other things. True, some of these features can be found in any enterprise-ready UNIX OS. But Solaris 10 integrates all of them into one well-tested package. Editor's note: This is the first of our published submissions for the 2008 Article Contest.
Thread beginning with comment 310938
To view parent comment, click here.
To read all comments associated with this story, please click here.
RE[3]: Comment by agrouf
by segedunum on Tue 22nd Apr 2008 23:05 UTC in reply to "RE[2]: Comment by agrouf"
segedunum
Member since:
2005-07-06

I too am impatient with the state of Linux storage management.

I don't see many people sharing your impatience in all honesty. The software RAID subsystem within Linux is pretty good, and has been tested exceptionally well over and over again over a number of years. You need to have spent a reasonable amount of money on a pretty decent hardware RAID set up if you really want something better. That extends to ZFS as well, as software RAID by itself can only take you so far.

The only perceived problem is that you don't get volume management, RAID and other storage management features for free in one codebase. If distributions started partitioning by using LVM by default, created userspace tools that had a consistent command line interface, as well as GUI tools that made LVM and software RAID much more visible and usable, then you'd see them more widely used on a wider variety of systems.

...but I find LVM and MD to be clumsy and fragile.

You're going to have to qualify that one, because LVM and MD software RAID were stable and being used before ZFS was even a glint in Bonwick's eye. Indeed, ZFS has yet to be proven in the same way.

I would love to see work on a Real Volume Manager (tm).

You'll probably see one come about when it becomes a requirement to do storage management on running desktop systems and other appliances. Until that day arrives those who require the features of ZFS are already using hardware RAID and some form of volume management, and beyond that, something like the Veritas Storage System.

That's part of the fallacy I see in some of the hype surrounding ZFS. It's useful and a nice thing to have, especially when compared with what you got in Solaris before, but its significance is overplayed. Most of what's in there is only really useful to people with some pretty large storage arrays, and if you have something that matters to you then you'll already be running hardware RAID of some description and volume management, and if you have money riding on it then you'll use Veritas as they have the tools that make the thing really useful, and it's very well proven having been around for the best part of two decades.

When we do get to a stage where desktop systems and various appliances need these storage capabilities (real home entertainment systems, whenever they actually happen) we'll have far better and affordable solid state or holographic storage, and the vast majority of what ZFS provides that is useful will have been transparently rolled into the hardware and not even software and kernel developers will need to worry about it.

To conclude, ultimately, when it comes to storage management today, we are limited by the hardware and the storage mediums that we have. You can't polish a turd by slapping ZFS on it, no matter how much self-healing goes on.

Reply Parent Score: 2

RE[4]: Comment by agrouf
by jwwf on Wed 23rd Apr 2008 02:21 in reply to "RE[3]: Comment by agrouf"
jwwf Member since:
2006-01-19

"I too am impatient with the state of Linux storage management.

I don't see many people sharing your impatience in all honesty. The software RAID subsystem within Linux is pretty good, and has been tested exceptionally well over and over again over a number of years. You need to have spent a reasonable amount of money on a pretty decent hardware RAID set up if you really want something better. That extends to ZFS as well, as software RAID by itself can only take you so far.

The only perceived problem is that you don't get volume management, RAID and other storage management features for free in one codebase. If distributions started partitioning by using LVM by default, created userspace tools that had a consistent command line interface, as well as GUI tools that made LVM and software RAID much more visible and usable, then you'd see them more widely used on a wider variety of systems.

...but I find LVM and MD to be clumsy and fragile.

You're going to have to qualify that one, because LVM and MD software RAID were stable and being used before ZFS was even a glint in Bonwick's eye. Indeed, ZFS has yet to be proven in the same way.
"

Clumsy: See your comments about consistent userland tools. Actually I think LVM is ok, but I am not a fan of MD's userland tools, and I am not convinced that the separation of the two subsystems is optimal. I believe this is for historical reasons, anyway. Regardless, this is a matter of taste.

Fragile: I can crash Ubuntu 6.06 pretty easily like this:

1. LVM snap a volume
2. Mount the snap read only
3. Read the snap, write to the parent, let a couple of hours pass.
4. Unmount the snap.
5. Sleep a little while.
6. Destroy the snap.
7. Panic.

The solution? Sync between each step, 4 to 6. This is not the only weirdness I have seen, but it stands out. Humorously, I was trying to use LVM snapshots in this case to reduce backup-related downtime.

The problem with hardware RAID is that most of them aren't very good, and the ones that are are really expensive. To approximate ZFS features like clones and checksumming, not to mention well integrated snapshots, you really need a NetApp or, less elegantly, block storage from EMC. The price per gig of these is about 10 times that of the dumb storage you would use with a software solution. I find it really ironic to hear that one establishment Linux perspective is "just use expensive, closed, proprietary hardware".

And before anyone says that if you have the needs, you'll have the budget, remember that overpriced storage is a problem with any budget, since it causes more apps to be crammed onto a smaller number of spindles with correspondingly worse performance. Plus, what is good about the little guy being stuck with a mediocre solution?

Reply Parent Score: 2

RE[5]: Comment by agrouf
by segedunum on Wed 23rd Apr 2008 09:15 in reply to "RE[4]: Comment by agrouf"
segedunum Member since:
2005-07-06

Clumsy: See your comments about consistent userland tools.

Well yer, but you hardly need to create a whole storage system to achieve that. One reasonable userland tool and multiple subsystems would do it. It probably hasn't been done because there's not much demand, and many people are using tools on top of the normal command line tools.

Fragile: I can crash Ubuntu 6.06 pretty easily like this: 1. LVM snap a volume 2. Mount the snap read only 3. Read the snap, write to the parent, let a couple of hours pass. 4. Unmount the snap. 5. Sleep a little while. 6. Destroy the snap. 7. Panic.

Yer. You're going to need to allocate enough space to the snaphot to take into account divergences and changes between the original logical volume and the snapshot. Which part of that did you not understand? In reality, the logical volume should just be disabled, so I don't know what you've encountered there.

This is the same for ZFS. Regardless of the space saving methods employed, snapshots and clones are not free (certainly if you're writing to a snapshot, or you expect to keep a RO snapshot diverging from an original) as proponents of ZFS want to make out. The only time they will be is when we get Turing style storage, but then, all of our problems will be solved. Do you see what I mean about polishing a turd? To go any further you need to solve fundamental problems with storage hardware.

Humorously, I was trying to use LVM snapshots in this case to reduce backup-related downtime.

Humourously, I do this all the time and I don't need an all-new file and storage system from Sun.

The problem with hardware RAID is that most of them aren't very good, and the ones that are are really expensive.

This debate of software versus hardware RAID has been done to death, and I'm sure you can find some adequate information via Google. ZFS is not adding anything to the debate, and software RAID in ZFS has been around an awful lot less than other software RAID implementations.

...you really need a NetApp or, less elegantly, block storage from EMC. The price per gig of these is about 10 times that of the dumb storage you would use with a software solution.

Yer, but if I need those features the data within them is worth more than any cost. Additionally, ZFS is still experimental and has been around for far less time than any of those solutions.

I find it really ironic to hear that one establishment Linux perspective is "just use expensive, closed, proprietary hardware".

I'm not entirely sure you understand why people use hardware RAID, and if you don't understand that then you've adequately demonstrated why ZFS is trying to solve problems that don't really need solving. Trying to label it as 'proprietary' to try and create a sensible usage scenario for ZFS is pretty desperate.

I have a choice of software RAID and various hardware RAID options, some better than others, proper hardware RAID allows you to do sensible hot swapping of drives and abstracts the RAID storage away from the system. Linux and Solaris support lots of 'proprietary hardware', but we're not tied specifically to any of them. That just seems like a pretty flimsy argument. "Oh, but it's proprietary!" is the usual last resort argument Sun themselves use, which is ironic in itself considering that various roadblocks make additional implementations of ZFS difficult.

If I'm worried about universally moving RAID arrays between machines then I have software RAID for that, and it doesn't stop me getting the data off and moving it off somewhere else.

And before anyone says that if you have the needs, you'll have the budget, remember that overpriced storage is a problem with any budget, since it causes more apps to be crammed onto a smaller number of spindles with correspondingly worse performance.

The market for that scenario is so small as to be insignificant. Overpriced storage is certainly not a problem today. People are not going to switch to ZFS because they think they're running out of space.

Plus, what is good about the little guy being stuck with a mediocre solution?

Nothing, but the problem is that the little guy has little need for all of the features available. He might use a volume manager and he might well use some form of RAID, but that's pretty much it. He's not going to get excited about ZFS if he's already using that stuff, and more importantly, he won't switch.

As I said in the previous post, when the little guy requires storage management of the kind provided by ZFS (and most of it will have to be entirely transparent) then storage hardware will have advanced far beyond any requirement for ZFS.

Reply Parent Score: 2

RE[4]: Comment by agrouf
by phoenix on Thu 24th Apr 2008 04:45 in reply to "RE[3]: Comment by agrouf"
phoenix Member since:
2005-07-11

The only perceived problem is that you don't get volume management, RAID and other storage management features for free in one codebase. If distributions started partitioning by using LVM by default, created userspace tools that had a consistent command line interface, as well as GUI tools that made LVM and software RAID much more visible and usable, then you'd see them more widely used on a wider variety of systems.


The biggest issue with all Linux tools are that they are not designed to work together, in that they don't harmonise the commandline options (or even the command names).

For instance, why is that you can't just use:
"mount -t <fs> -o <options> <device> <mount point>

to mount any filesystem? Some can be, if they have a mount.fs command installed, while others require running fsmnt commands.

Or, why can't one just use:
"mkfs -t <fs> <device>"

to format partitions with the filesystem of choice? Some have mkfs.fs command that work that way. Others don't and require you to run the separate command.

Or, why can't one use:
"fsck -t <fs> <device>

to check filesystems for errors.

Or, why can't one use:
"ifconfig <interface> <options>

to configure wired NICs, wireless NICs, vlans, bridges, and tunnels?

There are too many separate projects doing each little piece of the pie in their own corner of the world with very little regard for what the others are doing. This wouldn't be so bad ... if the layers between them were clearly defined, and the interfaces clearly defined, and didn't change at the drop of a hat.

"...but I find LVM and MD to be clumsy and fragile.

You're going to have to qualify that one, because LVM and MD software RAID were stable and being used before ZFS was even a glint in Bonwick's eye. Indeed, ZFS has yet to be proven in the same way.
"

LVM, DM, and MD are crap when you are working with large filesystems. On one 64-bit Debian Lenny box, I can't create multiple LVM volume if their total size goes over 1.2 TB. On a separate 64-bit Debian Etch box, I can create a single LVM volume that spans 7 TB, but creating a second one of 10 GB (yes GB) on the same box fails. On a third 64-bit Debian Etch box, I had to manually stitch together a bunch of 1.9 TB physical volumes into one large 10 TB volume group, but I can only partition it into smaller logical volumes if the total size of the volumes is less than 2 TB.

So much for trying to simplify things by using volume management for storage of backups and as storage for virtual machines.

"I would love to see work on a Real Volume Manager (tm).

You'll probably see one come about when it becomes a requirement to do storage management on running desktop systems and other appliances.
"

I'd much rather see something useful in the server room. LVM, MD, and DM have a long way to go before that happens.

Most of what's in there is only really useful to people with some pretty large storage arrays, and if you have something that matters to you then you'll already be running hardware RAID of some description and volume management


We're using hardware RAID, working with multiple TB storage arrays, and struggling with volume management, as the current crop of Linux tools really, really, really suck. I'm very tempted to turn our VM boxes into nothing more than NFS servers so that I can run FreeBSD or OpenSolaris on them and get some useful volume/fs management tools.

Reply Parent Score: 2

RE[5]: Comment by agrouf
by segedunum on Fri 25th Apr 2008 02:28 in reply to "RE[4]: Comment by agrouf"
segedunum Member since:
2005-07-06

LVM, DM, and MD are crap when you are working with large filesystems. On one 64-bit Debian Lenny box, I can't create multiple LVM volume if their total size goes over 1.2 TB. On a separate 64-bit Debian Etch box, I can create a single LVM volume that spans 7 TB, but creating a second one of 10 GB (yes GB) on the same box fails. On a third 64-bit Debian Etch box, I had to manually stitch together a bunch....

Your personal anecdotes on experiences with Linux storage management tools in a narrow set of circumstances count for very little. "Oh, this happened to me with LVM once!" I've created an absolute ton of multi gigabyte and terabyte logical volumes, and if what you described was the case generally then I think we'd have all heard about it. The very fact that you've mentioned 'Debian Lenny' sees your credibility get shot through the head right there.

Just because LVM does this or doesn't do that on a certain distribution, that doesn't mean that that is the way it works generally for everyone else. I daresay ZFS will have a few bugs and quirks once it gets more widely used, and if Sun actually gets a clue you might have an outside chance of finding the problem on their web site ;-).

I'd much rather see something useful in the server room. LVM, MD, and DM have a long way to go before that happens.

Well, they're used on a very, very, very wide basis these days, and I don't see people breaking down the doors and throwing out all their Linux boxes just to run ZFS. Just isn't happening.

I'm very tempted to turn our VM boxes into nothing more than NFS servers so that I can run FreeBSD or OpenSolaris on them and get some useful volume/fs management tools.

You'll do a cost/benefit analysis of that, like everyone else, and find out that it just isn't worth it. I don't know what you'd gain running FreeBSD, and the less-then-perfect userland in Solaris alone is enough to turn people off.

Reply Parent Score: 2