Submitted by Matthew Dillon writes: “I am going to start committing bits and pieces of the HAMMER filesystem over the next two months. Note that the filesystem will not be operational until we get closer to the 2.0 release in December so these bits and pieces will not be tied into buildworld/buildkernel until then.” Features: maximum size of half an exabyte, infinite snapshots, limited only by retention policy, streaming backups, asynchronous transactional support (no long fscks to check disk state). Dillon also explains why he chose not to use Sun’s ZFS.
it’s hammer time!
Nice goals set for his project. But while he expects to implement a working version ready for beta testing within record time, ZFS had more than one person working on it for like three years and it’s still missing some features. So I don’t think he can deliver within his intended timeframe.
Maybe, or it may go faster for him. There’s the old saying about too many chefs in the kitchen. Look at how many people were working on Vista and how long it took. The more people you have working on a project, the more you have to wait on others, more confusion, and more bickering.
Edited 2007-10-14 16:01
The concern for backwards compatibility is a huge reason for Vista taking as long as it did, making as many changes as it did: this filesystem only needs to be sufficiently compatible with the virtual file system interface, and implement according to that general contract: in no way can Vista and any single filesystem be comparable in scope, complexity, or manpower required to do things correctly. You might as well foolishly start comparing the development of this filesystem (or any other filesystem) that needs to have the VFS compatibility with say, KDE going between one major revision number and another, as that is a similar comparison to what you’re making.
Nonetheless, there’s often a great advantage to being only a single person in charge of a system, and also a disadvantage: a single person won’t (hopefully) end up creating something that reeks of “design by committee” but by the same token, it’s entirely possible that a single developer will have one or more huge blindspots that causes them to miss something needed or mess up in some spectacular way that’d be prevented by having more than one person involved.
On face value you might appear to be correct, but you have forgotten an important point. Matt is sitting in a very unique position when it comes to implementing this. He has free access to make the changes necessary in any part of the system he needs without much worry of requiring or maintaining backwards compatibility. Make no mistake, that can make all the difference when it comes to implementing such deep-rooted technologies quickly.
I also believe that this type of “filesystem” (I put quotes because this level of functionality goes far beyond a typical filesystem) has been his goal since the inception of DragonFlyBSD so all of the work to date has been done to facilitate the addition of HAMMER.
And more importantly, the situation he is in, he doesn’t have to contend with the office politics with marketing breathing down the programmers necks to implement zyx feature because it has foobah buzzword.
ZFS is also very much in its infancy; this isn’t a bash against it, but having had a look at the various bugs that have come up regarding it, I’d sooner feel more comfortable sticking with something like UFS.
ZFS is in operation since about 2003 by Sun. Some big companies and universities are using it already for servers. Maybe not as bullet-proof as UFS, but ready for prime-time.
I’m not an expert on the subject, but I’d say that ZFS consists of two parts, the FS and the “IO stack”. The FS is the actual filesystem (on disk data layout etc.) while the IO stack is the rest that makes ZFS special.
HAMMER is very similar in that it will also have two parts, the FS and the rest. The actual FS does not have to take particularly long to develop if you are familiar with FSes (like Matt) and do not try to make something truly revolutionary. The rest is probably the harder part, but the idea of a clustered file system is not new and Matt has been working a few years towards this goal and I would guess that a lot off the grunt work is already done.
As an example, the (FS independent) journaling was implemented as far back as 2005, and it is my understanding that it will play an important role when replicating across machines. Also, in February Matt posted an initial outline of HAMMER, so for those who claim that it will be done in record time: I’m not so sure, this is something he has been working on for quite some time now.
Sometimes smaller “teams” are more efficient !
Yeah, just look at windows vista.
But still, Matthew Dillon does tons of the work himself and his mentality to rewrite and redesign stuff instead of using what’s already available is pushing the date of an actually feature-complete stable release far into the future.
That maybe, but in the meantime there are other BSD operating systems that feature-complete and stable for everyone who needs that. Dfly is about scratching an itch Matt has about building a operating system around lightweight kernel threads that is also fully distributed.
Rewriting and redesigning stuff isn’t a bad thing. Sometimes conventions need to be challenged. I also don’t think there is much that they can share with other OSs, aside from userland tools and drivers. There really isn’t anyone else going where they’re going right now (VMS and Plan9 do the distributed computing thing, but I don’t think they can crib from those two), so rewriting stuff may also be more of a necessity if they want a correct rather then a hackish solution.
It is a nice approach since I have thought ZFS is THE file system of the next decade for *BSD machines. But will HAMMER be accepted by other *BSD distributions when it is ready? Apple is going to replace its default fs HFS by ZFS. I do not think that Apple will also port HAMMER when it is ready. But a file system nobody (beside DragonFlyBSD?) uses will remain just a niche product for tech enthusiasts. Nevertheless I think that the development of a next-generation fs can only benefit from such kind of projects and the mistakes were done.
I’m willing to bet that it will be accepted in all the other open source BSDs. There is no reason for them not to port it over, and i’m sure some developer will take interest and do it. I think that it is also very likely to be picked up in OpenBSD as soon as it is stable. If i recall correctly, OpenBSD doesn’t seem to be interested in ZFS, so this might be a good (sorta) alternitive for them
So far talking about ZFS replacing anything in OS X is total bullshit, it might happen, I hope it’s happening, but so far we don’t know this. What we do know is that support will probably be there ;D
I do not think that HAMMER will be supported by other BSDs. At least not fully, since it is more than an on disk format (just like ZFS) designed to fulfil a very special need not found in any of the other BSDs, namely supporting a fully clustered single system image OS (DragonFly).
He’ll be in jail for murder in 5 years.
The problem ZFS has is that it is TOO redundant. You just don’t need
that scale of redundancy if you intend to operate in a multi-master
replicated environment because you not only have wholely independant
(logical) copies of the filesystem, they can also all be live and online
at the same time.
I mean, he literally assumes all servers running DragonBSD will be live and replicated, thus not needing the extra redundancy built into ZFS.
What the hell has he been smoking? and then I’m not even considering the idea that it would be really REALLY nice to have a true cross OS filesystem that can be used on all *BSD, OS X and linux (then we only need ZFS for windows… anyone? ;-).
One of the main goals of dfbsd is to make a OS that is highly suited for clustering. Basically, he is making a file system that is better for this environment.
As for the cross platform FS, all the operating systems i’ve ever seen can use FAT, and the unix-like ones have UFS covered as well.
“As for the cross platform FS, all the operating systems i’ve ever seen can use FAT, and the unix-like ones have UFS covered as well.”
Just as a technical sidenote, I found the “tar file system” being one of the best cross platform file systems. It’s not a real file system in fact, but it can utilize any formatted storage media for data interchange among all UNIXes, Linusi, and Mac OS X.
For example, if you
% fdformat -y /dev/fd0
% tar cf /dev/fd0 /etc/rc.d/*
you can transfer this archive to a completely different system using
% tar xf /dev/rfd0
These systems do not even need to have any other than their own file system (e. g. UFS, ext2) represented as a part of the kernel (or a KLD / LKM). Only the desired device drivers are needed, of course.
You can access floppies (anyone still knows them?), CDs, DVDs, PDs, all kinds of memory cards and sticks and even tapes this way. Just imagine /dev/da0, /dev/sa1, /dev/rft0 or something similar in the example above. Compression (flags z or j) can be added if required by transfer sizes exceeding media size.
One advantage is the ability to work with directory hierarchies, file attributes, user and group settings, and selective access – things FAT cannot do.
Regarding FAT, the worst solution prevails, as usual. 🙂
Edited 2007-10-15 18:46
It really is not a bad assumption, DragonFly is a good system, but if you are really running a server there are other, better systems to use.
The whole idea with DF is to be a clustered single system image OS, anything but that is just a bonus.
Umm..of course he assumes this. That’s a driving goal of the entire project. The goal isn’t for a general purpose one size fits all file system. Rather it’s for a a file systems designed from the ground up to works in a multi-master, shared nothing cluster environment.
That a primary reason for rejecting ZFS initially for this aspect of the project.
“Dillon also explains why he chose not to use Sun’s ZFS.” I read his post and i don’t think this counts as a real explanation. This may count as a reason, but without him explaining details, it’s just a short quip.