Linked by Andrew Hudson on Mon 29th Nov 2010 21:50 UTC
Windows NTFS is the file system used by Windows. It is a powerful and complicated file system. There are few file systems that provide as many features and to fully cover them all would require a book. And in fact there is a book detailing NTFS, and it's already out of date. The purpose of this article is not to cover all of the features of NTFS, nor will it exhaustively cover NTFS features in detail. Instead we will cover its basic structure and then describe some of its more advanced features and provide use examples where possible. We will focus more on what it does, rather than how it does it. Trying to walk the line between informative and detailed is difficult and so this article contains a lot of references for people who hunger for more detail.
Order by: Score:
TxF
by Zifre on Mon 29th Nov 2010 22:17 UTC
Zifre
Member since:
2009-10-04

To me, by far the most interesting feature that sets NTFS apart from other great file systems like ZFS and Btrfs is the support for transactions.

I really don't understand why programs don't make greater use of it. I would really like to see it used for things like installs, etc.

Reply Score: 7

RE: TxF
by Lennie on Tue 30th Nov 2010 00:08 UTC in reply to "TxF"
Lennie Member since:
2007-09-22

I know atleast btrfs has transactions. I would guess ZFS does too.

Reply Score: 2

RE[2]: TxF
by panzi on Tue 30th Nov 2010 01:36 UTC in reply to "RE: TxF"
panzi Member since:
2006-01-22

Yes of course. Transactions is one of the big selling points of ZFS.

Reply Score: 2

RE[2]: TxF
by Zifre on Tue 30th Nov 2010 01:42 UTC in reply to "RE: TxF"
Zifre Member since:
2009-10-04

I know atleast btrfs has transactions. I would guess ZFS does too.

Both have transactions, but not like TxF. The transactions in Btrfs and ZFS is system level (like snapshots) and they block all other operations while they are running. (Correct me if I'm wrong, but this was how it worked, last I heard.)

Reply Score: 3

RE[3]: TxF
by Carewolf on Tue 30th Nov 2010 11:47 UTC in reply to "RE[2]: TxF"
Carewolf Member since:
2005-09-08

No most operating systems can support transactions if you block all other operations. The selling point of real transaction support is that you don't need to block everything else.

Reply Score: 3

RE[4]: TxF
by Zifre on Tue 30th Nov 2010 12:20 UTC in reply to "RE[3]: TxF"
Zifre Member since:
2009-10-04

No most operating systems can support transactions if you block all other operations. The selling point of real transaction support is that you don't need to block everything else.

That's my point. As far as I'm aware, ZFS and Btrfs transactions do block everything else, making them useless except in a few very specific scenarios.

Reply Score: 3

RE[5]: TxF
by phoenix on Tue 30th Nov 2010 17:57 UTC in reply to "RE[4]: TxF"
phoenix Member since:
2005-07-11

Uhm, what?

Every single write operation in ZFS is a transaction, and is atomic: either the write happened and the new data is available, or the write failed and the old data is available. You never get partially written data.

If something does go wrong, and there's a problem with the pool at import, you use zpool import -F <poolname> to roll back one transaction. If that still fails, you keep going, rolling back transactions until you get an importable pool.

Transaction support has been built into ZFS from day one, and is one of the core principals of ZFS.

Reply Score: 2

RE[6]: TxF
by Zifre on Wed 1st Dec 2010 20:56 UTC in reply to "RE[5]: TxF"
Zifre Member since:
2009-10-04

Maybe I haven't been very clear. The transactions in ZFS are system level. Individual applications can't create isolated transactions on different parts of the file system.

With TxF, an installer could, for example, create a transaction for the install process so that if you lost power, the install would not be half done. However, a file you saved between the start of the install and the power loss would still be there.

Reply Score: 2

RE: TxF
by Carewolf on Tue 30th Nov 2010 00:09 UTC in reply to "TxF"
Carewolf Member since:
2005-09-08

I am pretty sure the tree structures from filesystems like ZSH, btrfs or old Reiser4 enables transaction, but it might not be high on the list of things to implement.

If you support effecient snapshotting then transactions are not far behind.

Reply Score: 2

another thing done right
by renhoek on Mon 29th Nov 2010 22:35 UTC
renhoek
Member since:
2007-04-29

NTFS rocks, don't let anybody fool you. Transparent compression, encryption, rich acl and a lot of other stuff I don't know about. Too bad some features are not exposed to the interface (hard/soft links for example).

And only after 20 years the cracks start to appear. Throw a few hundred thousand files in a directory and it breaks down. But other than that, i have little to complain.

The absolute crap Microsoft build on top of it is an insult to the original designers. We need more metadata in NTFS, but this is not the way to go. NTFS did its job, it did it well but the time for plain filesystems have passed.

We need something like ZFS for the volume management and BeFS for the metadata and we are good for another 20 years. No rewrite needed ;)

Reply Score: 6

RE: another thing done right
by malxau on Mon 29th Nov 2010 23:33 UTC in reply to "another thing done right"
malxau Member since:
2005-12-04

And only after 20 years the cracks start to appear. Throw a few hundred thousand files in a directory and it breaks down. But other than that, i have little to complain.


Did you disable short name generation?

I don't think the article mentioned this, but NTFS generates two links per file: the name you use (long name) plus an autogenerated short name for DOS applications if the name you use is not 8.3 compliant.

Unfortunately the 8.3 namespace tends to congest pretty quickly, leading to painful times to insert and delete entries as NTFS searches for a free name.

Turning off shortname generation dramatically improves scalability at the filesystem level. Explorer et al may still struggle, but that's a different issue.

- M

Reply Score: 6

RE[2]: another thing done right
by renhoek on Mon 29th Nov 2010 23:59 UTC in reply to "RE: another thing done right"
renhoek Member since:
2007-04-29

Turning off short filename generation on a stable production server is a bit of a no-go. But you might be right, since the files started with the same characters. I suspected improper hashing or something like that in the first place, but it was a long time ago. Listing the directory took about 15 minutes before it even started printing in a dosbox, deleting the files was a pain.

Explorer struggles even with a few hundred files, which is what i meant with the crap they build on top of it. I really look forward for the day 8.3 is disabled by default and explorer is usable.

Now i'm a ZFS fanboy, which has different imperfections.

Reply Score: 2

RE: another thing done right
by UltraZelda64 on Tue 30th Nov 2010 03:24 UTC in reply to "another thing done right"
UltraZelda64 Member since:
2006-12-05

I admit, I view NTFS as a relatively nice and very reliable filesystem. I don't recall ever losing a file on it as a result of a problem with the filesystem itself. It beats the living hell out of FAT, which would always manage to "lose" critical system files and require copying them back over from the OS CD back in the Win9x days (though to be fair, this could have equally been caused by these OSes' tendency to crash and burn constantly, but IMO likely a combination of poor filesystem *and* operating system design...).

But one thing I will never praise NTFS for: performance. Sure, NTFS can take a hell of a lot more fragging than FAT ever could and slow down less due to it, but it still fragments to hell and back in very little time of normal use and will noticeable slow down in a short amount of time. Just like its predecessor, it constantly needs defragged--once every week (two at the very most) back when I used to run Windows XP on NTFS. And no, from what I've seen of Vista/7, it's not improved; the filesystem still seems happy to scatter pieces of files all over the disk.

When nice and clean (no or few excess fragments) NTFS is very snappy. It's just too bad it can't stay that way for much more than a week.

Reply Score: 2

RE[2]: another thing done right
by Liquidator on Tue 30th Nov 2010 06:45 UTC in reply to "RE: another thing done right"
Liquidator Member since:
2007-03-04

Yet, most DVB set-top-box manufacturers still ship their PVRs with FAT32 instead. Your recordings are split into 4GB chunks. Why don't they pay a license fee to use NTFS?! (or even an open-source filesystem)

Reply Score: 2

RE[3]: another thing done right
by TheGZeus on Tue 30th Nov 2010 15:49 UTC in reply to "RE[2]: another thing done right"
TheGZeus Member since:
2010-05-19

Because nearly every OS supports FAT32.
My guess is it makes it seem somewhat future-proof in the eyes of management.
"We can change everything and that drive will still work!"
I would never use NTFS in a set-top box, because most of those run some kind of Linux, and ntfs-3g is _usable_, but I avoid writing to NTFS volumes like the plague.

Most of the problems I had with it were years ago, but partly because I've only had to deal with NTFS once since them (Mac user bought a sata drive from me and wanted some data from me as well, but FAT32 didn't support many of the filenames).

Reply Score: 2

Huge article
by No it isnt on Mon 29th Nov 2010 22:38 UTC
No it isnt
Member since:
2005-11-14

Nice to see someone going into detail now and then. Thanks.

Reply Score: 3

OK, but
by lopisaur on Mon 29th Nov 2010 23:05 UTC
lopisaur
Member since:
2006-02-27

I enjoyed the article somewhat, I like technical stuff about filesystems. What I don't understand is what public the author is trying to reach... if you are talking about using file streams or file system journalling, is it really necessary to point out that there is a button in the lower left of the Windows desktop that is called "Start"?

Some comments about NTFS' problems would have been fair too. Like the fact that it is very prone to fragmentation and that it cannot handle directories with a large amount of files inside.

And I don't really think Stacker was big in the last decade... he's talking about the 90s.

Reply Score: 5

RE: OK, but
by flanque on Tue 30th Nov 2010 00:43 UTC in reply to "OK, but"
flanque Member since:
2005-12-15

I enjoyed the article somewhat, I like technical stuff about filesystems. What I don't understand is what public the author is trying to reach...

Perhaps the audience you're in? ;-)

Reply Score: 2

RE: OK, but
by OSbunny on Tue 30th Nov 2010 15:25 UTC in reply to "OK, but"
OSbunny Member since:
2009-05-23

Yeah if its so advanced why does it still suffer from extreme fragmentation? One of the biggest advantages to moving to Linux is that you don't have to defragment your hard drive.

Reply Score: 0

RE[2]: OK, but
by MamiyaOtaru on Fri 3rd Dec 2010 10:38 UTC in reply to "RE: OK, but"
MamiyaOtaru Member since:
2005-11-11

more like one of the (smaller) disadvantages to running linux is that you can't defragment your hard drive (short of copying the whole filesystem to another drive then back)

Reply Score: 2

RE[3]: OK, but
by djohnston on Sat 4th Dec 2010 17:45 UTC in reply to "RE[2]: OK, but"
djohnston Member since:
2006-04-11

If you use XFS, you can defragment.

xfs_fsr improves the organization of mounted filesystems. The reorganization algorithm operates on one file at a time, compacting or otherwise improving the layout of the file extents (contiguous blocks of
file data).

Reply Score: 1

The one thing I don't like about NTFS
by darknexus on Mon 29th Nov 2010 23:07 UTC
darknexus
Member since:
2008-07-15

I don't like the master file table, because it doesn't seem to be duplicated anywhere although it's possible to manually do that. The reason is that I've seen what happens when it gets corrupt, especially when the information about free and used space becomes inconsistent. This gives you the disappearing file effect, where by some of your files get accidentally replaced with newly created files because the mft had the wrong information. Usually this is fixable in the sense that you can stop the corruption and return the information to a consistent state, but no amount of fixing gets you those files back again. The mft should auto-duplicate every so often so that if the main one becomes inconsistent the fs can look to the backups and automatically bring it back into line. It's possible to manually do this, but it should be done constantly.

Reply Score: 1

malxau Member since:
2005-12-04

I don't like the master file table, because it doesn't seem to be duplicated anywhere although it's possible to manually do that. The reason is that I've seen what happens when it gets corrupt, especially when the information about free and used space becomes inconsistent. This gives you the disappearing file effect, where by some of your files get accidentally replaced with newly created files because the mft had the wrong information. Usually this is fixable in the sense that you can stop the corruption and return the information to a consistent state, but no amount of fixing gets you those files back again. The mft should auto-duplicate every so often so that if the main one becomes inconsistent the fs can look to the backups and automatically bring it back into line. It's possible to manually do this, but it should be done constantly.


This post seems to be confusing a lot of different issues.

Firstly, the MFT doesn't track free space; the volume bitmap does. If you have two files that exist on a volume that both believe they own the same cluster, that's corruption in the volume bitmap, not the MFT.

Making a file disappear requires lot of changes. It needs to clear an MFT bitmap entry, the MFT record, the index entry/entries, etc. There is some level of redundancy here: if an MFT bitmap entry is marked as available, NTFS will try to allocate it, then realize the MFT record is still in use and will abort the transaction.

If I had to speculate, I'd say you're seeing the effects of NTFS selfhealing, which attempts to resolve on disk inconsistencies automatically. If it really annoys you, you might want to turn it off.

Oh, and NTFS does duplicate the first four MFT records. Originally (back in Gary's time) this spanned the whole MFT, but that didn't seem like a valuable way to spend space. Most notably, there's always the issue of knowing when information is wrong - having two copies is great, but you still need to know which copy (if any) to trust. Reading both would be very expensive, so this really needs a checksum, and you start getting into substantial changes after that.

- M

Reply Score: 5

Tuishimi Member since:
2005-07-06

Very similar to Files-11/RMS on VMS.

Edited 2010-11-30 06:05 UTC

Reply Score: 5

NTFS is very very good
by poundsmack on Mon 29th Nov 2010 23:15 UTC
poundsmack
Member since:
2005-07-13

people can rip on windows and microsoft all they want. but throughout it's entire lifecycle NTFS has been a very good and stable file system.

Reply Score: 7

v I call BS...
by TemporalBeing on Mon 29th Nov 2010 23:21 UTC
RE: I call BS...
by Thom_Holwerda on Mon 29th Nov 2010 23:30 UTC in reply to "I call BS..."
Thom_Holwerda Member since:
2005-06-29

Is the title of this article "Features unique to NTFS"?

No?

Your comment is fail.

Edited 2010-11-29 23:30 UTC

Reply Score: 5

RE: I call BS...
by Carewolf on Tue 30th Nov 2010 00:19 UTC in reply to "I call BS..."
Carewolf Member since:
2005-09-08

File by file compression I think is intended for btrfs, but otherwise it is unknown in Linux.

Alternative streams are common in Mac. They sound useful, but without awareness of them, they are also quite scary.

I was mostly surprised by the description of hard link:
When you change the links attributes, the target attribute changes, ok
When you delete the link, the target is deleted.. Wait? This is not how it works in linux and posix. In posix filesystems the target is an inode and it is ref-counted, and only deleted when the last link to it disappear. Deleting a hard link doesn't delete the file. In fact you can delete the links 'target' and the so-called link will still work.

Reply Score: 2

RE[2]: I call BS...
by sukru on Tue 30th Nov 2010 04:48 UTC in reply to "RE: I call BS..."
sukru Member since:
2006-11-19

I think deletion depends on the api/tool you use.

If you use basic remove function, it probably only removed the link.

However if you delete it from a tool that does not recognize links (say explorer), it will try to "clean up" the directory first, by removing all the contents, then rmdir it, which might keep the original directory in its place, but now empty instead of full.

Reply Score: 2

v RE: I call BS...
by xaeropower on Tue 30th Nov 2010 08:44 UTC in reply to "I call BS..."
Comment by mtzmtulivu
by mtzmtulivu on Tue 30th Nov 2010 00:00 UTC
mtzmtulivu
Member since:
2006-11-14

Nice article.

btrfs and zfs now supports having multiple disks appearing as a single logical disk with data redundancy and integrity checks build in and disks can be added and removed on a "live system". Any idea of when NTFS will support this features if at all?

It is my understanding that linux started btrfs with above mentioned features among others because ext FS architecture pretty much started hitting the upper limits of its feature expansion capabilities.NTFS,like ext and HFS are file systems of yesterday, zfs and btrfs are file systems of tomorrow, when will NTFS offer features they offer? Will microsoft add these features to NTFS or will they start another one to try to catch up?

Reply Score: 2

RE: Comment by mtzmtulivu
by flanque on Tue 30th Nov 2010 00:45 UTC in reply to "Comment by mtzmtulivu"
flanque Member since:
2005-12-15

btrfs and zfs now supports having multiple disks appearing as a single logical disk with data redundancy and integrity checks build in and disks can be added and removed on a "live system".

Last I checked (a few months ago) the ZFS file system cannot shrink volumes. The ground work was in OpenSolaris but not anywhere near production ready. So as long as that hasn't changed and you didn't mean a concat volume then you are correct about ZFS.

Reply Score: 2

RE[2]: Comment by mtzmtulivu
by phoenix on Tue 30th Nov 2010 20:30 UTC in reply to "RE: Comment by mtzmtulivu"
phoenix Member since:
2005-07-11

Last I checked (a few months ago) the ZFS file system cannot shrink volumes.


Volumes (aka zvols) can be grown, shrunk, added, removed as needed. Volume size is just a set of properties (reservation, quota).

I believe you mean that a storage pool (aka zpool, or just pool) cannot be shrunk, in that you cannot remove root vdevs. This is true.

You can remove cache devices (aka L2ARC) and you can remove log devices (aka slogs) and you can remove spare devices.

But you cannot, currently, remove (or alter) root vdevs (bare device, mirror, raidz).

Someday, when the legendary "block-pointer rewrite" feature is completed, it will be possible to remove root vdevs; along with migrating between vdev types (mirror -> raidz; raidz1 -> raidz2), and with expanding raidz vdevs (4-drive raidz1 -> 5-drive raidz1).

Reply Score: 2

RE[3]: Comment by mtzmtulivu
by flanque on Tue 30th Nov 2010 20:39 UTC in reply to "RE[2]: Comment by mtzmtulivu"
flanque Member since:
2005-12-15

Yes I meant that.

Reply Score: 2

RE: Comment by mtzmtulivu
by AndrewZ on Tue 30th Nov 2010 13:58 UTC in reply to "Comment by mtzmtulivu"
AndrewZ Member since:
2005-11-15

"Any idea of when NTFS will support this features if at all?"

Microsoft would not reveal any future NTFS features.

Reply Score: 2

Comment by jwwf
by jwwf on Tue 30th Nov 2010 00:12 UTC
jwwf
Member since:
2006-01-19

Nice article. Never have had any real trouble with NTFS that wasn't related to hardware problems. I have seen some hair-raising MFT statistics though, eg an 800 GB MFT on an 7 TB volume, as reported by a sysinternals tool. The volume contained maybe 35 million files of a few dozen bytes each (among other things), and I believe they were being inlined into the MFT. I'm not complaining though since it did work despite these crazy numbers.

I don't know if the MFT buffering algorithm was smart enough to separate inlined, cold files from useful metadata (probably not), but I do believe that not being able to buffer much, if any of the MFT kills performance. Questions like these are why I really wish that the OS that runs most of the world's computers was open source: Windows is pretty good, but there is a lot of stuff you just can't easily find out about its internals.

Reply Score: 2

RE: Comment by jwwf
by Tuishimi on Tue 30th Nov 2010 06:12 UTC in reply to "Comment by jwwf"
Tuishimi Member since:
2005-07-06

I am sure you can; you can purchase an NTFS Internals Book. Most major operating systems have internals books... for the OS, file systems, etc. I used to have my VMS Internals Books, until I moved this last time (5 years ago or so) and I had too many books to move so I tossed them. ;) I gave away, or threw away, too many good books.

Reply Score: 2

Some comments
by Lennie on Tue 30th Nov 2010 00:16 UTC
Lennie
Member since:
2007-09-22

Shadow copy was actually first released with Windows XP.

The version that was included in Windows XP does not use NTFS for doing this. If I remember correctly, it just keeps some backup files and logfiles (so it can keep changes while a backup is running).

So it is not 'compatible' and the API is different.

__

I think it would have been a good idea to explain Juntion points

__

Just a quick comment about File screening, this is done by extension, not filetype. While Microsoft/Windows might be confused about the difference I would think OSNews would know the difference.

__

I wouldn't call NTFS advanced, though it has many features. I think most poeple would consider something like ZFS advanced.

__

I always felt speed was the problem of NTFS, I think that is why Microsoft wanted to replace it with winfs and that is why SQL-Server, Exchange and some other programs don't use the normal file-API.

__

Don't get me wrong, I applaud the effort. :-)

Reply Score: 3

RE: Some comments
by AndrewZ on Tue 30th Nov 2010 15:34 UTC in reply to "Some comments"
AndrewZ Member since:
2005-11-15

Some comments on your comments:

Re: Shadow Copy, Windows XP NTFS is the same as Windows Server 2003 NTFS.

My impression is that junction points are mostly deprecated in favor of links.

Under Windows, extensions pretty much equal file types.

While I would certainly agree that ZFS is an advanced file system, I don't think that you can argue that NTFS is not an advanced file system :-) I would also add that while ZFS is an excellent server file system, you wouldn't necessarily want it on your laptop.

And yes, databases like SQL Server and Oracle circumvent standard file systems for performance reasons.

Reply Score: 2

RE[2]: Some comments
by spiderman on Tue 30th Nov 2010 16:23 UTC in reply to "RE: Some comments"
spiderman Member since:
2008-10-23


And yes, databases like SQL Server and Oracle circumvent standard file systems for performance reasons.

I believe it is more about structure than performance. The file system is hierarchic whereas the Oracle and SQL Server are relational. They still rely on the filesystem but they store the tables and the records in one big file usually mapped to a tablespace. They can't map the tables and the records directly to the filesystem because they just are not hierarchic, not to mention the space wasted by the filesystem overhead. Interestingly, Oracle 11g implements a file system on top of its database: DBFS.

Reply Score: 2

RE[2]: Some comments
by Kebabbert on Wed 1st Dec 2010 11:20 UTC in reply to "RE: Some comments"
Kebabbert Member since:
2007-07-27

While I would certainly agree that ZFS is an advanced file system, I don't think that you can argue that NTFS is not an advanced file system :-) I would also add that while ZFS is an excellent server file system, you wouldn't necessarily want it on your laptop.

I dont agree with you.

The other day I screwed my Solaris 11 Express installation. I was toying with Zones (virtual machines) as a root, and by mistake typed the root command in the real computer.

I just rebooted and chose an earlier snapshot in GRUB and that was it. I deleted the old snapshot which was screwed, and continued to work.

What happens if you screw your Linux install? Or Windows install? Then you are screwed. With ZFS, you just reboot into an earlier snapshot, which takes one minute.

Yes, you certainly want ZFS on your desktop - lest you want to reinstall and reconfig, etc for hours.

And by the way, ZFS protects your data. Which no other filesystem does good enough (according to researchers).

Reply Score: 1

RE[3]: Some comments
by AndrewZ on Wed 1st Dec 2010 14:54 UTC in reply to "RE[2]: Some comments"
AndrewZ Member since:
2005-11-15

"Yes, you certainly want ZFS on your desktop"

Good point but let me put it this way. If you were selling inexpensive laptops to the uneducated masses, would ZFS be easy enough or transparent enough for Joe Average to use? Would people who don't know or don't want to know file systems be able to basic work without being impeded by ZFS features and functionality?

Reply Score: 2

RE[4]: Some comments
by phoenix on Wed 1st Dec 2010 19:01 UTC in reply to "RE[3]: Some comments"
phoenix Member since:
2005-07-11

"Yes, you certainly want ZFS on your desktop"

Good point but let me put it this way. If you were selling inexpensive laptops to the uneducated masses, would ZFS be easy enough or transparent enough for Joe Average to use? Would people who don't know or don't want to know file systems be able to basic work without being impeded by ZFS features and functionality?


If the "inexpensive" laptop has 2 GB or more of RAM, then you can get away with using ZFS without worrying too much about manual tuning, so long as the OS supports auto-tuning of ZFS memory settings (FreeBSD and Solaris do, no idea about Linux).

However, it's not the cost of the laptop that matters, it's the integration of the ZFS features with the OS. Currently, the only real integration is done in Solaris where GNOME (via Nautilus) provides access to the automatic snapshots feature via TimeSlider, and where system updates create bootable snapshots (Boot Environments), and stuff like that.

Until non-Solaris systems integrate ZFS features into the OS (especially the GUI layers), you probably won't be able to get "the unwashed masses" to use ZFS.

Edited 2010-12-01 19:02 UTC

Reply Score: 2

RE[5]: Some comments
by Kebabbert on Wed 1st Dec 2010 21:27 UTC in reply to "RE[4]: Some comments"
Kebabbert Member since:
2007-07-27

I have used ZFS on OpenSolaris on a 1GB RAM, Pentium4@2.4GHz. It worked, but not fast. I got something 30MB/sec. That is because ZFS is 128 bit, and P4 is 32 bit. If you use 64bit cpu, you get full speed.

Hence, you dont need 2GB RAM.


Regarding if ZFS is suitable for the average user. In my opinion: yes. Because ZFS is VERY easy to use. Much easier than, for instance NTFS. It is also far flexible.

In NTFS you have to partition your hard drive, that is static. In ZFS you have dynamically growing partitions (filesystems). You create a new filesystem whenever you want, and it takes less than a second. You create a zfs raid with ONE command. And there is no formatting that takes hours. No formatting at all, just use it right away. etc etc. There are numerous advantages, simplicity being one of them.

The most important, is that ZFS protects your data.

Reply Score: 1

RE[6]: Some comments
by AndrewZ on Wed 1st Dec 2010 23:18 UTC in reply to "RE[5]: Some comments"
AndrewZ Member since:
2005-11-15

It worked, but not fast. I got something 30MB/sec. That is because ZFS is 128 bit, and P4 is 32 bit. If you use 64bit cpu, you get full speed.

I don't think this is true. If you run ZFS on a 64-bit CPU, you get access to some 64-bit program counter registers, but the system bus, buffer size, etc is still the same.

Regarding if ZFS is suitable for the average user. In my opinion: yes. Because ZFS is VERY easy to use. Much easier than, for instance NTFS. It is also far flexible.

In NTFS you have to partition your hard drive, that is static. In ZFS you have dynamically growing partitions (filesystems). You create a new filesystem whenever you want, and it takes less than a second. You create a zfs raid with ONE command. And there is no formatting that takes hours. No formatting at all, just use it right away. etc etc. There are numerous advantages, simplicity being one of them.

The most important, is that ZFS protects your data.

That's good to know. Maybe some day we will see it in consumer devices. Although I don't see average consumers creating RAIDs :-)

Reply Score: 1

RE[7]: Some comments
by Kebabbert on Fri 3rd Dec 2010 18:41 UTC in reply to "RE[6]: Some comments"
Kebabbert Member since:
2007-07-27

"It worked, but not fast. I got something 30MB/sec. That is because ZFS is 128 bit, and P4 is 32 bit. If you use 64bit cpu, you get full speed.

I don't think this is true. If you run ZFS on a 64-bit CPU, you get access to some 64-bit program counter registers, but the system bus, buffer size, etc is still the same.
"
What is not true? That ZFS needs 64 bit cpu to reach full speed? Well, everybody says so. And I think it is also stated in the ZFS guides.

Many people (including me) reports low speed with 32 bit cpu. Many people (including me) reports much faster speed with 64bit cpu.

Reply Score: 2

Nitpicks
by malxau on Tue 30th Nov 2010 01:27 UTC
malxau
Member since:
2005-12-04

For those who don't know, I spent most of the last five years working on NTFS. This article is a good overview of this and other related technologies, but I thought I just just add my nits for the record.

Max FS Size

NTFS on disk allows for 2^63 cluster volumes, and files of up to 2^63 bytes. The current implementation is limited to 2^32 cluster volumes, which at default cluster size comes to 16Tb, although can go to 256Tb with a larger cluster size. The real point here is the 2Tb limit referenced in the article is a limit associated with MBR partitions, not NTFS.

File system files

One key omission here is the USN journal. It's a nice feature that has existed since Windows 2000 and does not have a direct equivalent on other platforms. Then again, it might warrant an article all on its own.

File compression

The on-disk representation of NTFS compression requires a lot of synchronization and reallocation, which leads to fragmented files that are very slow to access. This feature should only be used when space is more important than speed (which is becoming increasingly rare.)

Alternate data streams

ADSes do not have unique cluster sizes (which is an attribute of the volume); nor independent file type associations. In Windows, file associations are extension based, and the file name is shared between all streams. Again, in another article it might be valuable to consider different approaches OSes have taken to file type associations; I still really like OS/2's EA approach.

Hard links, soft links and junction points

Symlinks do not replace junction points; mklink allows creation of both. These have different properties with respect to security and how they are resolved over networks. Note that the \Documents and Settings heirarchy in Vista is comprised of junctions, despite the existence of symlinks.

Advanced Format Drives

Vista's support for these drives was a good first step, but not complete. If you use one of these and have Win7, please consider using either Service Pack 1 prereleases or the patch located at http://support.microsoft.com/kb/982018/ .

Transactional NTFS

Back when Vista was under development, we had Beta releases that allowed CMD scripts to start transactions, launch processes, and commit from a script. Alas, it was not to be.

Edit: Versioning

The NTFS version was 3.0 in Windows 2000, and 3.1 ever since. The version of the NTFS on disk format has not changed since XP. New features have been added in a purposely compatible way, so a Win7 volume should be usable in Windows 2000 and vice-versa.

- M

Edited 2010-11-30 01:31 UTC

Reply Score: 7

RE: Nitpicks
by AndrewZ on Tue 30th Nov 2010 14:06 UTC in reply to "Nitpicks"
AndrewZ Member since:
2005-11-15

Malxau,

Thanks for your input, you are 100% correct with your nits :-) One challenging aspect writing an article on NTFS is using 2nd and 3rd hand resources. Only when you get in to the actual development resources do you have access to the primary sources.

Reply Score: 2

RE: Nitpicks
by gus3 on Tue 30th Nov 2010 18:08 UTC in reply to "Nitpicks"
gus3 Member since:
2010-09-02

File compression

The on-disk representation of NTFS compression requires a lot of synchronization and reallocation, which leads to fragmented files that are very slow to access. This feature should only be used when space is more important than speed (which is becoming increasingly rare.)


The other instance I can think of, where compression can be beneficial, is to minimize the number of writes to the underlying storage medium. In the old days of sub-200MHz CPU's and non-DMA drive controllers, the time saved on disk I/O was a win over the extra CPU work of compression and decompression.

Today, the same might be useful for increasing flash longevity. Even with wear leveling, flash can take only so many writes. (Then again, one must question the wisdom of using any journaling FS on flash...)

Reply Score: 1

RE[2]: Nitpicks
by malxau on Tue 30th Nov 2010 20:39 UTC in reply to "RE: Nitpicks"
malxau Member since:
2005-12-04

The other instance I can think of, where compression can be beneficial, is to minimize the number of writes to the underlying storage medium. In the old days of sub-200MHz CPU's and non-DMA drive controllers, the time saved on disk I/O was a win over the extra CPU work of compression and decompression.


It really depends on the meaning of "I/O". In terms of bytes transferred, compression should be a win (assuming compression does anything at all.) The issue with NTFS (or perhaps, filesystem level) compression is that it must support jumping into the middle of a compressed stream and changing a few bytes without rewriting the whole file. So NTFS breaks the file up into pieces. The implication is that when a single large application read or write occurs, the filesystem must break this up into several smaller reads or writes, so the number of operations being sent to the device increase.

Depending on the device and workload, this will have different effects. It may be true that flash will handle this situation better than a spinning disk, both for reads (which don't have a seek penalty) or writes (which will be serialized anyway.) But more IOs increases the load on the host when compared to a single DMA operation.

(Then again, one must question the wisdom of using any journaling FS on flash...)


Although flash devices typically serialize writes, this alone is insufficient to provide the guarantees that filesystems need. At a minimum we'd need to identify some form of 'transaction' so that if we're halfway through creating a file we can back up to a consistent state. And then the device needs to ensure it still has the 'old' data lying around somewhere until the transaction gets committed, etc.

There's definitely scope for efficiency gains here, but we're not as close to converging on one "consistency manager" as many people seem to think.

Edited 2010-11-30 20:51 UTC

Reply Score: 1

RE[3]: Nitpicks
by gus3 on Wed 1st Dec 2010 00:11 UTC in reply to "RE[2]: Nitpicks"
gus3 Member since:
2010-09-02

The bigger problem with journaling on flash, is that so many FS operations need two writes: one to the journal, and then one to commit. Not good for flash longevity. In fact, early flash storage without wear-leveling got destroyed fairly quickly by journaling, when constant writing to the journal wore out the underlying flash cells in a matter of days.

Reply Score: 1

RE: Nitpicks
by phoenix on Tue 30th Nov 2010 18:15 UTC in reply to "Nitpicks"
phoenix Member since:
2005-07-11

Well, if you're going to be nit-picky, you might want to correct your usage of Tb to TB, since Tb is terabits. ;) ;) ;)

Reply Score: 2

Sparse files
by Drumhellar on Tue 30th Nov 2010 01:31 UTC
Drumhellar
Member since:
2005-07-12

I'm surprised he didn't mention sparse files.
I find those quite useful.

Reply Score: 2

rcsteiner
Member since:
2005-07-12

Microsoft's Gordon Letwin developed HPFS for the joint Microsoft and IBM OS/2 project, and I used it for years here with very few issues.

I've always found it interesting that MS allowed that filesystem to completely disappear from its radar. It was pretty good tech...

Reply Score: 3

sorpigal Member since:
2005-11-02

AFAIK it was IBM that owned HPFS. This probably explains why MS didn't want to use it: Why use a FS designed by people employed by your competitor when you have a perfectly good in-house FS?

Reply Score: 2

WinFS?
by quackalist on Tue 30th Nov 2010 02:50 UTC
quackalist
Member since:
2007-08-27

"WinFS was likely sacrificed to get a ship date for Vista."

Perhaps, but it was cancelled in 2006 and has yet to appear so I assume there must have been more to it then that. Pity, as I was rather looking forward to it at the time.

Seemingly, the more time passes and the greater the data even individuals have the more useful it would've been.

Edited 2010-11-30 02:55 UTC

Reply Score: 1

RE: WinFS?
by Bill Shooter of Bul on Tue 30th Nov 2010 07:48 UTC in reply to "WinFS?"
Bill Shooter of Bul Member since:
2006-07-14

It also didn't have very good performance. Longhorn in general was bad, but with winFS ... yikes, it was scary slow. At the time it was blamed on it being a debug build, then it was just dropped. Its basically been in development since the early 90s ( as a part of cairo).

I think its just a bad idea for an implementation. My 2 cents is that a better more specific, transparent database for specific filetypes is a better idea. Take a look at your typical Computer. Count all of the files on the Hard disks. How many of them have been created by the user, and how many are just for the OS and applications? If you could somehow apply indexing/database storage on the user created files, you'd have something scary fast and very useful. You can either buid some crazy intelligence into the FS, so every application has it with no new build ( Very difficult, IMHO) Or just create a new open file function in the OS, that designates it to be a "user generated file" and get all app developers to use that ( easier implementation, not going to work for a while...)

Reply Score: 2

RE[2]: WinFS?
by sorpigal on Tue 30th Nov 2010 13:08 UTC in reply to "RE: WinFS?"
sorpigal Member since:
2005-11-02

It would be trivial to mark files with metadata indicating that they are user-generated, then consume that metadata in any indexing tools. It's just that this would probably be so abused as to make the feature useless.

Reply Score: 2

RE[3]: WinFS?
by Bill Shooter of Bul on Tue 30th Nov 2010 15:46 UTC in reply to "RE[2]: WinFS?"
Bill Shooter of Bul Member since:
2006-07-14

When you are thinking of adding meta data, are you thinking that programs would do it or the OS? I don't think that would be trivial either way.

Reply Score: 2

RE[4]: WinFS?
by sorpigal on Wed 1st Dec 2010 12:00 UTC in reply to "RE[3]: WinFS?"
sorpigal Member since:
2005-11-02

Programs would do it. What's an OS? Certainly the kernel couldn't do this. It would be something each program would have to do, probably transparently to the user and programmer as part of a standard file API.

And, yes, it would be trivial.

Reply Score: 2

RE[5]: WinFS?
by Bill Shooter of Bul on Wed 1st Dec 2010 22:14 UTC in reply to "RE[4]: WinFS?"
Bill Shooter of Bul Member since:
2006-07-14

I was really wondering if you meant it woudl be transparent to the programmer or not. I don't think it would be trivial to have it Identify user created files in the api. In order for it to be transparent the function calls to save a config file ( non user created file) and a document ( user created file) woudl have to be the same while the api would have to figure out with some sort of algorithm which was created by the user and which was not.

As a programmer, some of the documents I create are indistinguishable from config files, because they are config files as well as documents I created. I'd want those to be all metadated up, but not any that were tweaked for me by an existing application I didn't write.

See the complexity of that? It would be easier to simply have a different api call for the two file writes, one with meta data and one with out, but then you have to get third party developer buy in.

Reply Score: 1

RE: WinFS?
by sorpigal on Tue 30th Nov 2010 12:44 UTC in reply to "WinFS?"
sorpigal Member since:
2005-11-02

WinFS was sacrificed because it's a good idea that just cannot be done practically.

Be tried to do the same thing with its early releases and found that they couldn't get it together, either, which prompted the rush to build BeFS before 1.0 shipped. It's unquestionably a hard problem and there is some question as to whether or not it can be solved in a way which is reliable and performant.

You will probably not see a database FS any better than BeFS any time soon.

Reply Score: 2

v Balls
by Luminair on Tue 30th Nov 2010 06:01 UTC
Ouch !
by Neolander on Tue 30th Nov 2010 07:22 UTC
Neolander
Member since:
2010-03-08

That's quite a heavy feature list ! Now I understand better why it took so much time before NTFS was properly supported on Linux ;)

In my opinion, a FS should focus on ensuring high performance and reliability, and the rest should be left to higher-level OS components as they are OS-specific features which all people who implement NTFS do not necessarily want to support. But well...

Reply Score: 2

RE: Ouch !
by AndrewZ on Tue 30th Nov 2010 14:09 UTC in reply to "Ouch !"
AndrewZ Member since:
2005-11-15

NTFS on Linux only supports a small number of basic NTFS features :-|

Reply Score: 2

RE[2]: Ouch !
by TheGZeus on Tue 30th Nov 2010 18:31 UTC in reply to "RE: Ouch !"
TheGZeus Member since:
2010-05-19

Indeed. I avoid write operations on NTFS volumes.
Well, I just avoid NTFS volumes...

Reply Score: 2

FAT32
by bazmail on Tue 30th Nov 2010 08:52 UTC
bazmail
Member since:
2005-07-25

Windows 95 OSR2 was the first consumer OS to support FAT32 (as opposed to Windows 95 vanilla as they were different products if essentially the same run time executive) OSR2 was OEM only and was also the first Windows version to support USB.

Good times.

Edited 2010-11-30 08:53 UTC

Reply Score: 1

RE: FAT32
by phoenix on Tue 30th Nov 2010 18:18 UTC in reply to "FAT32"
phoenix Member since:
2005-07-11

To be pedantic, it was Win95b that first supported USB. Win95a did not support USB. And Win95c was the best one of the bunch (aka OSR2.1 or something like that).

"OSR2" was an umbrella term that covered Win95a through Win95c. Supposedly, there was also a Win95d, but I never saw it in the wild.

Gotta love how MS names things, eh?

Reply Score: 2

Mistake in article
by bassbeast on Tue 30th Nov 2010 09:55 UTC
bassbeast
Member since:
2007-11-11

While it is a good and informative article, I thought I'd point out in the first graph that DOS was NOT written by Bill Gates, it was originally a CP/M clone called QDOS (Quick and Dirty Operating system) that was bought by MSFT for a one time payment of $50,000. While MSFT improved upon it later what they sold to IBM that became PC-DOS 1.0 was pretty much line for line QDOS.

Reply Score: 2

RE: Mistake in article
by sorpigal on Tue 30th Nov 2010 12:31 UTC in reply to "Mistake in article"
sorpigal Member since:
2005-11-02

I believe the table was saying not that DOS was written by billg but that FAT12 was, which is not something I can attest to but is reasonable.

Reply Score: 2

RE[2]: Mistake in article
by galvanash on Wed 1st Dec 2010 08:45 UTC in reply to "RE: Mistake in article"
galvanash Member since:
2006-01-25

I believe the table was saying not that DOS was written by billg but that FAT12 was, which is not something I can attest to but is reasonable.


Yes, Bill Gates wrote the original FAT12 implementation. FAT12 actually predates Tim Patterson's 86-DOS. FAT12 was implemented in Microsoft Disk BASIC-86 which he used for his S-100 micro that 86-DOS was built on, hence he used it for his file system.

Reply Score: 2

RE[3]: Mistake in article
by sorpigal on Wed 1st Dec 2010 12:02 UTC in reply to "RE[2]: Mistake in article"
sorpigal Member since:
2005-11-02

I know this, I just don't know if it was billg who wrote it or Paul Allen (or even someone else being paid by MS at the time).

Reply Score: 2

RE[4]: Mistake in article
by phoenix on Wed 1st Dec 2010 18:57 UTC in reply to "RE[3]: Mistake in article"
phoenix Member since:
2005-07-11

Uhm, "MS at the time" was just Bill G and Paul A. There were no "employees" until later.

Reply Score: 2

NTFS might corrupt your data
by Kebabbert on Tue 30th Nov 2010 10:27 UTC
Kebabbert
Member since:
2007-07-27

NTFS is not safe from a data integrity view. It might corrupt your data. The worst case of corruption is when the data is corrupted silently, i.e. the hardware nor the software never notices that data has been corrupted. Thus, you get no warning at all. You think your data is safe, but it has been silently altered on disk:
http://www.zdnet.com/blog/storage/how-microsoft-puts-your-data-at-r...



Using hardware raid is no better. It does not give you data protection. But ZFS gives you data protection. Read more on this, here:
http://en.wikipedia.org/wiki/Zfs#Data_Integrity

Reply Score: 1

NTFS: a wasted file system
by spiderman on Tue 30th Nov 2010 10:57 UTC
spiderman
Member since:
2008-10-23

It is technically good but it is wasted by closeness, patents and copyrights. Its value is greatly diminished by the fact that its structure is not published. You don't know if your disk will be readable in 10 years and you can't trust implementations based on reverse engineering. That is why FAT32 will remain the de-facto standard for removable drives. FAT32 is also closed but it is far simpler and reverse engineering is more reliable.
NTFS could have been much more useful but it has been wasted for very dumb business reasons. That is a shame.

Reply Score: 3

RE: NTFS: a wasted file system
by Drumhellar on Tue 30th Nov 2010 20:10 UTC in reply to "NTFS: a wasted file system"
Drumhellar Member since:
2005-07-12

You don't know if your disk will be readable in 10 years and you can't trust implementations based on reverse engineering.


Considering that it has been around for 17 years, and given Microsoft's long track record on providing backwards-compatibility, I don't think NTFS support will come anywhere near disappearing in the next 10 years.

Whether or not that's a good thing is a different argument.

Reply Score: 2

RE[2]: NTFS: a wasted file system
by spiderman on Tue 30th Nov 2010 20:27 UTC in reply to "RE: NTFS: a wasted file system"
spiderman Member since:
2008-10-23

If you had some documents written with the earlier version of MS Office you would say something different.

Reply Score: 1

Bill Shooter of Bul Member since:
2006-07-14

Comment deleted

Edited 2010-12-01 22:28 UTC

Reply Score: 2

NTFS could have been awesome
by Fergy on Tue 30th Nov 2010 11:39 UTC
Fergy
Member since:
2006-04-10

Every time I read about what NTFS can do I wonder why MS always tries to cripple it. It is limited by the software on top.
Apparently it could have:
- Longer than 255 characters(I have had to shorten a lot of directories and filenames)
- Automatic backup while the system is running
- File deduplication with hardlinks or something similar
- Smarter algorithms to avoid fragmentation
- Roll back the system to a clean install(without losing userdata)
- Grown the only partition on the system by just installing a new harddrive and linking it to it
- A tagging system built in(for an extra organization method on top of the filesystem)

I still find it hard to believe that MS could have solved 99% of the problems people have with Windows by using functionality already present in the filesystem...

Reply Score: 3

RE: NTFS could have been awesome
by Drumhellar on Tue 30th Nov 2010 20:14 UTC in reply to "NTFS could have been awesome"
Drumhellar Member since:
2005-07-12

The reasons why the best features in NTFS don't get used widely, I think, is mainly for the sake of compatibility.

For example, NTFS supports case sensitivity, but it is disabled by default because Win32 does not support case sensitivity for file names.

Reply Score: 2

AmigaRobbo Member since:
2005-11-15

'case sensitivity', a good thing?

Reply Score: 2

Drumhellar Member since:
2005-07-12

Actually, for anything other than passwords, probably not.

I think the way Windows handles it is best, from a usability standpoint, with case being preserved, but not mandatory.

Reply Score: 4

TheGZeus Member since:
2010-05-19

Allows two files with similar names to exist in the same folder, at the least.

Reply Score: 1

vivainio Member since:
2008-12-26

'case sensitivity', a good thing?


Yes.

If you insist on having case insensitivity for e.g. file names entered by user, do it in a library - open_file_case_insensitive("/TmP/Hello.TXt"). Don't break the whole operating system down to the very core for it.

While windows kernel probably is becoming ok these days, it's still crippled by various user-visible warts like:

- drive letters
- backslash recommended as path separator
- CR LF
- inability to delete/move files that are in use
- case insensitivity

Any pretense of elegance elsewhere is stained by these issues. Unix has warts too (terminal handling, whatever), but people rarely see those while the windows ones you hit several times a day.

Reply Score: 1

Um, what?
by sorpigal on Tue 30th Nov 2010 12:33 UTC
sorpigal
Member since:
2005-11-02

With Vista came linking features that were long available to the UNIX/Linux communities. A hard link looks and acts just like the target file or folder it points to. If you change an attribute on the link, you also change it on the target. If you delete the hard link, you delete the target.

Excuse me, but that does not sound like hard links as they have been known in Unix-land. What you have just said is that changes to the hard link file change the target, which is true since they are the same file, but it is certainly not true on Unix filesystems that deleting a hard link deletes the file its linked to.

Having just tested it on NTFS confirms my suspicions: NTFS doesn't do it this way either. Deleting all links to a file when you delete any one link would be crazy behavior. NTFS hard links work like Unix hard links.

Apart from inaccuracies like this I found the article usefully informative. I would like to see a comparison by someone who knows filesystems better than I do between NTFS and NSS, which was produces somewhat later, to contrast the different feature sets. I know that NTFS has relatively weak performance characteristics, possibly due to its complexity, and I know that NSS is pretty good in that area while having AFAIK an equivalent feature set.

Edited 2010-11-30 12:35 UTC

Reply Score: 3

I can't believe it!
by pmarin on Tue 30th Nov 2010 13:55 UTC
pmarin
Member since:
2006-12-30

An real article about Operating Systems in OSNews.

Congratulations!!!

Reply Score: 2

Windows 7
by Shannara on Tue 30th Nov 2010 20:59 UTC
Shannara
Member since:
2005-07-06

The chart on the front page is missing Windows 7 and the features added .. hmmm How long have window 7 been out? 2 years?

Is this a 2+ year article that was found a posted? I can't tell ...

Reply Score: 2

RE: Windows 7
by AndrewZ on Tue 30th Nov 2010 21:09 UTC in reply to "Windows 7"
AndrewZ Member since:
2005-11-15

Windows 7 has the same NTFS as Windows Server 2008, NTFS 6.0.

Reply Score: 2

Nice..
by Drunkula on Wed 1st Dec 2010 15:04 UTC
Drunkula
Member since:
2009-09-03

Good article, Andrew!

Reply Score: 1

Incorrect EFS Information
by ralish on Thu 2nd Dec 2010 12:36 UTC
ralish
Member since:
2009-05-13

I was looking at the EFS problems and most of these (none?) strike me as problems at all, but one of them is flat out incorrect:
# If your PC is not on a domain and someone gets physical access to it, it's very simple to reset your Windows password and log in.

This is true, however, resetting an account password from outside of the account (as opposed to logging in and changing the password) will invalidate the EFS keys bound to that account. The password for the user account is used to decrypt the relevant EFS keys, and thus, by changing it from outside of the account access to the EFS keys for that account can't be gained and will be permanently lost once the password is forcibly reset.

Windows will infact warn you of this when you try to do so, through, e.g. Local Users & Groups:
"Resetting this password might cause irreversible loss of information for this user account. For security reasons, Windows protects certain information by making it impossible to access if the user's password is reset. This data loss will occur the next time the user logs off."
This makes sense, as the EFS keys that are being held in their decrypted form in memory will cease to exist on log-off and their encrypted equivalents on disk will no longer be possible to decrypt.

In short, you'd have to bruteforce the password to guarantee access to EFS encrypted user data, not reset.

Disclaimer: Windows 2000 has some flaws in the EFS implementation and default settings that make some of the above less true. XP and newer are not vulnerable; Vista and newer even less so.

Edited 2010-12-02 12:36 UTC

Reply Score: 1

RE: Incorrect EFS Information
by AndrewZ on Thu 2nd Dec 2010 18:50 UTC in reply to "Incorrect EFS Information"
AndrewZ Member since:
2005-11-15

Thanks for that correction. It's good to know considering how easy it is to reset a Windows account password.

Reply Score: 2