posted by Andrew Hudson on Mon 29th Nov 2010 21:50 UTC

NTFS, 2/4


Alternate Data Streams

Alternate data streams have to be the single most under-used feature in NTFS. A data stream is simply the data that is retrieved when an application opens a file. For instance, if an image editor opens a JPEG image file it gets a data stream with the JPEG header information followed by compressed image data. This is a stream of data. The alternate data streams feature gives any file the ability to have multiple streams. Macintosh popularized this feature with its support of resource forks. Resource forks were used to store an application's icon, as well as the shape and positioning info for menus.

Under NTFS, each stream has its own opportunistic lock, file lock, cluster size, file size, and application association. But each stream shares the common file name and file permissions. Windows does actually use ADS to store file attributes. Right click on a file and select the Summary tab. You can add text to fields such as title, keywords, revision number, etc. These go into the alternate data stream called ?SummaryInformation.

Here are some examples you can try on your PC (input in bold, output regular):

C:UsersAndrew>Echo This goes into stream 1 > test.txt:stream1
C:UsersAndrew>Echo This goes into stream 2 > test.txt:stream2

C:UsersAndrew>More < test.txt:stream1
This goes into stream 1

C:UsersAndrew>More < test.txt:stream2
This does into stream 2

You can view stream properties of a file using dir /R

C:UsersAndrew>Dir /R
06/22/2010 11:01 AM 18 test.txt 19 test.txt:fork1:$DATA 17 test.txt:fork2:$DATA 1 File(s) 18 bytes

Data streams can be used to mix data and executables. For instance you can create a single file that looks like a data file but can also be executed:

C:UsersAndrew>Echo Data goes here >data.txt
C:UsersAndrew>Type test.exe>data.txt:data.exe

C:UsersAndrew>Type data.txt
Data goes here

C:UsersAndrew>Start ./data.txt:data.exe
Data.exe runs...

For a time malware creators used streams to hide their nefarious creations. It wasn't long before the anti-virus people caught on and added stream detection to their products. For an application to use an alternate stream, it must be written to do so. Most applications are not, and so alternate data streams are mostly unused on Windows. Sysinternals has a useful utility for this NTFS feature: streams.


File Screening

Strictly speaking, file screening isn't a file system feature, it's a file server feature. But it's an interesting feature so it gets a mention here. File screening makes it possible to create a server system policy making it possible to block specified file types. Don't want people downloading MP3 or video files on your system? Block ‘em. Here's how:

Open and expand the File Server Resource Manager, double click File Screen Management, Select the File Screens node, click on the Actions screen and then click on Create File Screen. In the File Screen Path dialog, select the folder where you want the screening to be enforced. E.G. C:Documents and Settings. Use Derive Properties from Template or Define Custom to define your screening parameters. Click Create to create the new screen.

You can create a variety of reports on file screening activities using the File Server Resource Manager.


Volume Mount Points

Volume mount points are useful mostly in server environments but are nonetheless interesting to know about and can have uses on PCs. A volume is just a partition that has a name. The most popular volume is of course the C: drive. While a home PC may have only one volume, a server environment may have dozens and dozens. What happens when you have partitions assigned to A: through Z: and you still need to mount more volumes? That's where mount points come in handy. In fact, mount points allow you to mount and manage many volumes without using the C: naming convention.

Using volume mount points allows you to transparently link multiple volumes together and here how: Click on the Window Start icon (lowest left), then Control Panel, System and Security, Administrative Tools, Create and Format Hard Disk Partitions. The Disk Management utility will appear. Select or create an empty partition. After selecting the size, a dialog will appear, select Mount in the following empty NTFS folder, and select the empty folder you want to mount it in. You can format the new volume and give it a name. After that the new volume is seamlessly mounted in the empty folder. The mount point is designated by an icon, and its explorer property is given as Mounted Volume.

Mounting a volume is a great way to get around the problem of outgrowing your C: partition. It's also a good way to extend the file system as a business grows. The mountvol utility provides the means of creating, deleting, and listing from command line and for scripts.


Hard Links, Soft Links, and Junction Points

With Vista came linking features that were long available to the UNIX/Linux communities. A hard link looks and acts just like the target file or folder it points to. If you change an attribute on the link, you also change it on the target. If you delete the hard link, you delete the target.

To create a hard link: Bring up the command line and type:

mklink /h New-Link Existing-Target

With the introduction of Vista, soft links replaced junction points. A soft link is a file that points to the target file or folder. A soft link can have access permissions different than the target, a soft link can link across disk volumes. A soft link can be deleted without affecting the target. A soft link can point to networked shares, a junction point could not. A junction point could only point to folders, a soft link can point to files.

Why use a soft link? Let's say you have some deep folder paths and you are tired of either typing long path names or clicking in Explorer. Create a link to where you want to go, and now you have a shortcut. More importantly you can use it to manage application versions or create a completely alternate folder system from an existing hierarchy. This is useful if you are developing code for two different platforms (E.G. Mac) and want to customize Windows.

To create a soft link: Bring up the command line and type:

Mklink New-Link Existing-Target

A soft link is listed in explorer as the white arrow overlay icon. Right click on the soft link and select Properties, then the Shortcut tab. The tab will list the soft link and list the path to the target.

Deleting Links: Caution is advised when deleting links. Use rmdir. Don't use Explorer to delete a link and don't use Del /s. Why not? Either of these will recursively delete the target and everything under it. Also be careful not to create a namespace cycle with links. This is essentially a circular folder path, and it could crash your anti-virus program or system utilities.


Fun With Logs

NTFS achieves a higher degree of error resilience using journaling, or logging. When a file is written to disk, several things must happen. Firstly, the space on the disk has to be allocated, then the data is written to the disk as a cluster of sectors, then the metadata is updated to reflect the location on the disk, the creation time, the file size, the owner, the permissions, etc. If the power goes out, or the system crashes before this process is completed several bad things could happen. The space could be allocated and written to but not updated. This would result in orphaned clusters. If the file existed and was being updated, it could be truncated and the metadata not updated. Leading to a file that was shorter than the listed file size. In the days of FAT these errors would be fixed with chkdsk. Constantly.

Journaling prevents these problems by always keeping the file system in a constituent state. It does this by logging file system updates to the $LOGFILE journal before each writing to the file system. The first thing a write does is set the volume dirty bit on. Each operation that modifies a file on an NTFS volume is processed as a transaction, stored in $LOGFILE, and separated by checkpoints. Each transaction is independent from any other, and is processed by the Log File Service. The Log File Service creates Redo and Undo entries in $LOGILE. A Redo allows a transaction to be rolled forward, or completed. An Undo allows a transaction to be rolled back without damage to the file system.

If the system crashes during a write operation, Windows detects the dirty bit on reboot, initiates the Log File Service and performs 3 passes on $LOGFILE. The first pass is an analysis to determine if any clusters need to be corrected, the second performs outstanding redos, and the third performs undos on transactions that cannot be completed.

The following operations modify an NTFS volume and create journal transactions: creating, deleting, extending, truncating, setting file information, renaming, and changing security.

While journaling prevents NTFS from becoming inconsistent with its metadata, journaling does not prevent data loss. If your PC powers down in the middle of a lengthy write operation, you are still likely to lose data. But at least you can now avoid a lengthy chkdsk on startup and the possibility of a corrupt file system.

There aren't that many practical things you can do with $LOGFILE, except maybe win points discussing it with other sysadmins. Nonetheless a useful utility for file system analysis is FSUTIL. FSUTIL is a Swiss arm knife that deals with update sequence number journals, volume management, quotas, hard links, file system info, and a wide variety of file system parameters.

Click on the Windows start icon (lowest left), type CMD into the Search Programs and Files line, Under the Programs line, right click on cmd.exe and select Run as Administrator. On the command line type:

fsutil fsinfo drives - list all drives
fsutil dirty query C: - check if the dirty bit is set on C:
fsutil fsinfo statistics C: - list stats on your C:
fsutil behavior set disablelastaccess 1 - a nasty hack that turns off updating of last access time and improves file system performance. Can affect backup programs.

Table of contents
  1. NTFS, 1/4
  2. NTFS, 2/4
  3. NTFS, 3/4
  4. Interview: Dr Gary Kimura, 1/4
e p (15)    91 Comment(s)

Technology White Papers

See More