The vfat file system and Linux

Recently I stumbled across a very nice article, written by Torsten Scheck, published on pro-linux.de, a German Linux site. This article proved to be so helpful to me that I decided it would be worthwhile to translate it into English and republish it. Comments of the translator will be added in italics. I hope a lot of people will find this little gem as useful as I did…

The vfat file system has been supported by Linux for years now. Still there are a lot of obstacles. I want to show, with the help of some practical examples, these obstacles and possible solutions and workarounds.

Table of contents

1. Introduction
2. Replication of ext3 and vfat partitions via rsync
   2.1. Mounting Parameters
   2.2. Adjusting the system time
   2.3. File size limits
   2.4. Using rsync
3. Problems with large vfat file systems
   3.1. Lost clusters
   3.2. dosfsck and high RAM Demand
   3.3. Executing dosfsck
   3.4. Formatting a large vfat file system
4. Conclusion

1. Introduction

Many Linux users dislike the use of vfat. There are ample – and technically superior – alternatives for Linux users. Additionally, there’s a judicial aspect: Microsoft’s attempt to demand license fees due to a patent on a component of vfat leaves a bad after taste in many mouths.

Sadly, vfat is still the easiest way to share data between Linux and the somewhat limited Windows-world. With the soaring popularity of mobile data carriers, like CompactFlash-cards, USB-sticks and external Firewire (or USB2.0)-hard drives the significance of vfat as the least common denominator even rose.

The following article is based on experience that I collected working with two external 250GB hard drives (Maxtor 5000XT). While one of these discs held an ext3 partition and served as my primary data storage, the other one, holding a vfat partition, was used as backup drive and as the connection to the "Windows-world". I pretty much do the same thing: I have an external 60GB USB2.0-hard drive that holds an ext3 partition for everyday backup and a vfat partition, also for backup and to have my data easily available on any Windows machine.

I recently decided to merge my experiences into an article because I believe that enough Linux users will have the same problems that I had. So I hope that this article will save many hours of difficult problem analysis. Special thanks to Uwe Menges who helped me throughout my little vfat odyssey.

2. Replication of ext3 and vfat partitions via rsync

The characteristics of the vfat file system that are discussed in this section are of course not limited to the work with rsync. But rsync serves as a particularly good example to show a lot of possible challenges with vfat.

For those who don’t know: rsync checks the date of the last change and file size to decide if a file that already exists on the destination folder has to be overwritten with the according file on the source folder. Using only Linux file systems, "rsync -av source/* destination" ensures that all files on the source are replicated identically on the destination. When running this command immediately again, no files should be transmitted.

To be able to use rsync between an ext3 and a vfat partition, there are some requirements to fulfill, and some of those are quite subtle.

2.1. Mounting Parameters

If you mount a vfat partition without any special parameters, a normal Linux user will not have write access. Also there will be a problem because vfat treates MS-DOS filenames (so called 8.3 filenames like FILENAME.EXT) differently:

	> mount /dev/sda1 /myvfat -t vfat
	> mount | grep sda1
	/dev/sda1 on /myvfat type vfat (rw)
	> grep sda1 /proc/mounts
	/dev/sda1 /myvfat vfat rw,nodiratime,fmask=0033,dmask=0033 0 0
	
	> cd /myvfat
	> touch ABCDEFGH
	> touch ABCDEFGHI
	> ls -l
	-rwxr--r--     1 root root     0 Dec 31 16:05 ABCDEFGHI
	-rwxr--r--     1 root root     0 Dec 31 16:05 abcdefgh
	
	Note the use of small letters - - - - - - - - - ^

Obviously, rsync will not like it when some filenames are suddenly written in small letters just because they coincidentally fit into the MS-DOS filename schema. These files will be transmitted on every synchronization attempt, even when they were not changed at all because rsync cares about upper and lowercase letters. Responsible for this problem is the standard mount parameter "shortname=lower".

The solution to this problem is to add the mount parameter "shortname=mixed". Now short MS-DOS-type filenames are displayed exactly the way they got created on the vfat partition. International (e.g. non-english) users may want to set their according codepage ("codepage=850" for German users as an example). This is important to keep correct special characters in the filenames of said shortname files. In practice, keeping the standard, "codepage=437" should be fine because the majority of the filenames are long ones anyway and these are stored in unicode. With the standard mount parameter "iocharset=iso8859-1" these are transformed into 8-bit filenames. If you use a unicode system (e.g. Fedora Core) you have to set "iocharset=utf8" explicitely.

File system permissions depend on the umask of the mount-user root and his user- and group-ID. To gain write access to the vfat partition also as normal user, set "umask=002" to give rights to read, write and execute to members of the group. All the other users do not have write access to the partition (can’t be bad to treat a multi-user system accordingly!) Now add the parameter "gid=100" so all the files stored on the vfat partition belong to the group "users" (at least on my Debian system). Additionally, we’ll add the parameter "uid=1000" to make sure that files copied from our source to the vfat partition don’t get "root" but the actual user as owner. (On my system, 1000 is the user-ID of the main user "t" who is member of the group "users"). On my Fedora Core 3 system, I used "uid=500" and "gid=500" which is my user and group-ID)

	> mount /dev/sda1 /myvfat -t vfat -o shortname=mixed,codepage=850,umask=002,uid=1000,gid=100
	> mount | grep sda1
	/dev/sda1 in /myvfat type vfat (rw,shortname=mixed,codepage=850,umask=002,uid=1000,gid=100)
	> grep sda1 /proc/mounts
	/dev/sda1 /myvfat vfat rw,nodiratime,uid=1000,gid=100,fmask=0002,dmask=0002,codepage=cp850,shortname=mixed 0 0
	
	> cd /myvfat
	> ls -la
	-rwxrwxr-x     1 t    users    0 Dec 31 16:05 ABCDEFGH
	-rwxrwxr-x     1 t    users    0 Dec 31 16:05 ABCDEFGHI

If you want to add these options to your /etc/fstab, you may want to add the parameters "noauto" and "user", so that the file system does not get mounted automatically at system start and can be mounted by a normal user. The parameter "user" implies, for security reasons, also "noexec", "nosuid", and " nodev". This, however, should not be a problem in our example, because we assumed to deal with an external hard drive for pure data storage. If you want to execute programs on the vfat partition, add the parameter "exec". As an optimisation you can turn off the updating of the last file access mit the parameters "noatime" and "nodiratime", if you don’t need this information. Personally, I do use this information, for example to find (with "find -atime -21") the audio files that I listened to during the last three weeks.

The resulting /etc/fstab-entry looks like this:

	/dev/sda1 /myvfat vfat shortname=mixed,codepage=850,umask=002,uid=1000,gid=100,noauto,user 0 0

	> mount /myvfat
	> mount | grep sda1
	/dev/sda1 on /myvfat type vfat (rw,noexec,nosuid,nodev,shortname=mixed,codepage=850,umask=002,uid=1000,gid=100)
	> grep sda1 /proc/mounts
	/dev/sda1 /myvfat vfat rw,nodiratime,nosuid,nodev,noexec,uid=1000,gid=100,fmask=0002,dmask=0002,codepage=cp850,
shortname=mixed 0 0

Further information can be found in the man-pages of the "mount"-command, especially in the sections "Mount options for fat" and "Mount options for vfat"

2.2. Adjusting the system time

A very sneaky problem occurs for everyone who uses a system time like "Europe/Berlin". Contrary to vfat, Unix file systems do take leap seconds in the past and daylight savings time into account. The time of the last file change of a file created in January would be deferred by one hour in June. As a consequence, rsync would transmit all files on every clock change.

An example:

	# On vfat during daylight savings time the date gets supplemented with the current time zone, thus being forged by one hour.
	> TZ=Europe/Berlin ls -l --time-style=full-iso /myvfat/testfile 
	-rwxrwxrwx [...] 2003-11-26 02:53:02.000000000 +0200 

	# On ext3 the date is displayed correctly also during daylight savings time.
	> TZ=Europe/Berlin ls -l --time-style=full-iso /myext3/testfile 
	-rw-rw-rw- [...] 2003-11-26 02:53:02.000000000 +0100 

As I did not find any mount parameter to change the time zone, I adjusted the system time to a zone that does not have daylight savings time (e.g. UTC) and set the local time zone "Europe/Berlin" for all users. As a consequence, however, all syslogd time stamps also use UTC instead of the local time zone.

Debian users can adjust this via "base-config". "Configure timezone" to "None of the above" and "UTC" gets the job done. Afterwards it should look like this:

	> cat /etc/timezone
	Etc/UTC

	> ls -l /etc/localtime
	[...] /etc/localtime -> /usr/share/zoneinfo/Etc/UTC

	# ~/.bash_profile
	export TZ=Europe/Berlin

2.3. File size limits

Linux only supports files up to 2GB size on vfat. You can at least backup bigger files by splitting them up into smaller parts. In the following example, I am backing up a 5GB-cryptoloop file to a vfat partition by splitting it into 2000MB pieces:

	> cd /myvfat
	> split -d -b 2000m /myext3/cryptfile cryptfile_backup
	> ls
	cryptfile_backup00 cryptfile_backup01 cryptfile_backup02 

Files bigger than 2GB must be excluded from rsync of course.

Reassembling the original file can be done via:

	# Linux
	> cat cryptfile_backup00 cryptfile_backup01 cryptfile_backup02 > cryptfile
	# DOS
	> copy /b cryptfile_backup00 + cryptfile_backup01 + cryptfile_backup02 cryptfile

2.4. Using rsync

The characteristics of vfat must also be considered when calling up rsync. As vfat does not support symbolic links, file permissions, owners, groups and devices, usage of the parameter "-a", which considers all of the above will not have any effect (aside from the error messages). Thus it’s best to use only the parameters that actually work:

	-r,	--recursive   treat folders recursively
	-v,	--verbose     show transmitted files
	-t,	--times       keep time settings
	-n,	--dry-run     test only
		--exclude     ignore files

As the date of the last file change is very important for rsync, the option "-t" is essential. It’s also very wise to test every rsync change with "-n" before.

The last roadblock to a successful synchronization with rsync is the time resolution of the vfat time stamp. It amounts to a little more than one second. You’ll have to set the parameter "–modify-window=1" to gain a tolerance of one second, so that rsync isn’t "on the dot".

In a nutshell, the command to efficiently transmit all files from an ext3 file system to a vfat file system is:

	> rsync -rvtn --modify-window=1 --exclude "lost+found" /myext3/* /myvfat

3. Problems with large vfat file systems

The rapid growth of hard drive size poses big problems for the rather neglected dosfstools and vfat-driver.

3.1. Lost Clusters

Under Linux Kernel 2.4.x., a limit of the cluster data type results in data loss, as soon as the vfat file system holds around 130GB. In Kernel 2.6.x., this problem was – rather accidently – solved, when many variables were consequently provided with a new type. A detailed description of this bug, including a testsuite and a patch (by Erik Andersen) can be found here. (The patch also allows for file sizes up to 4GB).

If you, however, work with a 2.4.x. Kernel and have a "full" vfat partition, be prepared to lose data: any written file in a new folder will be lost after unmounting the file system. When you mount the file system again, these files have a size of 0 and the clusters are in limbo. You can delete the unassigned clusters via dosfsck.

3.2. dosfsck and high RAM Demand

To conduct file system checks as efficiently as possible, dosfsck copies both FATs to RAM. With a very large file system on a 250GB drive, the high number of clusters yields a very high demand for RAM, that surpassed my 350MB (including swap). Thus dosfsck aborted with a malloc-error. Roman Hodek, the maintainer of dosfsck, proposed to convert the program to "mmap()", but also said that this change would be complex. As long as this situation has not changed, be sure top have sufficient RAM.

3.3. Executing dosfsck

As long as the vfat file system is mounted, dosfsck can be executed, but all repairs silently fail. Thus you should make sure that your partition is not mounted befora using dosfsck. In the following example, an unassigned cluster (due to the bug in Kernel 2.4.x.) is located and deleted. By the way, the command fsck.vfat is a symbolic link to dosfsck.

	> fsck.vfat -vr /dev/sda1
	dosfsck 2.10 (22 Sep 2003)
	dosfsck 2.10, 22 Sep 2003, FAT32, LFN
	Checking we can access the last sector of the filesystem
	Boot sector contents:
	System ID "mkdosfs"
	Media byte 0xf8 (hard disk)
		512 bytes per logical sector
		16384 bytes per cluster
		32 reserved sectors
	First FAT starts at byte 16384 (sector 32)
		2 FATs, 32 bit entries
	39267840 bytes per FAT (= 76695 sectors)
	Root directory start at cluster 2 (arbitrary size)
	Data area starts at byte 78552064 (sector 153422)
		9816944 data clusters (160840810496 bytes)
	63 sectors/track, 255 heads
		0 hidden sectors
		314295660 sectors total
	Checking for unused clusters.
	Reclaimed 1 unused cluster (16384 bytes).
	Checking free cluster summary.
	Free cluster summary wrong (641900 vs. really 641901)
	1) Correct
	2) Don't correct
	? 1 
	Perform changes ? (y/n) y
	/dev/sda1: 143 files, 9175043/9816944 clusters

3.4. Formatting a large vfat file system

When formatting with mkfs.vfat you have to add the option -F 32, so that a 32-bit file system is created. Without this option, a 12-bit or 16-bit file system is created, depending on the partition size, or the formatting process aborts (on an oversized partition). Fat16 only supports file systems up to 2GB, fat32 allows for up to 2TB (terabytes).

	> mkfs.vfat -F 32 /dev/sda1

4. Conclusion

Solving the problems described here cost me a lot of time. But to me, being able to perform my work exclusively with Free Software is a luxury that makes it quite worthwhile. Thus I want to thank all developers of these programs heartily. If the psychological strain of the aforementioned problems grows big enough, there will some volunteers who will approach the remaining problems.

© Torsten Schenk ([email protected])

License:

This text is subject to the GNU Free Documentation License (FDL). Free spreading in modified or unmodified form is allowed. Modifications must be marked unmistakeably and also distributed under the FDL.

Translated by Mag. Christian Paratschek. More of my work can be found on my website.

---

If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.