HOWTO: Increase write speed by 'aligning' FAT32

[SpeeDyLOCO]

Hi, very good description. You have done a good job:) But I have a question. Did you try this procedure but formatting to NTFS?

[kd8cgo]

NTFS I have not played with - I primarily use Linux at home, and maintain the USB stick in FAT32 format to keep a bootable drive that can be accessed on most all Microsoft operating systems - I use the flash drive mainly for computer repair work. NTFS would be great if I needed to transport large files regularly to Microsoft laden computers, I just don't have that need at this time.

I suspect that, if you have your partition tables aligned like in post #1, that NTFS itself will also be aligned, but I am not sure. NTFS does not have special reserved sectors or FAT tables to contend with before the clusters are made for the files system itself as far as I know. The MFT and other file system data are stored in files themselves inside a NTFS formatted partition, so one could assume that the clusters would start right at the beginning of the partition, making them aligned by default with the partition start sector.

[the_hand]

Good information. However, I have a problem calculating the reserved sectors for my SD. My FAT size is 6400 sectors, which means, after the math, I need 0 reserved sectors to align. Since -R 0 is not acceptable, I couldn't figure out how many sectors I need to reserve for the alignment.

[CitizenJohn]

Good information. However, I have a problem calculating the reserved sectors for my SD. My FAT size is 6400 sectors, which means, after the math, I need 0 reserved sectors to align. Since -R 0 is not acceptable, I couldn't figure out how many sectors I need to reserve for the alignment.

According to the mkfs.vfat man page:

-R number-of-reserved-sectors
Select the number of reserved sectors. With FAT32 format at least 2
reserved sectors are needed, the default is 32. Otherwise the default is 1
(only the boot
sector).

Since mkfs.vfat doesn't allow a reserve of zero sectors, you will have to move up to the next higher erase block boundary. (131,072 / 512 = 256 sectors per erase block) Try using -R 256. This should produce the same alignment as -R 0, just one block higher.

[CitizenJohn]

Even though his procedure did not work for me, I'd like to express my thanks to kd8cgo for the fine job of presenting the problem and proposing a solution. Without that I would not have stopped to consider the problem and come up with my somewhat different approach.

First, I'll tell you my experience with kd8cgo's process. I formated the drive using fdisk, exactly as kd8cgo so well described.

Then I turned to finding the correct number of reserved sectors to make the data align to the erase block boundaries. According to his calculations I needed to use 96 reserved sectors in the mkfs.vfat command, -R 96.

This is what my mkfs.vfat command produced. (Notice the slightly different numbers from those produced by kd8cgo's drive.)


john@Dell2400:~$ sudo mkfs.vfat -F 32 -R 96 -n MultiBoot -s 32 -v /dev/sdb1
mkfs.vfat 2.11 (12 Mar 2005)
/dev/sdb1 has 224 heads and 56 sectors per track,
logical sector size is 512,
using 0xf8 media descriptor, with 31271936 sectors;
file system has 2 32-bit FATs and 32 sectors per cluster.
FAT size is 7632 sectors, and provides 976768 clusters.
Volume ID is 4df59217, volume label MultiBoot .

After this I did what most of you would have done, tested to see the results. I copied a 457,713,753 byte file from my system's hard drive to my reconfigured flash drive. The results weren't encouraging: 97 seconds as I recall. 457,713,753 / 97 = 4.72 MB/sec.

I checked my work and believed it to be according to instructions. But what had gone wrong? It looked as if I would have to take a lot of time to relearn all that I had so gladly forgotten about the FAT32 file system, and I almost let the whole thing go. But one thing did come back to me from that distant past: the data portion of a VFAT partition starts immediately after the FAT tables.

I decided to check my alignment by recreating the file system again, using the same number of reserved sectors, 96. Then I wrote a short file with some fairly unique text in it. Since this would be the first thing written after creating a new file system it would end up as the first thing in the data area of the partition.

Then I would use the dd and grep commands to search for my fairly unique text. That would give me the address of the start of the data area, which I could divide by 131,072 to find out if it was located exactly on an erase block boundary. When I made those calculations I found the data portion of my file system started 32 sectors into an erase block.

I went back to my previous mkfs.vfat command and subtracted 32 from the "-R 96" I had used, to get "-R 64". After running that mkfs.vfat command, when I copied the same test file to my Patriot drive the results this time were impressive, 37 seconds. 457,713,753 / 37 = 12.37 MB/sec. A savings of one whole minute, and an increase in write speed from 4.72 MB/sec to 12.37 MB/sec, a 262% improvement.

Thanks to the work of kd8cgo I had been inspired to find my own solution to this important question. I'm perfectly willing to admit that I may have failed to accurately follow his procedure, but being actually able to see what was happening on the disk made me much more confident of success than the prospect of diving into the dark, murky waters in which the details of VFAT are shrouded.

In order to make this work it was necessary to use some pretty murky options to the grep command. I've already converted my drive to Linux ext2 file system, and don't want to undo that for the sake of a perfect demonstration. I have copies of some of my command line sessions and will quote those that I have. The rest I'll have to talk my way through.

It was not necessary to reformat the drive using fdisk.

In order to remove all traces of the testing I had done, it was necessary that I remake the file system using the mkfs.vfat command. I used the same command that I had used before with -R 96.

When that was completed, I removed the drive (It had been unmounted while making the new file system, using umount /dev/sdb1.)

I then reinserted the drive and allowed the system to mount it.

Then I went to the directory where the drive had been mounted. (Use the mount command to see where the system mounted it.)

On my system the drive ended up at /media/MultiBoot.


cd /media/MultiBoot
ls (Command should find no files.)
Create the first file on the file system as follows:
echo thisxxx > thisxxx.txt
ls (Command should show only this file: thisxxx.txt)

Next we want to find the address of the "thisxxx" text which should be at the very beginning of the data area for the sdb1 partition.

Since the whole drive is divided into erase blocks the search should start at the very beginning of the drive, so use /dev/sdb and not /dev/sdb1 in the dd command.

sudo dd if=/dev/sdb | grep -oba thisxxx

Very soon the following line should appear on screen (with different numbers, of course):
8011776:thisxxx
When it does use control-c to stop the command.

Use the dc command to enter the following information exactly as shown except for using your own number rather than the one above given as an example:


john@Dell2400:/media/patriot$ dc
3k
8011776 131072/p
61.125
61-p
.125
131072*p
16384.000
512/p
32.000
96r
-p
64.000
q
john@Dell2400:/media/patriot$

The 61-p isolates the remainder part of the number on the line above it.
The line 32.000 is the number of sectors past the erase boundary that the data starts. Subtracting that from the 96 sectors used to create this files system yields 64. So the following command should be used to remake the file system using your own calculated reserved sector number, in place of the example, 64:


sudo mkfs.vfat -F 32 -R 64 -n MultiBoot -s 32 -v /dev/sdb1

Remove the drive and reinsert it to have the system mount it.

Just in case there was a mistake, create another data area marker like this:

echo thiszzz > thiszzz.txt

Then run your tests. It should work. If not check again to make sure the text is at a boundary. Keep trying until it does. 262% improvement is worth the effort.

Good luck.

[CitizenJohn]

Please read The Ted Ts'o article on SSD alignment (http://thunk.org/tytso/blog/2009/02/20/aligning-filesystems-to-an-ssds-erase-block-size/) before you start, so you can understand a bit of what it is we are doing here! He has more information in there about LVM and EXT4 file systems, which we won't be using on a USB stick (we could use EXT2/3/4 on a stick if it is for Linux use ONLY)

The link to Ted Ts'o's article is broken. Click here (http://ldn.linuxfoundation.org/blog-entry/aligning-filesystems-ssd%E2%80%99s-erase-block-size) for a copy of the original article.

John Gunn
Coronado, CA

[c.cobb]

When using the Grub2 boot loader remember that, in addition to the MBR boot code, it also installs a 'kernel.img' file into the reserved sectors area. This file is around 30 KiB so be sure to allow for this extra space and, if necessary, increase the '-R' value by one sector as mentioned above (http://www.patriotmemory.com/forums/showthread.php?3696-HOWTO-Increase-write-speed-by-aligning-FAT32&p=41507#post41507).

If there's not enough space when you attempt to install Grub2, you will see an error that says the "Embedding area is too small," at which point your USB stick's MBR has already been updated but the Grub2 kernel (http://www.gnu.org/software/grub/manual/grub.html#BIOS-installation) has not yet been installed (that's -1 for Grub2).

[PATRIOTXT]

Why are all the default LBA Flash Block alignment values at 8064? Alignment doesn't seem to improve on anything for me.

[Timothyier]

NTFS, I have not played with - I use most often at home Linux and maintain the USB drive to FAT32 in order to create a boot floppy, which can be found on most all Microsoft operating systems to keep creating - I use the stick, especially for the computer repair work. NTFS would be great if I have to transport large files regularly loaded with computers, Microsoft took, I did not know that yet need.

I think if the partition tables aligned, as in post # 1 that NTFS be adapted themselves, but I'm not sure. NTFS has to deal with any specific areas or FAT tables cluster for the files themselves, as far as I know, are created. The MFT file and other information in the files themselves are stored in an NTFS partition, and could therefore be assumed that the cluster would be right at the beginning of the partition to start, so they are aligned with the boot sector of the partition.

[kd8cgo]

It seems my original 2nd post was deleted! I don't know how, but I had saved this in a Google Doc when I first created it, since the entire thing wouldn't fit in the 10,000 character maximum that Patriot Forums allows. I hope this information can be preserved, there are a lot of sites that link to this information across the Internet.

I have made the original Google Doc public. I also made a revision, as CitizenJohn pointed out I believe there was an error in the first version on how to calculate the reserved sectors - this has been changed in the Google Doc and -should- be correct now, I hope! I'm not sure, since the original seems to have disappeared!

For your viewing pleasure, the original document:

https://docs.google.com/document/d/1FBHLnaJJ1Ev5b90lrNpJ6BUaQFvAFrrUFYErlhtIuLM/edit