Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: HOWTO: Increase write speed by 'aligning' FAT32

  1. #1
    Join Date
    May 2010
    Posts
    6

    Post HOWTO: Increase write speed by 'aligning' FAT32

    Hello,

    I like many who visit this sub-forum, do so because they, or a program they used, formatted their USB stick at some point - afterwards noticing a substantial decrease in write performance on the drive. You can fix this by aligning the FAT32 file system to the 128K "blocks" or "cells" that flash memory uses. After much trial and error, and not being an expert in file systems, I was able to read up on the subject and figure out how to align my file system on the drive. The tools you need to perform these operations should be readily available in most any Linux distribution - don't let that scare you away, you can burn a Linux install CD and boot off of it to perform all of these commands without even needing to install Linux on your computer, for any that support a "Live CD" or "Try First before Install" feature, for instance Ubuntu is a popular distro that supports this feature. People who already understand aligning partitions, but want to align a FAT32 format to the aligned partition, can learn how in Step 3.

    Following this procedure will destroy all contents on your USB stick, back up important files BEFORE starting - you have been warned.

    Step One: Identify the drive and the device
    Insert the stick into your computer after booting into Linux. Most distributions will automatically mount the file system currently on the drive. Mounting is like when Windows detects your drive and gives it a drive letter, same thing. Linux does not use drive letters, instead it uses a system of folders, which is much cleaner and more organised. Depending on your distro, you fill most likely find the drive mounted under /media/USBstickname or /mnt/USBstickname. Many distros will name the folder of the USB stick whatever the volume label was set to, i.e. "Patriot" for a stock formatted USB stick.

    Next, open a terminal window if you are using a GUI, most modern Linux will be using a GUI, if you need this HOWTO: then you probably will be using a GUI. Find Terminal or Command Line or similar on the menus.

    using the command df, which stands for 'disk free' we can see our drives free space, but it is also a convenient way to find the device name and mount point for our drive. It is important to identify the proper disk here; you don't want to format your hard drive now do you?

    We will use df with a option, -h - this will produce so-called human readable information, in terms of Gigabytes and Megabytes, and not just long strings of bytes. Lets try:

    username@computername:~$ df -h

    Produces the following display: (simplified here)

    Code:
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda3              78G   37G   37G  50% /
    /dev/sdd1              15G  2.3G   13G  15% /media/MultiBoot  <<< My USB Drive
    Here we can see a few key things:

    1. My USB device is /dev/sdd (/dev/sdd1 is the first partition of /dev/sdd)
    2. My drive is mounted at /media/MultiBoot
    3. The drive size is reported as 15G, which matches up to my Patriot Xporter XT Boost 16GB

    This is all the information we need to move on to the next step.

    Step 2: Unmount and prepare the partitions

    First we unmount the drive, so that we don't have any files open and we can modify the drive at will.


    username@computername:~$ umount /media/MultiBoot

    If the command is a success, it will return you to the prompt without saying anything usually.
    Note:
    If you had a drive window open in the GUI with the files on the USB showing, it will close, or sometimes change to another path, since you just removed the path it was using.
    Now we can use a cool program called fdisk to modify the partition table on the usb drive - this is sometimes referred to as the Master Boot Record or MBR for short, although technically, the partition table is only one part of the entire MBR itself. fdisk is a powerful and somewhat advanced program, but don't let that scare you: the worst that could happen is you could inadvertently destroy the contents of your entire hard disk(s) in your computer if you are not careful. As long as you pay attention to what you are typing, and don't blindly type in commands without understanding what each command does, you are in little danger.
    Why are we changing partitions? Formatting doesn't usually change partitions!
    We are, because the standard partition table that Patriot has used is not 'aligned', at least not for our purposes. For instance, I checked my friends stock Patriot USB drive and it was quite 'odd' and none of the units lined up to a 128K boundary. To make our jobs easier, we will align the drive from the ground up so to speak, so that we KNOW where everything is while lining up data on our disks.
    Please read The Ted Ts'o article on SSD alignment before you start, so you can understand a bit of what it is we are doing here! He has more information in there about LVM and EXT4 file systems, which we won't be using on a USB stick (we could use EXT2/3/4 on a stick if it is for Linux use ONLY)

    A neat trick is to make our heads and sectors per track line up by using 224 heads and 56 sectors per track. A USB stick is not a hard drive and it does not have sectors, tracks, OR heads - but it pretends to be a hard drive, and so, it has a drive geometry just like any hard drive would. This geometry business of heads/sectors/tracks and so on has no physical relationship to actual physical properties on our USB stick, it just helps to make a convenient way to talk about how data is organised on the drive. Note that there are other combinations that also make 128K alignment easy, but this is the one mentioned in several SSD and flash articles around the web. You will get different speed results by using different geometries, so if you are the inquisitive type, try several and see what you like best.
    Understand the "Sector"
    The sector is something that you need to understand to proceed. It is the lowest or smallest size chunk of a disk that you access at the lowest levels while doing these things we are doing. A sector holds 512 bytes of information (very small). This small paragraph contains 372 bytes of information, and would just fit into one sector of a disk!
    We will be using sudo along with fdisk in order to give the program 'super user' or in Windows speak 'admin' privileges. Sudo will often ask you for your password to continue.

    user@host:~$ sudo fdisk -H 224 -S 56 /dev/sdxxx

    That command does:

    1. sudo: use admin rights
    2. fdisk: start the fdisk partition program
    3. -H 224 : set the heads geometry to 224 heads
    4. -S 56 : use 56 sectors per track
    5. /dev/sdxxx : the device we wish to change, in this case you fill in the xxx with your device from step one.

    The command will display something similar to:

    [sudo] password for username:

    WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
    switch off the mode (command 'c') and change display units to
    sectors (command 'u').

    Command (m for help): p

    We type 'p' at this prompt to display or 'print' the current partition table: You may want to type 'm' here so you can see a listing of all commands available in fdisk. Here's an example of 'p':

    Code:
    Disk /dev/sdd: 16.0 GB, 16030629888 bytes
    224 heads, 56  sectors/track, 2496 cylinders
    Units = cylinders of 12544 * 512 =  6422528 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O  size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier:  0x9a8001cf
    
    Device Boot      Start         End      Blocks   Id  System
    /dev/sdd1    *           1        2496    15654784    c  W95 FAT32 (LBA)
    
    Command  (m for help):
    Your display will be different, this shows a drive that has already been configured for the 224/56 alignment.

    You will need to perform the following steps in fdisk:

    1. Delete the existing partition
    2. Create a new partition that starts at cylinder 1 and ends at the end of the disk
    3. Mark the partition as a FAT32 (LBA) partition type, which is hex type "0x0C".
    4. Enter expert mode and change the beginning of data to sector 256. (Sector 256 x 512 bytes/sector = 128K (131,072 bytes)
    5. Return from expert mode and write the changes to disk.
    6. Press q to quit at any time while in fdisk before writing your changes if you feel uncomfortable or don't understand what you are doing.

    I don't want to provide a step by step here, since it would be too easy to just blindly type it all in and mess something up. Read, learn and understand what you are doing before you jump in! Research fdisk (man fdisk) and get to know the program. It's easy, but it takes a little learning to grasp the concepts. The information above will guide you easily along once you know fdisk.

    Once you have written the changes, your partition table is aligned and you can begin formatting...

    It seems someone deleted the 2nd post with the critical information about the FAT32 alignment process!
    Last edited by kd8cgo; 12-10-2012 at 05:24 PM. Reason: sabotage of the internets!?

  2. #2
    Join Date
    May 2010
    Posts
    6

    Thumbs up Step 3: The Final Frontier

    Step 3: Making an aligned FAT32 format
    Anyone can figure out alignment of the partition from many sources already posted on the web. One thing I could not find was a guide on how to align that pesky FAT32 format so that each cluster (allocation unit) is aligned inside the boundaries of erase blocks. Well, the Linux mkfs.vfat utility has all the options we need to make this aligned format possible!

    Please view this primer to FAT32 to get an idea of how the system is laid out on your disk. You normally have the first sector in the partition which is your Volume ID, which sits in the reserved space which is normally the first 32 sectors in the partition. This is followed by (2) copies of the File Allocation Tables, which vary in length when created depending on a variety of factors, including the chosen allocation unit size. The FAT size remains constant after creation. Please keep in mind for this discussion the physical size of a sector, which is 512 bytes.
    The Golden Nugget
    We will be changing the reserved sectors from the default value of 32, to a number that we will calculate from the reported size of the FAT tables after formatting. The goal will be to make the FAT tables end right at a 128K boundary, so each cluster of the file system will fall neatly within erase blocks on our disk!
    First we will format our disk using FAT32 paying no mind to reserved sectors. This will report to us our FAT size as so:

    user@host:~$ sudo mkfs.vfat -F 32 -n MultiBoot -s 32 -v /dev/sdd1
    mkfs.vfat 3.0.7 (24 Dec 2009)
    /dev/sdd1 has 224 heads and 56 sectors per track,
    logical sector size is 512,
    using 0xf8 media descriptor, with 31309312 sectors;
    file system has 2 32-bit FATs and 32 sectors per cluster.
    FAT size is 7641 sectors, and provides 977937 clusters.
    Volume ID is 40c250bd, volume label MultiBoot .


    The juicy bits are that 2nd to last line, it tells you the size on disk of 1 FAT table.

    7641 sector FAT x 2 FATs x 512 bytes/sector = 7,824,384 bytes

    The above formula shows you the exact amount of space the FAT tables are using at the beginning of your disk. This number is not usually going to be evenly divisible by 128K (131,072 bytes) as you can see 7,824,384 / 131072 = 59.695 erase block sized chunks. What we need to do is force the end of those FAT tables to end right at 60 blocks to do so we:

    131,072 x 60 = 7,864,320 bytes in 60 erase blocks

    7,864,320 - 7,824,384 = 39,936 bytes remainder

    39,936 / 512 = 78 sectors remain

    New reserved sector count for alignment = 78 + 32 original reserved sectors = 110

    Those are all the fundamentals required to align a FAT32 partition, so that clusters on the disk fall in line with the erase blocks of the physical cell medium.

    An example of the format command required:

    sudo mkfs.vfat -F 32 -R 110 -n MultiBoot -s 32 -v /dev/sddx

    Breakdown:

    1. sudo - super user privledges
    2. mkfs.vfat - create a FAT file system
    3. -F 32 - 32 bit FAT (FAT32)
    4. -R 110 - Use 110 Reserved Sectors (instead of 32)
    5. -n MultiBoot - drive label, up to 11 characters
    6. -s 32 - 32 sectors per cluster 32 x 512 - 16K allocation unit size
    7. -v /dev/sddx - Device to format
    Final Thoughts
    Larger allocation units seem to make for better large file write speeds, I had the best performance at 64K allocation unit size, at around 15-16MB/second. As long as the calculations are all correct, you will notice much better write speeds on your newly aligned drive. They may not reach the speeds of a brand new drive, I don't have a new drive to compare side by side with this one. I went from about 6MB/sec write speed to about 12MB/second speeds on a drive using 32K cluster sizes. Read speeds are always around, or slightly above 30MB/second.

    If a utility for Windows could be found that allows for advanced partition alignment that fdisk has, and the formatting options that mkfs.vfat allows for, a user could avoid having to use an alternate OS for this setup.
    Have fun,

    John C.
    Last edited by kd8cgo; 11-09-2012 at 06:13 AM.

  3. #3
    Join Date
    Sep 2005
    Posts
    1

    Default

    Hi, very good description. You have done a good job But I have a question. Did you try this procedure but formatting to NTFS?

  4. #4
    Join Date
    May 2010
    Posts
    6

    Default NTFS Formats

    NTFS I have not played with - I primarily use Linux at home, and maintain the USB stick in FAT32 format to keep a bootable drive that can be accessed on most all Microsoft operating systems - I use the flash drive mainly for computer repair work. NTFS would be great if I needed to transport large files regularly to Microsoft laden computers, I just don't have that need at this time.

    I suspect that, if you have your partition tables aligned like in post #1, that NTFS itself will also be aligned, but I am not sure. NTFS does not have special reserved sectors or FAT tables to contend with before the clusters are made for the files system itself as far as I know. The MFT and other file system data are stored in files themselves inside a NTFS formatted partition, so one could assume that the clusters would start right at the beginning of the partition, making them aligned by default with the partition start sector.

  5. #5
    Join Date
    May 2011
    Posts
    1

    Default

    Good information. However, I have a problem calculating the reserved sectors for my SD. My FAT size is 6400 sectors, which means, after the math, I need 0 reserved sectors to align. Since -R 0 is not acceptable, I couldn't figure out how many sectors I need to reserve for the alignment.

  6. #6
    Join Date
    Jun 2011
    Location
    Coronado, California
    Posts
    3

    Default

    Quote Originally Posted by the_hand View Post
    Good information. However, I have a problem calculating the reserved sectors for my SD. My FAT size is 6400 sectors, which means, after the math, I need 0 reserved sectors to align. Since -R 0 is not acceptable, I couldn't figure out how many sectors I need to reserve for the alignment.
    According to the mkfs.vfat man page:
    Code:
    -R number-of-reserved-sectors
                  Select  the  number  of reserved sectors. With FAT32 format at least 2
                  reserved sectors are needed, the default is 32. Otherwise the default is 1 
                  (only the boot
                  sector).
    Since mkfs.vfat doesn't allow a reserve of zero sectors, you will have to move up to the next higher erase block boundary. (131,072 / 512 = 256 sectors per erase block) Try using -R 256. This should produce the same alignment as -R 0, just one block higher.

  7. #7
    Join Date
    Jun 2011
    Location
    Coronado, California
    Posts
    3

    Default

    Even though his procedure did not work for me, I'd like to express my thanks to kd8cgo for the fine job of presenting the problem and proposing a solution. Without that I would not have stopped to consider the problem and come up with my somewhat different approach.

    First, I'll tell you my experience with kd8cgo's process. I formated the drive using fdisk, exactly as kd8cgo so well described.

    Then I turned to finding the correct number of reserved sectors to make the data align to the erase block boundaries. According to his calculations I needed to use 96 reserved sectors in the mkfs.vfat command, -R 96.

    This is what my mkfs.vfat command produced. (Notice the slightly different numbers from those produced by kd8cgo's drive.)

    john@Dell2400:~$ sudo mkfs.vfat -F 32 -R 96 -n MultiBoot -s 32 -v /dev/sdb1
    mkfs.vfat 2.11 (12 Mar 2005)
    /dev/sdb1 has 224 heads and 56 sectors per track,
    logical sector size is 512,
    using 0xf8 media descriptor, with 31271936 sectors;
    file system has 2 32-bit FATs and 32 sectors per cluster.
    FAT size is 7632 sectors, and provides 976768 clusters.
    Volume ID is 4df59217, volume label MultiBoot .
    After this I did what most of you would have done, tested to see the results. I copied a 457,713,753 byte file from my system's hard drive to my reconfigured flash drive. The results weren't encouraging: 97 seconds as I recall. 457,713,753 / 97 = 4.72 MB/sec.

    I checked my work and believed it to be according to instructions. But what had gone wrong? It looked as if I would have to take a lot of time to relearn all that I had so gladly forgotten about the FAT32 file system, and I almost let the whole thing go. But one thing did come back to me from that distant past: the data portion of a VFAT partition starts immediately after the FAT tables.

    I decided to check my alignment by recreating the file system again, using the same number of reserved sectors, 96. Then I wrote a short file with some fairly unique text in it. Since this would be the first thing written after creating a new file system it would end up as the first thing in the data area of the partition.

    Then I would use the dd and grep commands to search for my fairly unique text. That would give me the address of the start of the data area, which I could divide by 131,072 to find out if it was located exactly on an erase block boundary. When I made those calculations I found the data portion of my file system started 32 sectors into an erase block.

    I went back to my previous mkfs.vfat command and subtracted 32 from the "-R 96" I had used, to get "-R 64". After running that mkfs.vfat command, when I copied the same test file to my Patriot drive the results this time were impressive, 37 seconds. 457,713,753 / 37 = 12.37 MB/sec. A savings of one whole minute, and an increase in write speed from 4.72 MB/sec to 12.37 MB/sec, a 262% improvement.

    Thanks to the work of kd8cgo I had been inspired to find my own solution to this important question. I'm perfectly willing to admit that I may have failed to accurately follow his procedure, but being actually able to see what was happening on the disk made me much more confident of success than the prospect of diving into the dark, murky waters in which the details of VFAT are shrouded.

    In order to make this work it was necessary to use some pretty murky options to the grep command. I've already converted my drive to Linux ext2 file system, and don't want to undo that for the sake of a perfect demonstration. I have copies of some of my command line sessions and will quote those that I have. The rest I'll have to talk my way through.

    It was not necessary to reformat the drive using fdisk.

    In order to remove all traces of the testing I had done, it was necessary that I remake the file system using the mkfs.vfat command. I used the same command that I had used before with -R 96.

    When that was completed, I removed the drive (It had been unmounted while making the new file system, using umount /dev/sdb1.)

    I then reinserted the drive and allowed the system to mount it.

    Then I went to the directory where the drive had been mounted. (Use the mount command to see where the system mounted it.)

    On my system the drive ended up at /media/MultiBoot.

    cd /media/MultiBoot
    ls (Command should find no files.)
    Create the first file on the file system as follows:
    echo thisxxx > thisxxx.txt
    ls (Command should show only this file: thisxxx.txt)
    Next we want to find the address of the "thisxxx" text which should be at the very beginning of the data area for the sdb1 partition.

    Since the whole drive is divided into erase blocks the search should start at the very beginning of the drive, so use /dev/sdb and not /dev/sdb1 in the dd command.

    sudo dd if=/dev/sdb | grep -oba thisxxx

    Very soon the following line should appear on screen (with different numbers, of course):
    8011776:thisxxx
    When it does use control-c to stop the command.

    Use the dc command to enter the following information exactly as shown except for using your own number rather than the one above given as an example:

    john@Dell2400:/media/patriot$ dc
    3k
    8011776 131072/p
    61.125
    61-p
    .125
    131072*p
    16384.000
    512/p
    32.000
    96r
    -p
    64.000
    q
    john@Dell2400:/media/patriot$
    The 61-p isolates the remainder part of the number on the line above it.
    The line 32.000 is the number of sectors past the erase boundary that the data starts. Subtracting that from the 96 sectors used to create this files system yields 64. So the following command should be used to remake the file system using your own calculated reserved sector number, in place of the example, 64:

    sudo mkfs.vfat -F 32 -R 64 -n MultiBoot -s 32 -v /dev/sdb1
    Remove the drive and reinsert it to have the system mount it.

    Just in case there was a mistake, create another data area marker like this:

    echo thiszzz > thiszzz.txt

    Then run your tests. It should work. If not check again to make sure the text is at a boundary. Keep trying until it does. 262% improvement is worth the effort.

    Good luck.
    Last edited by CitizenJohn; 06-20-2011 at 12:54 PM. Reason: Clarified calculation of 262% write speed improvement.

  8. #8
    Join Date
    Jun 2011
    Location
    Coronado, California
    Posts
    3

    Default

    Quote Originally Posted by kd8cgo View Post
    Please read The Ted Ts'o article on SSD alignment before you start, so you can understand a bit of what it is we are doing here! He has more information in there about LVM and EXT4 file systems, which we won't be using on a USB stick (we could use EXT2/3/4 on a stick if it is for Linux use ONLY)
    The link to Ted Ts'o's article is broken. Click here for a copy of the original article.

    John Gunn
    Coronado, CA

  9. #9
    Join Date
    Nov 2011
    Posts
    1

    Default One other little detail. . .

    When using the Grub2 boot loader remember that, in addition to the MBR boot code, it also installs a 'kernel.img' file into the reserved sectors area. This file is around 30 KiB so be sure to allow for this extra space and, if necessary, increase the '-R' value by one sector as mentioned above.

    If there's not enough space when you attempt to install Grub2, you will see an error that says the "Embedding area is too small," at which point your USB stick's MBR has already been updated but the Grub2 kernel has not yet been installed (that's -1 for Grub2).

  10. #10
    Join Date
    Oct 2011
    Posts
    14

    Default

    Why are all the default LBA Flash Block alignment values at 8064? Alignment doesn't seem to improve on anything for me.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •