Slashdot Log In
Optimizing Linux Use On a USB Flash Drive?
Posted by
timothy
on Wed Dec 03, 2008 05:18 PM
from the right-here-in-my-pocket dept.
from the right-here-in-my-pocket dept.
Buckbeak writes "I like to carry my Linux systems around with me, on USB flash drives. Typically, SanDisk Cruzers or Kingston HyperX. I encrypt the root partition and boot off the USB stick. Sometimes, the performance leaves something to be desired. I want to be able to do an 'apt-get upgrade' or 'yum update' while surfing but the experience is sometimes painful. What can I do to maximize the performance of Linux while running off of a slow medium? I've turned on 'noatime' in the mount options and I don't use a swap partition. Is there any way to minimize drive I/O or batch it up more? Is there any easy way to run in memory and write everything out when I shut down? I've tried both EXT2 and EXT3 and it doesn't seem to make much difference. Any other suggestions?"
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Get a swap partition (Score:3)
Re: (Score:3, Interesting)
Well, laptop mode's an obvious. However, I would not enable swap at all, or at least put it on the FS somewhere. If you do that, you wont be able to hibernate properly. Nor will hibernation work properly if you encrypt swap (partition).
You're looking at standby or poweroff events due to encrypted partitions and systems. That's the pain you pay for. There is another way... TCPA, and IBM has writen the requisite linux tools to utilize it properly. Just everybody's too scared to use it properly.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Informative)
No, put it on flash. It'll be faster
No, it will most certainly be much slower. The memory page size is 4K, whereas the typical flash block size is much bigger. Flash can only be erased in blocks, so to write one small 4K page to flash memory, the controller has to read a whole block, erase the flash block and write back the modified block. That's why most flash storage devices suck so badly at storing many small files, even though the continuous write speed isn't too bad.
The performance penalty could be hidden by the OS: Instead of swapping o
Re: (Score:2, Interesting)
Or on a mostly-Windows machine, you can always use a local-drive-based swapfile (there's likely no swap partition). Mount NTFS drive, create a "myswapfile" somewhere innocuous, mount it as a loopback swap partition (-o loop). Delete it on unmount (as part of your shutdown process) -- if you're truly paranoid you can even take the time to scrub the sectors your swap was using.
Don't swap to the flashdrive -- you'll just hog USB bandwidth that you need for reading & writing real files off your root parti
He's running encrypted disk on the flash. (Score:2)
Delete it on unmount (as part of your shutdown process) -- if you're truly paranoid you can even take the time to scrub the sectors your swap was using.
He's running his flash drive using full-disk encryption. So he's concerned about leaving his data lying around. If that's the case he should not be swapping in the clear, then hoping it gets properly scrubbed later. The swapping should also be encrypted.
It should be possible to do the same loopback-though-encryption hack to the swap file on the Windows di
Re:He's running encrypted disk on the flash. (Score:4, Insightful)
Using encrypted swap is really easy on a modern Linux distribution. A random key is assigned to the swap partition on boot-up, so that after power-off, the data's almost as good as gone.
Parent
Re:Get a swap partition (Score:4)
Parent
Re: (Score:2, Informative)
Re: (Score:3)
Re:Get a swap partition (Score:4, Informative)
Apparently wear leveling is no panacea. Some devices wear-level only within a zone, and so a lot of activity in a hot zone can still burn that zone out. I guess such an algorithm only really works well when writes are distributed across zones evenly.
See page 5 in this document [micron.com] for a description of zone-based wear leveling.
Parent
Re:Get a swap partition (Score:4, Interesting)
Method doesn't work when improperly implemented. News at 11.
You're basically saying that wear-leveling is no panacea, because not all devices have a good wear-leveling algorithm. Wouldn't the solution, then, to be to demand devices which have a good algorithm, and to reward manufacturers who solve this problem with dollars and word of mouth?
Of course, the sad part is, the devices would probably be simpler, and cheaper to manufacture, if they didn't pretend to be block devices. Only problem is, then OSes (probably filesystems) would have to implement wear-leveling on their own, and Linux is the only one I know of which currently does that.
Still, it would be nice if we could fix that problem in software, rather than letting manufacturers get it wrong.
Parent
Re:Get a swap partition (Score:5, Insightful)
My point is that this seems to be more prevalent than people realize.
The problem with short product lifecycles is that you might not realize you bought a turd until after it eats your data, and there's no time for word-of-mouth to spread. And with so many no-name USB sticks out there, how can you really tell which ones are the turds and which ones aren't? There really isn't a good feedback mechanism here. You have some hope paying high dollar for a name brand, but I wouldn't put too much stock in it.
Also, there are conflicting product goals: Dynamic wear leveling, as described in the PDF I linked above, gives faster write performance. Static wear leveling gives longer silicon lifetime. Which algorithm will generate better word of mouth? It's not clear, although I suspect faster write performance will generate better word-of-mouth while the product is still on store shelves.
You would have to expose a number of low-level details, such as erase block size and other aspects of physical organization to the OS. And, you'd have to teach many OSes about how to talk to you, such as "block erase takes 5ms; device is unresponsive during erasure". Until there's a standardized way of doing this across platforms, faking yourself as a block device is an easy way to get broad compatibility with anything that understands the USB storage device (or SD card, or whatever format you're working with) profile.
You can still implement software wear leveling over the abstracted interface. Unless the device actively tries to work against you, it should help.
Parent
Re: (Score:2)
Re: (Score:2, Funny)
Does anyone actually use Readyboost??
Re: (Score:3, Informative)
Yes and it works quiet well in real world performance testing.
Even a 1gb or 2gb stick can improve overall responsiveness enough that makes a tight XP installation feel slow. This is especially true with gaming and applications that push the limits of your RAM shoving the HD cache down.
There are a few crap Flash drives, that just don't perform well, avoid them and it works like promised.
Re: (Score:3, Informative)
Meaning Vista+ReadyBoost is apparently 'faster' as far as responsiveness goes than a clean and properly tweaked windows XP installation.
I still think pics or it didn't happen, but never was ReadyBoost on XP mentioned.
Re: (Score:3, Informative)
Every access to FAT writes to the same area of the file system (the FAT), and this is what destroys old-style floppy disks.
That's why people use a flash file system http://en.wikipedia.org/wiki/File_system#Flash_file_systems [wikipedia.org]. http://en.wikipedia.org/wiki/YAFFS [wikipedia.org] is a good choice; it seems to be used in Android.
Re: (Score:3, Informative)
Re: (Score:2)
http://fixunix.com/kernel/373556-ubifs-vs-logfs-rfc-patch-ubifs-new-flash-file-system.html [fixunix.com] has a great discussion about using a flash file system on a block device that's really a flash device with built-in wear-leveling. While neither participant seems to decisively change the other's opinion, several good arguments are made.
Re: (Score:2)
Those chips are about as intelligent as a rubber band. There's currently just no way to implement sane wear filesystem-independent wear levelling in a tiny chip like that. In fact, ignoring the hints of the filesystem in general is a terrible approach, since the filesystem knows best how it can safely move its data around, and a flash-optimized file system is well worth using.
Re: (Score:2)
Re: (Score:2)
Mod parent as "funny"? (Score:3)
The best thing to do is make a ram disk and use that as swap space.
So the OS can use part of the memory as swap, rather than memory, and when the part it is using as memory is full it can copy it to the part it is using as disk? (Except it won't, because if I recall correctly the RAM disk, along with disk buffers, are all dynamically assigned in memory, so when you need to swap there's nowhere to swap to.)
Were you trying to be funny? Or just unintentionally managed it?
Re: (Score:2)
Re: (Score:3, Insightful)
Yert: It is not clear that the original post was a joke.
"Whoooosh" is fun for playing elitist games. But I'm more interested in helping people avoid being "whoooosh"ed into a lot of lost time, effort, and perhaps compromised data because they missed something that another poster thought was obvious and funny.
USB has high cpu use and encrypted does not help/ (Score:2, Informative)
USB has high cpu use and encrypted does not help a firewire based flash drive will be a lot faster. USB 3.0 may help but it too may still the limit of usb 2.0 high cpu use as well.
Hrm. (Score:5, Insightful)
Well, the sucky thing about USB is it requires an inordinate amount of CPU. Normally this isnt a worry, but if you're using a encrypted loopback.. well ouch.
One thing you could instead use is the SD card slot and a USB loader. If you choose a 8GB class 6 SD card, you could have plenty of room for whatever, and 6MB/s minimum speed. You're still going to take the CPU hit for encryption, but that is your choice. The big thing is to stay off the USB.
Re: (Score:2)
The internal SD card reader is typically connected to the USB bus so it probably won't change performance all that much.
Re:Hrm. (Score:4, Informative)
True, but some laptop and desktop designs have went away from USB connectedness.
For example, on my Thinkpad the SD reader is its own bus. The Bluetooth is a USB 1.1 (grr) device, so I need to rmmod the bt modules and remove usb old modules to be power efficient.
It really is a crapshoot on what the computer maker put on the USB bus. I just lucked out.
Parent
Re: (Score:2)
Lightweight software && ssh -x (Score:5, Interesting)
Another option, if you're booting on a box which has a good internet connection, is to ssh -X things over a network. Not only does this save a large amount of space, but I've found it's often faster to have a program like Firefox start on my snazzy box at home and ssh -X over than waiting for it to load off of my crappy usb drive.
USB1 vs USB2 (Score:5, Interesting)
It might be that the poor performance occurs when you're on a computer that only has USB1 support. On Dells this was added later than you might expect.
You might find you got better performance if you were to use a CD to hold most of the static software and the USB for just your home directory.
optimizing Linux on USB: multiple angles of attack (Score:5, Insightful)
I know a little bit about this because I am one of the developers for TurnKey Linux [turnkeylinux.org], a new opensource project which builds small installable live CDs (we're up to 9) optimized for various mostly server-related tasks. I've been investigating supporting live USB mode.
Your generic run-of-the-mill USB drive has about fourth-half the read/write performance of your hard drive nowadays (10-15MB/s). Since there are no moving parts (spinning platters), usually the seek times are very good.
There are several things you can do to optimize the performance of an operating system running live from a USB drive:
1) buy a faster USB drive: a good USB drive (e.g., Lexar JumpDrive [lexar.com]) can have 2-3 times the performance of a generic.
2) Use a Linux distribution with a smaller footprint such as DSL [damnsmalllinux.org] (50MB) or Puppy Linux [puppylinux.org] (standard edition is 68MB): the smaller the footprint, the less your drive has to read, the faster your system will load.
3) Try loading the operating system system into a ramdisk: many live USB distributions have the ability to load themselves into RAM. With some you have to add a cheatcode in the bootloader. Others do it by default if there is enough memory (usually not a problem with small distributions and modern computers).
4) Try turning on readahead: many distributions which are designed to run from a live CD or live USB have a feature that reads ahead various files important to the boot sequence sequentially. Whether or not this helps depends on the characteristics of the storage medium you are using, but you should investigate it.
Did you read the faq? (Score:5, Informative)
make sure you are on USB 2.0- interface can kill you.
Also did you check the faq-
No seriously:
http://www.linux-usb.org/FAQ.html#i5 [linux-usb.org]
especially the section on:
Q: What is max_sectors and how should I use it?
A:For USB Mass Storage devices (that is, devices which use the usb-storage driver) max_sectors controls the maximum amount of data that will be transferred to or from the device in a single command. As the name implies this transfer length is measured in sectors, where a sector is 512 bytes (that's a logical sector size, not necessarily the same as the size of a physical sector on the device). Thus for example, max_sectors = 240 means that a single command will not transfer more than 120 KB of data.
Buy a faster USB flash drive (Score:4, Interesting)
Most USB flash drives are very slow, and rely on heavy cacheing to make them usable. That doesn't help when you need to write large amounts of data (ie. in an apt-get update/install).
Some flash drives that advertise themselves as 10-15MiB/sec write capable peak out at only 1 or 2, and even less with small-block random I/O (since the erase-write cycle operates on relatively large blocks).
Several vendors make specialized flash drives that are somewhat more expensive (ie. 20-50% over average), but perform much better.
One is here: OCZ Turbo USB 2.0 [ocztechnology.com]
Re: (Score:3, Informative)
You could also skip flash entirely and buy a very small hard drive [apricorn.com]. I've got a 60-gig USB drive from Apricorn that I carry around in my pocket, with an AES-encrypted root filesystem. Performance isn't spectacular, but it's certainly usable.
Put i/o-intensive filesystems on a ramdisk? (Score:2)
Re: (Score:2)
Re: (Score:3, Informative)
Just to let you know, tmpfs ignores the device path, you can put whatever you want in it (and you aren't actually using /dev/ram* with that).
-:sigma.SB
Wearing of flash cells should be adressed (Score:2)
Try out different filesystems (Score:5, Insightful)
Try out different filesystems, NILFS [nilfs.org] seems to be optimized for FLASH usage.
Brtfs could also be worth a try.
use the "noop" IO/Scheduler with nilfs: /sys/block/sdX/queue/scheduler
echo noop >
Postmark benchs on an usb-stick (shameless copied from here [forumdeluxx.de]:
ext3 (mount -o noatime,noadirtime, normale Partition, scheduler cfq): 49 Transactions/s
nilfs2 (Partition aligned 128k, scheduler noop, protection_period 10s): 588 Transactions/s
Re: (Score:3, Funny)
I'm holding out for the MILFS filesystem.
what I do (Score:4, Insightful)
busybox (Score:5, Insightful)
Also, for any packages you build you should try to use the -Os option for gcc, and perhaps even strip the binaries to remove unused symbols and debug info.
Building the system as though this was an embedded system with a small disk should be a win in most cases since fewer things have to go over the wire to load a file and more of the binaries can fit into cache.
Journalling (Score:2)
I'm surprised no one mentioned this (or maybe someone has and it's just under my threshold), but not using a journalling filesystem can help tremendously. Having a whole system on a flash-based USB mass storage media formatted and mounted as ext3 is a great way to make sure the only bottleneck you'll ever have is disk I/O.
Dave Jones suggestions for the eee900 might help (Score:5, Informative)
Dave Jones recently posted elsewhere his notes for improving things on the eee900. Several of the steps focus on getting proper performance out of the flash, and so would also apply to booting from a USB thumb drive or other flash media. Here's what he had to say:
Use faster flash-drives (Score:3, Informative)
Re: (Score:2)
A read only filesystem for general use. That sounds like fun.