Linux Support for Hybrid Hard Drives?

christoofar asks: "HHDD (Hybrid Hard Disk Drive) technology has been receiving some buzz lately. The concept is not new, but Samsung has been working on a consumer version of HHDD that everyone can use. HHDDs are disk drives that carry onboard RAM (in this case, NAND flash) which is non-volatile and offers to speed boot times and writes to the disk. This carries enormous benefit to laptop users who need to keep their disk activity to a minimum in order to preserve battery life. Given that Microsoft is adding support for Hybrid Hard Drives in their upcoming Windows Vista release, what efforts are being undertaken in the Linux realm to use this new storage technology?"

It should just work

Why aren't the drives designed in a way that the drive technology worries about the details of retrieving and writing files, and sends them the way USB, IDE, SATA, or SCSI drives already do? Why would these drives need special drivers?

Re:It should just work

I would guess that it's because having a software-based queuing and caching system is more flexible. For example, if you want to optimize operating system startup, then you need some way to specify that the operating system is what you want in the flash memory, instead of some other data.

Or if you've got a laptop, you maybe want to cache writes to the flash if you're using the battery, but write directly to the disk if you're hooked up to the mains. Since the operating system knows about this but the drive doesn't, the operating system needs to control the drive.

Re:It should just work

For example, if you want to optimize operating system startup, then you need some way to specify that the operating system is what you want in the flash memory, instead of some other data.

Still sounds pretty much like disk caching. During bootup, the syst

Re:Simple solution ... still no special driver nee

Maybe someone doesn't want their operating system on the RAM-ified part. Maybe they want swap space there instead. Maybe they want the reserved journal space of a journaling filesystem that is frequently updated to be there. Of course all of these needs

Re:It should just work

I'm guessing that it's not so much that it can't work that way, as it is that it would kill performance.

Today's operating systems try to do exactly what this drive is doing by pumping all reads and writes through the paging system. The problem is that the OS can't take a full performance boost from this or data will be lost in a crash or power loss. You may remember that EXT2FS could easily lose all the data you'd recently written if it wasn't unmounted properly. (It ticked me off when I tried to move RedHat 5.2 packages to a special partition, then found out that they had all disappeared on reboot because I hadn't explicitly unmounted the disk. *grumble*)

Since this technology has little to no chance of losing data in a power failure, the OS can be modified to write the blocks immediately. This could easily result in a performance increase of 2 to 3 times what you normally see today. The improvements in writing meta-data alone could easily provide this increase.

sync

...then found out that they had all disappeared on reboot because I hadn't explicitly unmounted the disk. *grumble*

The SYNC command forces the system to flush all buffers to disk. Once it returns, you know that all currently cached data has been at least

Re:It should just work

There are a couple good replies to you already, but I'll add one more place the OS might be able to improve its performance if it knew about the hybridness of the disk.

I have a hunch (this is unsupported by data; you'd want to do some profiling before maki

Re:It should just work

I regularily see the amount of memory allocated to Firefox alone over 500 MB...

Currently Windows reports 920 MB pagefile usage. I think that's probably total memory use, not the amount that's swapped out, but I can almost guarantee that there's something o

What to put in there?

The question is not whether Linux will support the drive -- someone will eventually write one. The question is what files to put in there to make it boot faster. Perhaps should be done is to put the entire swap partition in there, and put those "hibernation" files in the swap. So it would resume fast, and would also make normal operations faster with the faster swap.

Why wait? Hack one up yourself!

You shouldn't have to wait for the drives to come out. Laptop drive controllers can address a master and a slave, they just don't have a slave drive connected in normal usage. I was just looking for an adapter [logicsupply.com] to let me put a pair of CF cards in place of a 2.5" hard drive. (I can only find the single-card version in a 2.5" form factor, all the dual-card ones are for 3.5" mounting.) I figure, put a solid-state card in one slot and a microdrive in the other, and I've got a hybrid-drive laptop, right?

I just got rid of an old toughbook cf-25 [wikispaces.com] that would've been perfect for this, as the drive mounting is gel and would easily accomodate an oddly shaped adapter instead of a regular drive. Or for the truly insane, a CF card piggybacked on a regular 2.5" drive! All I need is the ability to home-brew those little flex cables, and I'd be in business.

Nice to see

With all of the improvements in RAM, on-die cache and processor speed, the bottleneck in performence is HD speed. Anything that helps boot that is a welcome improvement.

Write caching in flash...

I'm not sure if write caching in the flash chips is a good idea...

Flash chips write at around 1Mb per second. Tops. Modern harddrives write around 50Mb per second. You'd need quite a lot of flash chips in parallel to top that.

Of course, you can cache acoup

Re:Write caching in flash...

Samsung's OneNAND design uses some interesting implementation details to run faster than is normally expected with flash. Their specs at http://www.samsung.com/Products/Semiconductor/OneN AND/index.htm [samsung.com] suggest writes at 9.3MB/s and reads at 108MB/s; plenty fast for many applications even without running multiples in parallel. It's certainly much slower than writing directly to the drive if the drive is active. However, if the drive isn't spun up at the moment and the amount of data to be written fits in the NAND flash cache, I could see this being a net performance boost over the spinup/seek time combination of the hard drive.

They also spec 100,000 erase cycles before it's worn out. As was noted by an underrated poster the last time [slashdot.org] this came up, intelligent flash controller designs like this can cope with bad bits and assuring level usage of the memory much better than what you normally see in random hunk of flash.
An analysis at http://www.sudhian.com/printdocs.cfm?aid=686 [sudhian.com] suggests 33 years of usage for a typical worker. When you run the numbers it doesn't sound that difficult to create a design that would likely outlast the mechanical parts of a standard hard drive.

Re:Write caching in flash...

If your OS is WRITING to swap in a "small random blocks" it should reevaluate its swap allocation strategy.

Other good uses for this

I think the real demand for flash-cache hard drives should be in the area of data loss prevention. Anyone who has any doubts about the fact that laptop hard drives are still way too fragile for the job needs to work at a help desk for a week. Any way to keep those drives spun down as much as possible should be seen as a good idea, since speed does not really matter much if you lose your data and/or need a new drive every three months simply because you like to move your laptop while it is running.

I know this just adds another point of failure to the mix with the addition of flash memory. However, with the apparent improvements in the quality of flash memory, I would expect one of these drives to outlast a current laptop drive by at least 50%. (Note: this is just idle speculation, but I don't know of any solid real-world statistics on laptop hard drive lifespan. I'm guessing laptop makers don't want us to know, either.)

I think that the point of drivers is, therefore, valid. There needs to be some sort of intelligence behind this system to allow frequently needed files to be held in cache in order for this to work effectively. Sure, you could build a drive that could try this on its own, but odds are you would totally throw out any performance or power advantage by doing so.

loss prevention

Most common source of data loss.

Laptop hard drives: the new floppy

Another use?

So I see a completely different picture here. There are a lot of files on the HDD that are never rewritten. System files, etc. This is where having hybrid drives really helps. Put the main boot files, executables and libraries which are accessed when staring things up in there. So no fast saves, but no load time.

It looks like the answer is "yes"

I've got the 2.6.15.4 kernel and there's a spot to enable support for "Memory Technology Devices".

Here's the description:
"Memory Technology Devices are flash, RAM and similar chips, often used for solid state file systems on embedded devices. This option will provide the generic support for MTD drivers to register themselves with the kernel and for potential users of MTD devices to enumerate the devices which are present and obtain a handle on them. It will also allow you to select individual drivers for particular hardware and users of MTD devices."

It's got options for a whole bunch of things, such as "FTL (Flash Translation Layer)", "NFTL (NAND Flash Translation Layer)", and "INFTL (Inverse NAND Flash Translation Layer)".

Given that it says the hard drives will use "a one-gigabit OneNAND flash chip" (according to the article), it sounds like it will work.

Re:Is it me or does this seem pointless?

But this is non-volatile memory we're talking about, so when it writes to the cache it is done writing the data. Subsequently transferring the data from the flash to the disk is more akin to defragmenting than "completing the write."

Re:Is it me or does this seem pointless?

Right like, perhaps in addition to the kernel, or other essential boot files, it would also store the journal for journaled filesystems - it will cache the writes and commit them the next time that power supply and demand warrant. Consider a 1 Gbyte flash

Re:Is it me or does this seem pointless?

Well, they only produce the isolinear chips for Star Fleet.
Seriously, the NAND flash memory stores the data immediately. It is a perfect write cache, because it does not forget the data. Almost perfect, because its write cycles are limited. But a more reli

Re:Is it me or does this seem pointless?

This could IMPROVE the situation you're talking about.

First of all, you know that right now your writes are cached in your computer's memory for up to 30 seconds before they are flushed to disk. This is done so that synchronous read calls are less disrupte

Re:Random writes needed?

i should add, if a couple million writes is true, even if you were to randomize or even out the bits algorithmicly, I still would be afraid of failure.

Windows doesn't open executables for writing

Windows doesn't open executables for writing when you run them. In fact, when you run a program, it prevents writing entirely. Windows loads executables through the memory-mapped file interface. It does this so it can share code pages between different

What I'd rather see anyway.

I'd like to see linux have support for (and have hardware makers create) what SUN used to call NFS Accelerators. Basically add-on NV (or battery backed) memory sufficent to transparently cache write information. On my laptop I'd put it in my ExpressCard(