Unionfs for Linux? 41
Lukey Boy asks: "A machine in my network is currently a large fileserver, and holds many hard disks full of media (namely my music and what not). Each drive is running a standard EXT3 filesystem with the same layout (/media, /media/mp3, and so on). My problem and question is how do I join these drives to look like a single hierarchy? I would like to, when I check /all/mp3, view the contents of each drive in this combined directory. FreeBSD has a unionfs filesystem type which supports the unioning of two drives - but only two is a fairly bad limitation, especially when I add a new drive. It appears that Al Viro is working on a unionfs for Linux 2.5, with again only two mount points supported. I was also considering using the Linux Volume Manager system, or possibly a software RAID striping arrangement; does anyone have any experiences doing anything similiar? Is there any decent inheritable filesystem (IFS) available for Unix machines?"
MOSIX (Score:2)
Of course, you could always port unionfs ;)
Use your own suggestion -- LVM (Score:4, Informative)
Re:Use your own suggestion -- LVM (Score:1)
Re:Use your own suggestion -- LVM (Score:4, Informative)
The only real problem is volume resizeing. Although LVM has great support for resizeing the actual volumes, there is always a little trickery involved with resizeing a filesystem ON those volumes =)
Anyways, I believe LVM is now part of EVMS.
I would give it a try, LVM is quite cool to use. I was using it when I had to switch all my drives around on a regular basis. Its nice to have a single device for my root partition and not have to make large changes to fstab and whatnot just to get my system to boot =)
(Now with a stable drive layout, I am back to using normal partitions... but thats just because I was too lazy to setup LVM again =)
BTW, nice nickname Lukey Boy
Re:Use your own suggestion -- LVM (Score:4, Informative)
Although LVM doesn't join two filesystems at the directory level, what it will do is allow you to have separate disk partitions and allocate and deallocate space from them from a pool made of of all your physical disks.
You can still have your
I'm using redhat 7.3 and lvm is included. The only problem was that I didn't know how to load the lvm module at boot time. In the end, I mucked with
Re:Use your own suggestion -- LVM (Score:3, Interesting)
Looks like you could do the same with ext2 and resize2fs if you wanted. If I recall correctly, when I set up my first box with LVM, resize2fs had lots of "beware, this is somewhat experimental code" warnings and I was reluctant to use it, which was one reason why I used reiserfs instead. Maybe a couple more years of testing have increased confidence in the utility, since those warnings aren't there now. I'd rather use reiserfs anyway.
Re:Use your own suggestion -- LVM (Score:2, Informative)
You can resize ReiserFS volumes online!
There's no need to unmount a volume before growing it. You still have to unmount it before shrinking it, though; resize_reiserfs will refuse to work if you don't do so.
Re:Use your own suggestion -- LVM (Score:1)
InterMezzo or Coda? (Score:1)
If I understand LVM correctly, it doesn't do what the poster wishes. It will allow him to choose which logical subdirs reside where, but not have all his files everywhere.
It seems like a distributed filesystem, like InterMezzo [inter-mezzo.org] or Coda [cmu.edu] would be a better match. That way his files are everywhere and the fs automatically manages updates regardless of where changes are made - including after disconnected operation.
Re:InterMezzo or Coda? (Score:1)
Actually, I just realized the poster's drives are all in a single file-server, not distributed across a server farm. (time to sleep I guess)
Re:Use your own suggestion -- LVM (Score:2)
Otherwise it's a nice system though. But I'd say use RAID5 to create redundant storage, then use LVM to put these together into one logical unit.
CXFS? (Score:1)
symlinks? (Score:1, Informative)
for f in
for g in
ln -s
done
done
Comment removed (Score:4, Informative)
Re:quick solution (Score:1)
I'm still waiting. :(
Re: (Score:1, Offtopic)
why does it hurt when i do this... (Score:2, Insightful)
don't do that.
unionfs with two fs's really isn't all that great idea. and no offence, but unionfs with more then two fs's is monumentally stupid. where do new files get created? how do you deal with conflicts? what happens with files that you edit? what if you remove a file that exists on more then one fs?
there are a slew of different answers to these questions so either the kernel inflicts (and yes, i mean inflicts) policy or you have to provide a way to configure all those options. should mounting fs's get as complicated as firewall rules?
there are lots of solutions to your problem. pick one of them rather then a bigger problem.
Re:why does it hurt when i do this... (Score:1)
Besides, telling me not to do it - what's with that? Everyone knows the Perl motto - there's more than one way to do it. Unix is the same - look at all the different variants out there.
Re:why does it hurt when i do this... (Score:3, Informative)
what he wants can easily be done with regular old mounts, eg he could sort his mp3s into 'classical', 'jazz' etc.. and have each one in a seperate dir and on a seperate disk. or he could symlink into the various disks. finally the most transparent way to merge the disks into one is LVM.
--paulj
Re:why does it hurt when i do this... (Score:1)
unionfs with two fs's really isn't all that great idea. and no offence, but unionfs with more then two fs's is monumentally stupid. where do new files get created? how do you deal with conflicts? what happens with files that you edit? what if you remove a file that exists on more then one fs?
I think a union/overlay/translucent filesystem is a natural and useful construct. I think having N readonly layers below one writable layer is as intuitive as for 1 readonly layer.
I dont think problems with name conflicts are necessarily much worse than having to map to 8.3 names, or names which differ only by case on a filesystem which is not case preserving.
Having to choose whether or not to have copy on write, or choose a z value for each layer, does not seem like a problem which makes the cost of overlay filesystems more than the benefit.
LarryRe:why does it hurt when i do this... (Score:3, Insightful)
I like it. Not.
If "there are lots of solutions" to the problem, showing at least one of them would perhaps turn outright discouragement into something useful.
The problems are obvious. And they have been solved for a long time - witness TVFS under OS/2. Does something similar exist for a free OS? LVM might be close.
Mounting filesystems should be no more nor less complicated than is needed to achieve the desired goal. If this means that it's "as complicated as firewall rules," so be it.
Seems that this is already the case, anyway. I've got three lines of firewall rules. The fstab on the same machine trounces that handily, without doing anything creative or silly - just mounting various partitions to various points.
(and before anyone asks how it can possibly be secure, I'll say this: It's -STABLE, and that's good enough for me.)
unionfs 2-fs limitation shouldn't be a big deal (Score:2)
For example, maybe you have
Mind you, I haven't looked at whether any given unionfs implementation would support this configuration, but loopback-type support typically isn't hard.
As as aside... wherever I type "space slash", slashdot eats the space, making the above a little harder to read than it should be. Anyone know why that is, and/or a workaround?
Re:unionfs 2-fs limitation shouldn't be a big deal (Score:1)
/ / / / / You could always use monospacing / / / /
/ / / / / The space eating has nothing to do with Slashdot, it has to do with font Slashdot uses (Times New Roman is the default, if I'm not mistaken). Go ahead and type " /" in your word processor of choise and it will have the same effect in TNR, Aral, Georgia, MS San Serif, etc. / / / / /
Thanks everyone. (Score:1)
organization (Score:1)
use LVM and make one large volume, create
very easy, very simple..
More uses for a union filesystem... (Score:1)
Re:More uses for a union filesystem... (Score:1)
You can do this with User-Mode Linux. Just put your root FS on a CD (you can even make it ext2 instead of iso9660), and then use UML's copy-on-write filesystem driver. Put the COW file in a ramdisk on the host OS. Voila!
I actually plan to implement just such a hack so I can use UML jails more safely on services, and still do useful things that need the occasional temporary file. (I don't think I'll put the COW file on a ramdisk specifically, but I will delete it post-use.)
--JoeRe:More uses for a union filesystem... (Score:1)
You could have a single image of the Windows filesystem, then keep multiple overlays, each containing the changed files for that instance. This would keep multiple virtual Windows systems going in their own jails, making it easier to recover from failures without having to trash unrelated projects.
Soudns like TVFS (Score:2)
Re:Soudns like TVFS (Score:1, Flamebait)
vinum (Score:1)
It does exactly what you want.
You can create virtual disks that span several hdd's, and create one huge filesystem.
D.
Re:vinum (Score:1)
AutoFS? (Score:1)
The nice thing is, you never have to su to root in order to mount anything - you just cd into the directory (on any machine) and it gets mounted in the background. You still get UNIX file semantics and permissions, the normal NIS & NFS behaviour.
On the machine with the DDS3 drive. I just kick of a script which traverses the NIS map and catches all the directories which I deem worth backing up. Works a charm, but really only on UNIX, which suits me. There are windows clients for much of this stuff (reflection, Solstice network client if you can still find it) but they're mostly crap.