Best Filesystem For External Back-Up Drives? 484
rufey writes "I've recently embarked on a project to rip my DVD and CD collection to a pair of external USB drives. One drive will be used on a daily basis to access the rips of music and DVDs, as well as store backups of all of my other data. The second drive will be a copy of the first drive, to be synced up on a monthly basis and kept at a different location. The USB drives that I purchased for this are 1 TB in size and came pre-formatted with FAT32. While I can access this filesystem from all of my Windows and Linux machines, there are some limitations." Read on for the rest, and offer your advice on the best filesystem for this application.
"Namely, the file size on a FAT32 filesystem is limited to 4GB (4GB less 1 byte to be technical). I have some files that are well over that size that I want to store, mostly raw DVD video. I'll primarily be using these drives on a Linux-based system, and initially, with a Western Digital Live TV media player. I can access a EXT3 filesystem from both of these, and I'm thinking about reformatting to EXT3. But on Windows, it requires a 3rd party driver to access the EXT3 filesystem. NTFS is an option, but the Linux kernel NTFS drivers (according to the kernel build documentation) only have limited NTFS write support, only being safe to overwrite existing files without changing the file size). The Linux-NTFS project may be able to mitigate my NTFS concerns for Linux, but I haven't had enough experience with it to feel comfortable. At some point I'd like whatever filesystem I use to be accessible to Apple's OS X. With those constraints in mind, which filesystem would be the best to use? I realize that there will always be some compatibility problems with whatever I end up with. But I'd like to minimize these issues by using a filesystem that has the best multi-OS support for both reading and writing, while at the same time supporting large files."
The solution.. (Score:5, Interesting)
is to stop being so diverse! Pick a platform and stick with it!
Ok, in all seriousness.. here's what you do:
- buy yourself a cheap (~200) box
- hook all your drives to it
- use whatever file system you want (JFS, XFS would be my recommendation)
- share it over your zoo of a network using nfs, samba, etc..
As a bonus, your file server box could double as a media center, and replace your WD TV Live dealie.. (probably not though.. right)
Irregardless, I think you're way better off with this approach vice trying to find the magical widely supported cross platform file system with large file capacity.
You also might want to consider RAID vs. your monthly sync. Yes, RAID isn't a backup.. but for something like this where
restoration would be possible, but just a pain in the ass.. mirrored raid can be a lot more convenient. You can always have
a third external to back up your irreplacable data on a semi-annual basis..
Uhm... (Score:1, Interesting)
Have you been living under a rock? NTFS has been writable on Linux for a long time now, using NTFS-3g - most distros come pre-bundled with it nowadays, and there's a Mac version as well.
I wouldn't.... (Score:5, Interesting)
network it (Score:4, Interesting)
Not if you use a network filesystem, such as Samba and NFS for the Windows and MacOS machines. Then on the Linux fileserver side, use whatever filesystem you want, and any OS can talk to that server.
Re:The solution.. (Score:4, Interesting)
Or get a cheap NAS like the D-Link DNS-321. While certainly far from the bee's knees in terms of performance or number of bays (2), it can be had for under $100 and has been hacked to death to run all sorts of other stuff on it. Plus it's nice and quiet and doesn't use much power. And it's kinda purdy.
I have a few suggestions (Score:2, Interesting)
If proprietary filesystems are on the table, how about VxFS [wikipedia.org] ?
Another possibility is to use FAT with cross-platform backup software. Maybe you don't need a filesystem at all: if this really is for backups... why not just create lots of extended partitions on the device and use TAR ?
AKA tar cf /dev/sdbXX -V 'VOLUME_A' /backup
That's crude and hard to keep organized, but also effective. Also, Some proprietary backup products that will work on a FAT filesystem, and not require large file support, even to backup large files
Or utilize a tool such as WINRAR that allows you to "split" a RAR file across multiple archives in chunks of a certain size, then store these files on a FAT filesystem.
FAT is the most cross-platform, oldest. But has known issues with fragmentation, and lack of journaling, effects reliability.
You could divide your backup volume into 2 partitions: one DOS/FAT partition with the bootable image and files required to 'load a virtual machine' that can see the files on the other partition in the preferred data format such as ZFS or FFS (e.g. pre-allocated eager zeroed thick VMware VMDK with 'split into 2gb files' enabled).
Then you just make sure the system you plug the drive into can boot a VM, with your "backup/file access environment"
What about a backup server? (Score:4, Interesting)
As an alternative to an external disk that goes to multiple machines, this might cost some, but perhaps consider a backup server?
The advantages to this setup:
1: The server initiates the backups, and can warn you in case something can't be read.
2: Most backup software stores snapshots, and some deal with the full/incremental/different cycle by using synthetic full backups. This makes restores to a certain point in time pretty easy.
3: More sophisticated backup software allows you to transfer backup sets to another media. This way, you just plug in a drive, do a transfer, and you have an offsite archive.
4: If one of the backup client machines gets hacked or malware installed, existing data stored on backup media cannot be altered.
The disadvantages:
1: You will need an active computer which is significantly more expensive than a hard disk.
2: Amanda/Zmanda for open source, Retrospect, Backup Exec, for commercial. The software costs a hefty chunk of change.
3: You have to make extremely sure that the backup server box is locked down tight. If someone compromises your backup server, they got data of every box you have. If you can, perhaps consider buying a router to put the backup server behind and only allowing the vital ports incoming.
4: Backup servers should have some redundancy for stored data. Because there is so much data stored from multiple boxes, a failure of a drive hurts more than on a normal machine.
5: Restoring a machine may vary in difficulty.
Re:network it (Score:3, Interesting)
I have been using freenas on old boxes aggregating old disks ( I always want to get every last hour of use out of hardware before throwing it away) for a few years now and I find that the windows clients (98/2000/XP/Vista) work really well, much better than ubuntu samba. Thus i use nfs sharing for linux and smb for windows from the same freenas.
I am rather crap at setting up samba and the freenas makes it all nice and simple for numpties like me.
Replace the WD TV thing (Score:3, Interesting)
Replace the silly little WD TV Live media player with a mITX system that's about the same size. Install Linux and XBMC and be done with it. You'll have the best possible media player on the planet, as much storage space in any configuration you want and the ability to expand everything when the time comes. No hassle, you'll have constant online backups available and you'll have a killer always-on media center.
Re:The solution.. (Score:1, Interesting)
Utter rubbish.
Irregardless is a perfectly cromulent word.
I've been there - NTFS all the way (Score:2, Interesting)
Re:NTFS (Score:3, Interesting)
As much as I love ZFS, one thing it's not good at is for a single removable disk. The inability for Apple to get this working without kernel panicking the machine was one of the many reasons they chose to drop it.
ext3 (Score:3, Interesting)
I format my external USB drives to ext3. Most of my machines are Linux anyway, and I can always plug the USB drive into my storage server and backup over Samba to any kind of drive supported by the storage server.
ext3 is pretty much stable and well understood. It just works. That's what I want for backup drives.
And my netbook has Ubuntu Linux on it, and ext3 performs well on the external USB drive there. I haven't tested NTFS over FUSE on the netbook, but I wonder about CPU overhead on the little Atom chip: it might be a little bit slow.
If you want a drive you can take over to your friend's house, and your friend just runs Windows or a Mac, then by all means NTFS.
steveha
turn your usb drive into a NAS (Score:3, Interesting)
I haven't tried it but it looks like a good idea. http://www.cyberguys.com/product-details/?productid=36218&sk=MC71419 [cyberguys.com]
Format it ext3 and then share it SMB for any OS.
Why all those ext3 recommendations? (Score:5, Interesting)
which is exactly what the submitter wants to use his setup for. Look into the crazy structures
("double indirect blocks") it uses. He should go with an FS that has sane data structures with
files >>4GB.
That kills most of the choices and leaves XFS, ext4, ZFS (only worth it if not used via FUSE,
i.e. in Solaris), and a couple more obscure ones.
I second the "forget OS portability, use a server" suggestion. There's great low-power, low-cost
hardware for this nowadays.
NTFS and Compress (Score:2, Interesting)
Re:ZFS, supported equally on your OSes (Score:4, Interesting)
Did you "zpool export tank" before you moved the drives over? If not, then the FreeBSD box saw the drives but said to itself "Those drive don't belong to me!" I've done a migration from FreeBSD to Solaris, but always with full drive devices ie /dev/ad0 - thus ignoring partitions tables and other such cruddyness.
Re:ZFS, supported equally on your OSes (Score:2, Interesting)
Oblig: XFS plug - it was made for media streaming (Score:3, Interesting)
XFS was designed with media streaming in mind -- and designed for large file, high performance. It had a defragmenter to keep disks in optimal condition before Windows98 had come out (was one at the request of a large, customer who had an especially pathological case -- before that there was normally not considered a need for it).
Files can be 'normal', have up to and addition 256K of resource-fork related into (extended attribute info), AND you can have a real-time section that can allow for completely bypassing the file system. It was sufficiently fast for video even back when disks were 1/10th the speed they are now.
On it's native OS, it could handle multiple streamed data to the same disk and keep it separate by allocating the separate channels out of disparate allocation groups on disk. I don't know how that works on linux. Unfornately, on linux even under x64, file block sizes AFAIK, are still limited to 4K. XFS has a 64K limit, but under linux is hamstrung to 4k. Of course Windows NT allows 64K block sizes. But not linux...hmmm....very weird. XFS minimizes impact of linux's tiny allocation block size by using a extents which can be at least 256k -- but believe the actual limit is in megabytes. Been a while since I read that stuff...
Of course -- not to be linux centric, but have heard ZFS is pretty good, but no idea of how it compares for anything.
my 2 cents...
Re:Check Tuxera NTFS (Score:3, Interesting)
Bzzt... NTFS can't handle filenames that ext3, XFS and other Linux-based filesystems can handle. I went through this dance with my Drobo (incidentally, do not EVER buy a Drobo, not if you care about your data; it's dangerous to store data on that device)
ext3 and the Windows-side e2fs-explorer style packages are fine, or use Samba/CIFS and serve it up that way. I use rsnapshot on Linux to back up my Linux and Windows machines to my NAS, which is ext3-formatted.
NTFS is fine, if you're only ever backing up or storing data that can be created on Windows machines, but not if you want to store data from other machines (i.e. back up a Linux machine for example).
Re:network it (Score:3, Interesting)
And Windows clients frequently have compatibility issues with Samba servers all the time, especially when Microsoft releases updates to the client software. Esp. when it comes to things like domain membership, and file permission.
Certainly not in my experience. Our main fileserver is RHEL 5.4 and we have approx 100 people over 19 locations connecting to it daily. It gets its data from our SAN. At 3 locations we have Fedora 8 systems serving up Samba shares (large local files that aren't good candidates for the WAN). The boxes are set to use domain credentials (Windows 2000 DC's) and everything passes through as you'd expect -- no prompting for credentials and new users' home folders are setup automatically the first time they connect. We also have mopiers in all 19 locations (some locations have several) accessing -- you guessed it -- the RHEL 5.4 fileserver over the WAN. No issues, files are owned by that user (except for folders with sticky groups). Security is a mix of both UNIX and Samba permissions.
Main Terminal servers are 2003 R2 (and all those users access data from this fileserver), user desktops are XP (and my MacBook Pro running 10.4) along with a handful of Vista and Windows 7 systems in use by MIS. Works very well for us. We've been doing this since RHEL 3. It's actually kinda boring since it requires such little maintenance. Several of these servers cross-replicate via rsync every night so the backup copies are always on at least one other location's server (not directly accessible by end users).
Yes, one of these servers uses NFS. I forget which one, since I set it up it's been uneventful. I remember there was a reason I wanted to use NFS but can't remember why, and I'm too lazy to login over the VPN to see which one. I do remember it's something critical we needed and NFS worked better.
Re:Why all those ext3 recommendations? (Score:2, Interesting)
I see quite many people here recommending ext3. Oh my. ext3 sucks for large files,
The reason so many people are recommending it is because it's the best choice, not the fastest choice.
XFS isn't popular enough. Yeah, it sucks, but unless there're other compelling advantages, I want to use code that has lots of other users to get rid of obscure bugs, especially for filesystem code.
ext4 is too new and doesn't offer sufficient advantages over ext3 in this use-case.
ZFS - Pretty much requires Solaris or FreeBSD. The poster obviously already knows Linux, whereas their Solaris/FreeBSD experience is an unknown quantity. Also, for the "generic" slashdot readers out there, Ubuntu/Fedora are a lot easier to install/setup than Solaris/FreeBSD (or Debian for that matter). Now, if ZFS was available natively under Linux, its snapshot/checksum/disk-pool features (all of which are so extremely cool and useful for a home-NAS) might provide sufficient advantage to overcome the fact it's less popular.
ext3 is a mature filesystem that can store large files (large enough for home use anyway) and is easily fast enough to flood a gigabit network connection. It's not that the other filesystems sucks, it's just that ext3 fulfills the role quite adequately and the minor advantages of the other filesystems isn't sufficient to overcome its advantage as the "de-facto" filesystem.
Hybrid? FAT32 and [your-choice-of-fs-here] (Score:5, Interesting)
I know this comment will get lost in the sea of other comments, but my recommendation to you would be a hybrid solution.
Create a small partition (1GB would be overkill) and format it FAT32.
Create another partition for the rest of the drive (or however you please) with your choice of FS (I prefer XFS, personally).
Store the drivers(/utilties) for the FS you chose and store them on the FAT32 drive.
Some popular drivers/utilties for Windows are:
ext2fsd for EXT2 - http://sourceforge.net/projects/ext2fsd/ [sourceforge.net]
rfstool for ReiserFS - http://freshmeat.net/projects/rfstool/ [freshmeat.net]
ltools for EXT2/EXT2/ReiserFS - http://www2.hs-esslingen.de/~zimmerma/software/ltools.html/ [hs-esslingen.de]
and so on and so forth (a simple google for "[FS] Windows Compatibility" usually works.)
Just my thoughts. :-)
btrfs & data checksums (Score:2, Interesting)
Bubba! (Score:2, Interesting)
I bought a BubbaII from http://www.excito.com/ [excito.com] its a small fan less linux box with 2x usb, 2x ethernet, and 2x extSata.
NB: NICs are gigEthernet, but they perform substantially slower than one expects. This according to the manufacturer, is by design to keen the temperature at a resonable level to accommodate the fan less design.
Failure on large files? (Score:3, Interesting)
I've been using this for years, but I do get solid lockups with very large files (over 4GB) which has made me very nervous.. Any idea where I screw up?
Desktop and laptop are still WinXP (home) until I make up my mind what I'll replace it with (it's narrowed down to OpenSuSE or Mandriva), but for some use a Windows desktop is still more suitable (I'd love Paint.NET to run on Linux, for instance).
Which File System? (Score:2, Interesting)
I'd choose the file system based on the system I plan to Host it on. If the host is Windows, I'd go with NTFS. If the Host is Linux, I'd go with EXT2. Regardless of the hosting system and the drive format, you want to make sure you can access the shared contents of the drive from your other systems on your home network. Using this approach, you won't need to worry about third-party drivers.