What Software Do You Use for Unix Backups? 216
jregel asks: "Linus has stated that
dump should not be considered a reliable backup program, and both tar and cpio have their limitations. So what are Slashdot readers doing for backing up Linux servers and workstations? (you do backup, right?)" Given this bit of news, have you used anything other than the standard Unix staple to back up your Linux boxes? If you were forced off of tar, cpio and dump, what would you use as a replacement?
Easy. (Score:4, Funny)
I'd use dd of course...
Re:Easy. (Score:2)
I was actually thinking, what happened to pax? Supports both cpio and tar formats, worked great last time I tried it.
taper (Score:2)
9 out of 10 network admins smack their tape monkeys when they forget about modprobe zftape after reboots.
Re:taper (Score:2, Interesting)
dump on solaris... (Score:5, Informative)
I found out that solaris has a very interesting command: fssnap
It creates a read-only snapshot of your filesystem intended for backup operations.
You create a snapshot, dump the snapshot, then delete the snapshot and the dump is consistent.
I wonder if there's something like this for linux...
Re:dump on solaris... (Score:2, Interesting)
Re:dump on solaris... (Score:3, Informative)
Re:dump on solaris... (Score:3, Informative)
The snapshot partition just has to contain enough space to hold the changes made to the original volume while the snapshot exists.
Re:dump on solaris... (Score:3, Interesting)
If you are using EMC Symmetrix storage, you can use the TimeFinder [emc.com] product to create a "Business Continuence Volume", or BCV. It deltas against your last backup (at the track level, not files or blocks), applies changes to a copy of the last backup to create a consistent image, then you can dump that to tape.
I wonder if there's something like this for linux...
So long as you have one host (Solaris, NT, whatever) to run the TimeFinder client
Re:dump on solaris... (Score:2)
One thing to beware: a lot of vendors ship the snapshot capabilities as an added cost option; if you intend to use it, make sure of your costings.
Re:Timefinder on EMC (Score:3, Funny)
I had thought we had found the answer to getting a six-month project done in 3 months - use "TimeFinder" by EMC.
-Peter
Re:dump on solaris... (Score:2)
I started wondering if there's something like this for Linux, but then realized that there's just way too many different filesystems to add the feature in any meaningful, practical way.
Re:Linux is about the only... (Score:2)
Unless the new kernel offers something you or your customers need; theres no need.
Re:Linux is about the only... (Score:2)
Wrong: EVMS (Score:2, Informative)
It's a simple patch you can add to any 2.4 kernel.
Re:Linux is about the only... (Score:3, Informative)
You do realise that dump doesn't give you a filesystem snapshot? Even on Solaris - the most venerable of modern UNIX - the manpage for ufsdump clearly states:
There's a good reason why nobody seriously uses dump anymore.
And Linux does sup
Re:Linux is about the only... (Score:2)
Roll Your Own (Score:4, Interesting)
I wrote my own (Perl) script, that copies all my "important" files (basically stuff in my home directory that can't be reconstructed by other means and all the system config files) to a new directory tree (using cpio) it then burns the copied tree to CD-RW and verifies the CD against the copied tree.
I operate a 4 disc system, so I always have the last four backups on CD and I keep the copied trees around (uncompressed) for as long as I have disk space. So far I've not needed the CDs (I store 2 of them offsite in case of disaster) but the copied filesystem trees have come in useful a couple of times.
The only drawback of this is it's not appropriate for backing up huge quantites of data (like lots of audio or video files) as the CD media is quite limited in size - but when rewritable holographic storage comes along I'll be able to just change my function that decides which files are "important".
Re:Roll Your Own (Score:3, Informative)
That's what I used to do, (wrappering tar) but the matanence of the script became a pain and I needed to add support for incremental backups, and exclusion lists.
After some web searching, I on google, fre
Re:Roll Your Own (Score:2)
I rsync my entire drive to another drive.
Incremental backups with rsync (Score:3, Informative)
No good answer (Score:2, Insightful)
Veritas also comes up a lot. Aside from cost, did you know Veritas can't back up single files larger than 2GB in size on Linux clients?
On paper, BRU looks pretty darned good. I haven't yet put that theory into practice.
Re:No good answer (Score:3)
If you really, truly need your data, no matter what, go with Tivoli Storage Manager
http://www-3.ibm.com/software/tivoli/solutions/ s to rage/
Sure, you have to pay for it, but it's really no more expensive than Veritas NetBackup, and certainly a better product!
Cross-platform (everything from Wintendo to OS/390, Solaris, Mac OS X, Linux
TSM is more of a hierarchical storage manager than more "traditional" backup programs.... but with things like Portable Backu
BackupEDGE vs. Taper (Score:4, Informative)
I'm actually doing a 100gb backup as we speak... so good timing on the Ask Slashdot.
My only beef with Taper (and I'd use it otherwise, on my home system) is that when you do an "e"xclude or "i"nclude of a directory, it scans the entire subtree, which can take *forever*, (like when excluding
mindslip
Taper limitations (Score:2)
rsync (Score:2)
Re:rsync (Score:4, Informative)
The rsync over ssh style of backup is so easy it's addictive!
Re:rsync (Score:2)
Re:rsync (Score:2, Informative)
Re:rsync (Score:2)
This makes rebuilding the system fairly trivial if the first hard drive fails.
It also means I can recover from mistakes if I notice them the same day.
Why not fix dump and/or Linux? (Score:3, Insightful)
Seems to me that Linus (or another kernel hacker) should fix the ext2 race condition reported in that thread, rather than blithely dismiss the problem with, "dump was a stupid program in the first place."
Re:Why not fix dump and/or Linux? (Score:2, Funny)
On the other hand, is anyone who wants to take a dump on Linux likely to contribute good code?
Re:Why not fix dump and/or Linux? (Score:3, Interesting)
It shouldn't be any major problem to build indexes over your tar files. What Linus argues that dump does incorrectly and tar does correctly is the reading of the filesystem. The way to read a live filesystem is through the filesystem API, not by blockwise reading of the device. All your arguments against tar and in favor of dump is about the output produced by the program. If you want output like dump produces it, then write a to
rsync (Score:3, Informative)
This backup machine keeps seven generations of daily backups on one disk (cp -al, so no duplicating of static data), and a few weekly ones on the other disk. Every night it rsyncs things off-site (to my home). That rsync has turned out to be unreliable (probably my adsl), so I have a script that does it in small bits and pieces. Takes a few hours in the early morning.
Re:rsync (Score:3, Informative)
It is about eight zillion times better than tapes. I have hot, random access to all versions of all my files. Thanks to the hard linking, space used is moderate. Since it backs up to a remote computer, backups are instantly off site. And if I want to verify my backups, I don't have to feed in eight million tapes; I
Re:rsync (Score:2)
Yes! rsync! (Score:2)
On the Mac, I use RsyncX [macosxlabs.org], which knows about resource forks, even when transferring them to systems which don't have them.
And on Windows, I use rsync [unimelb.edu.au] again.
I've tried every damn sync program for the Mac. I've tried tar and dump on UNIX. I've tried fancy network backup tools. I've not found anything that compares with rsync.
I hate the complexity of the command-line syntax, but it has the required functionality:
1. Automatically incremental.
2. Works locally from disk to disk or acro
cdrtools (Score:3, Informative)
cdbkup (Score:3, Interesting)
"CDBKUP is a professional-grade open-source package for backing up filesystems onto CD-Rs or CD-RWs."
quick way to lose files (Score:2)
Then there's the other things that don't translate well. Do you deference symbolic links? What about fifos and special devices?
If you want to be safe, you need to either check the directory tree first or put everything into a container without these restrictions. I've been developing some tool
Amanda (Score:2)
nice - can use tar or dump as the back end system. Works on *nix/ MaxOSX/windows via samba or cygwin.
Snapshots (Score:2)
I'm using LVM as a wrapper around Amanda to create and remove snapshots of every filesystem I'd like to backup.
Amanda! (Score:5, Informative)
I have been extremely happy with Amanda [amanda.org]. Single centralised backup server running amanda-server. Multiple workstations running the amanda-client. Amanda automagically schedules backups based on sensible heuristics. I just tell Amanda how many tapes I have, how many workstations I have, and Amanda does all the hard work of working out how much tape capacity is required and how often it should schedule incrementals/fulls.
The server/client protocol has been designed to avoid reliance on dangerous security holes like rsh. The server sends the client a "send me your dump" message. The client then connects back to the server and delivers it the output from dump or tar. You can configure exclusion lists on the client if you're worried about sending certain files or filesystems. You can also encrypt the data stream and/or use Kerberos for authentication.
If I forget to load a blank tape then Amanda plays it safe. It doesn't overwrite last night's backup: instead it stores incrementals into the "holding disk". Amanda will then flush the held backups to the next blank tape.
Amanda emails me reports after every backup with a neat summary of what went right/wrong. It also gives you several hours advance warning if you forget to load a blank tape or if any of the workstations are offline.
The only downside of Amanda is that it is fiddly to setup. The documentation is poor and the configuration files are cryptic. But if you're willing to invest some time and effort then you can't do much better (for free) than Amanda.
Re:Amanda! (Score:3, Interesting)
Yup, Amanda is great for small setups (I use it myself at home) but it lacks certain features to make it really usable. For example, I had to restore some files in Legato Networker; I was able to open up a GUI, navigate to the file and set the restore path (i.e. where it will restore to). With that done, it worked out which tapes the
Re:Amanda! (Score:2, Informative)
Amanda does the same thing, it's no problem. Yes, spanning tapes is a problem, but people might be working on it now. You can get around it by just backing up files, or directories, under the filesystem, in increments that are less than the tape size. I use it at a couple of different work locations, and it has worked r
afbackup (Score:4, Informative)
Features:
For those who don't know: AMANDA cannot append to tapes.
Every time you backup with AMANDA it must start from the beginning of the tape.
So, if you want backups every day, you must have a tape for every day.
(http://amanda.sourceforge.net/fom-serve/cache/29
Re:afbackup (Score:5, Informative)
Also tisk of appending is loss of tape or drive due to environmental factors - fire/flood (plane being driven into data centre).
Re:afbackup (Score:2, Informative)
In addition, unless you own a autoloader/robot unit, using a bac
Re:afbackup (Score:2, Insightful)
Arkeia (Score:3)
anyway, arkeia can back up windows, linux, unix, and mac osx.
Re:Arkeia (Score:2)
It's proprietary software, but has proven exceedingly reliable for backing up my entire network onto a tape library. Basically, it's cheap for what it does, and depending on how you use it -- it may be available at no cost.
~GoRK
Re:Arkeia (Score:2)
I evaluated this product and really liked it. I would be using it now except that I found out our Windows guy was using Veritas and had already had it paid for, so I installed the free Veritas Linux clients and pawned the backup job off on him
mysqlhotcopy, samba, tar, gzip, scp (Score:2)
First, I use mysqlhotcopy to get all sql data. Then, my backup server uses samba to tar and gzip up all data from various servers, Windows and Linux, into one place. Then it uses scp to send it all across the WAN to another backup server which keeps a business week rotation, and one month rotation. The other site does the same, and so far no problems at all.
This way you don't have
Not Free, but does work nicely. (Score:2)
Works for windows and Unix(AIX, Solaris, HP-Ux) but I don't see support for Linux but I would guess that if you can get it to work on the above there is a tweak to get it working.
Just anouther option, I know it is not going to be the flavor of the month because it is not free or OSS.
Enjoy,
Backup2L (Score:3, Informative)
Jason
Question on restoring .bxf files. (Score:2)
what tools will restore a backup done with Windows 2000/XP under Linux?
Under win95/win98, you can smbtar the entire remote drive into a compressed tarball. To restore, fdisk a new drive, format it, and tar -xjpf tarball.tar.bz2, and possibly sys C: it once it's back in the windows machine. Windows takes care of anything else that needs to be done.
Under Win2000/XP, obviously this won't work, so you need to use Windows's backup or other tools. But if you want to restore
Re:Question on restoring .bxf files. (Score:2)
you can connect to win2k/xp with smb just as easily as win95/98
what is the problem?
Re:Question on restoring .bxf files. (Score:2)
you're still stuck w/ the permissions thing, maybe you can convince tar to store permissions info, and use one of the ntfs's thats writable from linux to restore. Depending on your available network resources, you might consider having the user directores be remoately mounted (for ease of backup), so then you would only need to restore the os and installed programs if the hard drive fails.
I cannot be forced off tar (Score:5, Interesting)
They say tar has its limitations. I really dont understand.
Ive worked with different unixen and Linux distros, so I just dont want to be dependant on something that isnt installed by default everywhere. tar already has a VERY well known format and execution parameters.
Ive lost my fair share of data to buggy harddrives and dumb mistakes like pulling off the ide cable while the system is running. So cron does daily backups using tar cfj using a file that has a list of other files to be backed up. This way I dont have to backup the whole partition. To restore a certain file, just tar xvfj backup2.tar.bz2
The cron setup renames backup.bz2 to backup2.bz2 and removes backup2.bz2 so I have the data for the past two days. Beside incremental backup which I dont need due to this setup, what else could I need? And by the way the backup.bz2 is copied off onto an NFS share elsewhere incase my whole RAID setup crashes, or the XFS filesystem bombs out. This setup can be replicated onto FreeBSD Solaris and many others.
Re:I cannot be forced off tar (Score:2)
Also, if tar encounters a bad spot on a tape, it usually bombs out. cpio can be told to skip over the bad sector.
There are advantages and disadvantages to all the backup programs. I don't think that there is one program that is "perfect" for ever
Re:I cannot be forced off tar (Score:4, Insightful)
Not a criticism of your method (in fact, I use this), just a rant that the Linux MM system NEEDS TO BE FIXED. I'm sick of watching as some trivial process that will only read or write once gets the whole filesystem cached for it while programs I'm using interactively get swapped to disk. Video recording and playing programs (mplayer, ogle) have the same problem.
Let's hope 2.6 is better than 2.4. Can any kernel hackers comment on this? In 2.5 will tar cvjf /home /mnt/backup/home.tar.bz2 bring my system to its knees?
-- Bob
Re:I cannot be forced off tar (Score:2)
Re:I cannot be forced off tar (Score:2)
Perhaps because the NICE level dosn't impact anything but CPU timeshare? So a nice 19 tar -czvf /tmp/totape.tgz /home will still thrash the hell out of your system.
The semantics are fairly trivial: This process is generating a lot of disk cache that's only being hit once, so let's bound how much memory it uses.
The reality is much trickier. It'
Re:I cannot be forced off tar (Score:2)
It is the constant swapping-in and swapping-out that make the system unusable. Nice has absolutely no
Re:I cannot be forced off tar (Score:2)
Re:I cannot be forced off tar (Score:2)
Thinking like a bonehead, indeed. I got all focussed on keeping your interactivity up... and probably never finished the manpage to nice in the first place.
Hey, good thing I phrased that as a question, so that I'd get an intelligent answer as to why, rather than get my ass flamed off.
Off to blithely trash my system because I'm too retarded to boot it without pouring hot coffee in the power supply vents.
Re:I cannot be forced off tar (Score:2)
Umm, I use 2.5 myself and I dont know if the 50MB file brings it down. Its a pentium200 with 64MB ram, 256mb swap, and all backups occur at 4am. I remember testing it some time ago I think it finished the job clean while on the same lousy system I was running X and reading hotmail email using opera and twm. Maybe because its 2.5...
tar does not do incremental backups (Score:2)
A decent backup tool (as opposed to an archival tool) must absolutely have incremental backup support.
Re:tar does not do incremental backups (Score:4, Informative)
> backup, say gigabytes of data, daily.
>
> A decent backup tool (as opposed to an archival tool) must absolutely have
> incremental backup support.
Er?
tar --help
[snip]
Operation modifiers:
-G, --incremental handle old GNU-format incremental backup
-g, --listed-incremental handle new GNU-format incremental backup
[snip]
Local file selection:
-N, --newer=DATE only store files newer than DATE
--newer-mtime compare date and time when data changed only
[snip]
This is in tar (GNU tar) 1.12
(Which is really really old actually.. slackware 3.2 dist)
There are also tons of options to exclude directorys and files, to force it to span disks, and pretty much match in any way you need.
I've been making incremental backups (and even restored a few) for awhile now.
Re:tar does not do incremental backups (Score:2)
But, seriously. If you back up Gb of data and millions of files with tar periodically, I bow to you. Don't get me wrong, I like this tool (I happen to use it every day), it's just that the incremental backup support you mentioned is fairly primitive (it almost always needs custom helper scripts) and not at all adequate at that scale.
tar lacks things related to data management, which I kind of expect when it comes to periodic backup software. An example is file a
Re:Moderators on drugs again? (Score:2)
While we're at it, President Clinton is out of office now, Puff Daddy has changed his name two or three times, and the Florida Marlins are no longer the World Champions.
CD's and harddisks (Score:2)
Re:CD's and harddisks (Score:3, Informative)
(Obviously I don't like working in directories with thousands of entries, but some tools will produce them, it's easy to accidently hit numbers like that with mail or news spools, etc.)
As for the RW media, you do realize that they have a
Use dump. (Score:2)
* it's not better on other platforms
* the other tools aren't worse
Elizabeth Zwicky's classic Torture-testing Backup and Archive Programs [linuxfromscratch.org] will give a whole list of reasons why you should be suspicious of tar or cpio.
Heck, the FreeBSD Handbook [freebsd.org] answers the question "Which backup program is best?" by saying "dump(8). Period."
Re:Use dump and lose data (Score:4, Informative)
Dump works by reading the raw data partition. That works great with an unmounted partition, or if you have a very limited OS that does not perform any caching.
But Linux is different - it's now using the cached pages as the primary content, usually flushing them to disk only as the pages are dropped. This is the approach used by most mature OSes, but Linux doesn't yet have an interface for "dump" programs to query the OS for updated but unwritten sectors.
So dump is the worst of all possible things now. Not only will you get incomplete live files, you can get incomplete files even if the users have all terminated but the pages haven't been flushed to disk yet. That's non-deterministic, and there's simply no way for you to perform reliable dumps.
On the practical side, dump is specific to the filesystem. When everyone ran ext2, that wasn't a problem. But now people may have a mixture of ext2, ext3, reiserfs, xfs, jfs, and probably even other formats. Each requires their own dump and restore, and that requires a lot more effort.
mkisofs OR cp -ax (Score:2)
Small, important or irretrievable stuff gets mkisofs [even -J!] to CD-R.
Try star by Jörg Schilling (Score:4, Informative)
In addition to amanda, I have good luck with star [fokus.gmd.de] coded by Jörg Schilling [fokus.gmd.de]. star is very feature-rich, fast, standards compliant and has been around since 1985. Give it a try!
The star-users mailing list is here [berlios.de] . You can also look at the man page [fokus.gmd.de] and finally download it [berlios.de]
Hotswap IDE (Score:2, Interesting)
Making a backup is easy. I just plug in the bracket and start a homebrew script which:
- enables and inits the hotswap IDE channel
- mounts the partitions on the hotswap HDD
- removes system immutable flags on files on the hotswap HDD (so that they ca
BackupPC (Score:4, Informative)
Automated backups to an online disk server, open source, and a really nice web interface as well as command line interface.
It uses samba and ssh to backup and restore to windows and unix machines.
You can have it restore any files/folders in a backup you select, using the same methods (samba or ssh) as well as it can send the restore files to your browser in a tar or zip file.
I recently replaced a machine using amanda and a DLT drive with a fileserver using a raid 5 array and backuppc. Best switch ever.
We rolled our own... (Score:2)
Re:We rolled our own... (Score:2)
LoneTar... (Score:2)
TSM (Score:4, Interesting)
I've worked at places using Legato and Amanda, where restoring from backup was an unreliable and error-prone process more likely to be a waste of time than anything else.
TSM is not cheap, but is worth every penny. We have one full time and one part time employee handle the backup/restore jobs for about 2000 servers. Try that with Legato or Amanda.
The dangers of backing up live systems (Score:2, Interesting)
Companies with money can get a netapp box for critical data. Here you can absolutly use dump, tar or cpio. They create a "snapshot" of a file before backing it up.
Unfortunately we are talking a minimum of $40k for this type of solution.
If the snapshot comcept could be w
Re:The dangers of backing up live systems (Score:3, Informative)
In FreeBSD 5.0, you can dump(8) a snapshot [google.com]. I'm not sure if we're using snapshot in exactly the same way, but the point is that you're backing up a static "picture" of the filesystem, while the real filesystem can still be used read/write.
The best part is the FreeBSD costs considerably less than $40k.
Re:The dangers of backing up live systems (Score:3, Interesting)
Pax itself... (Score:2)
Backup Exec (Score:2)
For linux we create a SMB share with samba that the backup server has access to. All files are either tar-gzip'd or just copied over to the directory. Everything in the directory is backed-up.
Problems... (Score:3, Insightful)
I've got my own set of scripts (Score:2)
There are other functions to filter the files to add, and it can also include other files. If running as root it will switch to the user that owns the included file, and not allow including any files not owned by the file's owner. I use this to let people with an account on my system configure how their stuff gets backed up.
This simply generates a file
rsync (Score:2)
Good coding practices (Score:2)
Specifically, how many programmers routinely get advisory write locks on files they plan to update? How many home-brewed or ad-hoc backup solutions bother to get advisory read locks?
I've written
My Solution (Score:2, Funny)
The opensource solutions are pretty basic (Score:3)
At work, we use Veritas Netbackup. Having used both it and Tivoli Storage Manager, TSM is easily the better of the two.
[1] Estimates? Estimates? Just run the bloody backup.
NetVault (Score:2)
It's way! cheaper than BackupExec and kiskcs butt! Highly reccomended!
LK
Works for me ... (Score:2)
Who would take it away from me? And where do I ... (Score:2)
Before I start looking for new backup tools, I would look for the one responsible for removing my tools in the first place.
A possible(read: theoretical) form of backup would be to use the various online search engines as distributed backup mediums. Ie, convert your data into various web pages which are encoded. Since webcrawlers will crawl a site and attempt to store/cache te data(google, the wayback machine, etc), your data is, in theory, cached on those crawler databases.
The only problem with this idea
Re:Snapshotting (Score:2)
Re:ARCserve, unfortunately (Score:2)
For large scale systems Legato would be my choice but, the cost of Legato rules it out for anything but
Re:Lonetar and rsync (Score:2)