Recoverable File Archiving with Free Software? 80
Viqsi asks: "Back in my Win32 days, I was a very frequent user of RAR archives. I've had them get hit by partial hardware failures and still be recoverable, so I've always liked them, but they're completely non-Free, and the mini-RMS in my brain tells me this could be a problem for long-term archival. The closest free equivalent I can find is .tar.bz2, and while bzip2 has some recovery ability, tar is (as far as I have ever been able to tell) incapable of recovering anything past the damaged point, which is unacceptable for my purposes. I've recently had to pick up a copy of RAR for Linux to dig into one of those old archives, so this question's come back up for me again, and I still haven't found anything. Does anyone know of a file archive type that can recover from this kind of damage?"
where have you been? (Score:3, Informative)
Re:where have you been? (Score:5, Informative)
wow man (Score:3, Funny)
the mini-RMS in my brain
You really ought to have that looked at..
Re:wow man (Score:5, Funny)
:D
Re:wow man (Score:1)
Ah.... I've met RMS a number of times, and having once made the mistake of standing downwind, am familiar with this problem. Try Soap and Water, augmented with a long-handled stiff brush. Pardon the Pun, but Lather, Rinse, Repeat. Works wonders for your complexion, with immediate secondary positive effects on the `ol social life...
Re:wow man (Score:2)
Are you sure tar is unacceptable? (Score:1, Insightful)
Re:Are you sure tar is unacceptable? (Score:5, Informative)
then 2 years later you want the data back.
there's a read-error at some point within the
bunzip2 will actually be able to recover all other 900kB chunks of the original tar file, except for this missing chunk or part of it.
Tar will just choke at that point and you lost everything past the read error. bunzip2 was able to recover the data past the error, but tar can't use the data.
It's quite frustrating.
Re:Are you sure tar is unacceptable? (Score:2)
I've been there with
depends on the definition of lost (Score:1)
Ofcourse, when the big day came, and my hard drive broke, it turned out the other drive had bad sectors!
First, a comment: never ever ever ever use tar.gz to back up anything you'd like to have back.
You can recover stuff easily from tar past the break point - files in tar are basically concatenated together. So you miss the rest of the current file, but you can find the next header+file easily.
But gzip does not byte-align its data! That's, i
Re:depends on the definition of lost (Score:2)
Re:depends on the definition of lost (Score:1)
But rar would be better if it was more widely distributed and free (speech) - then i'd save a copy of the decompressor and its source code on every CD...
Re:depends on the definition of lost (Score:2)
Re:depends on the definition of lost (Score:1)
It seems that the rock-ridge extension deals with this by putting deeper directories into a dir called RR_MOVED, so I think if you use mkisofs with rock ridge all is fine (which could be another reason that I didn't encounter the depth limit)
Re:depends on the definition of lost (Score:2)
Try apio (Score:5, Informative)
Re:Try apio (Score:5, Informative)
Re:Try apio (Score:1, Funny)
Re:Try apio (Score:1)
Re:Try apio (Score:2)
o Passing a list of files to afio, by default, is a PITA; I much prefer tar / zip / rar for their convenience here, you can pass a wildcard at the command line *or* send them a list.
o I couldn't find a way to do a *partial* restore of a subset of files.
--Anyone got some tips on this? I did write a set of scripts for handling volume changes and the like; if anyone's interested in them, email me.
Re:Try apio (Score:1)
Par2 works great (Score:5, Informative)
Re:Par2 works great (Score:5, Informative)
good overview here: PAR2 files [slyck.com]
comparison between v1 and 2: here [quickpar.org.uk]
Re:Par2 works great (Score:2, Funny)
Re:Par2 works great (Score:1, Offtopic)
Look out for Ashcroft (Score:2)
Re:Par2 works great (Score:1)
Translation: pron moves now.
Re:Par2 works great (Score:2)
rar/par/par2 IP issues (Score:1, Interesting)
I wonder if rar, par and par2 infringe on this patent?
Re:rar/par/par2 IP issues (Score:2)
Re:Par2 works great (Score:2)
Errors would be detected and recovered automatically while PAR2 files scanned for recovery info. Heck, why not stream recovery packets right into the compression stream -- just like solomon-reed and CDROMs.
Can Par2 Do Binary Merge3? (Score:2)
A quick perusal of the QuickPar website [quickpar.org.uk] suggests that at least some Par2 clients can restore based on two damaged files and incomplete recovery files:
In the past, however, I've been dealing with getting remote files over a noisy connection where the remote server wasn't so thoughtful to create Par files or even
Yeah (Score:4, Insightful)
The format you're looking for is any format you like stored on reliable storage.
Why bother with all the intricacies of a pseudo-fault-tolerant data structure? Ultimately the best archive format for recovery will be one that just duplicates the whole archive twice over, doubling space requirements and improving immunity to lost sectors on drives. At which point one asks, "Why don't I just stick to simple files and archives, and use reliable storage that handles this crap for me, for all my data, automagically?" Storage of any sort just keeps getting cheaper and bigger. If you have any interest in the longevity of your data these days, there's almost no excuse for not using the data-mirroring built into virtually every OS these days and doubling your storage cost and read performance while preventing yourself from worrying about drive failure.
Re:Yeah (Score:4, Insightful)
I'm on a laptop. I like my laptop. It's a very nice laptop. However, it doesn't exactly support those kind of hardware upgrades, and I am still ultimately on a bit of a budget.
I kind of put forth the question not only out of the hope that a Magical Solution To All My Archival Problems would Mystically Appear (puff of smoke optional but appreciated) but because I want to find something I also feel like I can unreservedly reccomend to non-ideological friends who are looking for, say, something slightly more reliable than ZIP files. I could've mentioned that in the article post, but it was already getting long.
Re:Yeah (Score:5, Insightful)
Ignore this post. It's either a troll or an idiot.
PAR files substitute for missing pieces. They don't regenerate the whole file by themselves. Go look up how RAID 5 parity [somacon.com] works. They're not called PAR files for nothing [utk.edu].
Just because you don't understand how something works has no bearing on the fact that it does work. Except in certain performance-sensitive cases, doubling up is the least intelligent way of adding redundancy.
---
Dum de dum.
Re:Yeah (Score:2, Insightful)
Obviously you know nothing about error correction, so STFU.
Re:Yeah (Score:1)
There are WAY better (by better I mean take up less space and can detect more errors) methods of error detection (and correction), which have filled volumes of research publications and books, so I will not try to get into them here, but a (maybe not so) simple software trick will definitely sav
doesn't quit hold true (Score:2)
Mirroring HD's only protect against fatal failures of a single HD. Motor stops spinning? Then the other HD takes over.
It does NOT protect against failures on the disc. Errors while writing or reading or other fun stuff.
For true backup you need the following.
cpio (Score:5, Informative)
True, tar cannot handle a single error... all files past that error are lost.
On the other hand, cpio (and clones) can handle missing/damaged data without losing the undamaged portions that follow (you only lose the archived file that contains the damage). It is the only common/free format I can think of (from the top of my head) that is capable of this.
Re:cpio (Score:2, Informative)
ZIP also supports this (the command is "zip -F" with Info-ZIP, the standard zip/unzip program on Linux).
Re:cpio (Score:2)
Re:cpio (Score:2)
Nope, it's a design flaw - and a well known and documented one at that. See the O'reilly backup book (written by former co-workers) for more details (though it was well documented long before that book was written).
Just try to restore anything (past the error) using *any* version of tar, from a tar file (or tape) with an error in the middle. It will bomb out as soon as it hits the error.
Re:cpio (Score:3, Informative)
Comment removed (Score:3, Insightful)
Re:Tar options (Score:2)
I think an important feature of the issue here would be that life is a lot easier if you never get the junk data in the first place.
RAR isn't completely non-free (Score:4, Informative)
Re:RAR isn't completely non-free (Score:2)
The fact that that much exists does ease my mind about existing archives I've got (which is why I didn't mass-convert them ages ago). It's creating new, future archives that I'm worried about.
Re:RAR isn't completely non-free (Score:4, Insightful)
(Personally, I don't care about recovery records, I just keep two copies of everything, and I moved to 7-zip -- which can decompress RAR -- about six months ago.)
What if both copies develop a small fault? (Score:2)
No two files failing isn't likely to happen. We are however dealing here with disaster recovery. Disasters are always disastrous.
Of course error recovery won't work with a total failure like say a fire. Then your second copie is the better solution.
So two copies is a good idea. Error recovery is a good idea.
Two
Re:What if both copies develop a small fault? (Score:2)
At ho
Take a look at dvbackup/rsbep (Score:4, Interesting)
-- John.
Yes... (Score:4, Funny)
Basically you concatenate all the files together (cat should do), print it out on good 32lb paper, get a professor's signature and file it in a college lib...heard those things stick around for centuries
tar/gzip recovery toolkit (Score:5, Informative)
The gzip Recovery Toolkit
/tmp/tar.log 2>&1 & /tmp/tar.log
http://www.urbanophile.com/arenn/hacking/gzrt/gzrThe gzip Recovery Toolkit has a program - gzrecover - that attempts to skip over bad data in a gzip archive and a patch to GNU tar that enables that program to skip over bad data and extract whatever files might be there. This saved me from exactly the above situation. Hopefully it will help you as well.
[...]
Here's an example:
$ ls *.gz
my-corrupted-backup.tar.gz
$ gzrecover my-corrupted-backup.tar.gz
$ ls *.recovered
my-corrupted-backup.tar.recovered
$ tar --recover -xvf my-corrupted-backup.tar.recovered >
$ tail -f
RAR Archives (Score:4, Funny)
Back in my Win32 days, I was a very frequent user of RAR archives.
Bablefish translation: I was a huge warez kiddie.
On a related noted, were there any wide-spread, legitimate uses of
Re:RAR Archives (Score:2)
No, I got into RAR because a friend of mine (who was and still is into video game console emulation, especially music, which is where she discovered the format, I presume) used it to distribute her music compositions for a period, and I
Re:RAR Archives (Score:2, Insightful)
RAR was heavily used in Germany, among the gamer community. A lot of Descent
players for example distributed their custom levels, missions, textures,
hogfile utilities, savegame editors, and whatnot in RAR format. It was
annoying; I had to go hunt down and download a RAR extractor just to install
some of the stuff.
The usual argument was that RAR was "better" than ZIP either because of the
compression rates or because of the partial recoverability or wh
Re:RAR Archives (Score:1)
No, I was talking about an archive file format, and specifically about
selecting an archive file format for distributing stuff on the internet.
Using RAR or bzip2 for that is like sending richtext email or using
browser-specific markup in a web page.
Additionally, how many people *use* something is almost always irrelevant.
What I was talking about was how many people have software (any software)
that's capable of opening the fo
Re:RAR Archives (Score:2)
You do have a point, but on the other hand, everything has to start somewhere. Things have to evolve, we must move on to better things sometimes.
Just the fact that
Otherwise we'll be using the
Re:RAR Archives (Score:1)
Just make duplicates. (Score:2)
Total loss of the file seems more likely than bit flipping by themselves.
When your storage hardware/media starts flipping bits, it's probably going to die pretty soon.
And more often than not, your storage hardware/media just dies before you experience any bit flips.
You talk about your laptop computer and being on a budget. If you can't afford to make copies of your important files and store them elsewhere, then either your files aren't important, or successfully maintainin
Re:Just make duplicates. (Score:2)
Also, I almost never find the time in the week to do backups. I work for a nonprofit, so My Time Is Semiwillingly Not My Own.
That is an incorrect assumption (Score:1)
Re:That is an incorrect assumption (not always) (Score:1)
Re:That is an incorrect assumption (not always) (Score:2)
2. Ditch the old *nix box and switch to Linux / OS/X / *BSD
3. Switch to GNU tar or Joerg Schilling's "star"
4. ???
5. PROFIT!
tarfix (Score:3, Insightful)
may help some of those archive issues.
But, the archive format is not going to save you. Use multiple media. You need more than one physical archive for better safety, regardless of format. Hell, you'll probably die before some of today's media fails.
a different question (Score:2)
Re:a different question (Score:1)
originally designed to let you fit large files
onto floppies, but can be used for everything,
splitting dvds onto cds, etc. from the manual:
-v[k|b|f|m|M]
Create volumes with size=*1000 [*1024 | *1].
By default this switch uses as thousands (1000) of bytes
(not 1024 x bytes). You may also enter the size in kilobytes
using the symbol 'k', in bytes using the symbol 'b',
in megabytes - 'm', in millions of bytes - 'M' or select
one of several predefined
Re:a different question (Score:2)
Rar has one of the best Recovery methods (Score:2, Informative)
rar has one of the best recovery methods, as it has mutliple of them.
during compression:
Recovery Record (-rr option)
it has Recovery Record, this is data appended to the actual
rar file that lets you recover from errors within a file. The
default RR takes 1% of the archive and lets you recover 0.6%. You
can change this behaviour to going more recoverability by
specifying -rr[N]p and telling it larger percantage for recoverability.
Recovery Volume (-rv option)
further more, ra
Re:Rar has one of the best Recovery methods (Score:2)
Re:Rar has one of the best Recovery methods (Score:1)
winzip to winrar:
- same price
- winrar comes with full command line, and gui interface
- both support variaty of decompression schemes
- zip provides worse compression ratios than rar
- zip has virtually no recovery methods
- zip has no multi archive support (unless you
consider the current hack as a valid system)
- zip uses pathetic encryption (password breakers
exist for over a decade, rar still has not a
single password breaker)
So crappy Zip has same asking
Re:Rar has one of the best Recovery methods (Score:2)
--However, I *would like to* support Rar, as I perceive it to be a superior archiver. I could see my way to donating $10-$15 to help the guy make a living, but the current price is too much.
(I hardly ever use Windoze for anything
There is a Free Software extractor for RAR files (Score:1, Informative)
DAR & Parchive (Score:2)
From http://dar.linux.free.fr/ [linux.free.fr]:
dar is a shell command, that makes backup of a directory tree and files. It is released under the GNU General Public License (GPL in the following) and actually has been tested under Linux, Windows and Solaris. Since version 2.0.0 an Application Interface (API) is available to open the way to external independent Graphical User Interfaces (GUIs). An extension of this API (in its version 2) is in th
You can, too, recover tar archives!! (see: tarx) (Score:2, Informative)
It's been possible to do that for well over a decade, using various utilities such as tarx. I've successfully recovered files after a damaged point in a tarball many times. (Sigh, I used to use an old AT&T UNIX with a #$*@# broken tar, which occasionally created corrupt tarballs).
See this post [sunmanagers.org] on the Sun Managers list circa 1993, and the venerable comp.sources.unix collection, volume 24, for t