How Do You Store and Reconcile Email Archives? 380
heyitsjustme wants to know how you deal with old email. "I delete most of what I get but keep the stuff from friends and relations as an archive. Unfortunately I have these email archives from the late 80's through today in the form of macintosh, linux and windows mailboxes including AOL 1.0 mailboxes. What does everyone use to archive email across multiple platforms and non-standard mailbox formats? Is there an easy solution out there? Does anyone archive IM?"
Italian school of driving (Score:4, Funny)
Re:Italian school of driving (Score:4, Interesting)
American school of driving (Score:2, Funny)
Here's what I do... (Score:5, Funny)
Re:Here's what I do... (Score:5, Funny)
One day... someone... somewhere is going to invent some sort of mechanism for removing text you've already typed. It shall be called "back-one-space" and will remove the letter before it.
If this is impossible, surely they can keep a way of having all our text auto-submitted!
Re:Here's what I do... (Score:3, Funny)
rm -fR /var/spool/mail/* (Score:3, Funny)
and you are done!
Disk space is cheap. Why bother deleting? (Score:5, Insightful)
Thunderbird is able to import all my old mail archives (from years and years of Eudora) and search it effectively. If I were inclined to export all my archives from my Mac to my Windows machine, I could use Google Desktop Search to really search through it all.
Re:Disk space is cheap. Why bother deleting? (Score:5, Insightful)
Because if you delete early and often, you've committed no crime. If you wait to delete it until someone (feds, cops, *IAA, UN-black-helicopter troopers, whoever) demands you turn it over to them, you're screwed.
After all, you break laws too (everybody does, they are written that way). You just haven't been caught yet. (I know this because if you had, you wouldn't have all you email archived!)
Re:Disk space is cheap. Why bother deleting? (Score:3, Informative)
After all, you break laws too (everybody does, they are written that way). You just haven't been caught yet.
Instead of deleting all your e-mails "early and often" why not just delete the ones that have illegal activity in them? Or better yet, don't conduct illegal activity via e-mails. Those ar
Re:Disk space is cheap. Why bother deleting? (Score:3, Insightful)
For the majority of slashdotters this wouldn't be a problem, as I'm pretty sure American laws can't be retroactive. If I ate a chicken today and e-mailed someone saying I ate a chicken, and tomorrow it became illegal to eat chicken, I can proclaim to the world "I ate a chicken on the 12th of March" and I won't be able to be charged with any crimes.
However given your choice of words (regime change) I fig
Re:Disk space is cheap. Why bother deleting? (Score:3, Insightful)
Re:Disk space is cheap. Why bother deleting? (Score:3, Insightful)
ex post facto Law (Score:3, Interesting)
Dave's top ten (Score:5, Funny)
9. Disks fill up, no matter how cheap they are. Low cost doesn't excuse gluttony.
8. Backups take forever.
7. Restores take an eternity, especially if your not confident.
6. Mail client gets slower and slower.
5. Searches take too long.
4. Mail clients make mistakes, especially on big stores. See #7
3. Your CYA evidence may be used against you.
2. A mail store is not a file system and SMTP is not a file transfer protocol.
And the number one reason to delete your old email...
1. IT'S ALL A BUNCH OF USELESS CRAP JUST AS IT WAS WHEN YOU FIRST RECEIVED IT!!
Re:Disk space is cheap. Why bother deleting? (Score:2)
Awesome idea. I'm gonna be doing this.
Re:Disk space is cheap. Why bother deleting? (Score:3, Informative)
If you do this religiously, you will only ever have to worry about your current mail format, and how you're going to upgrade it all
Re:Disk space is cheap. Why bother deleting? (Score:3, Interesting)
Outlook + IMAP is the way I do it. You can drag messages between local storage and your mail server.
I work for Microsoft (Score:5, Funny)
I'm afraid... (Score:3, Insightful)
Re:I'm afraid... (Score:2)
And assuming you still have an app to read that: I've got some old netscape (2.x, I think) mail folders that I can read with a text editor, but I no longer have a program that will open it - there may be, ut I haven't looked that much really.
email archive (Score:4, Funny)
You must work for microsoft
Cyrus Imap... (Score:2, Interesting)
Re:Cyrus Imap... (Score:2)
It's still all available when I want it, but the day-to-day use is faster.
Unix mail format (Score:3, Interesting)
Re:Unix mail format (Score:3, Interesting)
Since mbox is
PDF (Score:4, Insightful)
Spotlight and Tiger (Score:5, Insightful)
Re:Spotlight and Tiger (Score:2)
Re:Insightful? (Score:2)
Re:Insightful? (Score:2)
PDFs more durable than mail, perhaps (Score:3, Insightful)
One Word (Score:5, Insightful)
Re:One Word (Score:2, Interesting)
I started running my own IMAP server on an old machine a year or so ago - and synced all my old mail archives to various folders.
My mailserver also solves another problem - multiple POP accounts. I have my IMAP server set up so that each one of my POP accounts gets automaticly tagged and sent to it's own folder.
A third common problem this solves is having multiple machines. Now my desktop's email client is always synced with my laptop's email client. Before I had run into problems when ev
Re:One Word (Score:2, Insightful)
Re:One Word (Score:5, Interesting)
Absolutely. I use no fewer than two mail clients on two different machines on any given business day. Every email I've sent since 1995 or something like that, and received since 1998 is available and searchable. Over this time, I've accessed this archive with the following clients:
* pine (lots of pine)
* mac mail
* thunderbird
* various netscapes/mozillas
* ML (some random IMAP reader)
* My phone (my old Sony/Ericcson speaks IMAP)
* My palm (two different apps)
* python
* a java webmail system I wrote
* three or four other webmail systems
* mutt
Re:One Word (Score:2)
Assume I have Linux or Windows boxes available (Would prefer Windows, yeah blah blah but other people don't like me 'wasting' hardware on *nix), and I use Outlook for my primary mail client.
Outport & recursive IMAP folder creation (Score:4, Informative)
Also, anyone know of a client program that will recursively create folders on an IMAP server (maybe a server issue. In which case, what server?)
I had gotten over translating my years of Outlook email into something more universally readable, but I have so many nested folders that the inability to have the client recirsively create IMAP folders is an issue. Suggestions?
Re:One Word (Score:3, Informative)
I used the UW IMAP server, which is a little easier to set up than the Cygnus one.
The UW IMAPd keeps its folders in mbox format, so it's a great tool for converting oddly formatted mail.
Moving email is pretty easy -- it's harder to move calendar entries, address books, notes, and the other sorts of data that ends up in a program like outlook. I think the easiest way to do it wo
It's simple: plain text (Score:5, Insightful)
I burn it to CD-Rs that I know won't get moved around or scratched. They stand a good chance of lasting the rest of my life.
Re:It's simple: plain text (Score:3, Informative)
No! Check those backups! I have lost data stored on CD-Rs (luckily I had copies), and many of my discs have started to turn yellow after about 2 years! Also, you can sometimes see these little spots of discolouration on the CDs, which makes me think there's a fungus of some sort that's eating them.
The lifespan of CD-Rs is unknown at this point. Don't trust them for more than
Log everything... (Score:2)
My email isn't in quite the mess yours is, (I used Eudora for almost all of my emailing since I first got on the net, and have just imported from one version to another, and now into Thunderbird.)
I would simply start copying & pasting, or see if you can try importing into excel.
Re:Log everything... (Score:5, Funny)
Hey B5_geek, here's a trick to free up a lot of disk space *and* raise the S/N ratio in your logs:
mv irclog.txt irclog.txt.fat && grep -vi lol irclog.txt.fat > irclog.txt && rm -f irclog.txt.fat
Upon Searching.. (Score:5, Informative)
More utilities than I want to bother with, but hopefully they'll have the converter(s) you need.
Good Luck!
gmail (Score:2)
Your favorite online storage (Score:5, Informative)
This might be useful, if they don't collapse under /.
Simple.. (Score:2, Funny)
Don't change e-mail clients (Score:4, Informative)
What I do at year end is move all of that year's messages to a new folder and reset my filters so that the new year's messages go into a new set of folders.
Periodically I just copy off previous year's messages to CD.
At least few times I have been able to back a couple of years and find information that I lacked.
Kinda Sorta OT (Score:5, Interesting)
Along these lines, is there an OSS package that can read the varied formats the Submitter is referring to, tag and drop them in a DB with a nice, friendly, web-enabled (secure) front-end for searching?
My former employer kept *all* of his email from the last 20 years in tar.gz files. Let's just say it wasn't easy to find an email from er, 15 years ago very easily.
Is there a package that can read the mbox, the other box-formats, plain text, pull from pop, old tar.gz bundles, categorize (sorta), tag and make such things searchable?
Totally a shot in the dark here, i'm not a mail guy at all
It is the "drink" that makes me wonder, sorry
Re:Kinda Sorta OT (Score:3, Informative)
Yes there is, check out
http://www.greenstone.org/cgi-bin/library [greenstone.org]
How I archive my mail (Score:2, Insightful)
Convert to MBOX format (Score:5, Insightful)
Almost every email client around can import and export mbox formats. Getting your email in a format that is going to be readable in 20 years is the first step, otherwise why bother?
Worse comes to worst mbox is readable as plain text.
Or use maildir (Score:5, Informative)
Personally, since 1999, I've been using a combination of maildir and procmail to archive and save my mail. Every message that comes in, goes to a folder called
I wish... (Score:2)
Closest thing to a helpful feature I've seen is the importing
One's things sure (Score:5, Funny)
Hmm that's easy (Score:2)
Every email client worth the name understands that, for the good reason that it's the format they receive emails in
For nonstandard forms of archives (perhaps old AOL clients and whatnot), you're probably left either (1) perl'ing a convert script or, if you can (2) fire up the old client in Win95 in VMWare or something and fwd the mails to yourself (tedious).
My technique (Score:2)
Messages are not sorted into separate subject folders. They are all in a single mailbox, the mailbox. Every month I back up this mailbox to the name of the previo
mbox or maildir (Score:3)
Re:mbox or maildir (Score:2)
Archive what? (Score:3, Interesting)
Sure, HDD space is cheap; but I tend to equate people who archive every single form of written communication to those who have an Obsessive Compulsive Disorder, in that they hoarde everything in sight: newspapers, snail mail, magazines, boxes, etc..
Commit to memory and destroy the evidence. Thats my way of handling archives.
Re:Archive what? (Score:4, Funny)
Re:Archive what? (Score:2, Informative)
Actually, yes. I did recieve a letter from my grandmom a week after she died. Snail mail works very slow indeed.
Reading the letter was strange. The content wasn't strange, just the feeling you get from recieving information from a dead person. That's all I'll say about it.
Cue the "I read dead people's email" jokes..
Re:Archive what? (Score:2, Funny)
Web Mail and Evolution (Score:2)
So how do I use evolution and a browser view without keeping double copies?
formail, mairix, and mutt (Score:4, Informative)
Use mairix [force9.co.uk] to search through email.
mutt [mutt.org] is the best mail client ever.
-rsw
Easy... (Score:2, Informative)
Archiving tool: ForKeeps (Score:3, Interesting)
http://www.fkeeps.com/whofor.htm
It's a bit of an old program and the interface is clunky, but it works reasonably well once you work through it.
Keep an archive on your current mail client (Score:2)
When you switch mail clients (you allways do in a few years), make sure you import all current *and* archive email in a new set of profiles. Backup from your current app.
Still have to figure gmail in the equation, but with pop3 access should be just a matter of importing it in a app and backing it up - but downloading up to a GB over pop
Delete it (Score:5, Insightful)
Re:Delete it (Score:3, Insightful)
Good point, though.
Re:Delete it (Score:3, Insightful)
same sort of thing.. (Score:2)
Google Desktop Search (Score:2)
Older versions of Eudora (Score:2)
How I do it (Score:5, Informative)
I use grepmail [sourceforge.net] to find old emails that I might need. Grepmail lets you use perl regular expressions to find messages and then outputs the entire message where a match was found. You can use grepm [barsnick.net] to open grepmail matches as a mailbox in mutt. grepine [www.dfki.de] does the same for Pine, which I use.
At the end of each year I clean the spam out of my archives using a procmail recipe and spamassassin. This recipe marks messages as deleted in the mailbox. I open these in pine, sort by deleted, and double check them. Once I'm sure they're all spam, I delete them:
The special spamassassin config turns off bayesian filtering and sets the threshold high:
The rest of the spam I clean out by hand.Re:How I do it (Score:4, Informative)
Put this in ~/bin/rotate-sent-mail.sh:
0 0 1 * * $HOME/bin/rotate-sent-mail.sh
convert to standard mailbox format (Score:2)
maybe run a nice script to have an index in a db on the sender/title
or standard mailbox, and then make a bootable minilinux with pine (mozilla/whatever) on the same media (use vmware or similar to access it from non unix systems
actually my mailbox is just a growing junk collection, and have to do something with that too
or just export th
Manatory ZOË plug (Score:3, Informative)
ZOË is a sort of an archiving proxy that sits between your mail client and your mail server. It stores and indexes everything, so you can pop open a browser window and do a search on anything you've ever sent or received. Naturally, this was created before gmail [google.com].
With ZOË you don't need to worry about those pesky email folders and waiting for long searches.
Naturally, spam filtering before ZOË is a good idea.
monthly mailbox format archives (Score:2)
Practical research applications (Score:2, Interesting)
If anyone has an idea of an open-source application similar to what the submitter is looking for, it would help my research quite a bit. There's practical research applications in this stuff, if someone's
CSV (Score:2, Interesting)
Admittedly, sometimes the column names didn't match up ("Sender" v "From"), etc., but for the most part that how I did it. I also made an effort to keep the number o
archiving IM (Score:2)
It's actually pretty convienient to be able to search through old conversations. It makes a better journal than trying to thoughts out to yourself.
My solution (Score:5, Funny)
I only keep... (Score:2)
IMAP and mailbox/maildir (Score:2)
IMAP allows synchronization between many different servers and clients.
Mailbox and maildir both are open formats. Maildir uses one file per message, which makes management a little easier but some less advanced file systems have trouble with so many files in a directory. Mailbox is a reliable standby. Either way, I keep annual archives.
I think it's inappropriate to keep permanent records of IM conversations. IM is more like a face-to-face talk and people don't stick a t
IMAP works (Score:2)
What I do is have an IMAP server running on a Unix machine, and archive my e-mail into IMAP folders. The server turns those into vanilla Unix mailboxes, so I can deal with the messages as plain text if I want to. Almost every e-mail client out there supports IMAP, so compatibility isn't much of a problem. It also lets me use different clients and different systems to access all my e-mail in one place.
Email archiving and tools (Score:2, Interesting)
I keep the emails in mailbox format (that is, in plain text as it is stored in most UNIX systems), in several files. The reason I do that is that most email readers (MUA) can read mailbox format. I keep them in several files to make it more manageable.
The tools that I use to manipulate emails are mostly "from", "procmail", "grep", and "less". There used to be tools from the "elm" era (still remember them?), such as "frm"
IMAP.. (Score:2)
I'd recomend for those without a limited budget to get a low-power machine, onboard video, slowest 90nm processor you can find, one har
AOL does.. (Score:2)
AOL does, don't you read
Best Archive Format (Score:2)
1. Print out emails.
Man, it's so nice not being burdened by the embarrasing history of all the emails I sent. Besides, that was a long time ago, way before Score: 5, Funny.2. Shred them.
save all ingoing and outgoing in YYYY-MM files... (Score:3, Informative)
I had multiple folders, sorted by people/project. I got in a complete mess and finally snapped when I spent half an hour looking for a simple message.
Use procmail to write all incoming messages to 'all-mail-YYYY-MM' and use Mutt hooks to write out to the same directory.
At the end of the year, cat them together and make 'all-mail-YYYY'. Accessing and reading this mailbox can be done with 'mutt -R -f all-mail-YYYY' as this opens read-only. Use 'l' to do 'limit' searches and use ~t, ~f, and ~b in AND combinations to limit on To: From: and body of messages. It's lovely only having to look in one place!
Procmail:
INCOMING=all-mail-`date +%Y-%m`
# now I want to keep a copy of EVERYTHING in a dated directory
$INCOMING
Muttrc:
set record="+all-mail-`date +%Y-%m`"
Works for me!
Dr Fish
Post it on /. (Score:3, Funny)
As a bonus, you can tell which emails are worth reading by how they get moderated. All your work related emails will probably be modded Troll, except for your performance review, which will be modded +5 Funny. Email from your illicit lovers will be modded Insightful, since that type of thing is new to most of us. Email from your family will be conveniently modded down so you will not have to deal with it. Your friends won't need to send you any email at all, since they are probably already on Slashdot, and therefore, know enough to post in your threads.
Problem solved. Ah, Slashdot... Is there anything it can't do?
I run my own personal IMAP server (Score:3, Insightful)
Offlineimap, of course (Score:3, Informative)
You open all your email with an email client and move all the disparate inboxes into a big IMAP store on your own computer or one provided by a joint like Fastmail.fm [fastmail.fm] or Runbox.com
Then, you keep a local backup on any computer that you move to with offlineimap [quux.org], a wonderful utility that doubles as a multi-inbox syncronizer and backup utility. I have been using it for the past two years and can attest to its reliability.
EnjoyArchiving IM ... (Score:3, Informative)
I actually have two backups of my mail:
Re:One word (Score:5, Insightful)
I don't know about you but I generate about 6GB of email archives per year. Besides that having my email potentially available for searching doesn't sit well with me. I'm not sure where it stands now but there were a lot of potential privacy issues with Gmail.
No I don't receive hords of email, just a lot of engineering related with source code,research, white papers attached. If you do anything business related it's important to keep all of the original emails received so there is an electronic paper trail.
Re:One word (Score:4, Funny)
Do you actually sign up to those free porn places?
Re:Since a month back (Score:2)
Until someone breaks into your house/someone who lives with you goes snooping.
Re:Since a month back (Score:3, Funny)
Re:IMAP all the way - agree 100% (Score:3, Insightful)
With an IMAP backend you can try it all without tying yourself into one format - that's Open Standards for you!