How Do You Store and Reconcile Email Archives?

heyitsjustme wants to know how you deal with old email. "I delete most of what I get but keep the stuff from friends and relations as an archive. Unfortunately I have these email archives from the late 80's through today in the form of macintosh, linux and windows mailboxes including AOL 1.0 mailboxes. What does everyone use to archive email across multiple platforms and non-standard mailbox formats? Is there an easy solution out there? Does anyone archive IM?"

Italian school of driving

[goombah99 (560566)]

No need for rear view mirrors. What is behind you is not important.

Re:Italian school of driving

[boybaha (584738)]

I also have email archives that stretch back to the early-1990s. I pretty much still have every email I've ever sent or received. When upgrading email clients, I often migrate my archives with me, converting them using whatever client's built-in importing and exporting functions I have available. I went from Eudora to Outlook Express to Thunderbird to Mac Mail. I also have programs that "pop" webmail off their sites (gmail, hotmail and yahoo) to consolidate them in whatever current mail client I'm using. I just keep them in neat folders ("Old Eudora Mail," "Old Yahoo Mail")..

Here's what I do...

[sub7 (187049)]

I archive all my pr0n on DVDs these days. It's really easy and oh wait... fsck!

Re:Here's what I do...

[The Amazing Fish Boy (863897)]

I archive all my pr0n on DVDs these days. It's really easy and oh wait... fsck!

One day... someone... somewhere is going to invent some sort of mechanism for removing text you've already typed. It shall be called "back-one-space" and will remove the letter before it.

If this is impossible, surely they can keep a way of having all our text auto-submitted!

Re:Here's what I do...

[sulli (195030)]

You probably don't need to check DVD file systems if you've burned them properly.

rm -fR /var/spool/mail/*

[xlyz (695304)]

and you are done!

Disk space is cheap. Why bother deleting?

[heypete (60671)]

Save it all. With the exception of some mail archives lost to catastrophic disk failures (I keep archives for my own convenience, not for any official purposes, so I don't back them up), I keep all my email.

Thunderbird is able to import all my old mail archives (from years and years of Eudora) and search it effectively. If I were inclined to export all my archives from my Mac to my Windows machine, I could use Google Desktop Search to really search through it all.

Re:Disk space is cheap. Why bother deleting?

[Libraryman (721151)]

Why delete?

Because if you delete early and often, you've committed no crime. If you wait to delete it until someone (feds, cops, *IAA, UN-black-helicopter troopers, whoever) demands you turn it over to them, you're screwed.

After all, you break laws too (everybody does, they are written that way). You just haven't been caught yet. (I know this because if you had, you wouldn't have all you email archived!)

Re:Disk space is cheap. Why bother deleting?

[aussie_a (778472)]

Because if you delete early and often, you've committed no crime. If you wait to delete it until someone (feds, cops, *IAA, UN-black-helicopter troopers, whoever) demands you turn it over to them, you're screwed.

After all, you break laws too (everybody does, they are written that way). You just haven't been caught yet.

Instead of deleting all your e-mails "early and often" why not just delete the ones that have illegal activity in them? Or better yet, don't conduct illegal activity via e-mails. Those ar

Re:Disk space is cheap. Why bother deleting?

[aussie_a (778472)]

On the other hand, laws are made retroactively, there could be regime change, or other unpleasentness.

For the majority of slashdotters this wouldn't be a problem, as I'm pretty sure American laws can't be retroactive. If I ate a chicken today and e-mailed someone saying I ate a chicken, and tomorrow it became illegal to eat chicken, I can proclaim to the world "I ate a chicken on the 12th of March" and I won't be able to be charged with any crimes.

However given your choice of words (regime change) I fig

Dave's top ten

[FreeLinux (555387)]

10 Your mother told you to stop being such a pack rat.
9. Disks fill up, no matter how cheap they are. Low cost doesn't excuse gluttony.
8. Backups take forever.
7. Restores take an eternity, especially if your not confident.
6. Mail client gets slower and slower.
5. Searches take too long.
4. Mail clients make mistakes, especially on big stores. See #7
3. Your CYA evidence may be used against you.
2. A mail store is not a file system and SMTP is not a file transfer protocol.

And the number one reason to delete your old email...

1. IT'S ALL A BUNCH OF USELESS CRAP JUST AS IT WAS WHEN YOU FIRST RECEIVED IT!!

Re:Disk space is cheap. Why bother deleting?

[cgenman (325138)]

The first rule of thumb is "always bring your mail with you." If you change clients, or you change OS's, there is always a way, however roundabout or painful, to get mail into a usable form. This may involve installing Outlook, exporting all of your mail to Outlook, and importing it all from outlook, but it is worth it. Worst comes to worst, redirect it all back to yourself.

If you do this religiously, you will only ever have to worry about your current mail format, and how you're going to upgrade it all

Re:Disk space is cheap. Why bother deleting?

[glesga_kiss (596639)]

This may involve installing Outlook, exporting all of your mail to Outlook, and importing it all from outlook, but it is worth it.

Outlook + IMAP is the way I do it. You can drag messages between local storage and your mail server.

I work for Microsoft

[Anonymous Coward]

...so I just delete everything after a major deal falls through.

I'm afraid...

[joNDoty (774185)]

the best way to consolidate various types of emails may be to email them to a common source. Then archive from there.

email archive

[Pompatus (642396)]

I delete most of what I get

You must work for microsoft

Unix mail format

[saccade.com (771661)]

I use the basic Unix mail format, essentially plain text series of messages. Eudora does fine with it; and most anything else can read/import it. I have email going back to the 80's in this format. The one time I had to convert was when I was working for a company that used "Quickmail" on the Mac. I wound up reverse engineering their format and hacking up a program to convert it to plain text.

Re:Unix mail format

[Zocalo (252965)]

Ditto, in my case the "mbox" format to be precise. I currently use Procmail to automatically CC all incoming messages to a dedicated archive file, one per month, each year in a seperate folder. Outgoing mail is also sent to the same file, although I could easily have an "infile" and an "outfile", break mail apart by topic, or whatever. For more robust long term backup purposes I simply tarball the dozen files within each directory into a file called "mail-yyyy.tar.gz" and backup as normal.

Since mbox is

PDF

[DisasterDoctor (775095)]

I delete almost everything, and only save a few very important or personal emails. For those I do keep, I print to PDF, and archive by date and person/subject. It works exceptionally well for me. It is all electronic, takes very little disk space, and keeps the clutter to a minimum, and eliminates most of the cross platform nightmares.

Spotlight and Tiger

[goombah99 (560566)]

Combine this with spotlight/tiger in mac os. Spotlight indexes PDF content. print it to pdf and it will be searchable. Assuming you have a Mac that is.

One Word

[Zone-MR (631588)]

One word: IMAP. If you can read your email using any decent email client, it should support moving it to an IMAP server. If you are using web-based email or some crappy client which can't export emails to a standard/raw format, you'll have to write a script to convert the messages.

Re:One Word

[pHDNgell (410691)]

One word: IMAP

Absolutely. I use no fewer than two mail clients on two different machines on any given business day. Every email I've sent since 1995 or something like that, and received since 1998 is available and searchable. Over this time, I've accessed this archive with the following clients:

* pine (lots of pine)
* mac mail
* thunderbird
* various netscapes/mozillas
* ML (some random IMAP reader)
* My phone (my old Sony/Ericcson speaks IMAP)
* My palm (two different apps)
* python
* a java webmail system I wrote
* three or four other webmail systems
* mutt ...who knows what else. I've got freedom to try whatever I want at any given moment without losing my current or past mail.

Outport & recursive IMAP folder creation

[ahbi (796025)]

I strongly recommend Outport [sourceforge.net]. It does an extremely good job of converting MSFT Outlook attachments into something more readable (mbox I think, it has been a while). MS Outlook usually mangles attachments into some wrapper called TNEF.

Also, anyone know of a client program that will recursively create folders on an IMAP server (maybe a server issue. In which case, what server?)
I had gotten over translating my years of Outlook email into something more universally readable, but I have so many nested folders that the inability to have the client recirsively create IMAP folders is an issue. Suggestions?

Re:One Word

[astrashe (7452)]

IMAP is the answer. I don't use IMAP on a regular basis, but it did let me export mail from outlook over to Evolution on linux.

I used the UW IMAP server, which is a little easier to set up than the Cygnus one.

The UW IMAPd keeps its folders in mbox format, so it's a great tool for converting oddly formatted mail.

Moving email is pretty easy -- it's harder to move calendar entries, address books, notes, and the other sorts of data that ends up in a program like outlook. I think the easiest way to do it wo

It's simple: plain text

[Faust7 (314817)]

Ever since I first got acquainted with e-mail on my Apple IIe in the '80s, I've used e-mail programs that offer plain-text storage as at least an option. It's one of the most universal formats in existence, and can be read one way or another on computers both decades old and brand new. I encountered some weird proprietary clients in the '90s that still stored e-mail in this format, because from a corporate perspective, this stuff was still in its infancy, plus HTML hadn't yet mucked everything up. To this day I still store in plain text from Eudora 6.2.

I burn it to CD-Rs that I know won't get moved around or scratched. They stand a good chance of lasting the rest of my life.

Re:It's simple: plain text

[Dwonis (52652)]

I burn it to CD-Rs that I know won't get moved around or scratched. They stand a good chance of lasting the rest of my life.

No! Check those backups! I have lost data stored on CD-Rs (luckily I had copies), and many of my discs have started to turn yellow after about 2 years! Also, you can sometimes see these little spots of discolouration on the CDs, which makes me think there's a fungus of some sort that's eating them.

The lifespan of CD-Rs is unknown at this point. Don't trust them for more than

Upon Searching..

[yuriismaster (776296)]

EmailMan [emailman.com] has the answers to your problem.

More utilities than I want to bother with, but hopefully they'll have the converter(s) you need.

Good Luck!

Your favorite online storage

[Alien54 (180860)]

Hriders.com gives unlimited free 1 Terabyte email accounts that include 500 Megabyte attachments. We have been asked why we would do such a thing. The answer is simple to help people store large amounts of information in a safe and secure environment. - - - We decided that yes a Terabyte of space may sound rather extreme to some, others will not think so. If you have a free membership with Hriders.com then you will receive a free 1 Terabyte 500 Megabyte attachment email account. You will be able to store over 40 million emails, videos, games, mp3s, or pictures.

This might be useful, if they don't collapse under /.

Don't change e-mail clients

[rueger (210566)]

That's one of the many reasons why I have stayed with Pegasus Mail [pmail.com]for many years. Because they were created in the same program I know that I can still access my old mail files without problems.

What I do at year end is move all of that year's messages to a new folder and reset my filters so that the new year's messages go into a new set of folders.

Periodically I just copy off previous year's messages to CD.

At least few times I have been able to back a couple of years and find information that I lacked.

Kinda Sorta OT

[hot_Karls_bad_cavern (759797)]

but ...

Along these lines, is there an OSS package that can read the varied formats the Submitter is referring to, tag and drop them in a DB with a nice, friendly, web-enabled (secure) front-end for searching?

My former employer kept *all* of his email from the last 20 years in tar.gz files. Let's just say it wasn't easy to find an email from er, 15 years ago very easily.

Is there a package that can read the mbox, the other box-formats, plain text, pull from pop, old tar.gz bundles, categorize (sorta), tag and make such things searchable?

Totally a shot in the dark here, i'm not a mail guy at all ... just wondering as the Submitter did what i like /. Submitters to do: make me think and look for new, better stuff ... or better ways to do old-stuff.

It is the "drink" that makes me wonder, sorry :)

Convert to MBOX format

[Jason Earl (1894)]

Almost every email client around can import and export mbox formats. Getting your email in a format that is going to be readable in 20 years is the first step, otherwise why bother?

Worse comes to worst mbox is readable as plain text.

Or use maildir

[suso (153703)]

Whatever you do, I think its best to keep it in an open and obvious format like mbox or maildir. The nice thing about maildir though, is that since all the messages are seperate, it might be a little easier to write a program to put them into a new format.

Personally, since 1999, I've been using a combination of maildir and procmail to archive and save my mail. Every message that comes in, goes to a folder called .saved-messages-YYYY-MM and also to my inbox. I simply don't touch the saved-messages folders and when I am done with the message in my inbox, I just delete it. This has worked well for me and makes it much easier to deal with archiving old mail. In the end, having categorized folders and such is just a waste of time. Its kinda like the wm2 (window manager) way of thinking, but for mailboxes.

One's things sure

[eddeye (85134)]

Don't ask Microsoft [slashdot.org].

mbox or maildir

[Xtifr (1323)]

I've been using mbox format since the eighties, and never had a problem with it on any platform. It's pretty much been THE standard for email for as long as email has existed. If I ever were to switch, I'd probably switch to maildir, which has nearly as wide-spread support these days.

Archive what?

[Mishura (744815)]

I never keep emails, or archive IMs or any other form of communication. Once a email is read, it is deleted. Same goes for normal old-skool mail, I read it and then trash it. The only exceptions are of letters/email of some importance such as information I need to keep handy, or if it has some kind of sentimental value (letters from deceased relatives for example.)

Sure, HDD space is cheap; but I tend to equate people who archive every single form of written communication to those who have an Obsessive Compulsive Disorder, in that they hoarde everything in sight: newspapers, snail mail, magazines, boxes, etc..

Commit to memory and destroy the evidence. Thats my way of handling archives.

Re:Archive what?

[Anonymous Coward]

If you throw out your mail as soon as you read it, how are you keeping letters written by deceased relatives? Are they sending you mail after they die?

formail, mairix, and mutt

[rsw (70577)]

Convert everything into mbox format. formail [procmail.org] will help you with that.

Use mairix [force9.co.uk] to search through email.

mutt [mutt.org] is the best mail client ever.

-rsw

Archiving tool: ForKeeps

[sstern (56589)]

I have several CDs worth of stuff archived with ForKeeps:

http://www.fkeeps.com/whofor.htm

It's a bit of an old program and the interface is clunky, but it works reasonably well once you work through it.

Delete it

[exp(pi*sqrt(163)) (613870)]

That way it won't be subject to a sub poena. You'll regret it one day if you don't. Do you realize how much incriminating stuff you have in there?

Re:Delete it

[man_ls (248470)]

Generally, most people don't have to fear subpoenas unless they're doing something illegal or nefarious in some way.

Good point, though.

How I do it

[Matt Perry (793115)]

I use a procmail recipe to archive my mail. I put it after filtering mailing lists and before I filter spam:

OLDMAILDIR = $MAILDIR
MAILDIR = $ARCHIVE_DIR
:0 cW: archive.lock
| /bin/gzip >>mailarchive-`date +%Y%m`.gz
MAILDIR = $OLDMAILDIR

I use grepmail [sourceforge.net] to find old emails that I might need. Grepmail lets you use perl regular expressions to find messages and then outputs the entire message where a match was found. You can use grepm [barsnick.net] to open grepmail matches as a mailbox in mutt. grepine [www.dfki.de] does the same for Pine, which I use.

At the end of each year I clean the spam out of my archives using a procmail recipe and spamassassin. This recipe marks messages as deleted in the mailbox. I open these in pine, sort by deleted, and double check them. Once I'm sure they're all spam, I delete them:

# vim:ft=procmail:

LINEBUF = 8192
SHELL = /bin/sh
MAILDIR = $HOME/mail

:0 fW: spamclean.lock
| spamassassin -e --prefs-file=/home/matt/.spamassassin/user_prefs-s pam_clean 2>/dev/null

# If the message was deemed to be spam, set the status to "deleted" so that
# we can delete it easily and optionally review it.
:0 e
{
:0 fhw
* ^^rom[ ]
| sed -e '1s/^/F/'

:0 f: formail.lock
| formail -I 'X-Status: D'
}

# Fix the mangled "From" line
:0 fhwE
* ^^rom[ ]
| sed -e '1s/^/F/'

# Remove the last of the SpamAssassin headers
:0 f: formail2.lock
| formail -I 'X-Spam-Checker-Version'

# File message in temporary mailbox
:0: sandbox.lock
z-cleaned_mbox

The special spamassassin config turns off bayesian filtering and sets the threshold high:

required_hits 15
clear_headers
fold_headers 0
use_bayes 0

The rest of the spam I clean out by hand.

Re:How I do it

[Matt Perry (793115)]

Almost forgot. I archive my sent mail as well. This might be harder for you if you don't use a single email client on a single machine. IMAP can help with that.

Put this in ~/bin/rotate-sent-mail.sh:

#!/bin/bash

# This script takes sent mail in $HOME/mail and moves them into
# $HOME/.mailarchives/sent. It will also rename the file to have the date of
# the log file included.

MAILDIR=$HOME/mail
ARCDIR=$HOME/.mai larchive/sent
year=`/bin/date +%Y`
month=`/bin/date +%-m`

# updating last months mail
month=$((month-1))

# if this is last years mail, set the date correctly
if [ $month -eq 0 ] ; then
month='12'
year=$((year-1))
fi

# if the month is less than 10, add the leading zero back
if [ $month -lt 10 ] ; then
month=0$month
fi

mv $MAILDIR/sent-mail $ARCDIR/sent-mail-$year$month
touch $MAILDIR/sent-mail && chmod 600 $MAILDIR/sent-mail
bzip2 -9 $ARCDIR/sent-mail-$year$month

Now add the following to your crontab:

0 0 1 * * $HOME/bin/rotate-sent-mail.sh

Manatory ZOË plug

[MCRocker (461060)]

Well, if you're going to be on this topic, a mention of ZOË [zoe.nu] is pretty much required.

ZOË is a sort of an archiving proxy that sits between your mail client and your mail server. It stores and indexes everything, so you can pop open a browser window and do a search on anything you've ever sent or received. Naturally, this was created before gmail [google.com].

With ZOË you don't need to worry about those pesky email folders and waiting for long searches.

Naturally, spam filtering before ZOË is a good idea.

My solution

[anon*127.0.0.1 (637224)]

I've found the easiest way to handle EMail when it's in multiple formats like that is to just print everything out and store it in boxes in my garage.

save all ingoing and outgoing in YYYY-MM files...

[Doctor Fishboy (120462)]

...and let mutt sort out.

I had multiple folders, sorted by people/project. I got in a complete mess and finally snapped when I spent half an hour looking for a simple message.

Use procmail to write all incoming messages to 'all-mail-YYYY-MM' and use Mutt hooks to write out to the same directory.

At the end of the year, cat them together and make 'all-mail-YYYY'. Accessing and reading this mailbox can be done with 'mutt -R -f all-mail-YYYY' as this opens read-only. Use 'l' to do 'limit' searches and use ~t, ~f, and ~b in AND combinations to limit on To: From: and body of messages. It's lovely only having to look in one place!

Procmail:
INCOMING=all-mail-`date +%Y-%m`
# now I want to keep a copy of EVERYTHING in a dated directory :0 c:
$INCOMING

Muttrc:
set record="+all-mail-`date +%Y-%m`"

Works for me!

Dr Fish

Post it on /.

[pyrrhonist (701154)]

Just post all your email on Slashdot. Since no one reads the articles anyway, no one will notice when they aren't on topic, and there's always the possibility that your email from Mom will actually be on topic. Furthermore, it will be easy to search using Google, and the Wayback Machine will automatically archive everything for you.

As a bonus, you can tell which emails are worth reading by how they get moderated. All your work related emails will probably be modded Troll, except for your performance review, which will be modded +5 Funny. Email from your illicit lovers will be modded Insightful, since that type of thing is new to most of us. Email from your family will be conveniently modded down so you will not have to deal with it. Your friends won't need to send you any email at all, since they are probably already on Slashdot, and therefore, know enough to post in your threads.

Problem solved. Ah, Slashdot... Is there anything it can't do?

Re:One word

[Padrino121 (320846)]

Gmail?

I don't know about you but I generate about 6GB of email archives per year. Besides that having my email potentially available for searching doesn't sit well with me. I'm not sure where it stands now but there were a lot of potential privacy issues with Gmail.

No I don't receive hords of email, just a lot of engineering related with source code,research, white papers attached. If you do anything business related it's important to keep all of the original emails received so there is an electronic paper trail.

Re:One word

[Anonymous Coward]

6GB yearly? Holy shit...

Do you actually sign up to those free porn places?

Re:Log everything...

[Rosco P. Coltrane (209368)]

I log and keep all my traffic including IRC logs going back to '94.

Hey B5_geek, here's a trick to free up a lot of disk space *and* raise the S/N ratio in your logs:

mv irclog.txt irclog.txt.fat && grep -vi lol irclog.txt.fat > irclog.txt && rm -f irclog.txt.fat

Re:Since a month back

[FLEB (312391)]

There are traps.