Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Communications Data Storage The Internet

How Do You Store and Reconcile Email Archives? 380

Posted by timothy from the translate-them-to-quipu dept.
heyitsjustme wants to know how you deal with old email. "I delete most of what I get but keep the stuff from friends and relations as an archive. Unfortunately I have these email archives from the late 80's through today in the form of macintosh, linux and windows mailboxes including AOL 1.0 mailboxes. What does everyone use to archive email across multiple platforms and non-standard mailbox formats? Is there an easy solution out there? Does anyone archive IM?"
This discussion has been archived. No new comments can be posted.

How Do You Store and Reconcile Email Archives? More

How Do You Store and Reconcile Email Archives?

Comments Filter:

  • Italian school of driving (Score:4, Funny)

    by goombah99 ( 560566 ) on Saturday March 12, 2005 @07:01PM (#11922512)
    No need for rear view mirrors. What is behind you is not important.

    • Re:Italian school of driving (Score:4, Interesting)

      by boybaha ( 584738 ) on Saturday March 12, 2005 @07:05PM (#11922555)
      I also have email archives that stretch back to the early-1990s. I pretty much still have every email I've ever sent or received. When upgrading email clients, I often migrate my archives with me, converting them using whatever client's built-in importing and exporting functions I have available. I went from Eudora to Outlook Express to Thunderbird to Mac Mail. I also have programs that "pop" webmail off their sites (gmail, hotmail and yahoo) to consolidate them in whatever current mail client I'm using. I just keep them in neat folders ("Old Eudora Mail," "Old Yahoo Mail")..

    • American school of driving (Score:2, Funny)

      by Anonymous Coward
      My car is bigger than yours. Move it or lose it!

  • Here's what I do... (Score:5, Funny)

    by sub7 ( 187049 ) on Saturday March 12, 2005 @07:03PM (#11922526)
    I archive all my pr0n on DVDs these days. It's really easy and oh wait... fsck!

  • rm -fR /var/spool/mail/* (Score:3, Funny)

    by xlyz ( 695304 ) on Saturday March 12, 2005 @07:03PM (#11922534) Journal

    and you are done!

  • Disk space is cheap. Why bother deleting? (Score:5, Insightful)

    by heypete ( 60671 ) <pete@heypete.com> on Saturday March 12, 2005 @07:04PM (#11922537) Homepage
    Save it all. With the exception of some mail archives lost to catastrophic disk failures (I keep archives for my own convenience, not for any official purposes, so I don't back them up), I keep all my email.

    Thunderbird is able to import all my old mail archives (from years and years of Eudora) and search it effectively. If I were inclined to export all my archives from my Mac to my Windows machine, I could use Google Desktop Search to really search through it all.

    • Re:Disk space is cheap. Why bother deleting? (Score:5, Insightful)

      by Libraryman ( 721151 ) on Saturday March 12, 2005 @07:32PM (#11922722)
      Why delete?

      Because if you delete early and often, you've committed no crime. If you wait to delete it until someone (feds, cops, *IAA, UN-black-helicopter troopers, whoever) demands you turn it over to them, you're screwed.

      After all, you break laws too (everybody does, they are written that way). You just haven't been caught yet. (I know this because if you had, you wouldn't have all you email archived!)
      • Because if you delete early and often, you've committed no crime. If you wait to delete it until someone (feds, cops, *IAA, UN-black-helicopter troopers, whoever) demands you turn it over to them, you're screwed.

        After all, you break laws too (everybody does, they are written that way). You just haven't been caught yet.

        Instead of deleting all your e-mails "early and often" why not just delete the ones that have illegal activity in them? Or better yet, don't conduct illegal activity via e-mails. Those ar

    • Dave's top ten (Score:5, Funny)

      by FreeLinux ( 555387 ) on Saturday March 12, 2005 @07:38PM (#11922760)
      10 Your mother told you to stop being such a pack rat.
      9. Disks fill up, no matter how cheap they are. Low cost doesn't excuse gluttony.
      8. Backups take forever.
      7. Restores take an eternity, especially if your not confident.
      6. Mail client gets slower and slower.
      5. Searches take too long.
      4. Mail clients make mistakes, especially on big stores. See #7
      3. Your CYA evidence may be used against you.
      2. A mail store is not a file system and SMTP is not a file transfer protocol.

      And the number one reason to delete your old email...

      1. IT'S ALL A BUNCH OF USELESS CRAP JUST AS IT WAS WHEN YOU FIRST RECEIVED IT!!
    • I could use Google Desktop Search to really search through it all.

      Awesome idea. I'm gonna be doing this.
    • The first rule of thumb is "always bring your mail with you." If you change clients, or you change OS's, there is always a way, however roundabout or painful, to get mail into a usable form. This may involve installing Outlook, exporting all of your mail to Outlook, and importing it all from outlook, but it is worth it. Worst comes to worst, redirect it all back to yourself.

      If you do this religiously, you will only ever have to worry about your current mail format, and how you're going to upgrade it all

  • I work for Microsoft (Score:5, Funny)

    by Anonymous Coward on Saturday March 12, 2005 @07:04PM (#11922540)
    ...so I just delete everything after a major deal falls through.

  • I'm afraid... (Score:3, Insightful)

    by joNDoty ( 774185 ) on Saturday March 12, 2005 @07:04PM (#11922541)
    the best way to consolidate various types of emails may be to email them to a common source. Then archive from there.
    • That's good for the text itself, but screws with the rest of the data (sender, date sent, etc)

      And assuming you still have an app to read that: I've got some old netscape (2.x, I think) mail folders that I can read with a text editor, but I no longer have a program that will open it - there may be, ut I haven't looked that much really.

  • email archive (Score:4, Funny)

    by Pompatus ( 642396 ) on Saturday March 12, 2005 @07:04PM (#11922543) Journal
    I delete most of what I get

    You must work for microsoft

  • Cyrus Imap... (Score:2, Interesting)

    by DaGoodBoy ( 8080 )
    ...with fetchmail / procmail / cyrdeliver for sorting and storing from other sources. How can 5GB of mail can't be wrong?! I can slice and dice my all my email (including about a gig of spam... [awtrey.com]) for choice bits of information.
    • Me too, though I add one little twist: a Perl script to archive older emails to subfolders, so my threaded sort of current email doesn't slow things down.

      It's still all available when I want it, but the day-to-day use is faster.

  • Unix mail format (Score:3, Interesting)

    by saccade.com ( 771661 ) on Saturday March 12, 2005 @07:05PM (#11922547) Homepage Journal
    I use the basic Unix mail format, essentially plain text series of messages. Eudora does fine with it; and most anything else can read/import it. I have email going back to the 80's in this format. The one time I had to convert was when I was working for a company that used "Quickmail" on the Mac. I wound up reverse engineering their format and hacking up a program to convert it to plain text.

    • Re:Unix mail format (Score:3, Interesting)

      by Zocalo ( 252965 )
      Ditto, in my case the "mbox" format to be precise. I currently use Procmail to automatically CC all incoming messages to a dedicated archive file, one per month, each year in a seperate folder. Outgoing mail is also sent to the same file, although I could easily have an "infile" and an "outfile", break mail apart by topic, or whatever. For more robust long term backup purposes I simply tarball the dozen files within each directory into a file called "mail-yyyy.tar.gz" and backup as normal.

      Since mbox is

  • PDF (Score:4, Insightful)

    by DisasterDoctor ( 775095 ) on Saturday March 12, 2005 @07:05PM (#11922550)
    I delete almost everything, and only save a few very important or personal emails. For those I do keep, I print to PDF, and archive by date and person/subject. It works exceptionally well for me. It is all electronic, takes very little disk space, and keeps the clutter to a minimum, and eliminates most of the cross platform nightmares.

    • Spotlight and Tiger (Score:5, Insightful)

      by goombah99 ( 560566 ) on Saturday March 12, 2005 @07:20PM (#11922649)
      Combine this with spotlight/tiger in mac os. Spotlight indexes PDF content. print it to pdf and it will be searchable. Assuming you have a Mac that is.
      • Store it in any format you like. If it isn't one of the supported file formats [apple.com] that Tiger automatically recognizes, then you (or some kindly programmer-type) can create a plug-in [apple.com] that supports your file format. Assuming you have a Mac...

  • One Word (Score:5, Insightful)

    by Zone-MR ( 631588 ) * <slashdot@NoSPam.zone-mr.net> on Saturday March 12, 2005 @07:05PM (#11922553) Homepage
    One word: IMAP. If you can read your email using any decent email client, it should support moving it to an IMAP server. If you are using web-based email or some crappy client which can't export emails to a standard/raw format, you'll have to write a script to convert the messages.

    • Re:One Word (Score:2, Interesting)

      by Murphy Murph ( 833008 )
      I second this.
      I started running my own IMAP server on an old machine a year or so ago - and synced all my old mail archives to various folders.

      My mailserver also solves another problem - multiple POP accounts. I have my IMAP server set up so that each one of my POP accounts gets automaticly tagged and sent to it's own folder.

      A third common problem this solves is having multiple machines. Now my desktop's email client is always synced with my laptop's email client. Before I had run into problems when ev

    • Re:One Word (Score:2, Insightful)

      by pasamio ( 737659 )
      IMAP isn't really a word, its an anocrym. But I agree, IMAP is the way I use, it helps for the relevant email and on my network I use both Linux and Windows (with a dedicated Linux box). I have Evolution set up to continually check and sort my email into IMAP folders and I generally read them off my linux box. If I need to click on links, I generally open up any Windows email client (from Thunderbird to Outlook) and it'll connect to IMAP and my emails will all appear (nicely sorted too!). If I need webmai

    • Re:One Word (Score:5, Interesting)

      by pHDNgell ( 410691 ) on Saturday March 12, 2005 @07:24PM (#11922678)
      One word: IMAP

      Absolutely. I use no fewer than two mail clients on two different machines on any given business day. Every email I've sent since 1995 or something like that, and received since 1998 is available and searchable. Over this time, I've accessed this archive with the following clients:

      * pine (lots of pine)
      * mac mail
      * thunderbird
      * various netscapes/mozillas
      * ML (some random IMAP reader)
      * My phone (my old Sony/Ericcson speaks IMAP)
      * My palm (two different apps)
      * python
      * a java webmail system I wrote
      * three or four other webmail systems
      * mutt ...who knows what else. I've got freedom to try whatever I want at any given moment without losing my current or past mail.
    • Can someone step me through getting an IMAP server running on my network, and moving existing mails to it?

      Assume I have Linux or Windows boxes available (Would prefer Windows, yeah blah blah but other people don't like me 'wasting' hardware on *nix), and I use Outlook for my primary mail client.

    • Outport & recursive IMAP folder creation (Score:4, Informative)

      by ahbi ( 796025 ) on Saturday March 12, 2005 @08:00PM (#11922864) Journal
      I strongly recommend Outport [sourceforge.net]. It does an extremely good job of converting MSFT Outlook attachments into something more readable (mbox I think, it has been a while). MS Outlook usually mangles attachments into some wrapper called TNEF.

      Also, anyone know of a client program that will recursively create folders on an IMAP server (maybe a server issue. In which case, what server?)
      I had gotten over translating my years of Outlook email into something more universally readable, but I have so many nested folders that the inability to have the client recirsively create IMAP folders is an issue. Suggestions?

    • Re:One Word (Score:3, Informative)

      by astrashe ( 7452 )
      IMAP is the answer. I don't use IMAP on a regular basis, but it did let me export mail from outlook over to Evolution on linux.

      I used the UW IMAP server, which is a little easier to set up than the Cygnus one.

      The UW IMAPd keeps its folders in mbox format, so it's a great tool for converting oddly formatted mail.

      Moving email is pretty easy -- it's harder to move calendar entries, address books, notes, and the other sorts of data that ends up in a program like outlook. I think the easiest way to do it wo

  • It's simple: plain text (Score:5, Insightful)

    by Faust7 ( 314817 ) on Saturday March 12, 2005 @07:05PM (#11922556) Homepage
    Ever since I first got acquainted with e-mail on my Apple IIe in the '80s, I've used e-mail programs that offer plain-text storage as at least an option. It's one of the most universal formats in existence, and can be read one way or another on computers both decades old and brand new. I encountered some weird proprietary clients in the '90s that still stored e-mail in this format, because from a corporate perspective, this stuff was still in its infancy, plus HTML hadn't yet mucked everything up. To this day I still store in plain text from Eudora 6.2.

    I burn it to CD-Rs that I know won't get moved around or scratched. They stand a good chance of lasting the rest of my life.
    • I burn it to CD-Rs that I know won't get moved around or scratched. They stand a good chance of lasting the rest of my life.

      No! Check those backups! I have lost data stored on CD-Rs (luckily I had copies), and many of my discs have started to turn yellow after about 2 years! Also, you can sometimes see these little spots of discolouration on the CDs, which makes me think there's a fungus of some sort that's eating them.

      The lifespan of CD-Rs is unknown at this point. Don't trust them for more than

  • I log and keep all my traffic including IRC logs going back to '94.

    My email isn't in quite the mess yours is, (I used Eudora for almost all of my emailing since I first got on the net, and have just imported from one version to another, and now into Thunderbird.)

    I would simply start copying & pasting, or see if you can try importing into excel.

  • Upon Searching.. (Score:5, Informative)

    by yuriismaster ( 776296 ) <{tubaswimmer} {at} {gmail.com}> on Saturday March 12, 2005 @07:06PM (#11922565) Homepage
    EmailMan [emailman.com] has the answers to your problem.

    More utilities than I want to bother with, but hopefully they'll have the converter(s) you need.

    Good Luck!
  • every email I get has a copy forwarded to a google gmail account. I also forward everything to an account on my server as well, but I like the gmail account because I can get at it anywhere.

  • Your favorite online storage (Score:5, Informative)

    by Alien54 ( 180860 ) on Saturday March 12, 2005 @07:08PM (#11922583) Journal
    Hriders.com gives unlimited free 1 Terabyte email accounts that include 500 Megabyte attachments. We have been asked why we would do such a thing. The answer is simple to help people store large amounts of information in a safe and secure environment. - - - We decided that yes a Terabyte of space may sound rather extreme to some, others will not think so. If you have a free membership with Hriders.com then you will receive a free 1 Terabyte 500 Megabyte attachment email account. You will be able to store over 40 million emails, videos, games, mp3s, or pictures.

    This might be useful, if they don't collapse under /.

  • Simple.. (Score:2, Funny)

    by billimad ( 629204 )
    I give mine to Microsoft for the safe storage and instant retreval they are renoud for. Oh wait..

  • Don't change e-mail clients (Score:4, Informative)

    by rueger ( 210566 ) * on Saturday March 12, 2005 @07:09PM (#11922590) Homepage
    That's one of the many reasons why I have stayed with Pegasus Mail [pmail.com]for many years. Because they were created in the same program I know that I can still access my old mail files without problems.

    What I do at year end is move all of that year's messages to a new folder and reset my filters so that the new year's messages go into a new set of folders.

    Periodically I just copy off previous year's messages to CD.

    At least few times I have been able to back a couple of years and find information that I lacked.

  • Kinda Sorta OT (Score:5, Interesting)

    by hot_Karls_bad_cavern ( 759797 ) on Saturday March 12, 2005 @07:10PM (#11922592) Journal
    but ...

    Along these lines, is there an OSS package that can read the varied formats the Submitter is referring to, tag and drop them in a DB with a nice, friendly, web-enabled (secure) front-end for searching?

    My former employer kept *all* of his email from the last 20 years in tar.gz files. Let's just say it wasn't easy to find an email from er, 15 years ago very easily.

    Is there a package that can read the mbox, the other box-formats, plain text, pull from pop, old tar.gz bundles, categorize (sorta), tag and make such things searchable?

    Totally a shot in the dark here, i'm not a mail guy at all ... just wondering as the Submitter did what i like /. Submitters to do: make me think and look for new, better stuff ... or better ways to do old-stuff.

    It is the "drink" that makes me wonder, sorry :)

  • How I archive my mail (Score:2, Insightful)

    by gpscc ( 315484 )
    I only use e-mail clients that store mail in ascii with standard headers. This means no Outlook mail. I still use the Netscape e-mail client to view and organize my mail. Also I have various perl scripts that can access the e-mail archive. I have 22 years of e-mail, archived on my PC. It gets backed up with the nightly backup onto a swapable firewire drive. I swap the backup every morning and have one of the drives with me.

  • Convert to MBOX format (Score:5, Insightful)

    by Jason Earl ( 1894 ) on Saturday March 12, 2005 @07:10PM (#11922597) Homepage Journal

    Almost every email client around can import and export mbox formats. Getting your email in a format that is going to be readable in 20 years is the first step, otherwise why bother?

    Worse comes to worst mbox is readable as plain text.

    • Or use maildir (Score:5, Informative)

      by suso ( 153703 ) on Saturday March 12, 2005 @07:24PM (#11922676) Journal
      Whatever you do, I think its best to keep it in an open and obvious format like mbox or maildir. The nice thing about maildir though, is that since all the messages are seperate, it might be a little easier to write a program to put them into a new format.

      Personally, since 1999, I've been using a combination of maildir and procmail to archive and save my mail. Every message that comes in, goes to a folder called .saved-messages-YYYY-MM and also to my inbox. I simply don't touch the saved-messages folders and when I am done with the message in my inbox, I just delete it. This has worked well for me and makes it much easier to deal with archiving old mail. In the end, having categorized folders and such is just a waste of time. Its kinda like the wm2 (window manager) way of thinking, but for mailboxes.
  • Well, my basic reaction to this topic is that I haven't ever found a really good way to do this, but I wish there was one. Actually, I wish it was part of a more general system to sort and manage personal files, restructuring things and getting rid of the duplicates. This is actually an indirect side effect of the cancerous growth of HDDs. Sort and clean up? Why bother? Just copy the universe to the new machine. Who cares about wasting a few GB?

    Closest thing to a helpful feature I've seen is the importing

  • One's things sure (Score:5, Funny)

    by eddeye ( 85134 ) on Saturday March 12, 2005 @07:13PM (#11922621)
    Don't ask Microsoft [slashdot.org].
  • Store everything in mbox format (that is, raw text with headers and all).

    Every email client worth the name understands that, for the good reason that it's the format they receive emails in :-)

    For nonstandard forms of archives (perhaps old AOL clients and whatnot), you're probably left either (1) perl'ing a convert script or, if you can (2) fire up the old client in Win95 in VMWare or something and fwd the mails to yourself (tedious).
  • I delete all mail that has not been directly addessed to me. Usually all mail from mailing lists, unless the message is really interesting or it is a thread I've been participating. I didn't use to do that, but when I changed to this method I deleted the old unimportant messages as well. It brings down the number of messages to a manageable level.

    Messages are not sorted into separate subject folders. They are all in a single mailbox, the mailbox. Every month I back up this mailbox to the name of the previo

  • mbox or maildir (Score:3)

    by Xtifr ( 1323 ) on Saturday March 12, 2005 @07:14PM (#11922628) Homepage
    I've been using mbox format since the eighties, and never had a problem with it on any platform. It's pretty much been THE standard for email for as long as email has existed. If I ever were to switch, I'd probably switch to maildir, which has nearly as wide-spread support these days.
    • Mbox is better for archival as it is only the one file, or a few larger files as opposed to many small files with special names. You can always tar Maildir, but then you might as well put the email in mbox which can be opened directly. There are a number of simple tools that will make a copy Maildir or Imap and store the result as mbox.

  • Archive what? (Score:3, Interesting)

    by Mishura ( 744815 ) on Saturday March 12, 2005 @07:15PM (#11922631)
    I never keep emails, or archive IMs or any other form of communication. Once a email is read, it is deleted. Same goes for normal old-skool mail, I read it and then trash it. The only exceptions are of letters/email of some importance such as information I need to keep handy, or if it has some kind of sentimental value (letters from deceased relatives for example.)

    Sure, HDD space is cheap; but I tend to equate people who archive every single form of written communication to those who have an Obsessive Compulsive Disorder, in that they hoarde everything in sight: newspapers, snail mail, magazines, boxes, etc..

    Commit to memory and destroy the evidence. Thats my way of handling archives.

    • Re:Archive what? (Score:4, Funny)

      by Anonymous Coward on Saturday March 12, 2005 @07:31PM (#11922721)
      If you throw out your mail as soon as you read it, how are you keeping letters written by deceased relatives? Are they sending you mail after they die?

      • Re:Archive what? (Score:2, Informative)

        by Mishura ( 744815 )
        If you throw out your mail as soon as you read it, how are you keeping letters written by deceased relatives? Are they sending you mail after they die?

        Actually, yes. I did recieve a letter from my grandmom a week after she died. Snail mail works very slow indeed.

        Reading the letter was strange. The content wasn't strange, just the feeling you get from recieving information from a dead person. That's all I'll say about it.

        Cue the "I read dead people's email" jokes..
    • I'd imagine once your relatives decease they stop sending you mail -- conveniently cutting down on the amount you need to archive. Nice!
  • I have a question along the same lines. I use Evolution to pull down my local email. But I would like to also be able to read it externally. If I set usermin to use the evolution mboxes, they become unsync'd in evolutions view.

    So how do I use evolution and a browser view without keeping double copies?

  • formail, mairix, and mutt (Score:4, Informative)

    by rsw ( 70577 ) on Saturday March 12, 2005 @07:17PM (#11922641) Homepage
    Convert everything into mbox format. formail [procmail.org] will help you with that.

    Use mairix [force9.co.uk] to search through email.

    mutt [mutt.org] is the best mail client ever.

    -rsw

  • Easy... (Score:2, Informative)

    by praetis ( 806293 )
    I archive my mail on /dev/null. Send it there daily.

  • Archiving tool: ForKeeps (Score:3, Interesting)

    by sstern ( 56589 ) on Saturday March 12, 2005 @07:25PM (#11922683) Homepage Journal
    I have several CDs worth of stuff archived with ForKeeps:

    http://www.fkeeps.com/whofor.htm

    It's a bit of an old program and the interface is clunky, but it works reasonably well once you work through it.
  • That's what I have been doing in recent years: keep your regular "recent" emails in one profile, and in another all your archive stuff, using the same client and formats.

    When you switch mail clients (you allways do in a few years), make sure you import all current *and* archive email in a new set of profiles. Backup from your current app.

    Still have to figure gmail in the equation, but with pop3 access should be just a matter of importing it in a app and backing it up - but downloading up to a GB over pop

  • Delete it (Score:5, Insightful)

    by exp(pi*sqrt(163)) ( 613870 ) on Saturday March 12, 2005 @07:29PM (#11922712) Journal
    That way it won't be subject to a sub poena. You'll regret it one day if you don't. Do you realize how much incriminating stuff you have in there?

    • Re:Delete it (Score:3, Insightful)

      by man_ls ( 248470 )
      Generally, most people don't have to fear subpoenas unless they're doing something illegal or nefarious in some way.

      Good point, though.

      • Re:Delete it (Score:3, Insightful)

        by pohl ( 872 )
        That's a rather optimistic viewpoint. It's entirely possible for someone to do nothing illegal and still find themselves at the blunt end of a civil suit.
  • Does anyone know what format and where outlook stores e-mails? I need to archive a friends e-mails and she only used outlook and I have no clue where to start with Windows 98 and outlook..
  • If you know basic programming (file access and working with a given API), then you can try writing a plugin for Google Desktop Search [google.com]. This way, you can add all your old files to the GDS database, and automatically archive the new ones.
  • I use Eudora 5.1 - I have every e-mail since 1995 at my disposal. It works very well. Upgrading doesn't hose mailboxes; you can move the entire subdirectory to a new machine and it works perfectly. I can't say much for the newer versions of Eudora. Version 6.2.1 seems to have some bugs in it, but if you can get earlier releases of this software, it's excellent.

  • How I do it (Score:5, Informative)

    by Matt Perry ( 793115 ) <perry DOT matt54 AT yahoo DOT com> on Saturday March 12, 2005 @07:44PM (#11922786)
    I use a procmail recipe to archive my mail. I put it after filtering mailing lists and before I filter spam:

    OLDMAILDIR = $MAILDIR
    MAILDIR = $ARCHIVE_DIR
    :0 cW: archive.lock
    | /bin/gzip >>mailarchive-`date +%Y%m`.gz
    MAILDIR = $OLDMAILDIR

    I use grepmail [sourceforge.net] to find old emails that I might need. Grepmail lets you use perl regular expressions to find messages and then outputs the entire message where a match was found. You can use grepm [barsnick.net] to open grepmail matches as a mailbox in mutt. grepine [www.dfki.de] does the same for Pine, which I use.

    At the end of each year I clean the spam out of my archives using a procmail recipe and spamassassin. This recipe marks messages as deleted in the mailbox. I open these in pine, sort by deleted, and double check them. Once I'm sure they're all spam, I delete them:

    # vim:ft=procmail:

    LINEBUF = 8192
    SHELL = /bin/sh
    MAILDIR = $HOME/mail

    :0 fW: spamclean.lock
    | spamassassin -e --prefs-file=/home/matt/.spamassassin/user_prefs-s pam_clean 2>/dev/null

    # If the message was deemed to be spam, set the status to "deleted" so that
    # we can delete it easily and optionally review it.
    :0 e
    {
    :0 fhw
    * ^^rom[ ]
    | sed -e '1s/^/F/'

    :0 f: formail.lock
    | formail -I 'X-Status: D'
    }

    # Fix the mangled "From" line
    :0 fhwE
    * ^^rom[ ]
    | sed -e '1s/^/F/'

    # Remove the last of the SpamAssassin headers
    :0 f: formail2.lock
    | formail -I 'X-Spam-Checker-Version'

    # File message in temporary mailbox
    :0: sandbox.lock
    z-cleaned_mbox

    The special spamassassin config turns off bayesian filtering and sets the threshold high:

    required_hits 15
    clear_headers
    fold_headers 0
    use_bayes 0
    The rest of the spam I clean out by hand.

    • Re:How I do it (Score:4, Informative)

      by Matt Perry ( 793115 ) <perry DOT matt54 AT yahoo DOT com> on Saturday March 12, 2005 @08:00PM (#11922863)
      Almost forgot. I archive my sent mail as well. This might be harder for you if you don't use a single email client on a single machine. IMAP can help with that.


      Put this in ~/bin/rotate-sent-mail.sh:

      #!/bin/bash

      # This script takes sent mail in $HOME/mail and moves them into
      # $HOME/.mailarchives/sent. It will also rename the file to have the date of
      # the log file included.

      MAILDIR=$HOME/mail
      ARCDIR=$HOME/.mai larchive/sent
      year=`/bin/date +%Y`
      month=`/bin/date +%-m`

      # updating last months mail
      month=$((month-1))

      # if this is last years mail, set the date correctly
      if [ $month -eq 0 ] ; then
      month='12'
      year=$((year-1))
      fi

      # if the month is less than 10, add the leading zero back
      if [ $month -lt 10 ] ; then
      month=0$month
      fi

      mv $MAILDIR/sent-mail $ARCDIR/sent-mail-$year$month
      touch $MAILDIR/sent-mail && chmod 600 $MAILDIR/sent-mail
      bzip2 -9 $ARCDIR/sent-mail-$year$month
      Now add the following to your crontab:

      0 0 1 * * $HOME/bin/rotate-sent-mail.sh
  • and just store the whole thing on cd/dvd/tape..

    maybe run a nice script to have an index in a db on the sender/title ....

    or standard mailbox, and then make a bootable minilinux with pine (mozilla/whatever) on the same media (use vmware or similar to access it from non unix systems ...

    actually my mailbox is just a growing junk collection, and have to do something with that too ... and hey, i even have stuff from the bbs times with quickmail, or bluemail .... or whatever it was called :)

    or just export th

  • Manatory ZOË plug (Score:3, Informative)

    by MCRocker ( 461060 ) on Saturday March 12, 2005 @07:45PM (#11922789) Homepage
    Well, if you're going to be on this topic, a mention of ZOË [zoe.nu] is pretty much required.

    ZOË is a sort of an archiving proxy that sits between your mail client and your mail server. It stores and indexes everything, so you can pop open a browser window and do a search on anything you've ever sent or received. Naturally, this was created before gmail [google.com].

    With ZOË you don't need to worry about those pesky email folders and waiting for long searches.

    Naturally, spam filtering before ZOË is a good idea.
  • I've kept all my mail for the last 10 years or so using stand unix mailbox files. Pine moves each read and sent folder into individual monthly folders which are very easy to grep through. In fact to use "modern" email programs I setup a cron script to do the same thing that pine used to do as well as move mail more than a year old into an "old" archive subdirectory. In addition I have procmail eat the binaries into an attachments folder. Its simple to setup all the directories for "viewing" via imap though
  • I do data mining research, most recently on the Enron email dataset [cmu.edu], and I've actually been having to roll my own multi-mailbox storage, access, and retrieval systems. It's taking way more time than I'd like, at this point I've gotten a database and web-based viewers [uiowa.edu] made up (beware, they're quite slow).

    If anyone has an idea of an open-source application similar to what the submitter is looking for, it would help my research quite a bit. There's practical research applications in this stuff, if someone's

  • CSV (Score:2, Interesting)

    by vnangia ( 730425 )
    Just about every email program that I've used has managed to export to CSV. A few web-based email systems didn't allow such imports and some hunting on the web found some sort of convertor (like YahooPOPS!, etc.) that converted to POP and then I exported them to CSV using Eudora or Outlook, or whatever program I was particularly enamored with.
    Admittedly, sometimes the column names didn't match up ("Sender" v "From"), etc., but for the most part that how I did it. I also made an effort to keep the number o
  • This is only practicle if you use something like Gaim that automatically saves conversations for multiple IM services. Every few months I'll copy my old IM logs to a backup location, usually when I upgrade the client software in case they change the format of the file or what not.

    It's actually pretty convienient to be able to search through old conversations. It makes a better journal than trying to thoughts out to yourself.

  • My solution (Score:5, Funny)

    by anon*127.0.0.1 ( 637224 ) <slashdot@baudkaM ... om minus painter> on Saturday March 12, 2005 @07:54PM (#11922831) Journal
    I've found the easiest way to handle EMail when it's in multiple formats like that is to just print everything out and store it in boxes in my garage.

  • one year of e-mails. I keep all important e-mails forever until I don't need them.
  • I use IMAP and mailbox/maildir.

    IMAP allows synchronization between many different servers and clients.

    Mailbox and maildir both are open formats. Maildir uses one file per message, which makes management a little easier but some less advanced file systems have trouble with so many files in a directory. Mailbox is a reliable standby. Either way, I keep annual archives.

    I think it's inappropriate to keep permanent records of IM conversations. IM is more like a face-to-face talk and people don't stick a t

  • What I do is have an IMAP server running on a Unix machine, and archive my e-mail into IMAP folders. The server turns those into vanilla Unix mailboxes, so I can deal with the messages as plain text if I want to. Almost every e-mail client out there supports IMAP, so compatibility isn't much of a problem. It also lets me use different clients and different systems to access all my e-mail in one place.

  • I archive most of my emails. Up to this point, my email archive is close to 2 GBytes.

    I keep the emails in mailbox format (that is, in plain text as it is stored in most UNIX systems), in several files. The reason I do that is that most email readers (MUA) can read mailbox format. I keep them in several files to make it more manageable.

    The tools that I use to manipulate emails are mostly "from", "procmail", "grep", and "less". There used to be tools from the "elm" era (still remember them?), such as "frm"

  • I've always found IMAP fantastic (properly configured, thousands of emails searchable instantly, from any email client anywhere in the world (if you use imaps or ssh tunnels), only downside is you need a machine to run it on, though if youve got more than one PC in the house, it's not a bad idea to put a 3rd one in to run a few network services, like email and file sharing.

    I'd recomend for those without a limited budget to get a low-power machine, onboard video, slowest 90nm processor you can find, one har
  • Does anyone archive IM?

    AOL does, don't you read /. ?

  • I use the Enron archive format for backing up my records. It's easy, but time consuming:

    1. Print out emails.
    2. Shred them.

    Man, it's so nice not being burdened by the embarrasing history of all the emails I sent. Besides, that was a long time ago, way before Score: 5, Funny.

  • save all ingoing and outgoing in YYYY-MM files... (Score:3, Informative)

    by Doctor Fishboy ( 120462 ) on Saturday March 12, 2005 @08:28PM (#11922986)
    ...and let mutt sort out.

    I had multiple folders, sorted by people/project. I got in a complete mess and finally snapped when I spent half an hour looking for a simple message.

    Use procmail to write all incoming messages to 'all-mail-YYYY-MM' and use Mutt hooks to write out to the same directory.

    At the end of the year, cat them together and make 'all-mail-YYYY'. Accessing and reading this mailbox can be done with 'mutt -R -f all-mail-YYYY' as this opens read-only. Use 'l' to do 'limit' searches and use ~t, ~f, and ~b in AND combinations to limit on To: From: and body of messages. It's lovely only having to look in one place!

    Procmail:
    INCOMING=all-mail-`date +%Y-%m`
    # now I want to keep a copy of EVERYTHING in a dated directory :0 c:
    $INCOMING

    Muttrc:
    set record="+all-mail-`date +%Y-%m`"

    Works for me!

    Dr Fish

  • Post it on /. (Score:3, Funny)

    by pyrrhonist ( 701154 ) on Saturday March 12, 2005 @09:07PM (#11923170)
    Just post all your email on Slashdot. Since no one reads the articles anyway, no one will notice when they aren't on topic, and there's always the possibility that your email from Mom will actually be on topic. Furthermore, it will be easy to search using Google, and the Wayback Machine will automatically archive everything for you.

    As a bonus, you can tell which emails are worth reading by how they get moderated. All your work related emails will probably be modded Troll, except for your performance review, which will be modded +5 Funny. Email from your illicit lovers will be modded Insightful, since that type of thing is new to most of us. Email from your family will be conveniently modded down so you will not have to deal with it. Your friends won't need to send you any email at all, since they are probably already on Slashdot, and therefore, know enough to post in your threads.

    Problem solved. Ah, Slashdot... Is there anything it can't do?

  • I run my own personal IMAP server (Score:3, Insightful)

    by EMR ( 13768 ) on Saturday March 12, 2005 @10:16PM (#11923551)
    I have a machine that runs a dedicated IMAP server with one account on it (mine) which has my 2GB+ of e-mail since 1996. (minus the spam of course). That way I can easily switch between different clients and not have to worry about converting my e-mail all the time.

  • Offlineimap, of course (Score:3, Informative)

    by LibrePensador ( 668335 ) on Sunday March 13, 2005 @12:56AM (#11924187) Journal

    You open all your email with an email client and move all the disparate inboxes into a big IMAP store on your own computer or one provided by a joint like Fastmail.fm [fastmail.fm] or Runbox.com

    Then, you keep a local backup on any computer that you move to with offlineimap [quux.org], a wonderful utility that doubles as a multi-inbox syncronizer and backup utility. I have been using it for the past two years and can attest to its reliability.

    Enjoy

  • Archiving IM ... (Score:3, Informative)

    by rocketfairy ( 16253 ) <(nmt2002) (at) (columbia.edu)> on Sunday March 13, 2005 @01:22AM (#11924314) Homepage
    ... is easy. gaim can do it automatically, plaintext or html, by recipient, and searching is easy. Opening up a nice big html file in a browser, complete with every conversation you've ever had with someone on AIM, can be quite handy.

    I actually have two backups of my mail:
    • raw mbox. procmail copies everything to a folder (on my mailsever) which fetchmail (on my box) grabs once in a while, usable with most civilized mail programs (want to copy everything to an imap server? Use t-bird or some crap) and searchable with mutt, or for that matter a text editor; and
    • gmail. Yes, you have an extra invite or 50 ... just procmail or otherwise autosend a copy of everything to a gmail account as backup. Gmail might not last forever (hence the mbox, and some cdr/tape), but while it does, it makes for handy searching. Not as nice as mutt though :)

Slashdot Top Deals

If all else fails, lower your standards.

Close