Forgot your password?
typodupeerror
Communications

Ask Slashdot: Best Way To Archive and Access Ancient Emails? 282

Posted by timothy
from the learn-them-like-the-ancient-memory-singers dept.
An anonymous reader writes "I started using email in the early 90s and have lost most of that first decade due to ignorance, botched backups, and so on. But since about 2000, I've got most — if not all — of my email in some form or other. I run Linux, so this has mainly been in a mix of various programs: Kmail, Evolution, Thunderbird. The past 2-3 years are still on the IMAP servers. My problem is that I only rarely NEED to look back to email of 5 years ago. But sometimes it's nice. Or I just want to reminisce about something...or find an old attachment that I was sent. But I do not want to be clogging my current email client of choice with vast backups and even more, I don't know if it will even easily convert. The file structures are different, some are mbox, others maildir, etc., and I would ideally like a way to 1) store and archive these emails, 2) access them, and 3) search by Sender, Subject, Date, Attachments. Is there anything I can do or do I just have to keep legacy applications on hand for this? Should I keep trying to upgrade and pull old files into the new applications? Any help or suggestions about what YOU do would be great."
This discussion has been archived. No new comments can be posted.

Ask Slashdot: Best Way To Archive and Access Ancient Emails?

Comments Filter:
  • IMAP (Score:4, Informative)

    by sylvandb (308927) on Friday March 29, 2013 @08:07PM (#43315563) Homepage Journal

    Just IMAP it all.

    I went IMAP in 1997 and have never looked back.

    I've also used IMAP as a temporary conversion measure for people switching e-mail clients so even if you aren't sure, it makes a good first step.

    I don't understand the concern about too many e-mails. I can access my email back to 1992. With multiple folders it shouldn't be a problem and with modern indexing a search shouldn't be an issue.

    • Re:IMAP (Score:4, Informative)

      by DNS-and-BIND (461968) on Friday March 29, 2013 @08:27PM (#43315707) Homepage
      When I search for my brother's name I don't want to wait 30 seconds for a search to complete, nor do I want to see his emails from 10 years ago. I just want to see his last email that he sent about the trip we're taking next week. That's the concern about too many emails.
      • by AK Marc (707885)
        That's when you sort by sender, then look for his name, since it's all alphabetical. What I don't like is the lack of a convenient multifield sort I want an email with "law" or "IRS" in it from between 1998 and 2000 sent directly to me, with nobody in the "cc" field.
      • Re:IMAP (Score:4, Insightful)

        by hobarrera (2008506) on Friday March 29, 2013 @10:49PM (#43316277) Homepage

        Archive old emails by year:


        Archives/2013
        Archives/2012
        Archives/2011
        Archives/2010
        Archives/2009 ...

        Only search in the appropiate ones. Easy, right?

      • Re:IMAP (Score:4, Interesting)

        by Yosho (135835) on Saturday March 30, 2013 @12:04AM (#43316531) Homepage

        For what it's worth, my personal mail server is an Athlon X2 3800+, still running Ubuntu 8.04.4 LTS. Pretty old by today's standards. I've got a dovecot server offering IMAP access and a Roundcube webmail server on it. My inbox has about 25,000 messages in it, and there are e-mails in there that go back to 2007.

        Doing a search by sender took maybe 1 or 2 seconds, and the most recent e-mails came up right away since they're sorted by date.

      • Then use Kmail and behold the power of akonadi and nepomuk. searching in the later versions is fast -- and I have a very large number of mails.

        It turns out that for large enough collections you _do_ need a DB :)

    • Re:IMAP (Score:5, Insightful)

      by kwerle (39371) <kurt@CircleW.org> on Friday March 29, 2013 @08:42PM (#43315771) Homepage Journal

      This.

      I fired up imap servers for all my old mail.
      I fired up a modern mail client (OSX Mail.app) and connected to all of 'em and also to gmail.
      I dragged all my old email into gmail. In a GUI. And it worked.

      Done.

      I no longer run mailservers. Too much of a headache. gmail is awesome (with imap access, even). Indexing, instant searching, etc.

      If you don't want/trust your email to the cloud, then this isn't for you. Unless you want to run your own imap server with whatever backend suits you - then you can dump it all there. I just can't be bothered to manage that after 15+ years of doing so.

      • It's a shame gmails has no real filters, like (sieve [wikipedia.org]).

        Yes, the webinterface CAN create filters. But if you're using a desktop clients, it's not confortabe to have to open a DIFFERENT client to configure filters.

        Gmail also lacks some imap features, notably, the sort command.

        • by peragrin (659227)

          That's why I do both.

          for every day I use gmail

          but once a year I fire up a current email client and download everything archived. I also purge the archive every couple of years of truly useless emails.

          I keep a a copy for safety and away I go.

      • Re: (Score:2, Funny)

        Hello kwerle,
        Thank you for your efforts, we appreciate it.

        Sincerely,
        The FBI.

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      IMAP offers "server side" storage with "client side" viewing.

      Besides the privacy implications, are you seriously suggesting that the OP
      a) find or create an IMAP server,
      b) force feed that server all his archived emails (presuming that there is some way to bulk import email into the IMAP server), and
      c) change his current email setup so that, from now on, his email is sent to the mail server on which the IMAP server runs?

      How is that any easier to manage than his current predicament? ISTM that your suggestion f

      • by hairyfeet (841228)

        Got a better idea? The guy has them in Lord knows how many programs splattered all over the place, he wants it all in one place and searchable and IMAP is good at that. I mean sure we would all prefer a "push button and its done" kinda deal but AFAIK no such thing exists that will let him store it locally. You could run it all into Gmail but some folks have privacy concerns and of course its not like Google hasn't lost stuff in the past, so given a complex situation like TFA I'd say IMAP is probably gonna b

        • Re: (Score:3, Informative)

          by arth1 (260657)

          Pretty much everything not made by Microsoft will support export to good old mbox. It's a good format to store in, because you can always import from it into other formats.
          And you can run simple scripts against the mbox files.
          More than once, I've done a grep against my mail archive, and more than once I've moved it to a new machine and new mail software.

          • Re:IMAP (Score:5, Informative)

            by Antique Geekmeister (740220) on Friday March 29, 2013 @10:04PM (#43316099)

            _NO_. Under no circumstances use "mbox" for mail storage, or anything other than a temporary stage on the way to transferring it to something contemporary and uable such as Maildir. If you lose that one mbox file, by file system corruption or by fat finger accident or overflowing a partition or in tht eprocess of merging new email with it, you've lost _all_ your mail in that mbox. And as you read, mark, or save mail, that file is constantly churning, making backup and replication of the mail spool far more dangerous and fragile, especially when the mail directory is bulky with years or decades of active mail threads or simply undeleted email.

            mbox was useful when the available inodes on a file system were limited programs benefited from using a single inode for transactions, and backups occurred on magtape, but there is simply no point to it in decasdes.

            • Re:IMAP (Score:4, Insightful)

              by icebraining (1313345) on Friday March 29, 2013 @10:27PM (#43316185) Homepage

              I agree with that for new emails, but for an archive file, none of it really applies. File system corruption and fat fingers should be handled by just restoring from backup, and merging / marking as read / etc is not really applicable for old mail, which should be accessed by either viewing it readonly or making a disposable copy.

              mbox might have its problems, but I don't think there's any good reason to spend time converting old files to Maildir.

              • Re:IMAP (Score:4, Interesting)

                by adolf (21054) <flodadolf@gmail.com> on Friday March 29, 2013 @11:24PM (#43316419) Journal

                What's wrong with MBOX? I've been using it with gigabyte-sized folders for over a decade and nothing bad has ---@@@ From MAILER-DAEMON Fri Jul 8 12:08:34 2011

                Seriously, though. Even when mbox gets trashed due to disk corruption, it is still every bit as useable as a trashed maildir: Messages get lost, not whole folders...and even then, that's what backups are for (right?).

                Or at least, that was my experience over the past many moons. As a static archive (Sent-mail-2012), I can't think of a single thing wrong with mbox. And it's easier to cat to tape than maildir.

                • by arth1 (260657)

                  Or at least, that was my experience over the past many moons. As a static archive (Sent-mail-2012), I can't think of a single thing wrong with mbox. And it's easier to cat to tape than maildir.

                  You don't even have to create a new tape entry, just append to the old one.

                  Not to mention that mboxes can b cat'ed together, as they get older. Like combining all dailys to a weekly, weeklies to a monthly, and monthlies to a yearly.
                  What it doesn't have is indices, but nothing prevents you from importing and converting a mbox to an indexed format. Or creating an index for that matter.

                  When archiving, the main point is that you want a format that is mature, long-term-supported, and easy to use tools again.

            • Re:IMAP (Score:5, Informative)

              by arth1 (260657) on Friday March 29, 2013 @11:19PM (#43316399) Homepage Journal

              _NO_. Under no circumstances use "mbox" for mail storage, or anything other than a temporary stage on the way to transferring it to something contemporary and uable such as Maildir. If you lose that one mbox file, by file system corruption or by fat finger accident or overflowing a partition or in tht eprocess of merging new email with it, you've lost _all_ your mail in that mbox.

              Thus speaks ignorance. If you write corrupt data to a mbox file, nothing prior to the corruption is affected at all. Unlike most formats that don't store each mail in a separate file, you can also very easily run recovery against a mbox file. Heck, a one-liner perl script can retrieve anything from before and after a corruption.

              And "overflowing a partition"? Um, run that by us again. If you mean disk full, that doesn't truly affect a format that's made for appending. You won't be able to append. Any other format you can come up with will have the same problem.

              And for archival purposes, this also does not apply. You don't make changes to your archive. Period.
              And you back it up. Period.
              Which is a heck of a lot easier to do with mbox than most other formats.

              But again, the main strength is that it is so simple, which means that pretty much every mail program out there will support it, one way or another.
              Choosing a more modern format leaves you with fewer options, and less certainty that it will be supported in the future. 20 years down the road, mbox will still be supported. It has an RFC - http://tools.ietf.org/html/rfc4155 [ietf.org]

              Can you say the same about ANY other format? Maildir doesn't work on systems that doesn't allow colon in file names, and hashes the filename based on the hostname which both isn't portable, and crashes badly for many implementations if you have a non-ascii hostname. Not to mention that the format has balkanized, to the point that it's no longer compatible betweeen implementations.

              Again, for archival purposes, simplicity is the key.

      • by skids (119237)

        a) cyrrus. done.
        b) there is. pine.
        c) also pretty easy.

        Personally I have 40K messages on an imaps store. A full text search does take a few minutes, but only because the hardware is utterly ancient -- less powerful than many modern cell phones. I should get around to upgrading that.

      • by raymorris (2726007) on Friday March 29, 2013 @10:11PM (#43316123)

        Force feed? WTF are you taking about? Dovecot can use any make mail format. Just set MAILDIR if it's in a non-standard directory. So the whole procedure is:

        yum install dovecot
        vim /etc/dovecot.conf (only if using a nonstandard mail location)
        service dovecot restart
        set username and password in GUI client

        I never will understand why some people feel the need to post on topics they don't have the slightest clue about.

      • Re:IMAP (Score:4, Interesting)

        by WillKemp (1338605) on Saturday March 30, 2013 @02:07AM (#43316831) Homepage

        [......] are you seriously suggesting that the OP
        a) find or create an IMAP server,

        Ridiculously simple. They're already running Linux, they just have to install dovecot and they've got a fully functional IMAP server (no configuration required) - which has access to all their local mail boxes.

        b) force feed that server all his archived emails (presuming that there is some way to bulk import email into the IMAP server)

        Ridiculously simple. Fire up Thunderbird, configure it to access your local IMAP server, select all, drag and drop.

        c) change his current email setup so that, from now on, his email is sent to the mail server on which the IMAP server runs?

        Why would they need to do that? Thunderbird (or other mail reader of choice) can access multiple accounts.

      • Re:IMAP (Score:4, Interesting)

        by Cajun Hell (725246) on Saturday March 30, 2013 @11:10AM (#43318279) Homepage Journal

        Besides the privacy implications

        There aren't any privacy implications. If you there were any, then you would have named one or mentioned an example. The situation prior to typing "sudo apt-get install dovecot" is that he had the data (so it's already subpoena-able or whatever you're trying to imply) and after that he'll also have the data. Nothing changes. Are you complaining that he's keeping the data rather than deleting it? I don't get your point at all.

        are you seriously suggesting that the OP a) find or create an IMAP server,

        Yes, because it's easy. This can be done in literally ten minutes. Maybe a little more if he doesn't already have good storage allocated for it (e.g. a Reiser formatted ~/Maildir, or whatever your own religion commands).

        b) force feed that server all his archived emails (presuming that there is some way to bulk import email into the IMAP server)

        Yes, because it's trivial. It's highly likely that whatever he is using to read each of his different mail archives, can also talk IMAP, because everything talks IMAP. You say "force feeding" as though literally selecting and dragging in a GUI, or picking "copy" or "import" off some menu, is hard. It's not.

        As for step c (changing how he receives email), I don't think that's being suggested but it may be a good idea. He can decide later, whether or not he wants his archive server to become his main/active server. That decision can wait and is not part of the scenario being discussed; it's an opportunity for the future.

        How is that any easier to manage than his current predicament?

        Because then he'll have his archive stored in a system that is specialized for handling the problem, accessible and searchable by any client he wishes to use, possibly even the very same tool he uses for his day-to-day non-archive mail reading. Or he can pick some other IMAP client if it handles mass/archive use case better than the routine use case. Everything Just Works, all together. All of his complexity and exceptions disappear. And at virtually no cost; there's no downside to counter any of the advantages.

        This is one of the easiest no-brainer Ask Slashdots, ever. There is one right objective best simple easy-and-fast-and-good(!) answer, and setting up an IMAP server is it. Probably because email storage is an old, very-solved problem.

  • Use the IMAP server - if you have control and/or space available.

    I just have a single large archive IMAP folder into which everything that isn't spam gets pushed. You could optionally create subfolders for time ranges (every 1-2 years, whatever works for you). Using dovecot with good indexing support on the backend quick searching has been great. If you do a sub-archive breakout on time the searches will be quicker, you could also then create a virtual mailbox combining them all for when search really needs

  • by the eric conspiracy (20178) on Friday March 29, 2013 @08:08PM (#43315573)

    I have all my personal email from 1998 in a Maildir directory with Dovecot as the server on a dual core Atom server running Centos. About 900 MB worth.

    Plenty fast.

     

    • Re:Maildir (Score:4, Interesting)

      by hedwards (940851) on Friday March 29, 2013 @10:24PM (#43316175)

      The main problem that I personally have is where I get emails from people that reference images and such from other servers. Most of the time it's commercial messages that I delete, but sometimes there's a newsletter that I want to save, and the images themselves turn out to be relatively important. Kind of annoys me to have to print them to PDF so that the formating gets preserved.

  • Trying to figure out what formats will be available in the future is pretty hard, it's easier to see what formats have been around a long time and are still in use.

    As such, two formats come up readily:

    mbox http://en.wikipedia.org/wiki/Mbox [wikipedia.org] and maildir http://en.wikipedia.org/wiki/Maildir [wikipedia.org]

  • Convert all your mail to maildir, and keep it on your home filesystem, whenever you need access from where ever connect to your home vpn, and connect to the filesystem, I have an account in thunderbird where I can search, or do whatever I want to it. Seems to work well.
  • Just dump them (Score:5, Interesting)

    by sk999 (846068) on Friday March 29, 2013 @08:13PM (#43315607)

    Had the same need 20 years ago when migrating from VAX/VMS to Unix. The old emails were saved in a not quite readable format, but I figured I could recover them if necessary. In the end, never bothered. Yes, there are a few (actually, only two) that I'd like to resurrect now, but life moves on.

    • Re: (Score:2, Insightful)

      by foniksonik (573572)

      This. Don't be a data hoarder. Go through them if you must and re-mail the best/important ones. Then dump it to dev/null and move on.

      Do the same with movies, books, bookmarks, photos, apps, docs, etc. you'll be happier without all that baggage. Music of course is another story. Keep that forever and only toss out the dreck (those extra songs on that album you bought because singles didn't exist yet).

      • by hedwards (940851)

        For mail and bookmarks that makes little sense. It would take me more time to make those decisions than it's really worth. Emails and bookmarks take up such a small amount of space that it's not really worthwhile to worry about.

        Now, general filesystem files, that's a different matter, I used to have a system where I only backed up things I cared about, and let filesystem crashes wipe out the rest of the data. Seemed to work out just fine.

    • by tutufan (2857787)

      Yeah, that's about where I'm at, too. I think of those Buddhist monks making the sand paintings (which they then sweep away). It's an exercise in recognizing the impermanence of all things.

  • Translate it _all_ to IMAP services, in MAILDIR format if available. I've repeatedly been faced with clients, partners, and colleagues who use their email as their insitutional memory and need to migrate to a new service. There are few technologies as straightforward, and robust, as a simiple IMAP server running a light, uncluttered IMAP daemon such as "dovecot", without the complex and nunnecessary requirements of aCyrus IMAP daemon, and most _definitely_ without the complex support requirements of an Exch

  • Use a database! (Score:5, Interesting)

    by cosm (1072588) <thecosm3 AT gmail DOT com> on Friday March 29, 2013 @08:14PM (#43315613)
    I'm a big fan of throwing together a DB when I want to store things categorically like that and want fast searches. If you are up to the task, hunt down some tools/roll your own so that you have a nice relational database and some stored procedures for getting what you want when you need it.

    You could export your emails to some parsable format, write an importer to extract the basics that you want to keep (from/to/subject/body,attachments/entire binary blob/etc) and then bulk insert that mess into on a mysql/sql server tucked away somewhere locally or "in the cloud" (EC2, Azure). Just another option as I'm sure you'll see here many here. At least with this route you are in full control of how you index, what you can search, encryption, performance, level of backups, etc. Maybe not the best way for some but I know if I had over 100000 emails that I wanted searchable very very quickly with advanced SQL like searching, this would be a cool way to do it (time permitting). Good luck! And to the pedantry to ensue...Yes. Good day.
    • Re:Use a database! (Score:5, Interesting)

      by Anonymous Coward on Friday March 29, 2013 @09:00PM (#43315853)

      And you could make a doilie, and a hat, and a casserole, and wallpaper with the headers, and knit the .signatures into a fancy flying cape.

      Just use IMAP and Maildir. Modern systems are fast enough to allow you to search the content directly, and not vulnerable to the database support wackiness this sort of "I can pre-organize it now and make my life better by wasting it pre-programming my queries" approach.

    • Indeed, databases are the perfect solution, though not relational ones; rather email database formats: like Maildir!

  • Gmail (Score:5, Informative)

    by lga (172042) on Friday March 29, 2013 @08:18PM (#43315659) Homepage Journal

    Best method of storing and searching old email? Gmail. It can import from pop and imap so you can point it at your other inboxes and let it get on with it.You can upload from other mail clients to Google's imap server. Obviously it's amazing at searching through the archives.

    Best method if you're concerned about Gmail's privacy? I'm still working on that one.

    • Re:Gmail (Score:4, Informative)

      by zekele2 (1556449) on Friday March 29, 2013 @08:55PM (#43315829)

      Best method of storing and searching old email? Gmail. It can import from pop and imap so you can point it at your other inboxes and let it get on with it.You can upload from other mail clients to Google's imap server. Obviously it's amazing at searching through the archives.

      Best method if you're concerned about Gmail's privacy? I'm still working on that one.

      The solution is Google Apps for your own domain. $5 a month per user, 25Gb space, IMAP, no advertising (which is where most of the privacy issues arise), and most importantly, no lock-in as you can switch your email to a different provider at any time without changing email address. As you said, Gmail is by far the best for searching old email. I haven't run an email server for years.

    • Better hope the algorithmic overlords don't decide you're a spammer and lock you out. Remember not to keep your eggs in a single basket.

    • Obviously it's amazing at searching through the archives.

      Regrettably, gmail's IMAP implementation does not support the sort command.

  • by 93 Escort Wagon (326346) on Friday March 29, 2013 @08:24PM (#43315687)

    Design a MySQL database for storing your mail messages, keying on sender, subject, date, and presence of attachments (bonus points for storing the attachments as blobs rather than as external files). Then write a perl script that'll automatically parse all your incoming email and convert it to database entries. I suppose if you're lazy the script could just monitor your mail spool, but it'd be better to just have it listen for incoming connections and handle the mail directly.

    Next, make copies of that script, modifying as necessary to process all your old mail archives.

    Oh, and you'll need to write another perl script to access all new mail - not from your mail spool, but from this database. You should probably name this system after some animal too. If you absolutely MUST have a graphical interface on it, don't use anything newer than TCL+Tk - but going with curses would be a better choice.

    Oh - it has to be GPLv3, or we'll hate you and probably mailbomb your machine.

    What - isn't that the Slashdot way?

  • by realmolo (574068) on Friday March 29, 2013 @08:26PM (#43315699)

    You don't need all those e-mails. Keep the few you actually care about (copy and paste the text into a regular file, and save any attachments you want), and get on with your life.

    People that keep every e-mail are weird. Quit living in the past.

    • by Ardyvee (2447206) on Friday March 29, 2013 @08:43PM (#43315777)

      It's kind of like photos, you know? Or letters, and such. People like to store those things, because they serve as a memory aid for what the mind no longer holds. It is also quite useful for history reconstruction/when you are old and have nothing else to do but a box full of photos/letters/etc.

      Not to say that you are wrong on your point, except on the weird part. Unless you are okay with double standards, or you also consider anybody who keeps photos of parties/graduations/etc weird... Just saying.

      • by houghi (78078)

        Not to say that you are wrong on your point, except on the weird part. Unless you are okay with double standards, or you also consider anybody who keeps photos of parties/graduations/etc weird... Just saying.

        I would. In the olden days you would have 24 photos of a whole week. I have been on holidays for 6 weeks and have a film of 36 photo's to prove it.
        Now you have 300 photo's of just packing your suitcase. So yes, if you keep all those thousands of photo's that are completely irrelevant, then you are weird

    • Re: (Score:3, Insightful)

      by FuzzNugget (2840687)

      Right, I don't need that client email from a few years ago to remind me about a detail on a project. It'd be better just to look like an idiot in front of them.

      Just because *you* don't need that archive doesn't mean everyone else doesn't need it.

      Why the hell *not* keep a conveniently categorized, organized, sorted, indexed and searchable database of all your important electronic communications? A few gigs is nothing these days.

    • by icebraining (1313345) on Friday March 29, 2013 @11:01PM (#43316325) Homepage

      But why would I waste time manually finding and copying individual emails, when I can just let the backup script archive them all for virtually no cost?

    • by ZeroPly (881915) on Saturday March 30, 2013 @01:21AM (#43316749)
      The problem is that a throwaway email might become critically important later on. There is no way to know in advance what is important and what is not.

      True story: while deployed in the Army, our communications guy could not find a piece of equipment which was very important and very pricey. He had been signing the monthly inventory forms saying he had it, assuming it was in a cabinet. He could not find any paperwork showing it was signed out - it had just disappeared sometime in the last 3 months and no one had seen it.

      On a long shot, I started searching my email - since I keep every last one. Sure enough, about 2 months prior, there was a throwaway email from him to the effect that he was going to turn in item X for repair since it was acting flaky. He checked at the contractor mentioned in that email, and it was sitting on the shelf waiting for pickup.
    • by CAIMLAS (41445)

      I recently had to go back 5 years to retrieve an email as evidence in my pending divorce. "I never said that!" - like hell you didn't, I've got it right here (and here, and here, and here). She wanted to play hardball, so hardball it was. :(

      I've had to go back 6+ months to retrieve important mail for myself for work and other personal matters as well. Every written correspondence with my ex-wife was in email - some damning, as above, but all of our "love letters" when we were courting. Wouldn't you want to

    • by Tom (822)

      It's not about living in the past, it is about not worrying what to keep and what to store. And the message you think most important today is likely to be completely worthless in five years, while you would love to still have that other message you thought unimportant back then.

      I keep everything so I can decide today which old message I consider important. It is very rarely that I venture into the archive, but I have needed messages two years old at times.

  • by Nexus7 (2919) on Friday March 29, 2013 @08:32PM (#43315727)

    I need to archive emails that I can search later - but with a twist. These are employees who've left the company. I can't keep 'em on at Google Apps 'cause I have to pay for that by user. So I use IMAP (making sure to set Chats to be shown in the IMAP list), create an account in Thunderbird, and slurp it all on to the local machine. It keeps all the folders, although I doesn't seem to be smart enough to figure out multiple labels, so it looks like it downloads the same email multiple times, once for it's folder, and once for "All Mail." Then I delete the account at Google. You just have to be sure to click through all the folders in Thunderbird and make sure it is done downloading before you blow the Google account away.

  • You can even read them in a text editor, every half decent email client can use them and there are free or cheap converters for the email clients that are not half decent.
  • notmuch (Score:3, Informative)

    by Anonymous Coward on Friday March 29, 2013 @08:34PM (#43315739)

    http://notmuchmail.org is Gmail for people that don't trust Google. Works great with your existing IMAP server using offlineimap.

  • Gmail. (Score:4, Insightful)

    by jtownatpunk.net (245670) on Friday March 29, 2013 @08:36PM (#43315745)

    As soon as gmail made IMAP available, everything went there. I used to get my stuff via POP and saved it all going back to the early 90s. When IMAP went live on gmail, I let it chug away for hours and hours until it was synced and all my archived stuff was stored on my gmail account. They've been bumping up the limit faster than my mail's built up so I'm now at 3.9 gigs used of 10.1 available, holding about twenty years of email. I have email clients on a desktop and couple laptops that I fire up every couple of months to sync with gmail and keep local stores in the event that google screws up and loses my data. (I like to think I'd be smart enough to disconnect from the internet before accessing the local clients if my gmail account ever went blank but I've got multiple copies just in case I forget.)

    I know that won't work for email fiends who pile up a gig a month but it works for me. I don't even bother sorting my email any more. It's faster to just search. Not like the old days when it would take my email client half an hour to slog through all the messages. :)

    • I did not know that gmail had something like that which would let you import your old emails into gmail. I'd just seen imap as a way to export mail out of gmail. Damn. They've got a time machine point of view into all the nooks and crannies of communication done by everyone who's bothered to imap-archive their ancient emails into google. That's even sicker than I'd ever anticipated or would have thought I'd have believed. ;>)
  • by chipperdog (169552) on Friday March 29, 2013 @08:36PM (#43315747) Homepage
    Set up a local courier IMAP server and copy mails there, and archive the Maildirs...each message will be a file and you can use tools like grep to search the Maildirs
  • Call it "Lexi Diamond - Ronda Rousey mud wrestle" and share it on a torrent and soon the whole world will back it up for you....

    Seriously though, even if you were a previous email hoarder, you will likely be able to comfortably archive all your emails *and* the tools needed to access them on a USB stick. Start by finding all the tools you need, source included, and place them on your storage medium. Compress it. Send it to the cloud.

    Mail files can be stored by year (easy enough to do with awk or other mail

  • Take your pick.

  • by Creosote (33182) on Friday March 29, 2013 @09:02PM (#43315867) Homepage

    I wouldn't posit this as the best way, but it's what I do. I keep my archival mail on a local filesytem arranged in directories, stored in the old-school mbox format. I run Dovecot under OS X for IMAP access to those messages from anywhere; when I need to search through the whole collection, I use mairix [force9.co.uk] (an indexing and retrieval system).

  • by certain death (947081) on Friday March 29, 2013 @09:04PM (#43315871)
    Just delete some goddamn email.. hoarder!
  • Simple. Archive mail by the year as it gets too big. Use mutt's search for the basic searching and maildir-utils for the heavy lifting.

    To those saying keeping email forever is hoarding: not if it's done right. You'd be surprised how useful it is to go back and find an email from four years ago.
    • Re: (Score:2, Interesting)

      by Anonymous Coward

      To those saying keeping email forever is hoarding: not if it's done right.

      That's like saying neck deep rooms of newspapers/magazines in a house isn't hording if you stack them neatly with little paths running through it.

    • by tqk (413719)

      Archive mail by the year as it gets too big. Use mutt's search for the basic searching and maildir-utils for the heavy lifting.

      Agreed, and add OfflineIMAP, grepmail, and ImapFilter. Best is grepmail output to a file creates a new mail file, and mutt can handle gzipped mail files. There's your backup.

  • by Vegan Cyclist (1650427) on Friday March 29, 2013 @09:08PM (#43315895) Homepage
    heh - i have all my email going back to '98 in Outlook Express. Best email program ever! It's nearly perfect for what i want. (Any way to get it to do inline spell checking, ie, underlines misspelled words as you type?) Still running it on an XP box. Been using Windows Live Essentials a bit for Win8, it's not horrific, but lacks some of the characteristics..hope MS injects some of the OE spirit into it..
  • by saccade.com (771661) on Friday March 29, 2013 @09:11PM (#43315913) Homepage Journal
    Eudora still runs on my Win7 box. I have email going back to at least the early '90s. All plaintext and easily searchable.
  • by Tihstae (86842)

    Upload it to one or several Google accounts and you have a permanent searchable archive.

    • No; Google's data mining algorithms will have a permanent searchable archive. You'll just have a temporary archive, until they decide to retire the service or ban you for some offense perceived by their automated "police".

  • I use Thunderbird.

    My mailboxes are all IMAP, so I found a use for the Local Inbox in Thunderbird that I always thought was a useless feature.

    At the end of the year, I create a subfolder labeled by Year, and I download all copies from the the year before the last (eg, my last download was of 2011 emails), then I purge them from the IMAP server to save space This way I still have universal access to my last years emails but easily searchable archives available at home.

    If you keep regular backups of your /home

  • by jrronimo (978486) on Friday March 29, 2013 @09:34PM (#43316007)
    I'd say follow the same rules as any archiving of media:

    Pick one format and migrate all of your messages to that: In this case, I'd say mbox. Thunderbird and most other mail programs read it and you can get most of your mail into mbox format via IMAP/Thunderbird from whatever mail client can read your old ones. You can store your mbox files locally in Thunderbird and gain Thunderbird's searching (for instance) without the need for an actual back-end. I was able to read some mail stored in Netscape Mail because it was just mbox files and opening them in Thunderbird was a breeze.

    Most importantly: Every 5-10 years, re-evaluate your storage choice. Is Thunderbird still around? Is mbox still pretty well regarded? If you find you need to migrate again, do it! If both are still active / supported, then hold onto 'em. The only way to perpetually maintain media access is to make sure your choices are still valid on a regular basis. This is true for any media: As the old formats go obsolete (cassette tape, VHS), you need to migrate that data to the next readily accessible format (CDs, DVDs; FLACs, MPEG(?)).

    I think the biggest problem is that you have a mish-mash of stored files right now. You'll save yourself a headache in the future by tearing the band-aid off now and taking the time to get all of your mail into one format. Then, in the future, when you need to convert, it'll be many steps easier since you won't have to visit Slashdot and find out what to do about your mail again next time. :)
  • Does there exist any program that's basically a lightweight Windows auto-starting (but 99.999% asleep and inert unless you're actively using it) background service that does nothing besides act like an abstraction layer between some kind of reasonable file-based mailstore roughly analogous to an Outlook .pst file (AFAIK, canonical Maildir is a physical impossibility under Windows) and any IMAP-compatible email client?

    I don't care about being able to access it from anywhere besides my local PC... binding to

    • The 500 Mhz Pentium and the Core i7 will have roughly the same performance in this use case because IO is the bottleneck. The speed is the speed of the disk and filesystem.

      To be more specific, a Pentium has a throughput of around 2 GB/s. Compare to 10 MB/s for a 7200 RPM drive doing random access on small files, 100 MB/s on large ones.

      So it's entirely reasonably to use a small low power Linux system like a Western Digital World Edition network drive or the ARM based stuff you mentioned for IO bound applic

  • by czth (454384) on Friday March 29, 2013 @09:45PM (#43316051) Homepage

    I run qmail [cr.yp.to] for sending/receiving mail (on Gentoo [gentoo.org]; netqmail package), using maildir, of course. On top of that, I run the Courier IMAP [courier-mta.org] server on my internal network (with TLS encryption). Until a few months ago I used Mutt [mutt.org] as a client (console-based), but I've moved to using Roundcube [roundcube.net] (web-based email), which I initially installed for my wife, and have been happy with it. I also have some automatic filtering to folders via Maildrop [courier-mta.org] (another Courier utility; it looks at a ~/.mailfilter file to route mail).

    Roundcube/the IMAP server's search is OK most of the time - I keep my inbox small and move older mail to sub-folders - when I want to do advanced searches or search large mailboxes I log in and grep through folders of interest; this works well with the maildir format with one file per message. Maildir was also quite resilient when I had a HD crash and needed to recover some lost mail (block scan for blocks that look like mail headers found most missing items, and I do better backups now - mail is under ~/.maildir and gets backed up automatically).

    I would move older messages to maildir (there are plenty of mbox converters, and almost anything non-proprietary should be convertible to mbox or maildir via existing programs or a short perl script) - even if at some point maildir dies off entirely, which seems unlikely, converting it to another format will always be trivial due to its simplicity and it has the advantages mentioned above of being able to search easily with grep etc.

  • I use PSTs and nightly backup.

    Sure, you can use GMail or the amorphous cloud for your purposes, but quite frankly, remember - if it's not in your possession, it's not as secure as it could be.

    No, I don't have world-ending secrets in my possession, but yes, I do get paranoid about my data.

  • MH stores each email as a plain text file, each folder as a directory. It uses the unix filesystem as its database. It's very quick and has tools to re-order a folder quickly.

    In addition, MH has tools to convert mail formats. It was designed in the days of low cpu power and small disks. It also lent itself well to being wrapped by other tools like xmh, exmh and mh-e so you don't have to learn the raw MH commands.

    Yes, IMAP is cool, but don't discount MH. Plus the O'Reilly MH book is free as a PDF.

    Oh, so

    • by RedLeg (22564)
      +1

      What he said, MH is the tool for this task. I have mail going back to early 90s, each message in a separate text file, sorted into directories by year. Once you're archiving in this format, you can then index the files for more rapid searches, or, if you're old school, just grep around when you're looking for something.

      Best thing is, once you have them organized this way, you're done, and can burn backups of the archive (by year) directories to CD or other long term storage, and not have to worry ab

  • I gathered up all my historical email records a few years ago, and used Aid4Mail to convert all the various mailbox formats to the common format I use today. Choose a format that's convenient for you, and standardize on it. Here's the product website: http://www.aid4mail.com/ [aid4mail.com]
  • I've got 16 years' worth of email in a multitude of formats, including all of those that are excoriated as unreliable, fatally flawed, or Satan's preferred meas of communication with our world.

    I have them on O-L-D CD-ROMS, DVDs, saved to two cloud services, on my personal server, and tar'd/zipped/RAR'd/Stuff'd in some of the same places, and up to three copies in each of these places. I wonder if the .sitf files will even decompress, but no point in deleting them.

    I've really only used a very few mail clien

  • http://www.mailstore.com/en/mailstore-home.aspx

    works well quick searches and its local .

    unfortunately its windows only but may work fine under wine.

  • I've got six years of archived email. I like Thunderbird's archiving scheme. You can have it automatically create archives as a calendar year goes by.
  • If you're not actually working with the old e-mails, and you don't mind waiting a few moments to search them, just keep the raw e-mails, in raw transmissible e-mail format, and be done.

    They are nothing more than a whack of text files at that point. And they are properly formatted with headers and everything.

    Want to seach? Full text search and you're done. Want to search by subject only" Simple regex search /^Subject\:.*?cucumber/ finds "cucumber" only on the subject line (yeah yeah, header folding exist

  • I've been using MailSteward on OSX. The starter version handles 15k or so entries using SQLite before it starts to bog, while the trade up is a front end to MySQL.

  • Parchment (Score:4, Funny)

    by SpaghettiPattern (609814) on Saturday March 30, 2013 @02:40AM (#43316915)
    Parchment -no less- does it for my ancient emails.
  • by gaspyy (514539) on Saturday March 30, 2013 @02:53AM (#43316943)

    I had to archive the emails since 1996. They were in multiple formats - Outlook Express dbx, Mailbox from Netscape Navigator and Thunderbird, Outlook.

    I converted all of them in .eml format. It's a simple, text format that can be read by the OS and easily parsed by any program and script. Much better than mbox or something else. Then I renamed all of them according to a rule - YYYYMMDDhhmm [From] [Subject]

    Now I can easily find any email. I can browse them using the file system, I can search them using the OS or via a script. Windows indexes them and extracts the metadata so any search is very quick.

A motion to adjourn is always in order.

Working...