Ask Slashdot: Best Way To Archive and Access Ancient Emails?

An anonymous reader writes "I started using email in the early 90s and have lost most of that first decade due to ignorance, botched backups, and so on. But since about 2000, I've got most — if not all — of my email in some form or other. I run Linux, so this has mainly been in a mix of various programs: Kmail, Evolution, Thunderbird. The past 2-3 years are still on the IMAP servers. My problem is that I only rarely NEED to look back to email of 5 years ago. But sometimes it's nice. Or I just want to reminisce about something...or find an old attachment that I was sent. But I do not want to be clogging my current email client of choice with vast backups and even more, I don't know if it will even easily convert. The file structures are different, some are mbox, others maildir, etc., and I would ideally like a way to 1) store and archive these emails, 2) access them, and 3) search by Sender, Subject, Date, Attachments. Is there anything I can do or do I just have to keep legacy applications on hand for this? Should I keep trying to upgrade and pull old files into the new applications? Any help or suggestions about what YOU do would be great."

IMAP

[sylvandb]

Just IMAP it all.

I went IMAP in 1997 and have never looked back.

I've also used IMAP as a temporary conversion measure for people switching e-mail clients so even if you aren't sure, it makes a good first step.

I don't understand the concern about too many e-mails. I can access my email back to 1992. With multiple folders it shouldn't be a problem and with modern indexing a search shouldn't be an issue.

Gmail

[lga]

Best method of storing and searching old email? Gmail. It can import from pop and imap so you can point it at your other inboxes and let it get on with it.You can upload from other mail clients to Google's imap server. Obviously it's amazing at searching through the archives.

Best method if you're concerned about Gmail's privacy? I'm still working on that one.

Re:IMAP

[DNS-and-BIND]

When I search for my brother's name I don't want to wait 30 seconds for a search to complete, nor do I want to see his emails from 10 years ago. I just want to see his last email that he sent about the trip we're taking next week. That's the concern about too many emails.

Thunderbird - for when you're done with GMail

[Nexus7]

I need to archive emails that I can search later - but with a twist. These are employees who've left the company. I can't keep 'em on at Google Apps 'cause I have to pay for that by user. So I use IMAP (making sure to set Chats to be shown in the IMAP list), create an account in Thunderbird, and slurp it all on to the local machine. It keeps all the folders, although I doesn't seem to be smart enough to figure out multiple labels, so it looks like it downloads the same email multiple times, once for it's folder, and once for "All Mail." Then I delete the account at Google. You just have to be sure to click through all the folders in Thunderbird and make sure it is done downloading before you blow the Google account away.

notmuch

[Anonymous Coward]

http://notmuchmail.org is Gmail for people that don't trust Google. Works great with your existing IMAP server using offlineimap.

The Email Mandala

[Irate Engineer]

Just sweep it all into the Trash Bin, breathe deep, and move on with your life confident in the impermanence of all things.

Namaste!

Re:Gmail

[zekele2]

Best method of storing and searching old email? Gmail. It can import from pop and imap so you can point it at your other inboxes and let it get on with it.You can upload from other mail clients to Google's imap server. Obviously it's amazing at searching through the archives.

Best method if you're concerned about Gmail's privacy? I'm still working on that one.

The solution is Google Apps for your own domain. $5 a month per user, 25Gb space, IMAP, no advertising (which is where most of the privacy issues arise), and most importantly, no lock-in as you can switch your email to a different provider at any time without changing email address. As you said, Gmail is by far the best for searching old email. I haven't run an email server for years.

Re:IMAP

[arth1]

Pretty much everything not made by Microsoft will support export to good old mbox. It's a good format to store in, because you can always import from it into other formats.
And you can run simple scripts against the mbox files.
More than once, I've done a grep against my mail archive, and more than once I've moved it to a new machine and new mail software.

Re:IMAP

[Antique Geekmeister]

_NO_. Under no circumstances use "mbox" for mail storage, or anything other than a temporary stage on the way to transferring it to something contemporary and uable such as Maildir. If you lose that one mbox file, by file system corruption or by fat finger accident or overflowing a partition or in tht eprocess of merging new email with it, you've lost _all_ your mail in that mbox. And as you read, mark, or save mail, that file is constantly churning, making backup and replication of the mail spool far more dangerous and fragile, especially when the mail directory is bulky with years or decades of active mail threads or simply undeleted email.

mbox was useful when the available inodes on a file system were limited programs benefited from using a single inode for transactions, and backups occurred on magtape, but there is simply no point to it in decasdes.

Clueless. Takes 10 minutes start to finish

[raymorris]

Force feed? WTF are you taking about? Dovecot can use any make mail format. Just set MAILDIR if it's in a non-standard directory. So the whole procedure is:

yum install dovecot
vim /etc/dovecot.conf (only if using a nonstandard mail location)
service dovecot restart
set username and password in GUI client

I never will understand why some people feel the need to post on topics they don't have the slightest clue about.

Re:IMAP

[arth1]

_NO_. Under no circumstances use "mbox" for mail storage, or anything other than a temporary stage on the way to transferring it to something contemporary and uable such as Maildir. If you lose that one mbox file, by file system corruption or by fat finger accident or overflowing a partition or in tht eprocess of merging new email with it, you've lost _all_ your mail in that mbox.

Thus speaks ignorance. If you write corrupt data to a mbox file, nothing prior to the corruption is affected at all. Unlike most formats that don't store each mail in a separate file, you can also very easily run recovery against a mbox file. Heck, a one-liner perl script can retrieve anything from before and after a corruption.

And "overflowing a partition"? Um, run that by us again. If you mean disk full, that doesn't truly affect a format that's made for appending. You won't be able to append. Any other format you can come up with will have the same problem.

And for archival purposes, this also does not apply. You don't make changes to your archive. Period.
And you back it up. Period.
Which is a heck of a lot easier to do with mbox than most other formats.

But again, the main strength is that it is so simple, which means that pretty much every mail program out there will support it, one way or another.
Choosing a more modern format leaves you with fewer options, and less certainty that it will be supported in the future. 20 years down the road, mbox will still be supported. It has an RFC - http://tools.ietf.org/html/rfc4155 [ietf.org]

Can you say the same about ANY other format? Maildir doesn't work on systems that doesn't allow colon in file names, and hashes the filename based on the hostname which both isn't portable, and crashes badly for many implementations if you have a non-ascii hostname. Not to mention that the format has balkanized, to the point that it's no longer compatible betweeen implementations.

Again, for archival purposes, simplicity is the key.