Ask Slashdot: Best Way To Archive and Access Ancient Emails? 282
An anonymous reader writes "I started using email in the early 90s and have lost most of that first decade due to ignorance, botched backups, and so on. But since about 2000, I've got most — if not all — of my email in some form or other. I run Linux, so this has mainly been in a mix of various programs: Kmail, Evolution, Thunderbird. The past 2-3 years are still on the IMAP servers. My problem is that I only rarely NEED to look back to email of 5 years ago. But sometimes it's nice. Or I just want to reminisce about something...or find an old attachment that I was sent. But I do not want to be clogging my current email client of choice with vast backups and even more, I don't know if it will even easily convert. The file structures are different, some are mbox, others maildir, etc., and I would ideally like a way to 1) store and archive these emails, 2) access them, and 3) search by Sender, Subject, Date, Attachments. Is there anything I can do or do I just have to keep legacy applications on hand for this? Should I keep trying to upgrade and pull old files into the new applications? Any help or suggestions about what YOU do would be great."
IMAP (Score:4, Informative)
Just IMAP it all.
I went IMAP in 1997 and have never looked back.
I've also used IMAP as a temporary conversion measure for people switching e-mail clients so even if you aren't sure, it makes a good first step.
I don't understand the concern about too many e-mails. I can access my email back to 1992. With multiple folders it shouldn't be a problem and with modern indexing a search shouldn't be an issue.
Re:IMAP (Score:4, Informative)
Re: (Score:2)
Re:IMAP (Score:4, Insightful)
Archive old emails by year:
Archives/2013
Archives/2012
Archives/2011
Archives/2010
Archives/2009
Only search in the appropiate ones. Easy, right?
Comment removed (Score:4, Interesting)
Re: (Score:3)
Then use Kmail and behold the power of akonadi and nepomuk. searching in the later versions is fast -- and I have a very large number of mails.
It turns out that for large enough collections you _do_ need a DB :)
Re:IMAP (Score:5, Insightful)
This.
I fired up imap servers for all my old mail.
I fired up a modern mail client (OSX Mail.app) and connected to all of 'em and also to gmail.
I dragged all my old email into gmail. In a GUI. And it worked.
Done.
I no longer run mailservers. Too much of a headache. gmail is awesome (with imap access, even). Indexing, instant searching, etc.
If you don't want/trust your email to the cloud, then this isn't for you. Unless you want to run your own imap server with whatever backend suits you - then you can dump it all there. I just can't be bothered to manage that after 15+ years of doing so.
Re: (Score:2)
It's a shame gmails has no real filters, like (sieve [wikipedia.org]).
Yes, the webinterface CAN create filters. But if you're using a desktop clients, it's not confortabe to have to open a DIFFERENT client to configure filters.
Gmail also lacks some imap features, notably, the sort command.
Re: (Score:3)
That's why I do both.
for every day I use gmail
but once a year I fire up a current email client and download everything archived. I also purge the archive every couple of years of truly useless emails.
I keep a a copy for safety and away I go.
Re: (Score:3)
Any web-based client *should* (unless they use a plugin or other weird config tool) save any filters, etc. to the web-based-profile (Like GMail does). Otherwise, if you choose Thunderbird (or any other sane client) you could just copy over the profile between installs. Even across OSes.
I've successfully used a singular Thunderbird profile on both a Windows and Linux boot off the same machine; granted Linux access to the NTFS partition Windows sat on and it used that profile directory. Been copying it forwar
Re: (Score:2, Funny)
Thank you for your efforts, we appreciate it.
Sincerely,
The FBI.
Re: (Score:2, Interesting)
IMAP offers "server side" storage with "client side" viewing.
Besides the privacy implications, are you seriously suggesting that the OP
a) find or create an IMAP server,
b) force feed that server all his archived emails (presuming that there is some way to bulk import email into the IMAP server), and
c) change his current email setup so that, from now on, his email is sent to the mail server on which the IMAP server runs?
How is that any easier to manage than his current predicament? ISTM that your suggestion f
Re: (Score:3)
Re: (Score:3, Informative)
Pretty much everything not made by Microsoft will support export to good old mbox. It's a good format to store in, because you can always import from it into other formats.
And you can run simple scripts against the mbox files.
More than once, I've done a grep against my mail archive, and more than once I've moved it to a new machine and new mail software.
Re:IMAP (Score:5, Informative)
_NO_. Under no circumstances use "mbox" for mail storage, or anything other than a temporary stage on the way to transferring it to something contemporary and uable such as Maildir. If you lose that one mbox file, by file system corruption or by fat finger accident or overflowing a partition or in tht eprocess of merging new email with it, you've lost _all_ your mail in that mbox. And as you read, mark, or save mail, that file is constantly churning, making backup and replication of the mail spool far more dangerous and fragile, especially when the mail directory is bulky with years or decades of active mail threads or simply undeleted email.
mbox was useful when the available inodes on a file system were limited programs benefited from using a single inode for transactions, and backups occurred on magtape, but there is simply no point to it in decasdes.
Re:IMAP (Score:4, Insightful)
I agree with that for new emails, but for an archive file, none of it really applies. File system corruption and fat fingers should be handled by just restoring from backup, and merging / marking as read / etc is not really applicable for old mail, which should be accessed by either viewing it readonly or making a disposable copy.
mbox might have its problems, but I don't think there's any good reason to spend time converting old files to Maildir.
Re:IMAP (Score:4, Interesting)
What's wrong with MBOX? I've been using it with gigabyte-sized folders for over a decade and nothing bad has ---@@@ From MAILER-DAEMON Fri Jul 8 12:08:34 2011
Seriously, though. Even when mbox gets trashed due to disk corruption, it is still every bit as useable as a trashed maildir: Messages get lost, not whole folders...and even then, that's what backups are for (right?).
Or at least, that was my experience over the past many moons. As a static archive (Sent-mail-2012), I can't think of a single thing wrong with mbox. And it's easier to cat to tape than maildir.
Re: (Score:3)
Or at least, that was my experience over the past many moons. As a static archive (Sent-mail-2012), I can't think of a single thing wrong with mbox. And it's easier to cat to tape than maildir.
You don't even have to create a new tape entry, just append to the old one.
Not to mention that mboxes can b cat'ed together, as they get older. Like combining all dailys to a weekly, weeklies to a monthly, and monthlies to a yearly.
What it doesn't have is indices, but nothing prevents you from importing and converting a mbox to an indexed format. Or creating an index for that matter.
When archiving, the main point is that you want a format that is mature, long-term-supported, and easy to use tools again.
Re:IMAP (Score:5, Informative)
_NO_. Under no circumstances use "mbox" for mail storage, or anything other than a temporary stage on the way to transferring it to something contemporary and uable such as Maildir. If you lose that one mbox file, by file system corruption or by fat finger accident or overflowing a partition or in tht eprocess of merging new email with it, you've lost _all_ your mail in that mbox.
Thus speaks ignorance. If you write corrupt data to a mbox file, nothing prior to the corruption is affected at all. Unlike most formats that don't store each mail in a separate file, you can also very easily run recovery against a mbox file. Heck, a one-liner perl script can retrieve anything from before and after a corruption.
And "overflowing a partition"? Um, run that by us again. If you mean disk full, that doesn't truly affect a format that's made for appending. You won't be able to append. Any other format you can come up with will have the same problem.
And for archival purposes, this also does not apply. You don't make changes to your archive. Period.
And you back it up. Period.
Which is a heck of a lot easier to do with mbox than most other formats.
But again, the main strength is that it is so simple, which means that pretty much every mail program out there will support it, one way or another.
Choosing a more modern format leaves you with fewer options, and less certainty that it will be supported in the future. 20 years down the road, mbox will still be supported. It has an RFC - http://tools.ietf.org/html/rfc4155 [ietf.org]
Can you say the same about ANY other format? Maildir doesn't work on systems that doesn't allow colon in file names, and hashes the filename based on the hostname which both isn't portable, and crashes badly for many implementations if you have a non-ascii hostname. Not to mention that the format has balkanized, to the point that it's no longer compatible betweeen implementations.
Again, for archival purposes, simplicity is the key.
Re: (Score:3)
Thank you, yes, I was speaking of file system corruption. And because any ongoing email management, such as deleting or even marking as read or unread the old messages, causes change in the content of the mbox file, from the message edited onward, the claim that "it's only risking messages after the point where you edit" is disingenuous. I'm afraid it's not "ignorance" speaking, it's lengthy and painful experience.
_If_ the mbox files re absolutely static, then mbox can be considered reasonably stable. But i
Re: (Score:2)
a) cyrrus. done.
b) there is. pine.
c) also pretty easy.
Personally I have 40K messages on an imaps store. A full text search does take a few minutes, but only because the hardware is utterly ancient -- less powerful than many modern cell phones. I should get around to upgrading that.
Clueless. Takes 10 minutes start to finish (Score:4, Informative)
Force feed? WTF are you taking about? Dovecot can use any make mail format. Just set MAILDIR if it's in a non-standard directory. So the whole procedure is:
yum install dovecot /etc/dovecot.conf (only if using a nonstandard mail location)
vim
service dovecot restart
set username and password in GUI client
I never will understand why some people feel the need to post on topics they don't have the slightest clue about.
Re:Clueless. Takes 10 minutes start to finish (Score:5, Funny)
I never will understand why some people feel the need to post on topics they don't have the slightest clue about.
Because it's a long standing Slashdot tradition!
Re:IMAP (Score:4, Interesting)
[......] are you seriously suggesting that the OP
a) find or create an IMAP server,
Ridiculously simple. They're already running Linux, they just have to install dovecot and they've got a fully functional IMAP server (no configuration required) - which has access to all their local mail boxes.
b) force feed that server all his archived emails (presuming that there is some way to bulk import email into the IMAP server)
Ridiculously simple. Fire up Thunderbird, configure it to access your local IMAP server, select all, drag and drop.
c) change his current email setup so that, from now on, his email is sent to the mail server on which the IMAP server runs?
Why would they need to do that? Thunderbird (or other mail reader of choice) can access multiple accounts.
Re:IMAP (Score:4, Interesting)
There aren't any privacy implications. If you there were any, then you would have named one or mentioned an example. The situation prior to typing "sudo apt-get install dovecot" is that he had the data (so it's already subpoena-able or whatever you're trying to imply) and after that he'll also have the data. Nothing changes. Are you complaining that he's keeping the data rather than deleting it? I don't get your point at all.
Yes, because it's easy. This can be done in literally ten minutes. Maybe a little more if he doesn't already have good storage allocated for it (e.g. a Reiser formatted ~/Maildir, or whatever your own religion commands).
Yes, because it's trivial. It's highly likely that whatever he is using to read each of his different mail archives, can also talk IMAP, because everything talks IMAP. You say "force feeding" as though literally selecting and dragging in a GUI, or picking "copy" or "import" off some menu, is hard. It's not.
As for step c (changing how he receives email), I don't think that's being suggested but it may be a good idea. He can decide later, whether or not he wants his archive server to become his main/active server. That decision can wait and is not part of the scenario being discussed; it's an opportunity for the future.
Because then he'll have his archive stored in a system that is specialized for handling the problem, accessible and searchable by any client he wishes to use, possibly even the very same tool he uses for his day-to-day non-archive mail reading. Or he can pick some other IMAP client if it handles mass/archive use case better than the routine use case. Everything Just Works, all together. All of his complexity and exceptions disappear. And at virtually no cost; there's no downside to counter any of the advantages.
This is one of the easiest no-brainer Ask Slashdots, ever. There is one right objective best simple easy-and-fast-and-good(!) answer, and setting up an IMAP server is it. Probably because email storage is an old, very-solved problem.
Re: (Score:3)
Run your own IMAP server. For the past decade or so, Dovecot [dovecot.org] has by far & away become the best choice. If you've set up any other daemons before it's really not very complicated software.
Use the IMAP (Score:2)
Use the IMAP server - if you have control and/or space available.
I just have a single large archive IMAP folder into which everything that isn't spam gets pushed. You could optionally create subfolders for time ranges (every 1-2 years, whatever works for you). Using dovecot with good indexing support on the backend quick searching has been great. If you do a sub-archive breakout on time the searches will be quicker, you could also then create a virtual mailbox combining them all for when search really needs
Maildir (Score:3)
I have all my personal email from 1998 in a Maildir directory with Dovecot as the server on a dual core Atom server running Centos. About 900 MB worth.
Plenty fast.
Re:Maildir (Score:4, Interesting)
The main problem that I personally have is where I get emails from people that reference images and such from other servers. Most of the time it's commercial messages that I delete, but sometimes there's a newsletter that I want to save, and the images themselves turn out to be relatively important. Kind of annoys me to have to print them to PDF so that the formating gets preserved.
Look to the past (Score:2)
Trying to figure out what formats will be available in the future is pretty hard, it's easier to see what formats have been around a long time and are still in use.
As such, two formats come up readily:
mbox http://en.wikipedia.org/wiki/Mbox [wikipedia.org] and maildir http://en.wikipedia.org/wiki/Maildir [wikipedia.org]
Re:Look to the past (Score:4, Interesting)
Convert it to one format (Score:2)
Just dump them (Score:5, Interesting)
Had the same need 20 years ago when migrating from VAX/VMS to Unix. The old emails were saved in a not quite readable format, but I figured I could recover them if necessary. In the end, never bothered. Yes, there are a few (actually, only two) that I'd like to resurrect now, but life moves on.
Re: (Score:2, Insightful)
This. Don't be a data hoarder. Go through them if you must and re-mail the best/important ones. Then dump it to dev/null and move on.
Do the same with movies, books, bookmarks, photos, apps, docs, etc. you'll be happier without all that baggage. Music of course is another story. Keep that forever and only toss out the dreck (those extra songs on that album you bought because singles didn't exist yet).
Re: (Score:2)
For mail and bookmarks that makes little sense. It would take me more time to make those decisions than it's really worth. Emails and bookmarks take up such a small amount of space that it's not really worthwhile to worry about.
Now, general filesystem files, that's a different matter, I used to have a system where I only backed up things I cared about, and let filesystem crashes wipe out the rest of the data. Seemed to work out just fine.
Re: (Score:2)
Yeah, that's about where I'm at, too. I think of those Buddhist monks making the sand paintings (which they then sweep away). It's an exercise in recognizing the impermanence of all things.
Re: (Score:2)
How large can a personal email archive really be? Even if you're storing the equivalent of a copy of War and Peace per day, for ten years, that's just a couple dozen gigabytes. Nowadays, it's hardly any trouble archiving that.
Transfer it all to imap (Score:2)
Translate it _all_ to IMAP services, in MAILDIR format if available. I've repeatedly been faced with clients, partners, and colleagues who use their email as their insitutional memory and need to migrate to a new service. There are few technologies as straightforward, and robust, as a simiple IMAP server running a light, uncluttered IMAP daemon such as "dovecot", without the complex and nunnecessary requirements of aCyrus IMAP daemon, and most _definitely_ without the complex support requirements of an Exch
Re: (Score:2)
If you want a single file, you can just use tar.
Directories or folders may seem primitive, but they're not necessarily going to stick around.
Abstracting the underlying storage (which differs anyway between systems), a directory is just a set of files. They'll exist for as long as files themselves exist, even if they have different names, and they'll certainly be here long after the SMTP headers (used by both Maildir and mbox) is long dead and buried.
GMail doesn't even use maildir.
They don't use mbox either.
Re: (Score:2)
um.. having one file per message is a lot more portable than some concatenated mess that needs parsing. directories and files aren't going anywhere. While they might be replaced by database structures....oh wait that's right, filesystems ARE databases.
Re: (Score:2)
A potentially excellent idea IF you can guarantee that one single file won't ever suffer from data corruption. Maildir or other multi-file formats will have a bit more overhead in terms of space and performance, but is far more resistant to data corruption. Good indexing eliminates most of the performance difference, and for my money I'll take data robustness over space any day, at least for personal files. Sure, regular backups are even better, but I know very few people that are actually good about doin
Use a database! (Score:5, Interesting)
You could export your emails to some parsable format, write an importer to extract the basics that you want to keep (from/to/subject/body,attachments/entire binary blob/etc) and then bulk insert that mess into on a mysql/sql server tucked away somewhere locally or "in the cloud" (EC2, Azure). Just another option as I'm sure you'll see here many here. At least with this route you are in full control of how you index, what you can search, encryption, performance, level of backups, etc. Maybe not the best way for some but I know if I had over 100000 emails that I wanted searchable very very quickly with advanced SQL like searching, this would be a cool way to do it (time permitting). Good luck! And to the pedantry to ensue...Yes. Good day.
Re:Use a database! (Score:5, Interesting)
And you could make a doilie, and a hat, and a casserole, and wallpaper with the headers, and knit the .signatures into a fancy flying cape.
Just use IMAP and Maildir. Modern systems are fast enough to allow you to search the content directly, and not vulnerable to the database support wackiness this sort of "I can pre-organize it now and make my life better by wasting it pre-programming my queries" approach.
Re: (Score:2)
Indeed, databases are the perfect solution, though not relational ones; rather email database formats: like Maildir!
Gmail (Score:5, Informative)
Best method of storing and searching old email? Gmail. It can import from pop and imap so you can point it at your other inboxes and let it get on with it.You can upload from other mail clients to Google's imap server. Obviously it's amazing at searching through the archives.
Best method if you're concerned about Gmail's privacy? I'm still working on that one.
Re:Gmail (Score:4, Informative)
Best method of storing and searching old email? Gmail. It can import from pop and imap so you can point it at your other inboxes and let it get on with it.You can upload from other mail clients to Google's imap server. Obviously it's amazing at searching through the archives.
Best method if you're concerned about Gmail's privacy? I'm still working on that one.
The solution is Google Apps for your own domain. $5 a month per user, 25Gb space, IMAP, no advertising (which is where most of the privacy issues arise), and most importantly, no lock-in as you can switch your email to a different provider at any time without changing email address. As you said, Gmail is by far the best for searching old email. I haven't run an email server for years.
Re: (Score:2)
Better hope the algorithmic overlords don't decide you're a spammer and lock you out. Remember not to keep your eggs in a single basket.
Re: (Score:2)
Obviously it's amazing at searching through the archives.
Regrettably, gmail's IMAP implementation does not support the sort command.
The obvious answer (Score:5, Funny)
Design a MySQL database for storing your mail messages, keying on sender, subject, date, and presence of attachments (bonus points for storing the attachments as blobs rather than as external files). Then write a perl script that'll automatically parse all your incoming email and convert it to database entries. I suppose if you're lazy the script could just monitor your mail spool, but it'd be better to just have it listen for incoming connections and handle the mail directly.
Next, make copies of that script, modifying as necessary to process all your old mail archives.
Oh, and you'll need to write another perl script to access all new mail - not from your mail spool, but from this database. You should probably name this system after some animal too. If you absolutely MUST have a graphical interface on it, don't use anything newer than TCL+Tk - but going with curses would be a better choice.
Oh - it has to be GPLv3, or we'll hate you and probably mailbomb your machine.
What - isn't that the Slashdot way?
Re:The obvious answer (Score:5, Funny)
Holy wheel reinvention, Batman.
Re: (Score:2)
You mock - but ...
Stop being a hoarder (Score:4, Insightful)
You don't need all those e-mails. Keep the few you actually care about (copy and paste the text into a regular file, and save any attachments you want), and get on with your life.
People that keep every e-mail are weird. Quit living in the past.
Re:Stop being a hoarder (Score:5, Interesting)
It's kind of like photos, you know? Or letters, and such. People like to store those things, because they serve as a memory aid for what the mind no longer holds. It is also quite useful for history reconstruction/when you are old and have nothing else to do but a box full of photos/letters/etc.
Not to say that you are wrong on your point, except on the weird part. Unless you are okay with double standards, or you also consider anybody who keeps photos of parties/graduations/etc weird... Just saying.
Re: (Score:3)
Re: (Score:3, Insightful)
Right, I don't need that client email from a few years ago to remind me about a detail on a project. It'd be better just to look like an idiot in front of them.
Just because *you* don't need that archive doesn't mean everyone else doesn't need it.
Why the hell *not* keep a conveniently categorized, organized, sorted, indexed and searchable database of all your important electronic communications? A few gigs is nothing these days.
Re:Stop being a hoarder (Score:5, Interesting)
But why would I waste time manually finding and copying individual emails, when I can just let the backup script archive them all for virtually no cost?
Completely missing the point... (Score:4, Insightful)
True story: while deployed in the Army, our communications guy could not find a piece of equipment which was very important and very pricey. He had been signing the monthly inventory forms saying he had it, assuming it was in a cabinet. He could not find any paperwork showing it was signed out - it had just disappeared sometime in the last 3 months and no one had seen it.
On a long shot, I started searching my email - since I keep every last one. Sure enough, about 2 months prior, there was a throwaway email from him to the effect that he was going to turn in item X for repair since it was acting flaky. He checked at the contractor mentioned in that email, and it was sitting on the shelf waiting for pickup.
Re: (Score:3)
I recently had to go back 5 years to retrieve an email as evidence in my pending divorce. "I never said that!" - like hell you didn't, I've got it right here (and here, and here, and here). She wanted to play hardball, so hardball it was. :(
I've had to go back 6+ months to retrieve important mail for myself for work and other personal matters as well. Every written correspondence with my ex-wife was in email - some damning, as above, but all of our "love letters" when we were courting. Wouldn't you want to
Re: (Score:3)
It's not about living in the past, it is about not worrying what to keep and what to store. And the message you think most important today is likely to be completely worthless in five years, while you would love to still have that other message you thought unimportant back then.
I keep everything so I can decide today which old message I consider important. It is very rarely that I venture into the archive, but I have needed messages two years old at times.
Thunderbird - for when you're done with GMail (Score:4, Informative)
I need to archive emails that I can search later - but with a twist. These are employees who've left the company. I can't keep 'em on at Google Apps 'cause I have to pay for that by user. So I use IMAP (making sure to set Chats to be shown in the IMAP list), create an account in Thunderbird, and slurp it all on to the local machine. It keeps all the folders, although I doesn't seem to be smart enough to figure out multiple labels, so it looks like it downloads the same email multiple times, once for it's folder, and once for "All Mail." Then I delete the account at Google. You just have to be sure to click through all the folders in Thunderbird and make sure it is done downloading before you blow the Google account away.
Mbox format and compress them (Score:2)
notmuch (Score:3, Informative)
http://notmuchmail.org is Gmail for people that don't trust Google. Works great with your existing IMAP server using offlineimap.
Gmail. (Score:4, Insightful)
As soon as gmail made IMAP available, everything went there. I used to get my stuff via POP and saved it all going back to the early 90s. When IMAP went live on gmail, I let it chug away for hours and hours until it was synced and all my archived stuff was stored on my gmail account. They've been bumping up the limit faster than my mail's built up so I'm now at 3.9 gigs used of 10.1 available, holding about twenty years of email. I have email clients on a desktop and couple laptops that I fire up every couple of months to sync with gmail and keep local stores in the event that google screws up and loses my data. (I like to think I'd be smart enough to disconnect from the internet before accessing the local clients if my gmail account ever went blank but I've got multiple copies just in case I forget.)
I know that won't work for email fiends who pile up a gig a month but it works for me. I don't even bother sorting my email any more. It's faster to just search. Not like the old days when it would take my email client half an hour to slog through all the messages. :)
Re: (Score:2)
Maildir (Score:3)
You know the old joke... (Score:2)
Call it "Lexi Diamond - Ronda Rousey mud wrestle" and share it on a torrent and soon the whole world will back it up for you....
Seriously though, even if you were a previous email hoarder, you will likely be able to comfortably archive all your emails *and* the tools needed to access them on a USB stick. Start by finding all the tools you need, source included, and place them on your storage medium. Compress it. Send it to the cloud.
Mail files can be stored by year (easy enough to do with awk or other mail
Gmail, Hotmail (Score:2)
Take your pick.
mbox + dovecot + mairix (Score:3)
I wouldn't posit this as the best way, but it's what I do. I keep my archival mail on a local filesytem arranged in directories, stored in the old-school mbox format. I run Dovecot under OS X for IMAP access to those messages from anywhere; when I need to search through the whole collection, I use mairix [force9.co.uk] (an indexing and retrieval system).
You are kidding, right? (Score:4)
maildir + mutt + maildir-utils (Score:2)
To those saying keeping email forever is hoarding: not if it's done right. You'd be surprised how useful it is to go back and find an email from four years ago.
Re: (Score:2, Interesting)
To those saying keeping email forever is hoarding: not if it's done right.
That's like saying neck deep rooms of newspapers/magazines in a house isn't hording if you stack them neatly with little paths running through it.
Re: (Score:2)
Archive mail by the year as it gets too big. Use mutt's search for the basic searching and maildir-utils for the heavy lifting.
Agreed, and add OfflineIMAP, grepmail, and ImapFilter. Best is grepmail output to a file creates a new mail file, and mutt can handle gzipped mail files. There's your backup.
Outlook Express! (Score:3)
Eudora (Score:3)
Google (Score:2)
Upload it to one or several Google accounts and you have a permanent searchable archive.
Re: (Score:2)
No; Google's data mining algorithms will have a permanent searchable archive. You'll just have a temporary archive, until they decide to retire the service or ban you for some offense perceived by their automated "police".
Thunderbird Local Inbox (Score:2)
I use Thunderbird.
My mailboxes are all IMAP, so I found a use for the Local Inbox in Thunderbird that I always thought was a useless feature.
At the end of the year, I create a subfolder labeled by Year, and I download all copies from the the year before the last (eg, my last download was of 2011 emails), then I purge them from the IMAP server to save space This way I still have universal access to my last years emails but easily searchable archives available at home.
If you keep regular backups of your /home
Same rules as any archiving: (Score:5, Insightful)
Pick one format and migrate all of your messages to that: In this case, I'd say mbox. Thunderbird and most other mail programs read it and you can get most of your mail into mbox format via IMAP/Thunderbird from whatever mail client can read your old ones. You can store your mbox files locally in Thunderbird and gain Thunderbird's searching (for instance) without the need for an actual back-end. I was able to read some mail stored in Netscape Mail because it was just mbox files and opening them in Thunderbird was a breeze.
Most importantly: Every 5-10 years, re-evaluate your storage choice. Is Thunderbird still around? Is mbox still pretty well regarded? If you find you need to migrate again, do it! If both are still active / supported, then hold onto 'em. The only way to perpetually maintain media access is to make sure your choices are still valid on a regular basis. This is true for any media: As the old formats go obsolete (cassette tape, VHS), you need to migrate that data to the next readily accessible format (CDs, DVDs; FLACs, MPEG(?)).
I think the biggest problem is that you have a mish-mash of stored files right now. You'll save yourself a headache in the future by tearing the band-aid off now and taking the time to get all of your mail into one format. Then, in the future, when you need to convert, it'll be many steps easier since you won't have to visit Slashdot and find out what to do about your mail again next time.
lightweight local IMAP server for Windows? (Score:2)
Does there exist any program that's basically a lightweight Windows auto-starting (but 99.999% asleep and inert unless you're actively using it) background service that does nothing besides act like an abstraction layer between some kind of reasonable file-based mailstore roughly analogous to an Outlook .pst file (AFAIK, canonical Maildir is a physical impossibility under Windows) and any IMAP-compatible email client?
I don't care about being able to access it from anywhere besides my local PC... binding to
It's IO bound, not CPU or RAM (Score:2)
The 500 Mhz Pentium and the Core i7 will have roughly the same performance in this use case because IO is the bottleneck. The speed is the speed of the disk and filesystem.
To be more specific, a Pentium has a throughput of around 2 GB/s. Compare to 10 MB/s for a 7200 RPM drive doing random access on small files, 100 MB/s on large ones.
So it's entirely reasonably to use a small low power Linux system like a Western Digital World Edition network drive or the ARM based stuff you mentioned for IO bound applic
maildir: qmail, courier-imapd, roundcube (Score:3)
I run qmail [cr.yp.to] for sending/receiving mail (on Gentoo [gentoo.org]; netqmail package), using maildir, of course. On top of that, I run the Courier IMAP [courier-mta.org] server on my internal network (with TLS encryption). Until a few months ago I used Mutt [mutt.org] as a client (console-based), but I've moved to using Roundcube [roundcube.net] (web-based email), which I initially installed for my wife, and have been happy with it. I also have some automatic filtering to folders via Maildrop [courier-mta.org] (another Courier utility; it looks at a ~/.mailfilter file to route mail).
Roundcube/the IMAP server's search is OK most of the time - I keep my inbox small and move older mail to sub-folders - when I want to do advanced searches or search large mailboxes I log in and grep through folders of interest; this works well with the maildir format with one file per message. Maildir was also quite resilient when I had a HD crash and needed to recover some lost mail (block scan for blocks that look like mail headers found most missing items, and I do better backups now - mail is under ~/.maildir and gets backed up automatically).
I would move older messages to maildir (there are plenty of mbox converters, and almost anything non-proprietary should be convertible to mbox or maildir via existing programs or a short perl script) - even if at some point maildir dies off entirely, which seems unlikely, converting it to another format will always be trivial due to its simplicity and it has the advantages mentioned above of being able to search easily with grep etc.
PSTs / nightly backup (Score:2)
I use PSTs and nightly backup.
Sure, you can use GMail or the amorphous cloud for your purposes, but quite frankly, remember - if it's not in your possession, it's not as secure as it could be.
No, I don't have world-ending secrets in my possession, but yes, I do get paranoid about my data.
Take a look at MH (Score:2)
MH stores each email as a plain text file, each folder as a directory. It uses the unix filesystem as its database. It's very quick and has tools to re-order a folder quickly.
In addition, MH has tools to convert mail formats. It was designed in the days of low cpu power and small disks. It also lent itself well to being wrapped by other tools like xmh, exmh and mh-e so you don't have to learn the raw MH commands.
Yes, IMAP is cool, but don't discount MH. Plus the O'Reilly MH book is free as a PDF.
Oh, so
Re: (Score:2)
What he said, MH is the tool for this task. I have mail going back to early 90s, each message in a separate text file, sorted into directories by year. Once you're archiving in this format, you can then index the files for more rapid searches, or, if you're old school, just grep around when you're looking for something.
Best thing is, once you have them organized this way, you're done, and can burn backups of the archive (by year) directories to CD or other long term storage, and not have to worry ab
Convert to a common format (Score:2)
Re: (Score:2)
I've wondered that (Score:2)
I've got 16 years' worth of email in a multitude of formats, including all of those that are excoriated as unreliable, fatally flawed, or Satan's preferred meas of communication with our world.
I have them on O-L-D CD-ROMS, DVDs, saved to two cloud services, on my personal server, and tar'd/zipped/RAR'd/Stuff'd in some of the same places, and up to three copies in each of these places. I wonder if the .sitf files will even decompress, but no point in deleting them.
I've really only used a very few mail clien
i like this for email archiving (Score:2)
http://www.mailstore.com/en/mailstore-home.aspx
works well quick searches and its local .
unfortunately its windows only but may work fine under wine.
In my case (Score:2)
Keep raw, search raw (Score:2)
If you're not actually working with the old e-mails, and you don't mind waiting a few moments to search them, just keep the raw e-mails, in raw transmissible e-mail format, and be done.
They are nothing more than a whack of text files at that point. And they are properly formatted with headers and everything.
Want to seach? Full text search and you're done. Want to search by subject only" Simple regex search /^Subject\:.*?cucumber/ finds "cucumber" only on the subject line (yeah yeah, header folding exist
MailSteward (Score:2)
I've been using MailSteward on OSX. The starter version handles 15k or so entries using SQLite before it starts to bog, while the trade up is a front end to MySQL.
Parchment (Score:4, Funny)
What I did for all about 50,000 emails (Score:3)
I had to archive the emails since 1996. They were in multiple formats - Outlook Express dbx, Mailbox from Netscape Navigator and Thunderbird, Outlook.
I converted all of them in .eml format. It's a simple, text format that can be read by the OS and easily parsed by any program and script. Much better than mbox or something else. Then I renamed all of them according to a rule - YYYYMMDDhhmm [From] [Subject]
Now I can easily find any email. I can browse them using the file system, I can search them using the OS or via a script. Windows indexes them and extracts the metadata so any search is very quick.
Re:I use gmail (Score:4, Insightful)
So can anyone with a subpoena. And you can bet Google would be running their advertising stuff on that.
There is no way I would put my life on a public server like that.
Re: (Score:3)
It is email. AKA over the web. AKA public.
And someone with a subpoena can get your records off of your ISP, or just come into your house and take it off your computers.
Re: (Score:2)
A subpoena won't get you into a house. That requires a search warrant which requires probably cause of a crime.
Completely different.
Re: (Score:2)
If you get a pissy enough opposing counsel and a judge to cooperate, warrants can be issued. Trust me, being on the wrong side of it. Of course, this will vary by your jurisdiction, IANAL, and especially not yours.
Re: (Score:2)
Except the gmail server and the ISP server might be located in
different places.
Re: maildir (Score:2)
Re: (Score:3)
Just sweep it all into the Trash Bin, breathe deep, and move on with your life confident in the impermanence of all things.
Plus that Trash Bin program has _great_ compression!