Infrastructure for One Million Email Accounts? 1216
cfsmp3 asks: "I have been asked to define the infrastructure for the email system for a huge company, which fed up of Exchange, wants to replace their entire system with something non-Microsoft. I have done this before, but not for anything of this scale. Suppose you are given a chance to build from scratch an email system that has to support around one million accounts. Some corporate, some personal, some free. POP, IMAP, webmail, etc are requirements. The system must scale perfectly, 99.9% uptime is expected... where would you start?"
qmail (Score:2, Insightful)
Highly paid consultants or....Ask Slashdot (Score:2, Insightful)
Re:Easy. (Score:2, Insightful)
If theyre using exchange (Score:3, Insightful)
If you're looking for a groupware replacement, then you've got a big job ahead of you. Scalix is a mess, bynari is a hack, etc. When you do get them running things end users end up buying like PDAs and apps that hook into outlook are going to cause more problems.
If its just pop/imap you really can't go wrong. A good webmail option is kinda a catch. Squirrelmail is nice, but compared to OWA its really out of its league.
If your post told us what they were fed up with and how they used their system you'd get some real advice. Expect the usual postfix vs qmail vs sendmail vs whoever mini-flamewars.
Unless It's A Very Old Exchange System... (Score:3, Insightful)
I'm sure someone, somewhere within the enterprise is using features of Exchange that they won't get anywhere else. Not to sound like a Microsoft fan-boy sock puppet, but there's some features that Exchange has that people in a business environment just love.
However, since you asked. I'd run Exim or Qmail and Cyrus IMAP.
Re:go to gmail (Score:5, Insightful)
Gmail does not have guaranteed uptime.
You do not pin your companies communications system on something you cannot sign a SLA agreement with.
need I go on?
Re:Split up the tasks (Score:4, Insightful)
Start with universities (Score:2, Insightful)
Then once you think you have a solution, budget plenty of time for extensive testing against simulated load. Make sure you simulate failures by, e.g., pulling plugs randomly. Buy the hardware and software *after* you're 100% sure it works, not before. And where possible, roll your solution out gradually, so that small problems don't turn into MCFs.
Re:Let the vendors do the work. (Score:5, Insightful)
DO NOT implement a half-assed solution. Unless you really know what you're doing (and if you were, you wouldn't be asking this question), don't assume that a million Linux servers strewn about a million offices and data centers is the best solution, even if it is easiest to set up and administer. Maybe it is, come up with a proposal with hard numbers and see how they compare to the vendors. A million dollars spent on a Sun E10000, and Oracle Grid subscription (scales perfectly, right?), or a million IBM engineers flown into your site when an emergency happens may be worth paying for.
Re:Obviously (Score:5, Insightful)
I work with Exchange, and think that the chances are better that they just had shitty architecture to begin with. Exchange is a great platform and scales well, so if the original people wouldn't do it, well then f*ck em.
Stilll convinced to migrate? Well, something with multiple datacenters, large scale, compressed SAN backend, and alot of clustering will do it. Shit, you could do the entire thing with MySQL if you REALLY wanted to. Moving the existing data over will be a huge pain no matter what you migrate to though.
My suggestion? Don't just jump off Exchange, do a proper requirements analysis and you might find it is alot cheaper to just redesign the existing architecture.
Re:Here's my plan and it's the best one you'll get (Score:5, Insightful)
This is the best advice he'll get? Sheesh.
Think this through -- a lot of e-mail programs check every 20 minutes. Assuming I actually hit any without duplications, I could potentially need 400 minutes or over six hours to get all my mail. Since it's random, it could take days.
And that's just for starters with this lame scheme. If I want to check mail, say, from the field on a dial-up once a day... hopefully you can see how badly this would suck.
What the guy should do is buy an e-mail system that can handle 1,000,000 users and not screw around trying to chewing gum his own solution.
Only 99.9% uptime? (Score:2, Insightful)
Re:Ask Slashdot? (Score:2, Insightful)
Re:Split up the tasks (Score:1, Insightful)
- Terrorists don't attack muslims.
- All terrorists are muslims.
- Those killing terrorists are not muslims.
- Only religious conversion and killing ends war.
It's hard to take advice on sendmail from someone who displays such an obvious lack of deep thought on other issues.Re:POP? (Score:2, Insightful)
As of six months ago, we have Exchange/Outlook, and no POP3/IMAP access to the server at all. You're stuck with Outlook or webmail based on how it's configured. After much reconfiguration, I finally got Outlook to behave mostly the way I want - including delivering mail locally rather than leaving it on some server a thousand miles away (literally, not joking here). Now if I didn't hate everything about Outlook...
All I want, and all I've ever wanted, is to be able to grab my messages easily and put them on my machine, not stored on a server somewhere. POP3 is great for that. It does absolutely everything I want and need for mail, and it's dead simple. Even if you don't make it the standard implementation, it'd be nice if admins everywhere left those of us who know what we're doing the option of using it.
Easy (Score:5, Insightful)
Re:NO Domino (Score:3, Insightful)
There are dozens of perfectly good mail servers out there. The more features they have the more likely you are to have problems. It's a pretty simple equation.
And if all else fails, you can write your own. I've written one, it's not very difficult (hacked it out in C# in a weekend). It's a very simple plain text protocol. But I wouldn't run the company on something I wrote in C# in a weekend. I don't even use it myself anymore. I'm running Exchange now for my personal mail server as that's what we run at work.
Re:Obviously (Score:2, Insightful)
Outbound queues (Score:3, Insightful)
them) so backed-up outbound queues don't interfere with normal outbound processing.
The FallbackMX hosts can use a file system optimized for directories with lots of files in them (and can of course themselves be tuned as the parent poster suggested.)
See what others are using (Score:2, Insightful)
That's a matter of doing an mx lookup, telneting to one of their gateways on port 25, and seeing if you can infer from their banners what mail system that they are running (for the inbound smtp gateways, anyway-- since there's nothing to prevent them from layering different products). Look to mailing list archives for messages sent from the various domains, and see what the headers tell you about their outbound mail path.
Example: Inbound Comcast HSI:
$ dig comcast.net mx
comcast.net. 250 IN MX 5 gateway-r.comcast.net.
comcast.net. 250 IN MX 5 gateway-r.comcast.net.
$ nc -vv smtp.comcast.net 25
Connection to smtp.comcast.net 25 port [tcp/smtp] succeeded!
220 comcast.net - Maillennium ESMTP/MULTIBOX sccrmhc14 #274
So, they use something claiming to be 'Maillennium'.
If you do this for AOL, you'll see some weird-looking, probably custom AOL gateway. Earthlink says something like:
'ESMTP EarthLink SMTP Server', AT&T WorldNet is also Maillennium, Verizon.net declares 'MailPass SMTP server v1.2.0', and so on.
If you really wish to probe to see if this is opensource-ish stuff with obfuscated banners, you can try fingerprinting them using smtpscan http://www.greyhats.org/outils/smtpscan/> to find out that it's really just Postfix or Sendmail hiding behind that custom 220 banner. Actually, it's the smtpscan fingerprint file is an interesting read all by itself...
Re:Still Have to Engineer it (Score:3, Insightful)
I currently run qmail in a small production environment, handling about 20k messages a day. It's small, but enough to point out the cracks.
qmail does many things well, but it also is a product of DJB-bizarroworld. The worst of the offenses, in my book, is that due to his security model, the smtp receiver will accept messages to any recipient, not just valid ones. Then, if it can't figure out what to do with it, it generates a bounce message - which usually bounces. This can kill a machine and a network connection during a dictionary spammer attack. Implementing SMTP-AUTH with qmail is a royal, gigantic, immense, overwhelming pain in the ass. It took me several hours to get it all patched together and working.
Want any of the above to work? Patch. Want a blacklist of users that shouldn't get mail? Patch. Want SPF support? Patch. Want the non-POSIX use of errno to be fixed? Patch. Usually, the patches don't go together smoothly, so you wind up spending hours figuring out the rejected chunks and how to properly patch them together. And this is a modern MTA?
While I've patched qmail to deal with a host of issues, there's no reason a modern MTA should need to be patched for most these. The rcpt authentication thing is just downright dumb, and smtp-auth is reasonably widely supported with the ESMTP standard.
I'm testing exim right now, and I'm pretty happy with it. It's fairly light, does everything I want and need, and isn't the configuration quagmire of sendmail. As soon as I rebuild the mail server, I'm switching the production environment away from qmail.
If you're a hard-core qmail adherent, that's great. It's fast and reasonably easy to configure in its basic form. However, I prefer something that's more standards-compliant and feature-rich right out of the tarball.
My advice to anybody considering qmail for the first time is to try it, but consider other popular MTAs like exim and postfix as well, including the 800lb. gorilla, sendmail. It's a pain, but get the O'Reilly book and you can do positively anything (and I do mean anything) you want with it.
Re:POP? (Score:4, Insightful)
Wal-mart has an estimated 1.6 million employees. (source [wikipedia.org])
General Motors, by contrast, has approximately 360,000 employees.
The post says "around one million accounts" which is very different from one million employees. I have over ten email accounts that I actively use for receiving mail and four to six for sending.
An ISP could easily have millions of accounts. But since he said "huge" company, they were using Exchange, and because he's asking Slashdot my guess is that he's not at an ISP. Instead, I'd guess he's at a medium-sized company that might offer email accounts to its customers or at a large company that also contains many subsidiaries (but wants one email domain for all of those).
Re:Still Have to Engineer it (Score:5, Insightful)
Re:Qmail!! (Score:5, Insightful)
Insert "imagine a beowolf of those" joke here, except it isn't a joke.
I think you might be underestimating the requirements for this large a project that "must scale perfectly". The "99.9% uptime is expected" requirement alone requires multiple internet connections, a large cluster of front end servers, and redundent database servers, preferably located in different states. (ie: "What do you mean our only server is in New Orleans?")
I don't think the average Dell dual Xeon box is up to the task for this large a project...
Re:Obviously (Score:2, Insightful)
Re:Simplicity is key. (Score:5, Insightful)
-Cyrus IMAP, while a monster to build and configure, can handle a pretty heavy load, and the latest versions can handle a lot of load-balancing internally.
-Exim's nice. I'm a Postfix man, myself. Sendmail is king, though. I'm not going to claim to like it, but it's up to the task, and there's something to be said with using a standard tool.
-While things like MD4 are okay for hashing, they're kind of CPU-intensive. Consider something like "second and third letter of username" that takes less CPU time. The right answer here depends a lot on the relative speed of CPU versus disk. If you can get dedicated hardware to do this (rare, but it exists), use whatever hashing the hardware supports.
-Consider some sort of cache (maybe even separate machines) between incoming SMTP and SpamAssassin/ClamAV. When the 2am spam run hits, your incoming SMTP machines can become overloaded. The downside: deciding what to do with mail that's not rejected the moment it's received.
-Set up a "mail machine" configuration with whatever OS and tools you use, and make it possible to create a disk image quickly. You're going to need a lot of hardware, which means that you'll have enough random failures to make building machines by hand impractical. This also means "have at least one extra built machine/disk array/etc. powered-on and waiting at all times" for those 4am hardware failures.
-You may find that things like NFS just aren't fast enough. Be ready to look at SAN or shared "direct-looking" storage. The tough part: this is hard to discover during testing. It may be overkill, but don't lock it out as a possibility.
-I/O is king. CPU speed won't matter as much as bus speed, disk speed, and memory speed. This is why a lot of companies use banks of big proprietary unix machines for their mail, even if they use commodity PCs elsewhere.
-I don't trust hardware load balancers. Sometimes they're necessary (and they do make life better when they work), but they're a big single point of failure. Consider other ways to split the load, or at least ways to work around the load balancer if it should fail. The Cyrus aggregator can handle some of this.
Re:Easy (Score:2, Insightful)
Unless the person wanted to start an Exchange flame war with his post, he clearly has no idea how to design an enterprise email infrastucture.
All the technology in the world can't help you if you don't understand what you are doing and based on his broad sweeping question, it would be easy to assume that he doesn't.
If he is the amateur email administrator that he has made himself out to be, no amount or advice or technology can help him.
If he can't design the email infrastructure he definitely won't be able to properly implement and manage it either.
Better leave this kind of work to the professionals.
Re:Obviously (Score:3, Insightful)
I am under the impression that if Hotmail were running clusters of Exchange servers Microsoft would be quite vocal in the enterprise scalability of Exchange.
Re:Obviously (Score:5, Insightful)
The reason Exchange uses a database can be summed up in three words: Single Instance Store.
Say you send one 1MB Word document to 100 of your colleagues. In a relational database-based, Single Instance Store-driven mail server, that document takes up exactly 1MB on the server. If somebody in the organization forwards the Word doc to the remaining 900 people in your organization, how much space does it take on the server? 1MB.
Send a 1MB document to 1000 users on a flat, mbox-style mail server, and how much space is taken up on the server? 1000MB.
I see your point about some things, sure. Being able to jump in and restore a mailbox from tape by just dumping a folder somewhere is nice, but it just doesn't scale in terms of storage the way a db-driven mail system does.
Don't flame me as an MS advocate. There are times when an SIS-based email system is good, and there are times when a flat email system is good. I've run Exchange environments for 500+ people, and I've run Linux-based mail systems for 1000+ people. I'm just saying that your particular argument is one-sided and flawed.
Re:Obviously (Score:5, Insightful)
Re:Obviously (Score:4, Insightful)
Say you send one 1MB Word document to 100 of your colleagues. In a relational database-based, Single Instance Store-driven mail server, that document takes up exactly 1MB on the server. If somebody in the organization forwards the Word doc to the remaining 900 people in your organization, how much space does it take on the server? 1MB. Send a 1MB document to 1000 users on a flat, mbox-style mail server, and how much space is taken up on the server? 1000MB.
Speaking of which, is there any filesystem around that "automagically" detects redundancy and avoids storing the same data twice (i.e. two files with the same content end up being stored only once)? (I don't mean hardlinks. Suppose I download some file for the second time without knowing the first instance exists). I suspect this would add a lot of overhead to the filesystem driver, but it'd certainly be a cool feature.
Re:Simplicity is key. (Score:4, Insightful)
80,000 is trivial. I was running a 12 node system with 87,000 users 12 years ago on hardware that was slower than a play station.
The complexity of going from 100,000 to 1,000,000 isn't just 10 times harder, you start to get into that area where sigma 4 system works with few problems with 100k but dies horribly with 1000k users. There is a line where instead of one machine being broken is unusual, you get this situation where at least one machine is always broken and it will often be broken in a way that is hard to diagnose.
You are wrong in every way. (Score:5, Insightful)
Second, people with rediculously frequent mail check times are not any more of a problem. Modern operating systems use file system caches. You do not have to touch the disk subsystem in any way, frequently accessed data will be in RAM.
And finally, a database has alot of extra overhead, and there is alot of deletes going on. Sure, such a select statement would work, but reading the files in one directory is an order of magnitude faster. And the deletes will really hammer your database. FFS+softupdates makes file deletion extremely fast. A relational database is not the answer for everything, stop trying to pretend it is. Use the right tool for the job, and for storing files, a filesystem is the right tool. Its not relational data, it doesn't need to be queried in arbitrary, complex ways, so it doesn't belong in a relational database.
Re:POP? (Score:4, Insightful)
Having never been near a computer, I have no idea. If I had to guess, I'd suppose that with a million users, 100,000 of them will have to be constantly reminded to delete their mail off the servers. 25,000 of them won't EVER delete their mail no matter what you do, and 5,000 will bitch and whine when you cap their fucking mailboxes. One of them will be the CEO, and he'll berate you in front of his smarmy suspender-wearing jerkoff golf buddies because you're a dumb hick that can't fit a terabyte of mp3s and porn (most of it redundant for chrissakes) into only 500 gigs of disk. You will also get to deal with countless issues involving different email clients. You would give almost anything to have a massive natural disaster wipe everything out so you didn't have to go to work tomorrow, but there's the wife and kids, so y'know, there it is.
Re:Obviously (Score:2, Insightful)
The biggest infrastructure problem plauging EDS right now is constructing a building large enough to hold all the money they are bilking out of us.
Re:Obviously (Score:1, Insightful)
Re:13 Million mail infrastructure (Score:2, Insightful)
Re:Qmail!! (Score:2, Insightful)
I'd take into consideration the fact that UNIX-based solutions are far more lightweight than ones of MicroSoft so you have no idea of what you're talking about unless you managed one yourself. Before debating on how long 0.01% downtime is, I'd rather you consider other numbers which are of much more importance to you now.
Quick setup (Score:5, Insightful)
my recommendations:
- Calculate with about 20-30 man days for the initial design. You'll need some software
development for about 30-50 man days, 100 man
days for setup, testing and fine tuning.
Figures may wary upon skill and LWF. Time
for integration into your backup service is
not included.
- Use a directory service with replication mechanism (preferred LDAP, we've done it with MySQL too). Every system except the load balancers will
get a replica.
- The user data is stored on machines with Cyrus . Depending on machine size, user profile, mbox size etc. you take between 5.000 and 50.000 users per system.
- The directory service knows which user is on which system. Prepare a script to move users from
one server to another (including the mbox).
- Incoming IMAP connects go through a loadbalancer to frontend systems with the perdition proxy. Those will relay thre requests
according to the directory to the responsible
IMAP server.
- Incoming HTTP requests will go through the
loadbalance to an Apache with Squirrel on the
frontend systems. Those will convert the requests
into IMAP requests and connect to the local
perdition.
- Generate a web frontend for the user to setup
auto reply, vacation and anti-spam settings.
- From those settings you can create SIEVE scripts for the user.
- Incoming and outgoing SMTP traffic is handled by systems with sendmail. Local delivery is handled by LMTP connects directly to the IMAP servers (cyrus can handle LMTP).
- Antivirus and Antispam is handled through the milter interface and appropiate plugins. Plan for individual settings per user (can be generated
from the data in the directory server).
- Loadbalancing SMTP us trivial.
- Add monitoring (e.g. Nagios), Backup and Restore (last one most important, nobody wants backup, all everyone wants is restore).
- If desired, use a cluster file system for those
IMAP servers to have even more redundancy.
- Make sure you have access to the internal DNS of your company. If you can setup "mail.acmecompany.com" to point to several ips (depending on location) this may ease your job
lot. If you cannot, this may be hard (and expensive) for your load balancers.
- You can scale everything horizontal in this concept. Choking point may be the load balancers.
- You can distribute the system easily onto several locations. Distribution over several continents is only recommended if you can either manage the DNS or the mail agent settings per continent.
Please forgive me, if i'm not completely correct. I'm only the sales repWith backup support you should be able to setup such a system in 6 to 12 months (the later more realistic for big companies).
Most probably users will complain about the lacking calendar.
Most troublesome will be the migration phase (hope you realized i didn't mention it above). This depends so much on your current scenario that it is very difficult to give a general advice.
> where would you start?
Contacting me ;-). Perhaps get a budget first. As i said, i'm sales....
Regards, Martin
Stop right now (Score:4, Insightful)
So, what you do right now is you go find someone who does know how to do it. And by that I mean someone who can demonstrate they know how. Which does not equate to having a low slashdot id; it equates to having done real projects of this scale.
So, how do you start? You ring IBM and get them to come in and talk to you. You ring Red Hat. You ring Accenture.
If you want impartial advice from someone who isn't a vendor (which is a good idea), then you go find some companies that has a million seat open source e-mail deployment in place and you see if you can get their messaging admin to talk to you.
Over here (Score:2, Insightful)
What a shame. (Score:3, Insightful)
Sad.
Re:Obviously (Score:3, Insightful)
Yup, that's a big problem with Microsoft Exchange's proprietary datastore.
Like the roach motel, data goes in, but you can't get it out.
Re:You are wrong in every way. (Score:3, Insightful)
I'm a lowly web programmer, not nearly as brilliant in the programming field as these other geniuses here, but I find it interesting that almost all web programming books tell you that if you can move processing into the database query instead of running it in the machine code, that it'll be faster.
This is so rarely the case. Unless you have a very powerful database server, odds are good that quite a lot of the various aggregate functions you might want to run will go much, much faster if you simple do a simple select in the database and then loop through the processing in the web app code. Not sure why this is true but it is.
A month or two ago I heard a great quote on Cartalk that I think should be plastered to every programmer, scientist, and engineer's bulletin board:
"Reality often astonishes theory."
In all honesty, though, I think that a database *would* be up to the task, even for 1M+ users. Consider Amazon, which probably gets several thousand simultaneous hits each second. And each page they pull up involves much more complex data searches than a simple mailbox.
I'd say the key concerns here aren't surrounding efficiency of processing. Mail servers, no matter how configured, are relatively low on the scale of computational complexity. It's more a size issue than anything else. The main problem will be determining how to store the data in a way that is safe, secure, fast, and reliable. Because the data needs to be redundant and widely dispersed (as in the New Orleans example someone pointed out above), it may be that a database, while not the fastest tool, may be the best tool for the problem.
I'll admit; I know nothing about how one would go about making identical file systems available simultaneously on many distant servers. But I'm guessing once you start doing that, you're starting to increase the complexity for the system in any case.
Well (Score:4, Insightful)
Re:Some info on how it works (Score:3, Insightful)
Not if you really want a copy.
For most normal users, disk space isn't a big problem. If it is, duplicate files aren't usually the cause of the problem.
When I make a copy of a file, I don't want the O/S to just add a link to the same file.
I want a frigging copy.
If there's a bad sector or something goes wrong the chances are higher that I can recover the data if I have a _real_ copy.
I use a file system for storing data. If disk storage was such a big problem, Google etc wouldn't be giving out GBs to users for _free_.
I/O is a bigger problem. Disks store a lot more nowadays, but are not that much faster.
Re:Obviously (Score:4, Insightful)
Re:You are wrong in every way. (Score:2, Insightful)
So you took the nearest LART and Enlightened him.
Seriously, though - is there ever a reason to stick 1,000,000 objects into one container without any regard whatsoever to the type of objects or container? (Ignorance doesn't count.)
Re:Obviously (Score:3, Insightful)
I'm pretty sure that both Veritas and Legato can scale to a million exchange mailboxes, but as it happens Wallmart used Tivoli (which should scale that large as well, given its mainframe background). It's strange that they didn't have Exchange backups with a high-end backup product in place corporately - but I know next to nothing about Tivoli. Was Wallmart just being cheap?
Re:Obviously (Score:4, Insightful)
No they cannot. Microsoft does not want you backing up mailboxes. You backup mailstores, which are several (hundred - however many will fit on a single disk partition) mailboxes. This works great for disaster recovery, you restore the failed disk.
It is worthless for a single user who just deleted some important message. You end up building a new exchange server, and then restoring the entire mailstore, than going into that box and grabbing the one message. Veritas (I presume Legato as well) has an option to go in an grab each message from the mailbox one at a time. However this is slow - 1/5th the speed of a normal backup.
I work for, a company that competes with Veritas and Legato (though we try for much smaller accounts, big enterprizes need things we don't provide). We do Exchange backup, and are pretty sure that Veritas is doing it exactly like us. I strongly doubt anyone can scale mailbox level backup to millions of users.
Re:~ 320K accounts (Score:3, Insightful)
1 million total users at 99.9% uptime as per the original request? Not exactly "child's play", but honestly, not much harder
Domino on iSeries does seem to be a reasonable option for a deployment of this size, especially given the rather generous uptime allocation that is being offered..."3 nines" being EXTREMELY generous for an iSeries shop (you'd even be able to schedule monthly downtme on purpose and still meet this uptime goal.)
I do note that IBM has benchmarked Domino on a 16-way Power5-based iseries at a 33ms response time for 175,000 concurrent users (details here: http://www-03.ibm.com/servers/eserver/iseries/domi no/scalerecord.html [ibm.com])...and given the limited usage pattern of POP3 (yuck!), a properly-deployed solution should be able to meet the published needs with just one server. AND provide backup. AND enable the user to restore an individual mail store, mail box, or object on-demand. If high-availability or higher performance is necessary, 2 servers could be deployed in several different configurations (mirrors, clusters, HA failover, etc.).
And if the moans of "Outlook-only users" get to be too much of a problem, IBM offers a "connector" that can offer MAPI access to Domino's mail store.
Hell yes, I'm an iSeries fanboy. Those machines have proven themselves to be reliable, capable, economical systems over the long haul. Now, while (due to price) I wouldn't suggest deploying an iSeries to be a simple file, print, web, or small-database server, true...but when you need to move freight and *lots* of it, but you don't want to spend hours every week in operating and administering the system, it's hard to beat the venerable System/38 ne AS/400 ne iSeries systems.