Forgot your password?
typodupeerror
Communications Software

Infrastructure for One Million Email Accounts? 1216

Posted by Cliff
from the better-have-good-spam-filters dept.
cfsmp3 asks: "I have been asked to define the infrastructure for the email system for a huge company, which fed up of Exchange, wants to replace their entire system with something non-Microsoft. I have done this before, but not for anything of this scale. Suppose you are given a chance to build from scratch an email system that has to support around one million accounts. Some corporate, some personal, some free. POP, IMAP, webmail, etc are requirements. The system must scale perfectly, 99.9% uptime is expected... where would you start?"
This discussion has been archived. No new comments can be posted.

Infrastructure for One Million Email Accounts?

Comments Filter:
  • Obviously (Score:5, Funny)

    by SpiffyMarc (590301) on Thursday September 08, 2005 @07:43PM (#13514109)
    I'd start by submitting a question to Ask Slashdot.
    • by CDMA_Demo (841347) on Thursday September 08, 2005 @07:46PM (#13514140) Homepage
      I'd start by submitting a question to Ask Slashdot.

      Upon which the global "wankfest" will commence, leading to solutions ranging from Novell to qmail based solutions, upon which the OP will look for someone else for advice, upon which the OP will end up paying an IBM consultant [huhcorp.com] to set up his company's email.
      • by AKAImBatman (238306) * <akaimbatman@gmai[ ]om ['l.c' in gap]> on Thursday September 08, 2005 @08:19PM (#13514440) Homepage Journal
        upon which the OP will end up paying an IBM consultant to set up his company's email.

        At which point the highly paid consultant will post a question to Ask Slashdot...
      • Re:Obviously (Score:4, Insightful)

        by Tuna_Shooter (591794) on Friday September 09, 2005 @10:40AM (#13518624) Homepage Journal
        One BIG issue between what people are running now and what they will HAVE to run soon is the little item of SOX compliancy. Be VERY careful that your little million user mail system is compliant or the implementation costs will double. Believe me i do this for a living and just saw one of our financial clients get stung big time.
    • by WarPresident (754535) on Thursday September 08, 2005 @07:47PM (#13514159) Homepage Journal
      I'd start by submitting a question to Ask Slashdot.

      Ah, a proof by contradiction, eh?
    • Re:Obviously (Score:5, Insightful)

      by whackco (599646) on Thursday September 08, 2005 @08:12PM (#13514390) Journal
      Actually, I was going to use "Obviously" as my subject line... so I'll just respond to yours.

      I work with Exchange, and think that the chances are better that they just had shitty architecture to begin with. Exchange is a great platform and scales well, so if the original people wouldn't do it, well then f*ck em.

      Stilll convinced to migrate? Well, something with multiple datacenters, large scale, compressed SAN backend, and alot of clustering will do it. Shit, you could do the entire thing with MySQL if you REALLY wanted to. Moving the existing data over will be a huge pain no matter what you migrate to though.

      My suggestion? Don't just jump off Exchange, do a proper requirements analysis and you might find it is alot cheaper to just redesign the existing architecture.
      • Re:Obviously (Score:5, Informative)

        by Karl Cocknozzle (514413) <kcocknozzle@hotmail . c om> on Thursday September 08, 2005 @08:50PM (#13514645) Homepage
        I work with Exchange, and think that the chances are better that they just had shitty architecture to begin with. Exchange is a great platform and scales well, so if the original people wouldn't do it, well then f*ck em.

        Your point about putting more effort up-front into design is well taken, but thhat advice applies to any platform...

        WIth that said, and without turning this thread into an Exchange bitchfest...

        Why in the hell can't you restore a mailbox from backup using only the tools you already have if the user is no longer present in Active Directory? You can't even export the mailbox with EXMERGE... Your choices are 1) 3rd party recovery tool (like Quest Recovery for Exchange) or 2) Build an ENTIRE OTHER SERVER and do a normal, full restore of the entire mail store so you can extract one measly mailbox.

        OBviously, the "Recovery Storage Group" feature is a VAST improvement over the old Exchange 5.5 way of bringing back just one mailbox (that being setup another server) but this is a MAJOR duh situation on Microsoft's part. They seem to think that since their "best practice" is to never ever erase any user account ever ever ever, that its okay to leave this gaping flaw in their enterprise groupware product. Sorry, but I think that sucks. We paid out the ass for "Enterprise" edition (to avoid the arbitrary 16gb limit on the mail store) and goddammit, I should be able to bring back a mailbox without its corresponding AD account without wasting a whole day setting up another server... I've only had to do it once (today) but the whole time I Was thinking how much esaier a mailbox restore on my OS X Server at home would be... Just restore the frickin' files and move on with your life.
      • Re:Obviously (Score:5, Interesting)

        by jrockway (229604) * <jon-nospam@jrock.us> on Thursday September 08, 2005 @09:25PM (#13514841) Homepage Journal
        > you could do the entire thing with MySQL if you REALLY wanted to

        I am so tired of people shoving everything into relational databases. What queries are you going to run against your database, anyway? SELECT * FROM messages WHERE read=0? Try "ls new" in your maildir. The reason things never scale right is because people design things to be "new" and "cool" like putting their e-mail into a relational database. No. Just use the filesystem. It, and its supporting tools, have been around for 30 years! It Just Works! It doesn't use any userspace memory! There are no permissions issues, because the kernel controls the permissions. It's the optimal solution.

        The filesystem is really really efficient (for e-mail) and really really reliable.

        Please, don't use a database!
        • Re:Obviously (Score:5, Insightful)

          by jerkychew (80913) on Thursday September 08, 2005 @10:22PM (#13515158) Homepage
          Since you've taken things off topic, I'll grab the wheel and pull it right off a cliff.

          The reason Exchange uses a database can be summed up in three words: Single Instance Store.

          Say you send one 1MB Word document to 100 of your colleagues. In a relational database-based, Single Instance Store-driven mail server, that document takes up exactly 1MB on the server. If somebody in the organization forwards the Word doc to the remaining 900 people in your organization, how much space does it take on the server? 1MB.

          Send a 1MB document to 1000 users on a flat, mbox-style mail server, and how much space is taken up on the server? 1000MB.

          I see your point about some things, sure. Being able to jump in and restore a mailbox from tape by just dumping a folder somewhere is nice, but it just doesn't scale in terms of storage the way a db-driven mail system does.

          Don't flame me as an MS advocate. There are times when an SIS-based email system is good, and there are times when a flat email system is good. I've run Exchange environments for 500+ people, and I've run Linux-based mail systems for 1000+ people. I'm just saying that your particular argument is one-sided and flawed.
          • Re:Obviously (Score:5, Insightful)

            by AnyoneEB (574727) on Thursday September 08, 2005 @10:34PM (#13515225)
            Or you could just use a filesystem that supports hard-linking files (see: man ln [google.com]), so you do not have to worry about that even when using a filesystem for this purpose. Since such a file is read-only, it could just be linked to all of those people's mail boxes. If you do not know what a hard link is, it is basically the same thing you are describing, except done in the filesystem and handled transparently by the kernel. Basically, every "file" you see in an Ext 2/3 filesystem is really just a pointer to where the file is stored, and any actual file can have as many as these links as you want. When there are no remaining links to a file, it is allowed to be deleted.
          • Re:Obviously (Score:4, Insightful)

            by doshell (757915) on Thursday September 08, 2005 @10:34PM (#13515226)

            Say you send one 1MB Word document to 100 of your colleagues. In a relational database-based, Single Instance Store-driven mail server, that document takes up exactly 1MB on the server. If somebody in the organization forwards the Word doc to the remaining 900 people in your organization, how much space does it take on the server? 1MB. Send a 1MB document to 1000 users on a flat, mbox-style mail server, and how much space is taken up on the server? 1000MB.

            Speaking of which, is there any filesystem around that "automagically" detects redundancy and avoids storing the same data twice (i.e. two files with the same content end up being stored only once)? (I don't mean hardlinks. Suppose I download some file for the second time without knowing the first instance exists). I suspect this would add a lot of overhead to the filesystem driver, but it'd certainly be a cool feature.

    • by 88NoSoup4U88 (721233) on Thursday September 08, 2005 @08:33PM (#13514543) Homepage
      The obvious answer is of course : Send all those thousand employees an Gmail invite !
    • by einhverfr (238914) <chris.travers@gmai[ ]om ['l.c' in gap]> on Thursday September 08, 2005 @08:35PM (#13514550) Homepage Journal
      First, you need to start by drafting real requirements. What do you need exactly? Antispam? Antivirus? Try to have it fill up at least a page.

      Once you have that done, you can start looking at solutions. You will have two parts to your solution:
      1) The DMZ email relays (possibly including other antispam/antivirus functions) You really need high availability here.

      2) Your email storage and retrieval systems. These may be a little more tolerant to downtime on an individual basis. But if you need to have redundancy here, there are ways to do it.

      I think Hotmail did fine with BSD and Qmail.* I am sure Postfix is equally capable.

      * Although Qmail itself has never had a security vulnerability discovered, you should be careful. TCPRules (on which qmail relies) has a vulnerability that can lead to root access for local users. This is not a problem on systems with no local users, however. I am not aware of any patch for the TCPRules vulnerability.
  • Easy. (Score:5, Funny)

    by Chess_the_cat (653159) on Thursday September 08, 2005 @07:43PM (#13514113) Homepage
    gmail.google.com
  • by technoextreme (885694) on Thursday September 08, 2005 @07:44PM (#13514127)
    bashing my head up against a desk.
  • Um... (Score:4, Informative)

    by Stevyn (691306) on Thursday September 08, 2005 @07:44PM (#13514128)
    I'd start by contacting people who know how to do it and can actually help you. A few responses on slashdot aren't going to help you along the entire process. Maybe even bring in a consultant.
  • For starters... (Score:3, Interesting)

    by cached (801963) on Thursday September 08, 2005 @07:45PM (#13514136)
    For starters, uptime should usually be higher than 99.9% for this large a site. 99.9% uptime means 40-45 minutes of downtime a month. Try going for 99.99% at least, though this usually increases the cost by about 250% according to what I have seen a few years back.
  • POP? (Score:5, Funny)

    by lseltzer (311306) on Thursday September 08, 2005 @07:46PM (#13514149)
    A million users and they want POP3? Add a gun and a single bullet to your administration requirements.
    • Re:POP? (Score:5, Funny)

      by JoshWurzel (320371) on Thursday September 08, 2005 @07:49PM (#13514185) Homepage
      I'd ask for six bullets. Why would you want to risk getting the empty chamber?
    • Re:POP? (Score:5, Funny)

      by tktk (540564) on Thursday September 08, 2005 @08:00PM (#13514295)
      I'd ask for enough bullets to handle the department thats making you to do this.
    • Re:POP? (Score:5, Interesting)

      by mre5565 (305546) on Thursday September 08, 2005 @08:10PM (#13514371)
      A million users and they want POP3? Add a gun and a single bullet to your administration requirements.
      No doubt a well deseved +5 for humor, but for those of us less in the know (and a chance at another +5 for informative), what is so bad about POP3? Thx.
      • Re:POP? (Score:4, Insightful)

        by Anonymous Coward on Thursday September 08, 2005 @10:52PM (#13515320)
        what is so bad about POP3

        Having never been near a computer, I have no idea. If I had to guess, I'd suppose that with a million users, 100,000 of them will have to be constantly reminded to delete their mail off the servers. 25,000 of them won't EVER delete their mail no matter what you do, and 5,000 will bitch and whine when you cap their fucking mailboxes. One of them will be the CEO, and he'll berate you in front of his smarmy suspender-wearing jerkoff golf buddies because you're a dumb hick that can't fit a terabyte of mp3s and porn (most of it redundant for chrissakes) into only 500 gigs of disk. You will also get to deal with countless issues involving different email clients. You would give almost anything to have a massive natural disaster wipe everything out so you didn't have to go to work tomorrow, but there's the wife and kids, so y'know, there it is.
      • Backups (Score:5, Informative)

        by Craig Ringer (302899) on Friday September 09, 2005 @02:03AM (#13516467) Homepage Journal
        Backups.

        With POP3, the client downloads mail and deletes it off the server. Without a significantly butchered POP3 server there's no way to hold copies of that mail for a period of time (say, to ensure it goes on to your archival tapes, or to make sure you can recover files the user deleted accidentally). It's one less thing to worry about if their workstation / laptop dies, too - just give 'em another one. If more mail clients supported LDAP address books and WebDAV calendars this would be even nicer; as it is I still have to keep their mail folders in their network home dir so I can back up their address book.

        You can back up POP3 boxes if you're on a corporate network, by forcing the client to keep its spools on the user's homedir. That tends to be slow and inefficient, though, and it doesn't let you do things like transparently split out attachments and store only one copy of an identical attachment for everybody.

        It's also easy to lose mail with POP3 if your client does something silly. Most clients seem pretty decent now, but I remember old Eudora versions used to DELE mail off the server then crash, corrupting their mailboxes. Woohoo.

        IMAP gives admins much more control over user mail. You can back up their mail folders, including their outbox and filed mail. You can enforce mail lifetime limits if your information retention policy requires it. You can store single copies of duplicate messages and attachments. You can give users access to shared mailboxes, and to each other's mailboxes where necessary. You can manage their mail folders remotely ("I can't delete $message, help!"). You can set up filters that deliver mail into sub-mailboxes automatically. Good clients automatically sync the IMAP mailbox so it can be used when the client is offline, like POP3. You can have your anti-spam software learn from their mail client's Junk folder. It's just much saner for business environments, in much the same way that network home directories and thin clients are much saner than a bunch of desktops with local storage are.

        IMAP also permits you to give the user a single view of their mailboxes from their desktop and when they're on the road, or accessing their mail from home. Don't even talk about "leave mail on server" for POP3 - users WILL misconfigure it and suck all their mail down onto one of their machines, then come to you looking for help cleaning up the resulting awful mess.

        Now, for an ISP, things are the opposite. You want to get the users' mail through your system and get rid of it. Most ISPs only offer POP3 and have small mailbox caps, so the user can't set their client to never delete mail off the server. They don't want to be responsible for user mail, they want it off their hands ASAP. An ISP can just tell a user who deleted a message then wants it back "well, that was silly then wasn't it?". An ISP doesn't want to back up 5 years worth of mail for 500,000 users.

        My point is that for corporate environments IMAP is so superior that it's almost nuts to offer anything else, but for an ISP POP3 is a much more viable option. So what's so bad about POP3 depends entirely on what your needs are.
  • ~ 320K accounts (Score:5, Informative)

    by Anonymous Coward on Thursday September 08, 2005 @07:47PM (#13514152)
    At IBM we use Lotus Notes which has saved us LOTS of virus hassles. Every employee has an account and we're something like 320,000 worldwide. The mail "databases" are spread among Domino servers but I don't know what platform these run on, or what hardware specs they have. I imagine it's either Windows or Linux... but who knows, maybe we're using some of our PowerPC-based iSeries servers. These are the boxen formerly known as AS/400.
    • by DaveCar (189300) on Thursday September 08, 2005 @08:17PM (#13514428)
      The mail "databases" are spread among Domino servers

      Yeah, but we all know what happens when one of these Domino servers falls over ...
    • by Greyfox (87712) on Thursday September 08, 2005 @08:33PM (#13514538) Homepage Journal
      I've been subjected to bloated goats every time I've contracted out to IBM and I've hated the experience every time. There are a number of projects going on inside the company to try to avoid having to use it, but no one's ever had a whole ot of success at it. IT steadfastly refuses to enable imap on the servers, ostensibly because the mail servers would not be able to handle the load of EVERY SINGLE IBM employee on the planet saying "OH THANK GOD!" at once and migrating to a mail client that doesn't SUCK DONKEY BALLS.

      Don't get me wrong. Notes isn't just a crappy E-mail client. It's also a crappy database access client that provides user-definiable forms which can be used to populate rows in the database. When you start getting a LOT of rows, the performance really goes to shit unless you replicate the database down to your local hard drive.

      Rather than the Notes based solution, I would suggest an old 386 running BSD and Sendmail. That'd save you a lot of pain in the long run, versus dealing with Notes.

    • by Anonymous Coward
      Of course, everyone should note that recommendation is coming from an IBM employee.

      Sorry, but Lotus Notes sucks; it's an abomination in almost every way. It's bloated, slow, buggy and has what is arguably the worst user interface ever (The User Interface Hall Of Shame said they could have based their entire site on this one app!) Sure, it does group meeting notes and can let you check other people's calendars but it falls flat as an email system. If it can't do the basics, who cares about the "advanced" fea
  • It's obvious (Score:4, Informative)

    by gulfan (524955) on Thursday September 08, 2005 @07:47PM (#13514155)
    Your first bet would be Ask Slashdot.

    However, I'd personally ask Google [google.com]. They've done it and even their search engine has information. I found an interesting link from there detailing the deployment of a large hundred thousand user mail system, from the architecture to the software located on Linux Journal [linuxjournal.com].

  • Who to talk to (Score:3, Informative)

    by Effugas (2378) * on Thursday September 08, 2005 @07:47PM (#13514156) Homepage
    I've heard surprisingly good things about Communigate Pro, though I have no idea if it scales that high.

    Mirapoint is probably _the_ vendor to speak to, though.
  • by Serveert (102805) on Thursday September 08, 2005 @07:47PM (#13514160)
    I'm sure other commercial vendors have it but I do know that large companies like ATT et al use it to handle their email. It's a shrinkwrap product that does it all and then some but it's very pricy.

    I'm sure you could hack together something to do this much like what google did. Might take some time but it's totally doable.
  • Vendors (Score:5, Interesting)

    by XorNand (517466) * on Thursday September 08, 2005 @07:48PM (#13514170)
    I'd start with talking to vendors. Consult with some sendmail gurus, Notes guys, etc. Any of these people/companies would salvate at the thought of being a part of a project this large. First, talk to the client and hammer out the real needs with solid performance requirements, timeframes, growth expectations, (meaning real numbers) etc. Put together a well thought-out Request For Proposal and send them out to as many applicable vendors that interest you. Then just stand back and play the role of ringmaster. The vendors will give you all the ideas you need.

    Just do one thing, please: make sure that the client is honest-to-goodness serious about this. I absolutely hate getting pie-in-the-sky RFPs from people who are just kicking the tires. It's a good way to burn bridges by not looking professional.
    • Re:Vendors (Score:4, Funny)

      by This is outrageous! (745631) on Thursday September 08, 2005 @10:41PM (#13515259)
      hammer out the real needs with solid performance requirements, timeframes, growth expectations, (meaning real numbers)
      Integers, kid. INTEGERS.

      Those newfangled "real numbers" are nothing but bullet-point creeping featuritis. Integers, on the other hand, have been around since at least Kernighan & Richie. They do one thing and do it well. Keep true to the Unix philosophy! Real numbers in information technology? Just say NO.

  • Split up the tasks (Score:4, Informative)

    by jgardn (539054) <jgardn@alumni.washington.edu> on Thursday September 08, 2005 @07:49PM (#13514183) Homepage Journal
    There are three parts to your system: sending mail, receiving mail, and storing mail. Keep them separate.

    Your receivers will be a bank of servers running sendmail. They will do appropriate spam processing to reduce the amount of mail actually received. They feed the data into the storage servers.

    The storage system has the data partitioned out so that all the data for one user would go to one server while all the data for another will go to a different one. The storage system also has to provide POP and IMAP access. You may want a special setup where the IMAP or POP service known which server to go to. Investigate having one giant virtual filesystem so that the system isn't too complicated.

    Your webmail access will use IMAP to access the actual mail. It can be a completly different system.

    The sending system will be a chokepoint for all outgoing mail. You are going to scan it as it goes out to look for virus-sent emails or unauthorized messages. For instance, you may want marketing email to be processed differently than inter-office email and such.

    All of these systems will be running sendmail. I know sendmail has a bad rap for being insecure, but the insecurities have been found and since fixed. It is by far the most manageable system when it comes to large-scale deployments with heavy customization.
    • by michelcultivo (524114) on Thursday September 08, 2005 @07:59PM (#13514291) Homepage Journal
      And please don't forget to use Maildir for email storage, it's very good for backup and very easy to manage.
    • You wrote: Your receivers will be a bank of servers running sendmail. They will do appropriate spam processing to reduce the amount of mail actually received. That's 2 tasks. This requires absolutely robust, absolutely lightest weight email servers, with serious caching. Sendmail can do it: Postfix can do it, and is vastly easier to manage. The syntax of configuring sendmail configurations is just too arcane for most of us to deal with. Definitely add blacklist filtering and SPF on the front end, to r
    • by fwc (168330) on Thursday September 08, 2005 @09:06PM (#13514733)
      This is right on the mark. I would differ in a few implementation details (aka I hate sendmail with a passion), but this is the way we do it at a medium-size ISP with a mail server "cluster" running in the thousands of mailboxes category.

      In short, we have mail servers accepting the mail and dropping it on a shared NFS server which stores all the mail. The incoming servers run spam and virus filtering and is responsible solely for delivering the mail to the customer's mail directory which lives on the NFS server.

      On the client side, we run IMAP and POP3 servers which access the stored mail on the NFS server to deliver it to the clients.

      The exact software used for both of these functions are somewhat irrelevant. Once you split this up this way, you can also split the selection process. I.E. which is the best server for accepting SMTP mail and dumping it in customer's mail directories. Which can be answered with a completely different answer than the question of "what is the best NFS (or SANS) server to use to store the mail", or "what IMAP server should we be using", or "what webmail front end should we be using", or so on.

      It also makes changing your mind down the road on any piece easier since you can actually run and test any one of these components in the live system as a final test before moving a replacement into the system.

      FWIW, I would *love* to consult on something this scale.

    • by Doktor Memory (237313) on Thursday September 08, 2005 @11:13PM (#13515487) Journal
      All of these systems will be running sendmail.

      You're high. Building a massive production email system on Sendmail 9 is slow-motion suicide. If the security holes don't get you, the terrible configuration methods and complete lack of scaleability will, nevermind the fact that Sendmail Inc is trying desperately to replace the product.

      "Most managable with [...] heavy customization?" I'd laugh if I wasn't crying. And I'm crying because I used to work for a company that deployed a massively customized sendmail infrastructure -- and I was one of the poor bastards who had to maintain it. Trust me, you don't want to do this. Ever.

      Yes, milter is cool. No, it's not cool enough to justify burning CPU cycles on sendmail in 2005.

      Even Sendmail Inc tacitly admits that Sendmail's design is garbage: take a look at the design document [sendmail.org] for Sendmail X, and note carefully how much it resembles Postfix and Qmail. There are very good reasons for this.
  • by gad_zuki! (70830) on Thursday September 08, 2005 @07:50PM (#13514193)
    they're probably using the groupware too. Are they also willing to ditch outlook?

    If you're looking for a groupware replacement, then you've got a big job ahead of you. Scalix is a mess, bynari is a hack, etc. When you do get them running things end users end up buying like PDAs and apps that hook into outlook are going to cause more problems.

    If its just pop/imap you really can't go wrong. A good webmail option is kinda a catch. Squirrelmail is nice, but compared to OWA its really out of its league.

    If your post told us what they were fed up with and how they used their system you'd get some real advice. Expect the usual postfix vs qmail vs sendmail vs whoever mini-flamewars.
  • by zentec (204030) * <.moc.liamg. .ta. .cetnez.> on Thursday September 08, 2005 @07:50PM (#13514196)
    ...they need to think about this very carefully.

    I'm sure someone, somewhere within the enterprise is using features of Exchange that they won't get anywhere else. Not to sound like a Microsoft fan-boy sock puppet, but there's some features that Exchange has that people in a business environment just love.

    However, since you asked. I'd run Exim or Qmail and Cyrus IMAP.
  • by DavidDPD (885638) on Thursday September 08, 2005 @07:54PM (#13514234) Journal
    I'm not sure that there is any commerical solution that can support 1 million emails well. Hence why Yahoo and Google have built there own custom systems. Some engineering may need to be required.

    For pop3 & imap4rev1, look at:
    http://www.dbmail.org/index.php?page=overview [dbmail.org]

    Still need an MTA, I think qmail is the fastest, best, but I'd used exim, as its easier.

    Database - not sure if MySQL and PostgreSQL will scale with dbmail.

    I'd say use FreeBSD, because of the ports collection (Don't linux Flame me). However, something like Solaris 10 x86 (or Solaris+Sun Hardware) might provide a bit better scaling, and HA hardware, SAN support, support in general, etc. Though, a bit tougher on the OSS software installs (In My Experience)

  • by slashname3 (739398) on Thursday September 08, 2005 @07:59PM (#13514282)
    I have several gmail accounts I can give you. Once you have serveral of these you can assign gmail accounts to the rest of your users. :)
  • by ejoe_mac (560743) on Thursday September 08, 2005 @08:00PM (#13514294)
    1) It'll run on anything - Win32, Linux, BSD, Solaris, x86, XServers, Alphas, Power5
    2) It'll scale as big as you can dream - over 5 million accounts with clustering
    3) MAPI support
  • My number one suggestion is hire someone who has built scalable mail systems, and written tons of code to support them: Matt Simerson

    You can learn about him, and his mail projects at http://www.tnpi.biz/internet/mail/toaster.shtml [tnpi.biz]

    -Chris Knight
  • by shub (88921) * on Thursday September 08, 2005 @08:03PM (#13514320) Homepage
    Try Googling for "Scalable E-mail Systems" and "Scalable IMAP services". Of course, I'm biased since most of the top hits are from the slides from the presentations that I've done at LISA 2000 [usenix.org], LISA 2002 [usenix.org], etc....

    My slides relevant to this discussion can be found at http://www.shub-internet.org/brad/papers/dihses/ [shub-internet.org] and http://www.shub-internet.org/brad/papers/sistpni/ [shub-internet.org].

    And yes, Nick Christenson [jetcafe.org] has been a long-time friend and co-author of mine.

    Feel free to contact me directly if you want some referrals.

  • by naoursla (99850) on Thursday September 08, 2005 @08:06PM (#13514343) Homepage Journal
    I bet Google would be willing to sell you a solution.
  • by MattW (97290) <matt@ender.com> on Thursday September 08, 2005 @08:10PM (#13514372) Homepage
    (1) Plan an server setup which can handle the load. The requirements may change, but one million users is a fair bit. How much average incoming and outgoing emails is that? Figure that out, using a network sniffer or sniffers on existing traffic if need be (although logs should work). Then use this to calculate a number of servers needed for an outgoing smtp farm, an incoming MX farm. Figure out how much storage space is to be provided per user, and then figure out how you want that storage space to be accessible. Probably your best bet is to have a round-robin DNS farm of imap/pop servers which proxy connections based on the users login to a backend farm of actual mailservers responsible for storage. Plan the ability to move users from server to server to rebalance as needed. Outgoing smtp is a lot easier since you're not really storing things long term. Plan a web farm for webmail. (And pick software) Don't forget to plan some sort of backup, and make sure your system is flexible as far as email retention; chances are the email retention policy will change at some point and your setup should be able to change with it.

    (2) Test. For each server, hammer it. Test it's load under as close to real world circumstances as you can. Then create unreal punishing loads and see how it handles it. Plan in advance for how your server farm handles something like virus-generated mass emails causing 1000% spikes in load.

    (3) Using your testing results, spec out the actual hardware. RAID, cheap hardware, redundancy, etc. If you have control over the network choice, plan a location with multiple fiber trunks coming into the building and provider redundancy. Remember backhoes in concert? Don't get hit by that. Plan for server failures, drive failures, network failures, power failures, and security compromises.

    (4) Deploy! If you did the rest right, this is the easy part. You'll have redundant network connections, HSRP, redundant switches, a proxy farm, an imap/pop farm the proxies connect to, an smtp farm for outgoing emails, and a web server farm for serving up webmail (depending on how you choose to architect the disk space, the web farm and the pop/imap farm may be one and the same; depends on how you set things up.)

    Here's a starter link to a setup which is smaller but, in principle, fairly similar:

    http://www.itd.umich.edu/umce/features/2004/cyrus. html [umich.edu]

    Finally, if you don't want to screw it up, ask someone who has done it before. Paying someone $300/hr for a 10-30 hour review of your plan is dirt cheap compared to horking the setup. Someone who has worked in huge email environments (a la, hotmail) could show you gotchas before they bite you. (If you need help figuring out who to ask, I could even point you to some of the appropriate people)
  • by Dark Coder (66759) on Thursday September 08, 2005 @08:12PM (#13514383)
    Gee whiz... I'm surprised that the groupware is getting tossed out. If as small as 20% of the user is accustom to Outlook Calendaring, they'll represent 95% of the complaints in a new system. An advance warning to all existing account should be mailed out (both paper and email) so that nothing falls through the cracks.

    Now to the mega-infrastructure that I set up for an undisclosed company for under 50K (and also didn't want groupware).

    1. Transport Sender (sendmail). That's right! Good ol' plain sendmail scales. It does require some pretty savvy tweaking so get Sendmail.Com consultant onboard just for this. Use SleepyCat DB for speed for all sendmail setups. For one million, I had about 23,000 transaction per minutes during the day. You'll require 10 servers for this for cushion (against some idiots sending an ISO attachment).

    2. Payload receiver (sendmail). A second group of machine to handle the reception of SMTP payloads.

    3. IMAP4S/POP3S - Hey what's with the "S"? Nothing like sending your user's password in the clear. Unless you enforce VLAN in your corporate environment and limit all IMAP4/POP3 to VLAN, the "S" is a mandatory security feature, inside and outside. Guess what "S" stands for?

    4. Webmail - SquirrelMail - Yet another dedicated server (in which I had to add two more load-balanced server to handling the growing pain). Use https for login only.

    5. AntiVirus (ClamAV) - It was the best back then, now its just running in the middle of the pack. sendmail has milter that allows extensibility such as MIMEDeFang, wilter, rureal (reverse-DNS check), spamassasin, and SPF.

    6. Support - Half the effort is put into those webpages that would 'hand-hold' these newbies into reconfiguring their machine. Worth the effort if you have over 20 expert PC users that can do their boxens. Otherwise do it yourself at each PCs. These pages should cover Thunderbird, Evolution, as well as Outlook and Outlook Express.

    7. Learn to spin 11 plates, one on each pole. Keep them spinning... If they start to drop and break, bring in some more Unix dudes.

  • Simplicity is key. (Score:5, Informative)

    by chrome (3506) <chrome.stupendous@net> on Thursday September 08, 2005 @08:28PM (#13514509) Homepage Journal
    My job is building systems like this. Current mailserver system I designed and built is hosting 80,000 email accounts, and will scale out to a million quite cheaply by just adding more machines.

    OpenLDAP

    You need a central configuration repository to store the email accounts, their passwords, etc. OpenLDAP is perfect for this, and you can replicate it out for scalability. Be prepared to learn about LDAP schemas.

    Exim

    Use Exim because it has a simple process model (a single binary that does all the work, like sendmail) but has a human readable configuration file and has to be the most flexible MTA out there. You will have customers with weird requirements sometimes, and Exim will be able to meet those. Plus, it has Exiscan-ACL built-in these days, which allows you to do virus scanning and spam scanning at the DATA stage, before the mail is actually accepted by the MTA. It means you can make the sending MTA deal with the bounces if the mail is a virus or is obvious spam.

    Courier-IMAP for POP3 and IMAP access.

    Yeah its written by a sociopath, but nothing else works as good in the field. It works out of the box with sensible LDAP schemas and is fast, reliable and secure. Handles SSL, all the different authentication methods, what have you. Maildir compatible.

    Maildir message store.

    Store the mail in maildirs. Don't put them in /maildirs/domain.com/user/Maildir - split the domains up with a 2 level deep hashing algorithm (if you're virtual hosting domains, which is what it sounds like to me), so make it something like /maildirs/xx/xx/domain.com/user/Maildir, where xx/xx might be something like 3f/6b (depending on the hash). Use MD4 for the hash because its more balanced than MD5.

    NFS mount the maildirs from a fast NFS device like a Netapp. Netapps are recommended because you can plug them in, and they just work, plus they are easy to scale by adding more trays.

    Linux NFS servers set up with heartbeat and shared disk also make a nice HA NFS, and would be cost effective, but you'll have to buy an array anyway (probably fiber channel) so it might be better just get something thats completely integrated like the Netapp.

    Spamassassin.

    Can be configured to scan make at DATA time in the SMTP conversation. A LOT of configuration work here to make it play nice on a massively scaled platform, but it can be done. Mostly it needs to have things like the auto whitelisting and bayseasn filtering turned off, as the extra DB file work is a bit excessive.

    Actually, I'm sure there is a way to make it work with a less resource intensive repository, but using the standard SA rules seems to work well for my environment. *shrug*

    ClamAV.

    Free antivirus, it works, and integrates well with Exiscan-ACL. Set it up to scan via the daemon, and configure it to update every couple of hours from cron, and bob's your uncle.

    Scaling out

    Make every box the same. Make every box an MTA, a POP3/IMAP server, etc. Use something like Kickstart to automate builds so that you can build a machine in 10 minutes, and all you have to do is configure the IP address and plug it in. If you want to be REALLY sexy, you could make the machines boot off the network, and mount / from a shared NFS area, and make /var/spool/exim the internal mirrored disks. DHCP them, then all you do is plug a machine in and set it to PXE boot. Pretty trivial to do.

    Load balancing

    Hardware load balancers are pretty much a necessity. Don't touch cisco stuff. Its not very good. Go with Foundry Networks ServerIrons. The XLs can handle 1 billion requests/day if you configure them in Direct Server Return mode (also known as DSR/Foundry switchback). Use it. It makes all the return traffic go directly out to the net, meaning your ServerIrons have to switch less traffic and track less sessions. I would recommend however for a million users a pair of the ServerIron 450GTs, or bigger. Maybe one per VIP/Service.

    Now, if this is all looking pretty daunting, you could always hire me to build it for you :)
    • by PapaZit (33585) on Thursday September 08, 2005 @10:18PM (#13515134)
      All of the paren't suggestions are decent, but there are a few alternatives that may make sense:

      -Cyrus IMAP, while a monster to build and configure, can handle a pretty heavy load, and the latest versions can handle a lot of load-balancing internally.

      -Exim's nice. I'm a Postfix man, myself. Sendmail is king, though. I'm not going to claim to like it, but it's up to the task, and there's something to be said with using a standard tool.

      -While things like MD4 are okay for hashing, they're kind of CPU-intensive. Consider something like "second and third letter of username" that takes less CPU time. The right answer here depends a lot on the relative speed of CPU versus disk. If you can get dedicated hardware to do this (rare, but it exists), use whatever hashing the hardware supports.

      -Consider some sort of cache (maybe even separate machines) between incoming SMTP and SpamAssassin/ClamAV. When the 2am spam run hits, your incoming SMTP machines can become overloaded. The downside: deciding what to do with mail that's not rejected the moment it's received.

      -Set up a "mail machine" configuration with whatever OS and tools you use, and make it possible to create a disk image quickly. You're going to need a lot of hardware, which means that you'll have enough random failures to make building machines by hand impractical. This also means "have at least one extra built machine/disk array/etc. powered-on and waiting at all times" for those 4am hardware failures.

      -You may find that things like NFS just aren't fast enough. Be ready to look at SAN or shared "direct-looking" storage. The tough part: this is hard to discover during testing. It may be overkill, but don't lock it out as a possibility.

      -I/O is king. CPU speed won't matter as much as bus speed, disk speed, and memory speed. This is why a lot of companies use banks of big proprietary unix machines for their mail, even if they use commodity PCs elsewhere.

      -I don't trust hardware load balancers. Sometimes they're necessary (and they do make life better when they work), but they're a big single point of failure. Consider other ways to split the load, or at least ways to work around the load balancer if it should fail. The Cyrus aggregator can handle some of this.
    • by thogard (43403) on Thursday September 08, 2005 @10:39PM (#13515248) Homepage
      Current mailserver system I designed and built is hosting 80,000 email accounts, and will scale out to a million quite cheaply by just adding more machines.
      80,000 is trivial. I was running a 12 node system with 87,000 users 12 years ago on hardware that was slower than a play station.

      The complexity of going from 100,000 to 1,000,000 isn't just 10 times harder, you start to get into that area where sigma 4 system works with few problems with 100k but dies horribly with 1000k users. There is a line where instead of one machine being broken is unusual, you get this situation where at least one machine is always broken and it will often be broken in a way that is hard to diagnose.
  • by hellfire (86129) <deviladv.gmail@com> on Thursday September 08, 2005 @08:32PM (#13514536) Homepage
    ... Is anyone wondering what's going on at Microsoft right now?

    It starts with a slashdot geek working in the email department spitting up his coffee, followed by a few rumors which make it up to a guy in accounting and customer service, followed by frantic management emails, including some inappropriate language, from Steve and Bill. Then a few good geeks start tracing who this cfsmp3 guy is and try to trace him to a company while the salesreps begin coldcalling any customers running around 1 million customers.

    And Microsoft will botch it because they have no experience in cowtowing and bootlicking, which are important skills for any company who wants to humbly keep its customers.
  • Easy (Score:5, Insightful)

    by xihr (556141) on Thursday September 08, 2005 @08:42PM (#13514602) Homepage
    Resign. You're obviously in way over your head if you have to resort to asking Slashdot readers for advice like this.
  • by Anonymous Coward on Thursday September 08, 2005 @08:46PM (#13514623)
    Hi Cliff;

    Sounds like a fantastic design opportunity here. The 5% of the project that is Enterprise architecture is what I enjoy the most as well. I'm assuming money probably isn't an object in terms of how much gear and bandwidth you may have to feed to this.

    I'm happy to let my fingers type away below, I'd love to keep in touch and see how you end up shaping this system. my email is allowmx at hotm...

    Before I ask, are there actually a million accounts? Or is that just a ceiling that you have to show proof of concept with?

    I've only implemented up until about 250,000 accounts of any kind, as I'm sure you're probably aware, the base transactional resource costing is essentially the same..

    For me, I would look at this for sure from at least these two angles:

    1) knowing your transactional costs (how much of your hard resources, bandwidth, cpu and disk space) will each type of transaction in your system take?) I mostly use this approach to get not an exact number, but an idea of magnitude, and detail where it happens on it's own to make sure the proper attention is applied to them.

    2) Failsafe intelligence & capacity in the infrastructure, as well as the failsafe intelligence & capacity in at the application layer. You have to know that your hardware, software, os, business logic and applications are all monitorable internally, externally for availabilty and actual "can I use it". Transactional logs, etc, of having information available when the inevitable problems come up.

    Also, having a capacity for as many of these layers to be self-healing, and fungible to the point that your service delivery is homogenous in as many ways possible. If your network finds something doesnt work or route, with mail, you can find another way to route it. Having a transactional manager of some kind, direct or not, could be useful in this case depending on what the client wants.

    99.9% uptime equates to about 526 minutes, or 87.6 hours you _could_ be down each year. Thats about 7.3 hours a month, or one day a month.

    Based on that, having flexible, redundant tools setup in a high-availabily arrangement at their respective operating capacities is key. I'm not sure if your current exchange problems are being aided by not enough equipment, bandwidth, or other stability issues, so I'll just assume that it's all of them :)

    I apologize if anyone else has already mentioned some of this, but here's some of what I've found to help me where email has become as crucial to a business as their cell phone.

    On the hardware level:

    - STORAGE: Everything goes on a SAN, if not more than one. Don't waste your time with anything less.
    - SERVERS: All servers have redundant hot swappable parts in the very least, power and hard drives. I'd even suggest making the servers Iscsi bootable so they can boot off the backbone. Beyond this, I like to buy my servers in piles of identical ones. Have 1-2 spare serevrs of each kind sitting there, ready to throw hot swap drives into from a failed server. That way if a server dies, you can address the power supplies, or get the HD's in that machine into another identical server and get it up and running while you diagnose the hardware problem independantly. My approach to any kind of problem is FIX, DETECT and REPAIR. Get it up and running, find out what was wrong, make sure it's fixed for good. Too many of us stop at the first too ;)

    The idea I have in mind is a smaller scale of a google beige box army. linux/bsd offer so much more transcations for each piece of hardware, so that works very much in your favor. Obviously something enterprise grade to satisfy the client such as the Compaq/HP Proliants, etc. I feel these Servers ahve the best overall support, manageability and information tools, and their openlinux drivers interface wonderfully with open source operating systems)

    Networking/Communication level:

    - Entire mail processing architechture communi
  • by Temkin (112574) on Thursday September 08, 2005 @08:53PM (#13514664)

    One million email accounts is quite a lot. You getting into the big league ISP category with something like this. It's not a one person operation to put something like this together. You're going to need a substantial number of well trained people to do this. There's only a couple players in the field at this level. Sun's JES Messaging system owns a sizeable chunk of the market, followed by OpenWave and a small gaggle of fly-by-nights with unproven track records.

    Some of the larger email systems however are homegrown using open source parts. Yahoo and Google immediately come to mind, and they do work quite well. But you probably don't have the resources that they do to engineer & test something like this. Yahoo is rumored to have more than 200 people working on email alone.

    Sun has a deployment like this canned, sitting on a shelf in Santa Clara. Tell them what you need, write a check, and they'll show up with the kit. 99.999% uptime if you write a big enough check. Make them to throw in the Waveset stuff.
  • I/O (Score:5, Informative)

    by Graymalkin (13732) * on Thursday September 08, 2005 @09:35PM (#13514904)
    While not quite a million users, HEC Montréal switched from Netscape Messaging Server running on AIX to Postfix/Cyrus/SquirrelMail running on Linux. Linux Journal [linuxjournal.com] ran a really nice article and a follow-up [linuxjournal.com] about their transition.

    One of the first things the school did was figure out how exactly their current system was failing them. Their old AIX boxes were being stressed just by the volume of mail coming through the system, they had little power left over to do any sort of filtering. This led to users getting drowned in unwanted e-mail which only exacerbated the existing load issues. This is one of the first things you need to do, figure out why your current system isn't working properly. You'll be better equipped to fix the problems when they've actually been identified.

    HEC Montréal also went for heavy redundancy and specialization. Instead of a handful of servers sharing all of the tasks equally each node in the cluster has its own job with every class of job having a backup server. Every job is going to take a beating with so many users, even if only a fraction of them are using the system at any given time.

    I'd say the most important part of what you're doing will be modeling your current use. Are you getting a ton of traffic from viruses and worms spreading over your internal network? Do you get huge amounts of spam traffic to users? In such cases filtering at your SMTP servers will relieve the rest of the system from extraneous traffic. While you might need really beefy external SMTP servers you won't need nearly as much storage space on a SAN or NAS.
  • by kenblakely (768899) on Thursday September 08, 2005 @10:24PM (#13515167)
    AKO (www.us.army.mil) is the Army's official intranet portal. We provide email for over 1.72M users, and we move almost 3 million messages a day. We do it all with Sun Messaging Server ver5.2 (soon to be Jes3) and we have exactly 2 (count 'em) two mail administrators. Sun mail is rock solid and scales great. We offer POP, SMTP, enterprise SPAM and Virus filtering as well as personal address books besides. We don't get the rich Outlook fat client, but then we want to be all web-based anyway. Can't say enough about Sun mail. If we had to do this with Exchange, I'd have to hire prolly 50 admins and deploy order of magnitude more machines.
  • by StarsAreAlsoFire (738726) on Friday September 09, 2005 @01:54AM (#13516435)
    From ASR ( http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.ht ml [xnet.com] )
    Re : Mail Transfer Agents

    Qmail : a small office of neatly dressed clerks, delivering short clipped remarks to queries, and handling mail with a rude impersonality, except in the case of failiure where they let their hair down and have an after-hours beer and let you know about it, pointing to the pertinent header sections.

    MMDF: A jumped up mailroom boy with a chip on his shoulder. Loves the bureaucracy and takes great pride in stamping "illegal address" in red ink on any mail it passes. Unpacks all the mail and repacks it in his own special envelopes before delivery to end users.

    PP: MMDF gone mad with standards fever. Think "Brazil".

    No, PP is... well, see, when it receives a letter, it chops it into small pieces, then translates bits of it using an English-Hungarian phrasebook and puts all the bits into various pigeon-holes. When it gets round to delivering the message, it collects all the bits, translates them back using a Hungarian-English phrasebook, tapes them together, and loses the letter. Some time later, you get a bounce message:

          ----- The following addresses had permanent fatal errors -----

          ----- Transcript of session follows ----- ... while talking to bloat.example.com.:
    >>> RCPT To:
      550 My hovercraft is full of eels

    PP is John Cleese.

    Sendmail: Shiva as a postman. Many arms delivering mail, dancing, taking drugs, destroying as it sees fit. Often makes creative changes to the mail for kicks, but ultimately can be persuaded to do anything with the right incantation...and that includes giving you other people's mail.

    VMail: No experience yet, but I'd guess something like a wisened old man sitting on the porch outside the postoffice. Looks at everyone who passes by with deep suspicion, but turns out to be friendly and helpful once he realises you're not there to rob the place.

    Micro$oft IMC: The Scarlet Pimpernel of postmen. Hard to find, impossible to order about, but every once in a while it saves a piece of mail from disaster. Sometimes even with it's head(ers) intact.

    cc:Mail SMTPLINK: A 5 year old child left in charge of a large sorting office. Can't reach over the counter properly, can't handle more than one letter at once and has to go looking for a grownup whenever it wants to deliver to mail to other towns. Often opens parcels to look for shiney things inside then just delivers the wrapping paper onwards.

    cc:mail UUCPLINK: an insane madman sitting in a box. Mail is thrown into a box where unknown things happen to it.. sometimes mail actually leaves the box.. usually to be delivered to the administrator of a totally unrelated postoffice and containing a complaint that the madman could not find the recipient in his dark box and would you please contact the person with the key of the box. Of course, the only way to reach that person is by mail and even if the box is opened the madman cannot be pursuaded to actually send mail to unknown addressees to the person with the key anyway...
    Gus, Pete Bentley, Malcolm Ray, Perry Rovers
  • Quick setup (Score:5, Insightful)

    by mseeger (40923) on Friday September 09, 2005 @02:44AM (#13516606)
    Hi,

    my recommendations:

    • Calculate with about 20-30 man days for the initial design. You'll need some software development for about 30-50 man days, 100 man days for setup, testing and fine tuning. Figures may wary upon skill and LWF. Time for integration into your backup service is not included.
    • Use a directory service with replication mechanism (preferred LDAP, we've done it with MySQL too). Every system except the load balancers will get a replica.
    • The user data is stored on machines with Cyrus . Depending on machine size, user profile, mbox size etc. you take between 5.000 and 50.000 users per system.
    • The directory service knows which user is on which system. Prepare a script to move users from one server to another (including the mbox).
    • Incoming IMAP connects go through a loadbalancer to frontend systems with the perdition proxy. Those will relay thre requests according to the directory to the responsible IMAP server.
    • Incoming HTTP requests will go through the loadbalance to an Apache with Squirrel on the frontend systems. Those will convert the requests into IMAP requests and connect to the local perdition.
    • Generate a web frontend for the user to setup auto reply, vacation and anti-spam settings.
    • From those settings you can create SIEVE scripts for the user.
    • Incoming and outgoing SMTP traffic is handled by systems with sendmail. Local delivery is handled by LMTP connects directly to the IMAP servers (cyrus can handle LMTP).
    • Antivirus and Antispam is handled through the milter interface and appropiate plugins. Plan for individual settings per user (can be generated from the data in the directory server).
    • Loadbalancing SMTP us trivial.
    • Add monitoring (e.g. Nagios), Backup and Restore (last one most important, nobody wants backup, all everyone wants is restore).
    • If desired, use a cluster file system for those IMAP servers to have even more redundancy.
    • Make sure you have access to the internal DNS of your company. If you can setup "mail.acmecompany.com" to point to several ips (depending on location) this may ease your job lot. If you cannot, this may be hard (and expensive) for your load balancers.
    • You can scale everything horizontal in this concept. Choking point may be the load balancers.
    • You can distribute the system easily onto several locations. Distribution over several continents is only recommended if you can either manage the DNS or the mail agent settings per continent.
    Please forgive me, if i'm not completely correct. I'm only the sales rep ;-). But we've done it several times for ISPs. OSS software usually does the biggest part of the work. Usually some components (depending on existing contracts and knowledge) are commercial software (e.g. anti virus, load balancers, cluster file system). Typical operating systems are Solaris or Linux.

    With backup support you should be able to setup such a system in 6 to 12 months (the later more realistic for big companies).

    Most probably users will complain about the lacking calendar.

    Most troublesome will be the migration phase (hope you realized i didn't mention it above). This depends so much on your current scenario that it is very difficult to give a general advice.

    > where would you start?

    Contacting me ;-). Perhaps get a budget first. As i said, i'm sales....

    Regards, Martin

  • Stop right now (Score:4, Insightful)

    by biglig2 (89374) on Friday September 09, 2005 @04:54AM (#13517059) Homepage Journal
    What you have here is an opportunity for a tremendous open source win against exchange, and you are about to stuff it up because you do not have a clue how to do it.

    So, what you do right now is you go find someone who does know how to do it. And by that I mean someone who can demonstrate they know how. Which does not equate to having a low slashdot id; it equates to having done real projects of this scale.

    So, how do you start? You ring IBM and get them to come in and talk to you. You ring Red Hat. You ring Accenture.

    If you want impartial advice from someone who isn't a vendor (which is a good idea), then you go find some companies that has a million seat open source e-mail deployment in place and you see if you can get their messaging admin to talk to you.

  • Well (Score:4, Insightful)

    by Shads (4567) * <shadus@shMENCKENadus.org minus author> on Friday September 09, 2005 @08:54AM (#13517958) Homepage Journal
    In my opinion you're going to need a cluster of servers or at least round robin'd mx records for the servers. I personally think sendmail scales the best of the mta packages and offers the best set of features and ease of maintenance, although alot of people would argue it's intrinsicly insecure... I've never had problems, but I kept our mail servers up to date. I would seperate the smtp machines the outside world uses to deliver mail to your space from the servers used by users of your service to deliver mail. I would also move delivery services (imap, pop, webmail) to their own machines instead of having them on the smtp machine and you would probally be best to use a nas for the actual storage medium. This is actually a really interesting project. Good luck and let us know how it turns out :)
  • Notes/Domino (Score:4, Interesting)

    by hey! (33014) on Friday September 09, 2005 @09:35AM (#13518183) Homepage Journal
    Of course nearly everyone who uses it hates it, because it seems unnecessarily complicated. But this is precisely the kind of situation Domino was designed to handle: scaling. If you can get by with Sendmail, you don't need or want Domino, but if you want to manage a million email accounts, this is one of the first places I'd look.

    This is exactly what Notes was designed to do: scale. People have been building systems on this scale with notes for nearly twenty years. You can not only scale it by moving parts of your email system onto mainframe class iron, but you can distribute it and provide all kinds of flexibility and redundancy into your system to meet virtually any messaging requirement (e.g. choose an alternate MTA for high priority traffic when there are Internet disruptions). Naturally there's some complexity involved, but if you can get by with sendmail you probably shouldn't be using Notes.

    What's more important is that management of accounts and identity, which is distributed, delegatable, and backed up by robust cryptographic certificate management. You can let a subsidiary manage it's own accounts, they can subdelegate that to a division and the division can subdelegate that to the IT staff on site; at each level policies can be set, enforced, and changed for lower levels.

You can do more with a kind word and a gun than with just a kind word. - Al Capone

Working...