Infrastructure for One Million Email Accounts? 1216
cfsmp3 asks: "I have been asked to define the infrastructure for the email system for a huge company, which fed up of Exchange, wants to replace their entire system with something non-Microsoft. I have done this before, but not for anything of this scale. Suppose you are given a chance to build from scratch an email system that has to support around one million accounts. Some corporate, some personal, some free. POP, IMAP, webmail, etc are requirements. The system must scale perfectly, 99.9% uptime is expected... where would you start?"
Kerio (Score:2, Informative)
Um... (Score:4, Informative)
for spam... (Score:2, Informative)
NO GMAIL (Score:2, Informative)
1 Million Users! (Score:2, Informative)
~ 320K accounts (Score:5, Informative)
It's obvious (Score:4, Informative)
However, I'd personally ask Google [google.com]. They've done it and even their search engine has information. I found an interesting link from there detailing the deployment of a large hundred thousand user mail system, from the architecture to the software located on Linux Journal [linuxjournal.com].
Who to talk to (Score:3, Informative)
Mirapoint is probably _the_ vendor to speak to, though.
openwave's email server does this but it's $$$ (Score:3, Informative)
I'm sure you could hack together something to do this much like what google did. Might take some time but it's totally doable.
CommuniGate (Score:2, Informative)
Is able to run clusters, and clusters of clusters, and theoretically scale into the hundreds of millions of accounts. Offers all the things you want, and more. LDAP, ACAP, etc, etc, integrated webmail. Intelligent directory creation structures, etc.
Split up the tasks (Score:4, Informative)
Your receivers will be a bank of servers running sendmail. They will do appropriate spam processing to reduce the amount of mail actually received. They feed the data into the storage servers.
The storage system has the data partitioned out so that all the data for one user would go to one server while all the data for another will go to a different one. The storage system also has to provide POP and IMAP access. You may want a special setup where the IMAP or POP service known which server to go to. Investigate having one giant virtual filesystem so that the system isn't too complicated.
Your webmail access will use IMAP to access the actual mail. It can be a completly different system.
The sending system will be a chokepoint for all outgoing mail. You are going to scan it as it goes out to look for virus-sent emails or unauthorized messages. For instance, you may want marketing email to be processed differently than inter-office email and such.
All of these systems will be running sendmail. I know sendmail has a bad rap for being insecure, but the insecurities have been found and since fixed. It is by far the most manageable system when it comes to large-scale deployments with heavy customization.
Novell? (Score:2, Informative)
For the lazy... (Score:5, Informative)
---
ok i work for a large uk isp in the messaging (email) operations dept. we currently have 2.5-3 million active accounts (and a load of suspended), and manage anywhere upto 12-16million mails per day
our setup is like this (this is simplistic though):
front line - anti abuse mta's - these do dnsbl type lookups (spamcop, spamhaus and sorbs). we have 9 incoming
next we have mta's. they farm mail off to brightmail servers, which do similar to spamassassin. we have 6 incoming mtas, and 8 brightmail servers (not enough - high load)
after that they farm off to vscans (6)
after that any mail that gets through is delivered to mail stores (8 + 2 hot spares)
what you want to be doing is similar to this above - chaining hte mail from one level to the next. the first level should be the rbl's - these are less processor intensive, and can remove a fair whack of your mails in one swoop. spamassassin is going to be more cpu intensive, since it has to open each mail and read the first x many bytes
id have separate machine(s) holding your master directory, and if you can get directory caches then do that too (to take the load off the master directory) - ours run oracle
i dont know what your budget is, but split up hte different tasks as much as possible. that way if you need to add more to any pool (rbl lookups, spamassassin etc) you just add another machine..
one last thing - we also have a separate box just for postmaster mail (with exim + spamassassin funnily enough) - it tends to get busy
Last edited by Slidey on 09-08-2005 at 11:19 PM
--
(end of quote)
Groupwise doesn't suck (Score:1, Informative)
It's not free, but it's not dependent on the Linux community either. There is a concentrated and very very dedicated support and development crew. Message store size can be up to 1/3 the size of Exchange, and moving servers around is a cinch.
I'm not a Groupwise admin or anything, but I have been and Exchange guy, and I feel your pain.
When all else fails... goto spec.org (Score:2, Informative)
I recommend CommuniGate [stalker.com].
Didn't We Just Have This Question? (Score:2, Informative)
Re:CommuniGate (Score:3, Informative)
I would use postfix (Score:2, Informative)
CommunigatePro from Stalker.com (Score:5, Informative)
2) It'll scale as big as you can dream - over 5 million accounts with clustering
3) MAPI support
Hire Matt Simerson, the creator of MailToaster (Score:4, Informative)
You can learn about him, and his mail projects at http://www.tnpi.biz/internet/mail/toaster.shtml [tnpi.biz]
-Chris Knight
13 Million mail infrastructure (Score:1, Informative)
The keys were:
- qmail (but postfix will work better nowadays)
- smtp (4 machines)
- pop/imap (4 machines)
- separated webmail 1 is enough (2 to high availability)
- NDS (Netscape Directory Server) which is now owned by RedHat and opensourced.
Hope that helped.
Scalable e-mail systems? (Score:3, Informative)
My slides relevant to this discussion can be found at http://www.shub-internet.org/brad/papers/dihses/ [shub-internet.org] and http://www.shub-internet.org/brad/papers/sistpni/ [shub-internet.org].
And yes, Nick Christenson [jetcafe.org] has been a long-time friend and co-author of mine.
Feel free to contact me directly if you want some referrals.
IBM Z990 (Score:2, Informative)
99.9% reliabilities is more then normal for those machines. It is modular enough to expand to what ever you may need in the future, and it has the dataprocessing horsepower to actually hand the 20k or so concurrent users at a time and have the harddrive space to match that many users as well.
Run linux or unix on top of VM and you should be fine.
Product Page for Z990:
http://www-03.ibm.com/servers/eserver/zseries/z99
Re:Kerio (Score:4, Informative)
disclaimer: I work for one of those companies.
Plan. Test. Spec. Deploy. (Score:5, Informative)
(2) Test. For each server, hammer it. Test it's load under as close to real world circumstances as you can. Then create unreal punishing loads and see how it handles it. Plan in advance for how your server farm handles something like virus-generated mass emails causing 1000% spikes in load.
(3) Using your testing results, spec out the actual hardware. RAID, cheap hardware, redundancy, etc. If you have control over the network choice, plan a location with multiple fiber trunks coming into the building and provider redundancy. Remember backhoes in concert? Don't get hit by that. Plan for server failures, drive failures, network failures, power failures, and security compromises.
(4) Deploy! If you did the rest right, this is the easy part. You'll have redundant network connections, HSRP, redundant switches, a proxy farm, an imap/pop farm the proxies connect to, an smtp farm for outgoing emails, and a web server farm for serving up webmail (depending on how you choose to architect the disk space, the web farm and the pop/imap farm may be one and the same; depends on how you set things up.)
Here's a starter link to a setup which is smaller but, in principle, fairly similar:
http://www.itd.umich.edu/umce/features/2004/cyrus
Finally, if you don't want to screw it up, ask someone who has done it before. Paying someone $300/hr for a 10-30 hour review of your plan is dirt cheap compared to horking the setup. Someone who has worked in huge email environments (a la, hotmail) could show you gotchas before they bite you. (If you need help figuring out who to ask, I could even point you to some of the appropriate people)
YIKES! Tossing out the groupware?! (Score:5, Informative)
Now to the mega-infrastructure that I set up for an undisclosed company for under 50K (and also didn't want groupware).
1. Transport Sender (sendmail). That's right! Good ol' plain sendmail scales. It does require some pretty savvy tweaking so get Sendmail.Com consultant onboard just for this. Use SleepyCat DB for speed for all sendmail setups. For one million, I had about 23,000 transaction per minutes during the day. You'll require 10 servers for this for cushion (against some idiots sending an ISO attachment).
2. Payload receiver (sendmail). A second group of machine to handle the reception of SMTP payloads.
3. IMAP4S/POP3S - Hey what's with the "S"? Nothing like sending your user's password in the clear. Unless you enforce VLAN in your corporate environment and limit all IMAP4/POP3 to VLAN, the "S" is a mandatory security feature, inside and outside. Guess what "S" stands for?
4. Webmail - SquirrelMail - Yet another dedicated server (in which I had to add two more load-balanced server to handling the growing pain). Use https for login only.
5. AntiVirus (ClamAV) - It was the best back then, now its just running in the middle of the pack. sendmail has milter that allows extensibility such as MIMEDeFang, wilter, rureal (reverse-DNS check), spamassasin, and SPF.
6. Support - Half the effort is put into those webpages that would 'hand-hold' these newbies into reconfiguring their machine. Worth the effort if you have over 20 expert PC users that can do their boxens. Otherwise do it yourself at each PCs. These pages should cover Thunderbird, Evolution, as well as Outlook and Outlook Express.
7. Learn to spin 11 plates, one on each pole. Keep them spinning... If they start to drop and break, bring in some more Unix dudes.
Novell Groupwise or Lotus Notes (Score:2, Informative)
Commercial Package Options (Score:2, Informative)
If you are into benchmarks, the folks at SPEC [spec.org] have published results from several packages.
Simplicity is key. (Score:5, Informative)
OpenLDAP
You need a central configuration repository to store the email accounts, their passwords, etc. OpenLDAP is perfect for this, and you can replicate it out for scalability. Be prepared to learn about LDAP schemas.
Exim
Use Exim because it has a simple process model (a single binary that does all the work, like sendmail) but has a human readable configuration file and has to be the most flexible MTA out there. You will have customers with weird requirements sometimes, and Exim will be able to meet those. Plus, it has Exiscan-ACL built-in these days, which allows you to do virus scanning and spam scanning at the DATA stage, before the mail is actually accepted by the MTA. It means you can make the sending MTA deal with the bounces if the mail is a virus or is obvious spam.
Courier-IMAP for POP3 and IMAP access.
Yeah its written by a sociopath, but nothing else works as good in the field. It works out of the box with sensible LDAP schemas and is fast, reliable and secure. Handles SSL, all the different authentication methods, what have you. Maildir compatible.
Maildir message store.
Store the mail in maildirs. Don't put them in
NFS mount the maildirs from a fast NFS device like a Netapp. Netapps are recommended because you can plug them in, and they just work, plus they are easy to scale by adding more trays.
Linux NFS servers set up with heartbeat and shared disk also make a nice HA NFS, and would be cost effective, but you'll have to buy an array anyway (probably fiber channel) so it might be better just get something thats completely integrated like the Netapp.
Spamassassin.
Can be configured to scan make at DATA time in the SMTP conversation. A LOT of configuration work here to make it play nice on a massively scaled platform, but it can be done. Mostly it needs to have things like the auto whitelisting and bayseasn filtering turned off, as the extra DB file work is a bit excessive.
Actually, I'm sure there is a way to make it work with a less resource intensive repository, but using the standard SA rules seems to work well for my environment. *shrug*
ClamAV.
Free antivirus, it works, and integrates well with Exiscan-ACL. Set it up to scan via the daemon, and configure it to update every couple of hours from cron, and bob's your uncle.
Scaling out
Make every box the same. Make every box an MTA, a POP3/IMAP server, etc. Use something like Kickstart to automate builds so that you can build a machine in 10 minutes, and all you have to do is configure the IP address and plug it in. If you want to be REALLY sexy, you could make the machines boot off the network, and mount / from a shared NFS area, and make
Load balancing
Hardware load balancers are pretty much a necessity. Don't touch cisco stuff. Its not very good. Go with Foundry Networks ServerIrons. The XLs can handle 1 billion requests/day if you configure them in Direct Server Return mode (also known as DSR/Foundry switchback). Use it. It makes all the return traffic go directly out to the net, meaning your ServerIrons have to switch less traffic and track less sessions. I would recommend however for a million users a pair of the ServerIron 450GTs, or bigger. Maybe one per VIP/Service.
Now, if this is all looking pretty daunting, you could always hire me to build it for you
Re:Split up the tasks (Score:3, Informative)
Where to start (seriously) (Score:5, Informative)
Once you have that done, you can start looking at solutions. You will have two parts to your solution:
1) The DMZ email relays (possibly including other antispam/antivirus functions) You really need high availability here.
2) Your email storage and retrieval systems. These may be a little more tolerant to downtime on an individual basis. But if you need to have redundancy here, there are ways to do it.
I think Hotmail did fine with BSD and Qmail.* I am sure Postfix is equally capable.
* Although Qmail itself has never had a security vulnerability discovered, you should be careful. TCPRules (on which qmail relies) has a vulnerability that can lead to root access for local users. This is not a problem on systems with no local users, however. I am not aware of any patch for the TCPRules vulnerability.
suggestions (Score:2, Informative)
in order to increase reliability, you want to adopt a clustered design - if a machine or two fail, nothing should happen to the service.
in order for all the machines to be able to find the user preferences/passwords/etc, you'll want some sort of common storage for them. it could be on a shared filesystem, in ldap, mysql, etc. ldap is common and a good choice (it has very fast read/query performance) - make sure you use replication so an ldap server failure doesn't take you down (or better yet, a multi-master setup). if you use ldap or sql, make sure you are indexing correctly on the data you most commonly pull up.
in order for all the machines to access the user's mail, you'll want some sort of shared message storage. a shared filesystem is easiest, you could choose from nfs, redhat gfs, veritas cluster fs, etc. if you use nfs, make sure the nfs server can failover to a backup system if the nfs master dies (netapps are great for this).
rather than using round-robin dns, i'd invest in a load balancer. there are some free options for bsd and linux, but the commercial products are very nice and easy to use. f5 labs bigips are very nice, cisco CSSes are garbage.
other suggestions about breaking the services into different groups are spot on. personally, i'd have 3-4 inbound smtp servers inside a loadbalanced pool that handled inbound mail and passed the messages to virus and spam scanning services before delivering them to the shared message store (your load might dictate you need more servers, but if you design right you can just add more as time goes on). i'd probably put pop3 and imap services on those hosts as well, and possibly only allow pop3s and imaps (the ssl encrypted varients).
i'd also have a set of outbound mail servers that users would connect to to relay outbound mail. they would require smtp auth, and possibly only allow connections on smtps ports. spam/virus scanning would be performed before the message was accepted by the server, so users would get immediate feedback if their message didn't go through. the outbounds would not do any local delivery, so they would not mount the shared message store (you'll get proper bounces for all invalid mail addresses this way, instead of smtp rejections for invalid email addresses in local domains).
i'd have another set of servers that did virus and spam scanning for both the inbound and outbound smtp servers. you'd want these machines to have faster cpus than the rest, and virus and spam scanning are usually quite cpu intensive. again, if your load increased (or was more than you had anticipated), the system is easy to grow just by adding more machines.
another set of servers would handle the shared filesystem (if nfs, or gfs exported via gnbd), and possibly also the shared preferences store (ldap).
the final set of servers would handle webmail.
each set of servers should be firewalled from the others (especially the webmail servers, which are probably the most vulnerable to attack), with only the neccessary allowed traffic going through.
qmail and postfix can easily read ldap, i'm sure sendmail can also (as can commercial solutions). anything will work for the smtp daemon.
since you are supporting pop3 users, maildir is a better choice over mailbox for your message stores. courier or cyrus would be a good choice, and come with pop3, imap, and MDA (message delivery agent) components.
i'd have the inbounds accept mail from remote sources immediately (assuming the user being delivered to was valid) and have them hand off the message to an MDA, which would perform spam scanning, virus checking, and any user filtering configured before delivering the message to the user's mailstore. (scanning after the message is accepted uses more resources, but grants you more flexibility - users can have their own spamassassin settings, or you can add any number of filtration steps).
for virus scanning, check out ClamAV. for spam scanning, look at spamassassin (
Re:~ 320K accounts (Score:3, Informative)
Argue for your favorite all you want, but friends don't specify Lotus Notes to friends.
Re:Split up the tasks (Score:2, Informative)
Products such as Ironport [ironport.com], Openwave Edge Gx [openwave.com], and Symantec Mail Security Security [symantec.com] use technologies such as traffic shaping, reputation services, directory harvest attack detection, etc. to help keep spam out of your network.
Re:I worked at a company that did this... (Score:3, Informative)
We have account data stored in an LDAP store, mirrorred to a second (read-only) store for redundancy/scaling when busy. LDAP scales wonderfully for read-heavy tasks such as this one.
As has been mentioned separately, separating recipient (edge), storage, and outbound mail servers is really important. Our edge servers perform RBL checks, greylisting (on some domains that want it), SPF (ditto), reject various attachment types, perform a reverse-MX check to try to accept from valid addresses only, and perform a recipient address check to quickly reject incorrectly addressed messages. That cuts down 80% of incoming mail (with very few false positives). Mail is then forwarded to a second set of edge servers that run SpamAssassin (set to flag spam, not stop it) and ClamAV on attachments. Finally, it goes into the storage servers. POP3/IMAP/Webmail points at the mail directories on these servers. Our outgoing servers are quite a simple setup, with SMTP Auth (also hooked to LDAP). We also have a few listservs setup, but they are a side issue.
Qmail is a bear to setup, and asking the author for advice is a good way to get flamed. Other than that, it works very well, we haven't had any security issues, and it's adequately fast - especially if you apply the "silly qmail todo" patch, fixing concurrency problems under high load. It's part of the Qmail-LDAP distribution (as is almost everything else I listed).
For servers, we use FreeBSD. I'm sure other OSes would do a fine job, but FreeBSD has been rock solid for us.
Intelligent Architecture (Score:5, Informative)
Sounds like a fantastic design opportunity here. The 5% of the project that is Enterprise architecture is what I enjoy the most as well. I'm assuming money probably isn't an object in terms of how much gear and bandwidth you may have to feed to this.
I'm happy to let my fingers type away below, I'd love to keep in touch and see how you end up shaping this system. my email is allowmx at hotm...
Before I ask, are there actually a million accounts? Or is that just a ceiling that you have to show proof of concept with?
I've only implemented up until about 250,000 accounts of any kind, as I'm sure you're probably aware, the base transactional resource costing is essentially the same..
For me, I would look at this for sure from at least these two angles:
1) knowing your transactional costs (how much of your hard resources, bandwidth, cpu and disk space) will each type of transaction in your system take?) I mostly use this approach to get not an exact number, but an idea of magnitude, and detail where it happens on it's own to make sure the proper attention is applied to them.
2) Failsafe intelligence & capacity in the infrastructure, as well as the failsafe intelligence & capacity in at the application layer. You have to know that your hardware, software, os, business logic and applications are all monitorable internally, externally for availabilty and actual "can I use it". Transactional logs, etc, of having information available when the inevitable problems come up.
Also, having a capacity for as many of these layers to be self-healing, and fungible to the point that your service delivery is homogenous in as many ways possible. If your network finds something doesnt work or route, with mail, you can find another way to route it. Having a transactional manager of some kind, direct or not, could be useful in this case depending on what the client wants.
99.9% uptime equates to about 526 minutes, or 87.6 hours you _could_ be down each year. Thats about 7.3 hours a month, or one day a month.
Based on that, having flexible, redundant tools setup in a high-availabily arrangement at their respective operating capacities is key. I'm not sure if your current exchange problems are being aided by not enough equipment, bandwidth, or other stability issues, so I'll just assume that it's all of them
I apologize if anyone else has already mentioned some of this, but here's some of what I've found to help me where email has become as crucial to a business as their cell phone.
On the hardware level:
- STORAGE: Everything goes on a SAN, if not more than one. Don't waste your time with anything less.
- SERVERS: All servers have redundant hot swappable parts in the very least, power and hard drives. I'd even suggest making the servers Iscsi bootable so they can boot off the backbone. Beyond this, I like to buy my servers in piles of identical ones. Have 1-2 spare serevrs of each kind sitting there, ready to throw hot swap drives into from a failed server. That way if a server dies, you can address the power supplies, or get the HD's in that machine into another identical server and get it up and running while you diagnose the hardware problem independantly. My approach to any kind of problem is FIX, DETECT and REPAIR. Get it up and running, find out what was wrong, make sure it's fixed for good. Too many of us stop at the first too
The idea I have in mind is a smaller scale of a google beige box army. linux/bsd offer so much more transcations for each piece of hardware, so that works very much in your favor. Obviously something enterprise grade to satisfy the client such as the Compaq/HP Proliants, etc. I feel these Servers ahve the best overall support, manageability and information tools, and their openlinux drivers interface wonderfully with open source operating systems)
Networking/Communication level:
- Entire mail processing architechture communi
Re:Obviously (Score:5, Informative)
Your point about putting more effort up-front into design is well taken, but thhat advice applies to any platform...
WIth that said, and without turning this thread into an Exchange bitchfest...
Why in the hell can't you restore a mailbox from backup using only the tools you already have if the user is no longer present in Active Directory? You can't even export the mailbox with EXMERGE... Your choices are 1) 3rd party recovery tool (like Quest Recovery for Exchange) or 2) Build an ENTIRE OTHER SERVER and do a normal, full restore of the entire mail store so you can extract one measly mailbox.
OBviously, the "Recovery Storage Group" feature is a VAST improvement over the old Exchange 5.5 way of bringing back just one mailbox (that being setup another server) but this is a MAJOR duh situation on Microsoft's part. They seem to think that since their "best practice" is to never ever erase any user account ever ever ever, that its okay to leave this gaping flaw in their enterprise groupware product. Sorry, but I think that sucks. We paid out the ass for "Enterprise" edition (to avoid the arbitrary 16gb limit on the mail store) and goddammit, I should be able to bring back a mailbox without its corresponding AD account without wasting a whole day setting up another server... I've only had to do it once (today) but the whole time I Was thinking how much esaier a mailbox restore on my OS X Server at home would be... Just restore the frickin' files and move on with your life.
You need a staff of 10 to 20 to run this... (Score:3, Informative)
One million email accounts is quite a lot. You getting into the big league ISP category with something like this. It's not a one person operation to put something like this together. You're going to need a substantial number of well trained people to do this. There's only a couple players in the field at this level. Sun's JES Messaging system owns a sizeable chunk of the market, followed by OpenWave and a small gaggle of fly-by-nights with unproven track records.
Some of the larger email systems however are homegrown using open source parts. Yahoo and Google immediately come to mind, and they do work quite well. But you probably don't have the resources that they do to engineer & test something like this. Yahoo is rumored to have more than 200 people working on email alone.
Sun has a deployment like this canned, sitting on a shelf in Santa Clara. Tell them what you need, write a check, and they'll show up with the kit. 99.999% uptime if you write a big enough check. Make them to throw in the Waveset stuff.
Oh, dear God, you RECOMMEND Notes? (Score:3, Informative)
Sorry, but Lotus Notes sucks; it's an abomination in almost every way. It's bloated, slow, buggy and has what is arguably the worst user interface ever (The User Interface Hall Of Shame said they could have based their entire site on this one app!) Sure, it does group meeting notes and can let you check other people's calendars but it falls flat as an email system. If it can't do the basics, who cares about the "advanced" features.
Doubt me? Okay. Let's try a little experiement.
First, sort your inbox by subject. Oh, I forgot. YOU CAN'T. Well, let me take that back. You can if you simply follow these simple instructions...
First, you need to have Domino Designer installed. In Designer, open Folders in left pane, then open folder $Inbox, highligh the Subject column. In the window with Columns properties in second tab you can check-in the "Click on column header to sort..." checkbox. Close $Inbox folder window. To prevent design refresh, in Folders view, right-click on $Inbox folder, choose Design properties and on third tab check-in "Prohibit design refresh or replace to change".
[blinks eyes in disbelief]
Un. Fucking. Believable.
Oh, and the feature I like the best is the pop-up dialog that tells you you have new mail. So you click to make that go away, switch over to LN to read the new mail and it's not there... Oh, yes, that's right, you have to press F9 to actually download the email to your client, even after being notified by an obnoxious popup that you have new mail.
Want to know another neat little feature related to that F9 key? According to our LN System Admin, get a few dozen people to all press and hold the F9 key for a few seconds at the same time and you can crash the Domino server backend requiring the server to reboot. Nice.
I could go on but I think I've made my point. I have never, ever, encountered anyone who has switched from Notes and been pleased with the change.
Re:Easy. (Score:2, Informative)
Re:Split up the tasks (Score:4, Informative)
In short, we have mail servers accepting the mail and dropping it on a shared NFS server which stores all the mail. The incoming servers run spam and virus filtering and is responsible solely for delivering the mail to the customer's mail directory which lives on the NFS server.
On the client side, we run IMAP and POP3 servers which access the stored mail on the NFS server to deliver it to the clients.
The exact software used for both of these functions are somewhat irrelevant. Once you split this up this way, you can also split the selection process. I.E. which is the best server for accepting SMTP mail and dumping it in customer's mail directories. Which can be answered with a completely different answer than the question of "what is the best NFS (or SANS) server to use to store the mail", or "what IMAP server should we be using", or "what webmail front end should we be using", or so on.
It also makes changing your mind down the road on any piece easier since you can actually run and test any one of these components in the live system as a final test before moving a replacement into the system.
FWIW, I would *love* to consult on something this scale.
Cyrus IMAP (Score:3, Informative)
For authentication, of course you have choices among LDAP, Kerberos (both of which are usable even if you're stuck with a Windows domain for authentication), PAM and other things. Very flexible; too flexible for some and it can be a bit confusing.
I've been working on rewriting the HOWTO, although I haven't made a ton of progress, it may still be useful to you: http://nakedape.cc/info/Cyrus-IMAP-HOWTO [nakedape.cc] and here's a presentation I put together for Linuxfest Northwest: http://nakedape.cc/info/Cyrus-IMAP-Intro [nakedape.cc].
You mention a million mailboxes, but that doesn't really mean much--that is just an estimate of storage requirements. What is more important to determine is how many concurrent users you will have and how much actual traffic--storage is cheap, memory not so much.
Re:Split up the tasks (Score:2, Informative)
You might give serious consideration to outsourcing your spam and virus filtering.
Re:Split up the tasks (Score:3, Informative)
I would argue that should be sending, receiving, accessing, storing. I'm not so sure sending and receiving need to be separated either.
The storage system has the data partitioned out so that all the data for one user would go to one server while all the data for another will go to a different one.
Uh, sounds to me like you're suggesting a storage system for every user. I'm sure that's not what you meant, but it's what you wrote :).
The storage system also has to provide POP and IMAP access. You may want a special setup where the IMAP or POP service known which server to go to. Investigate having one giant virtual filesystem so that the system isn't too complicated.
You should separate where the mail is stored from where the mail is accessed. Ie: your IMAP and POP servers access a mail store on a SAN or NAS. Depending on load, things like Webmail might require yet another layer of separation (ie: Webmail <-> IMAP <-> Storage).
It's really important to separate the mail access system(s) from where the mail is actually stored, otherwise you are building a system with single points of failure and performance bottlenecks.
All of these systems will be running sendmail.
That would be an absolute nightmare. Postfix is just as functional and orders of magnitude easier to administer.
Although, as I've said elsewhere, if this palce really does have a million-seat Exchange environment, they're almost certainly not going to be able to replace that with Squirrelmail, IMAP and Postfix. Exchange does a hell of a lot more than just sending emails back and forth.
Re:Simplicity is key. (Score:3, Informative)
We did something similar at my last job, where we had to maintain 9 million+ smallish files. We originally had one level of indirection and NTFS choked (huge amount of time spent in the kernel, not enough time running our app). Adding another directory level made it happy again.
There may yet need to be another level of indirection at the folder level to handle those few people who never deleted any email over their 15-year career with the company.
Chip H.
I/O (Score:5, Informative)
One of the first things the school did was figure out how exactly their current system was failing them. Their old AIX boxes were being stressed just by the volume of mail coming through the system, they had little power left over to do any sort of filtering. This led to users getting drowned in unwanted e-mail which only exacerbated the existing load issues. This is one of the first things you need to do, figure out why your current system isn't working properly. You'll be better equipped to fix the problems when they've actually been identified.
HEC Montréal also went for heavy redundancy and specialization. Instead of a handful of servers sharing all of the tasks equally each node in the cluster has its own job with every class of job having a backup server. Every job is going to take a beating with so many users, even if only a fraction of them are using the system at any given time.
I'd say the most important part of what you're doing will be modeling your current use. Are you getting a ton of traffic from viruses and worms spreading over your internal network? Do you get huge amounts of spam traffic to users? In such cases filtering at your SMTP servers will relieve the rest of the system from extraneous traffic. While you might need really beefy external SMTP servers you won't need nearly as much storage space on a SAN or NAS.
Re:Split up the tasks (Score:3, Informative)
Good point. One thing to be aware of when using Maildir, though, is that since each message is stored in its own file you'll have to make sure you configure your filesystem so that it can handle holding a massive number of files/messages. If you configure an ext3 partition with the default number of inodes, for example, then with one inode per message you might find yourself running out of inodes before you run out of disk space.
Try Backup Exec for Single Mailbox Restorations (Score:2, Informative)
Army Knowledge Online does it for 1.72 million use (Score:5, Informative)
Re:Simplicity is key. (Score:3, Informative)
Exim: I've tried the rest, Exim's the best.
MD4: You do it once, when the account is created, and put the location of the Maildir into the LDAP directory. No CPU hit.
Spam/ClamAV: I've found separating this stuff out makes it worse, not better. Having all the machines equal, and having lots of them, seems to work better. Don't ask me why, I'm not a professor at this stuff, I just know what works and what doesn't.
Disk images: Don't do it. Its a dark road. I use Fedora Core 4 and Kickstart. I build RPMs of everything, including configs, and build it all with the kickstart. You could do something similar with Debian if that's your poison.
NFS: NFS is good. Get a fast NFS server and you won7t have problems. Use gig for the interconnect. SAN based Global File Systems are not their yet. They are too buggy and unreliable.
IO: CPU does help a lot, actually, if you're doing the spam/antivirus thing. If you don't do that, then fine.
Hardware load balancers: Foundry kit is trustworthy. I've been using their stuff for years and never had any major problems with it. I've got ServerIrons that have been running for 3 years without a reboot and without a problem. The key is: understand how they work, and you won't have any problems.
Re:Qmail!! (Score:5, Informative)
365 days * 24 hrs/day = 8760 hours per year
0.1% downtime = 0.001 downtime
8760 * 0.001 = 8.76 hrs
You're off by two orders of magnitude.
8.76 hrs / 12 months = 0.73 hrs/month = 43.8 minutes/month
One 45 minute scheduled downtime (assuming its scheduled) per month isnt terrible. It's not great, but costs really start to go up as you add nines beyond those 3.
Re:Qmail!! (Score:3, Informative)
Re:Qmail!! (Score:3, Informative)
Check your match before telling him he doesn't know what he's talking about.
It's 8.76 hours of outage a year.
Try this... (Score:2, Informative)
First, the most important is the backend storage.
- I would try using a SAN for storage, like a small Clarion for example. I would carve the storage for the mail there on a volume.
- I would create a set of export servers that would connect directly to the SAN and re-export the volumes to a set of front end servers using a combination of gndb, gfs, etc...
See this document:
- http://www.redhat.com/magazine/008jun05/features/
- configure a set of servers that would act now as the mail servers themselves (frontends). I would strongly suggest using maildir. CourrierIMAP for the pop3/imap accounts is great. Install this on all the machines. For the SMTP agent you could use courrier but I usually prefer Exim.
- run both the IMAP/POP/SMTP servers on all the servers, using maildir only.
- use a mysql database to store the users information (passwords, email addresses, etc...). You might want to configure 2 mysql servers. One as the Master slave that will receive only the writes, and the other that would be accessed for read and balanced with the first one as reads to access user information and accounts will probably be 99% of the database activities.
- use a load balancer to put in front of all the frontend servers, do a load balancing for all the services (POP3/IMAP/SMTP) with sticky session that will try to keep the same users on the same machines when they try to download their mail.
When you are running out of capacity, simply adds new frontends, put them behind the load balancers and voila...
of course I would advise going right away with powerfull 2x3.6GHZ P4 servers and like 4GB of memory. That is powerfull and can certainely serve a LOT of users already per server.
my 2c, written quickly. I apologies if not complete but I am pretty sure the general idea is there and sound.
open to comments
Re:Obviously - RTFM!!! (Score:2, Informative)
Re:go to gmail (Score:3, Informative)
Really want to know why the client isn't as slick? (Score:3, Informative)
It has this thing called a seperation layer. All the code except the ui is the same on all the platforms. Clients used to be for os/2, mac, win16, win32, and solaris. Client side that got scalled back because nobody paid for the others -- client is win32 and mac now -- soon with code under linux as part of the next generation client. Lots of people are using on Wine.
Now, the server is still cross platform. Win32, Linux, Aix, iseries (as/400), zseries.
The problem with making something cross platform is, you don't use all the nifty little Windows specific integration and custom pretty things. You don't get something for nothing -- you have to make all those bits.
Oh, the other thing? Outlook feels integrated because everything automatically does the windows automatica launch active-x thing. Just highlight a message subjet, bingo! Embedded code launches! that's why viruses and worms.
If stuff wants to run in Notes, it has to be have a signature. OHHH, public/private key signatures and encryption. When? 1991. Hunh? Yeah, since 1991.
If something wants to run in Notes -- It need PERMISSION to run. Thus, no viruses or worms unless you're stupid enough to tell them "OK, sure, go screw up my machine".
Yes -- the development environment is weird and pretty unsophisticated. It takes a lot of time to learn because its not like other things. BUT -- I can make it do cool, secure, reliable things at a tenth of the cost you can in J2EE or MS
Excited about JSR170? Ah, me too. The Notes database internals match it almost perfectly. Domino will make a great JSR170 back end. Hell, its almost that already.
Meantime, you trolls are whining about a product that runs in Linux as a server and (using Wine) as a client. Runs on Mac. Has a fully functional JAVA environment for development and a remote API through CORBA and DIIOP. No no, instead you'll use a proprietary only -- Windows Only, Active Directory Only, Virus Distribution Engine from Microsoft.
ahahahahahaha. Enjoy it!
Re:You are wrong in every way. (Score:2, Informative)
I'm not arguing that relational db's are the way to store everything; I'm totally about the right tool for the job. But file systems are good for storing files, they're not intended for the level of data updates (new files / deleted files) that a high use email server generates. Databases are. Also disk writes from databases are also optimized if your database is well designed and resistant to paging. If you don't want a RELATIONAL database, fine. There are other types of databases you know. Mail servers don't have anything in common with file servers in terms of resource usage.
Re:Obviously (Score:3, Informative)
The question is bogus. Hypothetically..... (Score:3, Informative)
If it were me starting from scratch -- the model for a million uses is the internet itself. SMTP, DNS, and mabe a big LDAP directory tool. For calendaring, you're SOL, but nobody calendars with a million poeple. That's meaningless. Calendaring is only useful at the workgroup level anyway. Look to any good workgroup calendaring tool and let users define thir own working groups.
Now, backing off the big million user stupid number. In the real corporate world, you have two real players and a ton of also-rans. The two real players are IBM/Lotus with Notes and Microsoft with Exchange.
The market is split roughly evenly. In the US Microsoft leads a bit, in Europe and EMEA IBM/Lotus leads. How much and actual numbers are hard as hell to track down. IBM doesn't release them and Microsoft likes to count every copy of Office as an Outlook seat. Suffice it to say both companies own about a hundred million actual users.
The basic trade off between the two - With Exchange you get tighter integration with Active Directory and smooth look and feel integration on windows. It feels like all part of the operating system. On purpose. On the other hand, you're forced to use Active Directory, forced to use Win32, and all that integration without any real security means viruses are unstopable. With Notes you get a bulky client that many users find hard to understand. You also get almost 100% prevention of virus spread (it has built in security) and other goodies. Its also a development platform and its cross platform. The client is Win32 and Mac, and users have writen howto docs for WINE. The server is linux, win32, AIX, ZSeries, and iSeries (as/400).
You may not know this, but BOTH can use the Outlook client. Yes, the outlook client is supported with a Domino mail infrastructure. Who'd have thunk it?
Oh, and Domino supports other mail clients too. Pop3, IMAP, and a very good Web Browser -- all at once for the same person if you like. Its got native SMTP support, as well.
What Notes isn't, its pretty. Most people say Outlook is prettier. Ok. Easy to do if you own the OS and make software that only runs in one environment.
So, I hear rants about Notes. I hear trolls whining about a product that runs in Linux as a server and (using Wine) as a client. Runs on Mac. Has a fully functional JAVA environment for development and a remote API through CORBA and DIIOP.
No no, instead they'll use a proprietary only -- Windows Only, Active Directory Only, Virus Distribution Engine from Microsoft.
You gotta love that. Why? Well, its pretty.
Google It! (Score:2, Informative)
Re:Split up the tasks (Score:3, Informative)
However sendmail isn't one of them. Just because there was an issue with m4 (also something that should be gone) doesn't mean the core app is broken. M4 use started over a decade ago when even awk wasn't consistent on all unix systems.
On one setup that has both sendmail and postfix, I know postfix loses far more mail than sendmail (which has lost none).
I use sendmail because I can make it do everything I want it to and sometimes I have to have an MTA that does odd or unusual things. I've spent time and learned how to make it do very unusual things (at the
Also a complete rewrite of sendmail is being done right now. Too bad its taking away all the cool low level macros but I expect most people will find that an advantage.
Re:Obviously (Score:4, Informative)
Re:Obviously (Score:2, Informative)
Re:You are wrong in every way. (Score:1, Informative)
I would disagree with the filesystem approach as high availability and disaster recovery aren't as good. If you think the relational database approach won't work, you're not using Oracle 10g RAC.
Billion row tables are common. Deletes on these tables aren't an issue as you're deleting by primary key and you can store that in an index-organized table partitioned however you want. I manage TB size database daily, I know.
Using a real relational database (not lame ass mySql) you can take advantage of all of the high availability & disaster recovery features of Oracle too, like backups, RAC and standby databases.
Re:Obviously (Score:5, Informative)
Re:You are wrong in every way. (Score:3, Informative)
Yes, there is. Try some kerneltrap articles to learn more about Linux (and OS internals in general
2^32 = 4 "GiB". That's all you can address with 32 bits.
On ia32, Linux allocates 1GiB of virtual address space to the kernel, and the remaining 3GiB to user space.
Thus, the maximum amount of physical memory that can be mapped to a stock ia32 kernel is 1GiB.
There are awkward things you can do at kernel compile-time to get more than 1GiB accessible to the kernel on ia32, but it's not as pretty as you seem to be thinking.
Re:Obviously (Score:3, Informative)
http://asg.web.cmu.edu/cyrus/download/imapd/overv
http://doc.powerdns.com/powermail/indepth.html#AE
Re:You are wrong in every way. (Score:3, Informative)
There are awkward things you can do at kernel compile-time to get more than 1GiB accessible to the kernel on ia32, but it's not as pretty as you seem to be thinking.
Well, if setting CONFIG_HIGHMEM=y counts as "awkward". The kernel docs say that's the correct setting for machines with between 1 and 4 GiB of RAM.
From my laptop (with 1.5GB RAM and a Linux 2.6.13 kernel, with CONFIG_HIGHMEM=y, which is actually the default setting on most distros these days):
1228596 KiB == 1199.8 MiB == 1.1717 GiB
So my kernel is currently using more than 1GiB for caching disk storage. In fact, my kernel can address up to 4GiB of RAM. I have another 1GiB DIMM on the way (which will push my laptop to 2 GiB RAM), so in a few days I'll be able to show my machine caching around 1.7GiB. (Yes, there is a reason I need 2 GiB RAM in my laptop, and it's actually not file caching).
For machines with more than 4GiB of RAM you have to use PAE. That will allow the system to use up to 64GiB of RAM, but each process (including the kernel, even though it's not really a process) can only access 4GiB. So, your argument holds some water in the case that:
In that case, the database can cache more than the kernel. I'm not aware of any database engine that has such cache daemon processes. IMO, if you're putting more than 4 GiB in the box, you should probably go ahead and buy an Opteron for it also, avoiding the whole issue.
Plain Text (Score:3, Informative)
What you need to do is support STARTTLS for these protocols. That lets the client connect then negotiate an encrypted connection with the server before sending passwords. It's easy to configure the server to refuse to authenticate the client unless an SSL session has been set up if that's what your security policy dictates. It's also possible to have the server demand a client certificate from the client before setting up the SSL connection, adding an extra layer of authentication.
You'll probably also have to support the old IMAPs, POP3s, and SMTPs standards, but they should be considered deprecated and only in place for crap clients that don't know about STARTTLS.
Backups (Score:5, Informative)
With POP3, the client downloads mail and deletes it off the server. Without a significantly butchered POP3 server there's no way to hold copies of that mail for a period of time (say, to ensure it goes on to your archival tapes, or to make sure you can recover files the user deleted accidentally). It's one less thing to worry about if their workstation / laptop dies, too - just give 'em another one. If more mail clients supported LDAP address books and WebDAV calendars this would be even nicer; as it is I still have to keep their mail folders in their network home dir so I can back up their address book.
You can back up POP3 boxes if you're on a corporate network, by forcing the client to keep its spools on the user's homedir. That tends to be slow and inefficient, though, and it doesn't let you do things like transparently split out attachments and store only one copy of an identical attachment for everybody.
It's also easy to lose mail with POP3 if your client does something silly. Most clients seem pretty decent now, but I remember old Eudora versions used to DELE mail off the server then crash, corrupting their mailboxes. Woohoo.
IMAP gives admins much more control over user mail. You can back up their mail folders, including their outbox and filed mail. You can enforce mail lifetime limits if your information retention policy requires it. You can store single copies of duplicate messages and attachments. You can give users access to shared mailboxes, and to each other's mailboxes where necessary. You can manage their mail folders remotely ("I can't delete $message, help!"). You can set up filters that deliver mail into sub-mailboxes automatically. Good clients automatically sync the IMAP mailbox so it can be used when the client is offline, like POP3. You can have your anti-spam software learn from their mail client's Junk folder. It's just much saner for business environments, in much the same way that network home directories and thin clients are much saner than a bunch of desktops with local storage are.
IMAP also permits you to give the user a single view of their mailboxes from their desktop and when they're on the road, or accessing their mail from home. Don't even talk about "leave mail on server" for POP3 - users WILL misconfigure it and suck all their mail down onto one of their machines, then come to you looking for help cleaning up the resulting awful mess.
Now, for an ISP, things are the opposite. You want to get the users' mail through your system and get rid of it. Most ISPs only offer POP3 and have small mailbox caps, so the user can't set their client to never delete mail off the server. They don't want to be responsible for user mail, they want it off their hands ASAP. An ISP can just tell a user who deleted a message then wants it back "well, that was silly then wasn't it?". An ISP doesn't want to back up 5 years worth of mail for 500,000 users.
My point is that for corporate environments IMAP is so superior that it's almost nuts to offer anything else, but for an ISP POP3 is a much more viable option. So what's so bad about POP3 depends entirely on what your needs are.
here's a mail from someone doing this on cyrus now (Score:1, Informative)
> > > plan to put that number in production in a couple of months here.
> >
> > Ouch, 750k? How many concurrent accesses?
> >
>
> We currently have 1.6M, 1.2M and 940k mailboxes in 3 boxes with fiber to a single emc storage, all boxes
> dual Xeon 3.4Ghz EMT64T with 4G.
We tend to have quite large mailbox lists, but not as large as this. The biggest issues we've found with large mailbox lists are:
1. Number of concurrent connections.
If you support/encourage IMAP usage, then you tend to end up with quite a few more connections than POP.
Although technically IMAP can be very long lived, we find there are lots of short connections (mostly due to things like Outlook Express which when doing a "sync" pass does a logout and login for each *folder* in a users account!) and some long ones. With about 650,000 folders on one machine (about 130,000 users) and at peak times we see about 3500 imapd processes. We use linux 2.6, and find that this is a good number of maximum processes to have. Although the kernel is just about O(1) for everything these days, we find that there does seem to be a bit of an elbow point around the 5000 process mark where things just seem to start showing higher latency and average loads on the server
2. Size of mailboxes.db file
With a large mailbox file, you probably want to use the skiplist format. Part of the implementation of the skiplist db however is that the entire file is mmap'ed into memory. While this is generally fine since each process shares the same mmap file backing, with really large mailboxes.db files you can end up with just huge page tables.
For instance, the above 650,000 folders mailboxes.db is about 100M is size. With pages being 4k each, that means each process needs 25,600 pages just to mmap that file into it's process space. If you have > 4GB of RAM, you have to use x86_64 or PAE mode in linux. Both of these mean that each page requires a 64bit page table entry (8 bytes). If you have 3500 process then...
3500 * 25600 * 8 = 716800000 = 683M
Yes, that's 700M of memory just to hold the memory map of all your processes, no actual real data at all!!
This also means that you MUST use the high-PTE option in linux, or else you'll have lots of low memory pressure.
3. IO
CPU isn't an issue. IO definitely is. Cyrus uses minimal CPU on todays hardware, but it still is an IO hog.
That's part of the reason we sponsored the meta-data split patches that have gone into 2.3 so that you can separate out the email store part and the cyrus.* files onto separate partitions/spindles to improve overall performance. Where possible, split out:
user.seen state files
quota files
cyrus.* files
email spool files
Onto separate spindles/partitions. At least that way you'll be able to use something like "iostat -p ALL 120" to see which parts of your system are generating the largest IO.
Re:Stop right now (Score:3, Informative)
The main challenge when I was doing it 5 years ago (I designed and wrote most of the prototype of a free webmail system, and managed the development team that completed it) was lack of good open source webmail solutions and lack of scalable mail storage systems, and hardware limitations.
Today there's a huge number of GOOD IMAP based webmail packages, such as IMP, and mail storage isn't much of a problem anymore - you can get a couple of TB of storage relatively cheaply.
Today, if I was going to do this in a corporate setting, I'd buy 3-4 small cheap servers to process inbound/outbound mail, 2-3 reasonably high powered machines with good IO capacity and RAID5 to split the users mail storage, POP/IMAP access over (IO is more or less the ONLY thing that really matters - whenever you need to make a choice, always choose higher IO capacity over almost anything else), 2 machines for an LDAP directory of which server the user is on, 2-3 cheap servers to run the web frontend on.
All in all for that kind of scale, if your total cost pans out to more than 20-30 cents per user in hardware these days you're doing something very,very wrong (and you can manage for MUCH less depending on usage patterns of your users and how much time you're willing to spend on tweaking the software).
Re:Army Knowledge Online does it for 1.72 million (Score:3, Informative)
Hula (Score:2, Informative)
Hula [hula-project.org] claims to scale pretty well, integrate with ClamAV and SpamAssassin, and have lots of other cool gimicks for calendars and such. For 1 million accounts, I'd get some sort of dedicated [barracudanetworks.com] spam/virus filter, though.
Re:Army Knowledge Online does it for 1.72 million (Score:2, Informative)
Re:For the lazy... (Score:3, Informative)
1. Storage - \Disks, lots of Disks\ - we use EMC DMX3000's for the stateful machines (~180TB raw) which work very nicely.
Your back end needs to handle lots of small random writes - this makes storage vendors cringe when mentioned, as it makes a mockery of their lovely benchmarks.
2. Clustering - you'll need that also on your master directory and message stores's. Veritas is nice.
3. Load balancing - For the front end boxes (pop, imap, web). Cisco CSS's are pretty good for this.
4. OS - We run Solaris. It might not be the fastest thing around, but it works pretty much non-stop; has good vendor support and is very mature. RedHat might be on the horizon as well as Solaris for x86. Windows? don't be daft.
5. Test environment. Have a scaled down exact copy of the production system to test things on. i can't stress how important this is.
6. Proper automated server build procedure. One word - Jumpstart. All OS and application configs and builds in Jumpstart. So if you loose a box, it's no big deal about rebuilding it at 3am on Saturday morning when you've had a bevvy or two the night before, and all you feel like doing is chundering (i speak from experience - a SunFire 6800 does not respond well to projectile vomit)
One correction of Slideys post, we now have 16 brightmail boxes (10 in, 6 out) and it's not enough.
Cheers.
Enterprise-wide scalability (Score:1, Informative)
You never mentioned what your RTO and RPO were. If you can lose 24 hours worth of data, there are fairly standard methods. 12 hours is doable. Less than that and you need to spend a ton more money. SRDF/RA is interesting when you get down to the 5 minute area and don't want to write across the WAN for both locations.
Probably the easiest solution is to get 4 mainframes, 2 per site, create linux partitions on them and use some commercially supported MTA. Use all the mainframe replication facilities to do the remote replication daily.
Or you could use email like it was meant with federation and each dept or location having their own local server.
Don't forget about spam filtering, SOX compliance, and automatic encryption of external communications. IronMail merged with a PGP product can do this. The free PGP implementations make the data the individual's, not the companies. I'll just say that commercial PGP has "other solutions available" so the company still can get access to encrypted information.
NMCI Blows (Score:4, Informative)
When it works at all it's slow. Sometimes you can hit the Send button and just sit there and wait a while.
When we have to work on a Navy project we had to start bringing our own equipment and hubs. Even their developer machines come loaded with 10 year old software and you can't get your email and be logged in as a developer at the same time. To check mail you have to log out, log back in under a different account, then log back in as a developer. The NMCI machines are boat anchors.
NMCI is the worst defeat the US Navy has ever suffered.
Cyrus (Score:2, Informative)
Re:Qmail!! (Score:2, Informative)
1 yr = 24 * 365 h = 8760 h
99.9% reliability => 8760h * 0.999 uptime = 8751.24 hours uptime or 8.67 hours downtime similarly 99.99% leads to 0.867 hours downtime = 52.56 minutes
you're off by one magnitude!
Re:CommunigatePro from Stalker.com (Score:3, Informative)
This so called time bomb applies only to FORMER customers who upgrade without a current license. Sounds fair to me.