What Setup Is Best For Large Mailing Lists? 19
Super_Frosty asks: "I run a large mailing list that has expanded beyond my means. Perhaps you have heard of it - its headquarters is at
this site. I currently have about 10,000 subscribers, which I am running from a small group of sad Macintoshes. I can hardly send out the daily message anymore.
Anyhow, I can't find anybody who knows how to administer a list this large. I am beginning to have embarassingly close calls with people flooding and spamming the list due to security problems. My hardware can't handle it, and most software packages aren't meant for it, either. Can anybody reccommend a hardware and software solution for this list? Surely, some Slashdotter has administered a large list - I hope!"
More information, please (Score:1)
qmail + ezmlm (Score:3)
Every other list I'm on, run on a variety of setups (the most common appears to be sendmail + majordomo), has really poor latency of around a couple of hours. For a daily email that might not be so bad, except to realise that the latency transfers directly into list delivery speed.
There's not much more to say about it. Try it - I never regret it.
Don't really need list software? (Score:2)
If the subscribers do need to have discussions, then I would suggest qmail + ezmlm. If I recall, it too allows lists to be split amongst many locations, albeit with more trouble.
Now, none of these suggestions are going to help if the problem is bandwith, or anything other than server speed/power/etc. What exactly seems to be the bottleneck?
sendmail + SmartList (Score:1)
I use basically the same software to do a job very similar to what you're doing, just not on a daily basis... more like every 2 or 3 days. I've only got 3Kbps of bandwidth available to it; and it finishes 10K emails to lists of 10,000 to 15,000 addresses in under 12 hours, usually.
Hardware isn't that hard (Score:2)
Mailman has a great user and admin interface (all web based, very intuitive to use), but it gets kind of hard to admin once the mailing list reaches any size at all (it's kind of a pain to work with only 150 members on one list I admin). I don't know much about MTAs; sendmail seems to work OK but I've heard that postfix or qmail are faster. YMMV.
Running big lists (Score:1)
I built a large scale mail cluster for our company. We send mail to ~1M addresses daily.
MTA doesn't really matter that much, I've found; people will argue with me here, but sendmail is fine, and I use it becuase I know it best. What is important for bulk outgoing mail is an optimized injection agent - the beast that feeds the MTA (sendmail or what have you). Initially, we used bulk_mailer, but found it didn't scale well for really big lists - it would choke a machine. What we ended up doing was writing a limited MTA in perl that talked to a list of mail servers, sending one message to a mass of people (still tuning, but about 1000 seems right) - 1000 RCPT to: lines, one message - then switching to a random mail server and repeating. This might not be clear - we have a client/server cluster, one master sends mail through a lot of slaves. We do several different lists, so the master has many instances of the custom script running, each picking a slave at random from which to send mail.
Slaves can fall over and they'll just be skipped.
The master is still a single point of failure. I'm hoping to modify things such that they all randomly get lists and use eachother as slaves, but getting file distribution right in the face of a dead host is still a problem. Partial failures are a HUGE problem (if a machine blows up, how do you ask it how many people it sent to?), as is centralized logging. Even knowing when something is going wrong is hard... Mass mail is difficult to debug. Plus, I tend to have other things to do, and this works fine, now. I just want it perfect...
As a benchmark, we send about 7 million messages a week. Delivery starts shortly after midnight; everything is done by 5AM.
I haven't gotten in to handling bounces and bad addresses yet... let alone forged subscriptions. Suffice to say we have a complicated procmail config and a script that talks to mysql to record delivery failures, and a complicated transfer/update arrangement to update our oracle-based master list when too many delivery failures happen... and a blacklist for forgeries.
Whew. Running mass mail servers is a mess. I don't wish that was all I had to do, but if it were, I could probably get it right.
-Abulafia, not logged in.
Re:Don't really need list software? (Score:1)
i'm not really sure what to do, but please email me at security@63.225.139.3 so that we can have this conversation in provate.
Postfix + mailman (Score:2)
Re:Running big lists (Score:1)
Sendmail + Majordomo (Score:2)
Neither myself, nor the guy who actually owns the machine, are sendmail experts, so our problems stem from sendmail taking so long to deliver messages to places having trouble recieving mail. Creates a small backlog, but messages are delivered to subscribers in under an hour usually. Subscription requests and admin things by email usually take under a minute to get confirmation.
We also keep archives using MHonArc and glimpse, searching is a little slow, but that's to be expected considering the machine speed, and tens of thousands of messages.
It really isn't a bad setup, considering all the software is free, the machine was sitting in the basement collecting dust, and it's run off a static IP cable modem.
qmail + ezmlm (Score:2)
When I last checked, Hotmail and Yahoo mail used qmail. I doubt your load is more than theirs.
For discussion style mail lists you can use ezmlm. But if it's just one way, one to many broadcast, just qmail will do.
However if the problem is bandwidth, then you need to either get a larger pipe, OR find someone willing to redistribute the mail for you- e.g. you send them a single email with tons of addresses and they handle the delivery. Do NOT use someone's mail server for that without their permission.
Good luck,
Link.
PetiDomo (Score:1)
http://www.cys.de/deutsch/produkte/index-petido
The previous release PetiDomo II has been released under the GPL. Their website is in German, but if you send them an email they should be able to help you in english.
Good luck,
Heiko
lyris? (Score:1)
Re:Don't really need list software? (Score:1)
I do need some kind of list software - I can't bcc: the joke to a thousand people using microsoft lookout or Eudora!
Anyway, the subscribers "don't" get to post ot the list - but I've done experiments, I've heard from other people - my list package doesn't have the best security, and it might be possible for an unauthorized person to send a message to the list. Which would be bad.
I'm running it on a 90MHz Macintosh (yes, it sucks). The package I'm using - I probably shouldn't say what it is - but it's meant for a list of about a hundred people. It sucks.
Besides the potential security issue, it is damn slow. There are barely enough hours in a day to send out all of the messages. I need more speed.
I do have a very fast DSL pipe. It's just a hardware and software problem. Basically, if you had a list, what s/w and h/w would you use?
P.S. I just went to Best Buy today and bought Red Hat 6.2. Woo hoo! So, linux solutions are now acceptable.
Thanks for the help!
Bounce handling (Score:1)
I definitely second the choice of qmail + ezmlm[-idx]. (-idx for the moderation features) ;)
But for a list of any size you probably want the software to handle subscriptions and bounces automatically and securely, and ezmlm does just that by sending cryptographically secure confirmation requests and handling bounces by encoding the subscribers address in the Return-Path.
See http://cr.yp.to/qmail.html [cr.yp.to] and http://www.qmail.org [qmail.org] for more advertising.
If I were you.... (Score:2)
Say for example Listbox [listbox.com] or there's always Egroups [egroups.com]. My own company's mailing list services [catalystinternet.com] aren't really ready for prime-time yet, but I do operate the announcements-only mail list for the Philadelphia Eagles, and that is about 11000 subscribers although it is one-way only.
--
18,000+ subscribers with RH5.2, qmail, Majordomo (Score:1)
One of the lists was similar to yours in that it was used for broadcast and not really for discussion. We sent out a computer security newsletter once a week to about 18,000 subscribers. I think when I left the company the total had risen as high as 20,000. This same machine also handled several discussion lists. Although they were small, one of them was quite active and had about 7,500 to 8,000 subscribers.
My experience has been that for lists this size you don't need a fast machine as much as you need RAM. Also, Sendmail doesn't seem to be as efficent as qmail. There was a noticable increase in delivery times once we moved to qmail from Sendmail. For the 18,000+ subscriber list, I kept my personal email address at the top and bottom of the mailing list addresses and there would be about a four hour delay between receiving the first message and the last one. Not too bad. I don't remember the times for Sendmail but it wasn't quite that fast.
Re:If I were you.... (Score:1)
Re:Don't really need list software? (Score:2)
Since you are rightly worried about security, I'd recommend going out and reading up on everything you can. bugtraq archives, dejanews, risks archive are a few of the places to start. You've obviously put some time and effort to grow a lame joke list to 10k subscribers, so put some effort into protecting it.
I noticed your emails coming into my
the AC