Replacing SMTP? 539
dousette asks: "In reading over one of the RFC's governing the SMTP protocol, and other RFC's as well, it's interesting to note that you see some big names and big companies from time to time. With all the loopholes in the current SMTP specification, is it possible for the Slashdot collective to come up with another one? Would it stand a chance in making it into a standard, or do they just listen to Cisco, AT&T, etc? I realize that a lot of people have a lot of ideas how things should be done (and they haven't been shy about posting them to Slashdot), but has anyone tried to write the RFC for a replacement protocol? As a side note (where I won't be shy about posting how things should be done), if there were a replacement trusted protocol, one could have mail received via that protocol bypass spam filtering, id checking, or whatever checks might be in place (saving processor cycles, etc). The regular checks could still be done on other mail received via the 'older' SMTP protocol. If more and more ISP's make use of this, SMTP could be gradually phased out... or if you are one for a sudden cut-over, just cut to the new one at the same time as the IPv6 upgrade!"
What a silly article. (Score:1, Insightful)
Do we really need a new standard? (Score:1, Insightful)
slashdot (Score:5, Insightful)
Not a chance. The slashdot collective taken as a whole, is a very stupid group of people. Even the few intelligent people wouldn't be able to get anything useful done because they'd be shouted down by the teaming masses of idiots.
We hate Sony's recording arm, but we'll sell our souls to them for the next cool gadget. We hate MS, but 90% of us use windows on our main home machine. No to mention all the idiots who use words like boxen.
Re:Check out Internet Mail 2000 (Score:5, Insightful)
I wish people would stop inviting rate increases or new charges as an answer to spam. It's not the answer. It might be inexpensive for you, but many of us DO send a lot of email and it'd get expensive really quick. You'd get rid of a lot of good and valid email communication along with the spam.
I'm even opposed to the "pay a dime, but I'll give it back if I wanted to hear from you" approach. Those of us running a mailing list would run the risk of having some idiot sign-up a bunch of accounts only to have that person say "No, I didn't want that" and collect the money.
I believe we need a trusted protocol. This might be as simple as having all emails PGP signed and everything else being sent to the bit-bucket (if you want to be aggressive) or only passed through to the user if the unsigned message had an extremely low spam score.
But if everyone were to use Bayesian I swear we wouldn't even have to propose a new protocol, talk about new legislation, etc.
*SIGH*
Think how many devices (Score:5, Insightful)
Doesn't mean it shouldn't be done, but don't be fooled into thinking it's just a "propose a new spec, step 2?, profit" type of deal....
--D
Re:Check out Internet Mail 2000 (Score:2, Insightful)
you don't realize that IM is a form of email? you are just sending packets of text... the second someone charges for SMTP, i'll just run my own. you could just charge the end users for data transfered instead of flat monthly fee, but most wouldn't go for that.
Re:What a silly article. (Score:5, Insightful)
Re:Check out Internet Mail 2000 (Score:5, Insightful)
With the current system, an smtp server can go down, and no one would notice because no one was received their email yet, but with im2000, if the sending machine goes down, then no one can read their mail from there. This would create a lot of unknowns, "why can't I read my email?". Also what about people that don't have a full on connection, you don't want to require those people to be connected just to read their mail. Sure you can queue it for downloading offline somehow, but that's going to be much slower than normal because you have to connect to say 30 different servers where your email is hosted.
Also there's the case of somesmallcompany.com sending out a mailer/advertisement to millions of people, because the email is hosted on their machine, their connection/server might become overwhelmed, causing heaches for everyone wanting to read their mail. "Why does my mail load so slow?"
It's a nice try, but it'll never work.
Another thing, what happens when the message is done being read? Is it deleted on the sender's machine? If so, then how will the user remember that they sent the email to check if it's been read. If not, when will the message get deleted? Obviously it can't stay there forever.
The great thing about the current system, is that you just send and forget. If it bounces, you get a new email message saying hey, something went wrong. But with im2000, if the message hasn't been read yet, WHY? Did the user just not check their mail yet? Is there connection/routing problem where they suddenly occurred after the hosting server sent the notification, etc.
Answer: No. (Score:1, Insightful)
Can we, as a community, do that? No. Why not? Because it's really, really hard.
The IETF, as a community, is not dumb or lazy. (Some of us individuals are.) Many contributors to the IETF read Slashdot, as a matter of fact. But the reason spam is a problem is not that SMTP is flawed; it's because Internet email is successful. It's successful because it ignores the problems of authentication and authorization and just lets anyone send mail to anyone else.
Would a new protocol (or an SMTP extension) fix the problem? Of course not. Spam happens because you typically want people you don't know, with whom you don't have a relationship, to be able to send email. It's easy to solve the spam problem: it's a trivial special case of either a public key system, general text classification algorithms, or micropayments. We've been waiting for each of these for decades, but I'll leave it up to you as to which one you want to solve first.
The ideas mentioned for a "trusted" protocol do not require a new protocol. SMTP, perhaps with extensions, can be used to handle the vague ideas in this story.
SMTP over TLS (Score:5, Insightful)
Working out the details of an appropriate certificate policy is not trivial, though.
Waste of time and effort. (Score:2, Insightful)
It's simple. Don't bother.
The problem will remain, it will just shift tactics. By 'fixing' SMTP you're not addressing the problem, you're addressing a symptom of the problem.
Anything we do on the technology side to fix this problem will ultimately do nothing.
That's not to say that SMTP can't be improved on... but improving on it purely to 'stop spam' is a waste.
I have been working on another one (Score:5, Insightful)
Actually, I've been working on a broader based piece of infrastructure than a new mail protocol, but the first problem I intend to attack is mail.
RFC 822 is fine for messages, but the transport needs a big upgrade. Also, envelope senders and receivers are non-verifiable, and therefor broken. One day, spammers are going to start using mailing lists and message boards to construct a profile of people you talk to, and send you mail that appears to come from them, thereby making whitelists useless.
The basic premise of my general transport is that all messages are addressed to a public key and come from a public key. All messages are signed by their supposed source ID, and most messages are encrypted to the destination ID.
A public key ID plays a similar role to an IP address in an IP packet. There will be distributed databases that hold (signed) mappings between public key IDs and their locations using other networking mechanisms.
I'm trying to design this protocol and its implementation so its easy to encapsulate it in almost anything. My first connection to an outside protocol will be IMAP/SMTP.
It's far from being ready for even a public alpha yet, but I do have preliminary code for creating certain kinds of messages at https://svn.generalpresence.com:5131/repos/trunk/C ++/pract_crypto/ [generalpresence.com]. I'm borrowing heavily from Bruce Shcneier and Niels Ferguson's latest book, Practical Cryptography. The initial implementation is in a mix of Python and C++. It requires Swig and the GMP library. I haven't designed the implementation itself to be in the least robust against attacks by someone who has root on your machine.
I am calling the protocol 'CAKE' for now. CAKE stands for Key Addressed Crypto Encapsulation. It is a layered protocol, since I intend it to be layered on top of any other protocol you can think of. :-)
One intention of mine is to publish a hash collision problem along with information mapping a public key to a mailbox. First time senders will have to solve the hash collision problem to avoid having the mail thrown away. I'm planning on simply wrapping an RFC 822 message in a CAKE shell.
Re:Will receive email for work. (Score:3, Insightful)
Most examples of "takes a second or two" are very processor dependent. You'd then also have the problem of running code on another machine, DOS attacks, all that fun.
Re:Will receive email for work. (Score:3, Insightful)
It might work to allow the reciever to have a whitelist of addresses -- so that you only have to do the work if you are sending to someone you don't know. Still, I sure wouldn't want to have to let my computer crunch numbers for five minutes just to send an email to someone I had never contacted before.
However, as far as I can tell, this is really no different than the pay-per-email solution, but less straightforward.
Re:Check out Internet Mail 2000 (Score:2, Insightful)
The key words there are "a couple of years ago." Yes, a few years ago he published that web page. And you know what happened as a result of that?
Absolutely nothing.
Why?
Because it makes no sense whatsoever.
Bernstein came up with a lot of stuff over the years which did make sense, and which did take off, and became accepted. This isn't one of them.
So what if the mail's contents are now stored on the sender's system? How's that going to prevent a spammer from keeping a copy of the spam stashed somewhere, and then spamming a few hundred million mailboxes with a link to the spam. So, what have we accomplished here?
The problem with spam is not where the bits and bytes that make up the spam are physically stored. The traditional definition of spam is "unsolicited junk E-mail". Jiggering the bits around, and saving them in a different directory, or on a different server, will not magically turn unsolicited junk into solicited E-mail.
Re:Difficult Problem (Score:1, Insightful)
In peer-to-peer comunication (which is what email is best for) privacy is really what is important, and privacy and authentication can go hand in hand.
Can anyone think of some good examples of where both privacy and anonymity are required at the same time.
Re:What loopholes in SMTP? (Score:5, Insightful)
In particular, lack of authentication is a strength of SMTP, just as it is with IP. It means, for example, that I can implement my own authentication (or plug in PGP or whatever), and don't have to use the mail-transfer layer's after it turns out to have a serious hole that lets the spammers and con-men through.
Protocols that try to do everything for you have the inherent problem that, when a serious problem arises, you have to put up with it until the idiots at the vendor decide to solve it.
SMTP is simple enough that even a relatively incompetent programmer can do it correctly. You can type it yourself via a telnet connection.
And adding features in the higher layers is easy.
Why replace it? (Score:3, Insightful)
If you exent it two servers with the extensions will send mail in the new more secured format.
Frst things first keep it friendly you should be able to do everything via telnet because it jsut makes testing easy.
So what do you need a little bit on how to get into this new mode once your connected to port 25.
The server must offer what encryption methods it allows as a list more perfered methods first. Unencrypted should be an option all senders and receivers MUST allow it as encryption is nice but CPU heavy and you can allays depreciate enencrypted senders.
There should be a DNS entry for the sending mail servier in the domain that the from address and the reply to address originate (some new DNS field well it's a nice big distributed DB with cache so why not?) This needs more work and it sorta outside of scope.
If the sender domain is part of the servers domain of responcibility the server must use the from and replyto addresses to authenticate the user(s) passwords via CHAP, Kerebros etc this MUST be done after the encrypted state is up and can NOT allow unencrypted passwords and perferable uses a CHAP like system where the password is never on the wire.
The receiving server must include all options specified by the sending server as message headers.
The server MUST only accept mail that is destined to it's domains or source from one of it's domains if accompinied with valid credentials.
A server to server intermediary authentication may be implemented.
OK thats no where near completed but it's a start.
Re:Check out Internet Mail 2000 (Score:3, Insightful)
But making spam costly will indeed stop the majority of spam that is sent today, which is useless, annoying stuff that far less than 1/100 of 1 percent of people actually make a purchase based on.
That is the kind of spam that I really want to stop, and I think that making spamming cost money will have the desired effect.
In my mind I see a smooth graph of how much it costs to send junk email versus how many companies will send junk email. Right now we are at the end of the scale where the cost to send spam is almost nothing, and so the number of "companies" that will send out spam is HUGE, and not-concidentally the content of that spam is very poor.
We need to work our way to the end of the scale where spam costs real money to send, and only a moderate number of legitimate businesses then find sending spam worthwhile. And the kind of spam which is sent is then would be less annoying (to me anyway) as well. Spam would be more like advertising of actual products than make-money-rich schemes based on trying to sell snake oil.
Be wary of 'trusted' protocols (Score:5, Insightful)
That's not the answer either. Microsoft, Yahoo, et al have been lobbying for this approach, and for good reason. They want to function as the certificate authority (CA). They want to determine who can or cannot send email. They can use that power to literally sell the ability to send spam. They can also use that ability to censor their opponents.
Microsoft also wants a new patented standard that can't be legally implement with open source software.
Re:What a silly article. (Score:3, Insightful)
Re:not the answer - you got that right! (Score:5, Insightful)
---
Aye, there's the rub. I do the same thing, what with having about 5 domains and various e-mail addresses for each.
There needs to be:
1) A way of verifying if you're allowed to use said mail server. Easy. Simple login/password over encrypted connection - technology already in place.
2) A way of verifying what e-mail addresses & domains are allowed on outgoing e-mails from said mail sever. That would be new, but should be easy to develop.
3) A way of destination server contacting the originating server and verifying the e-mail it received is from an authorized and stated e-mail account. That would be new, too, and would be a bit more complicated, but still fairly simple.
4) While you're at it, you might as well encrypt the whole frigging process. The saying, "E-mail is not like a letter; it's like a postcard." should be obsolete. It should be like a letter written in a language only the sender and receiver can understand. By default. Every time. The technology is around, but needs to be standardized and integrated and something the user never has to set up or think about. I loved a recent commercial that said that something was really private, "as secret as your e-mail password." Yikes. People _really_ don't understand e-mail technology.
Also, a way to have a mail server respond to a confirmation request only by servers it's sent mail to recently would be a good thing - that would cut down on trying to scan a server with a dictionary attack to get valid e-mail addresses to spam to.
The problem is not that these are difficult technical challenges - they're not, and the technologies exist in fairly decent form already. The problem is getting this done in a standard and accepted way and out into the field for everyone to use. _That's_ gonna be a real bitch.
Re:Check out Internet Mail 2000 (Score:4, Insightful)
Anyway, it's just an idea. I think that the whole question of "should we come up with a new mail protocol" is kind of misguided because obviously the hard problems here are not technical, they're in dealing with the huge momentum built up around the current mail technologies. It seems to me that we already have all the technology that we need to solve this problem, it's more a matter of enforcement by ISPs. I was just posting the link because it was relevent.
If you're the same Mr. Sam that made Courier, then thanks
Problem definition? (Score:2, Insightful)
The simplest such policy is a whitelist -- but this means you can't just give your email address to a friend, and expect her to be able to send you mail. It means that if your friends change email addresses, they can't just send you email saying "This is my new address -- my old ISP stinks!"
A more complex policy might include some public key infrastrucure, where a user needs to have a valid key to sign their messages, in order to send mail. This brings up "who do you trust" in terms of who can sign message-signing keys, but more importaintly, who is going to have more trouble with this, a spammer that wants 100 throw-away accounts with keys per week, or someone's Mom who just wants to type a message and click "send?"
I think that it will be little use in trying to come up with a protocol solution to spam, without first defining what security policy you need to enforce. Once you do that, you can design exactly what you need to make your new policy work. (You might even find that SMTP is not at fault -- you might just need a layer above that transport to secure your mailbox.)
Not much wrong with SMTP, just use teergrubing (Score:3, Insightful)
Judicious teergrubing (intentional slowing of responses; teergrube is German for tarpit) can alleviate many problems.
For example, let's examine the Rumplestiltskin attack (a form of dictionary attack to guess e-mail addresses). The trouble here is that most mail servers send back their "No such account" response immediately, so an attacker can try about 5-15 addresses a second. If the mail server was programmed to wait 5 seconds before sending back the response, then the Rumplestiltskin attack would be slowed down by about 50 times. Even better would be to make the delay longer and longer for repeated attempts from the same IP. This way, a normal user with a couple of dud e-mail addresses is not harmed much, but the Rumplestiltskin attack eventually gets bogged down in the tarpit. We have a 3 second delay at the login prompt if we enter the wrong password, so why not a delay at the mail server for incorrect e-mail addresses?
Another way to slow the spam is to teergrube *all* e-mail connections so all email takes a few minutes to send. Legitimate users aren't harmed much by this, but spammers are hurt a lot. Spammers rely on speed to send all their e-mail, and if we slow them down we can hurt them.
Then there's the question of what happens if a spammer sends another RCPT or other similar packet before receiving the response from the first? SMTP can legally drop the connection because such command buffering may be "unsupported". So the spammer must be teergrubed or must experience a *lot* of dropped connections.
There's no need to replace SMTP yet. Instead, we use the tools we have in a slightly different way, and the spammer can be inconvenienced a lot.
For more information on teergrubing, go here [iks-jena.de].
Re:Does IM2000 fight SPAM? Not really. (Score:1, Insightful)
Nor does IM2000 impose a lot of storage requirements on the spammer's ISP, since that 1Kb message, even with 100,000 recipients, is still stored by that ISP as a single 1Kb message. It is only duplicated when picked up by the recipient. So the storage space required for a mass mailing is reduced to such a miniscule level, that it really doesn't significantly impact the spammer's own ISP to accept this duty.
In fact, the only way in which I can think of that IM2000 makes things tougher on spammers, is that when their ISPs are notified of their shady activity, the ISPs can terminate their accounts and nip all of their pending spam in the bud before all of the recipients have picked them up. Whereas with SMTP you can't close the barn door after the horses have left, with IM2000 you possibly could. But this is not that important an advantage. By the time enough users have complained for the ISP to actually take this action, the overwhelming majority of recipients will have already checked their mail.
So the verdict: IM2000 is essentially spam-neutral. But it is very ISP-friendly (in terms of storage not bandwidth) because as I mentioned in an earlier post it delays the cloning of an email for all its recipients until the moment each one retrieves it
TO RECAP: IM2000 does lift the storage burden of all that spam from the recipient's ISP. But only a very tiny percentage of that burden is shifted to the sender's ISP. The rest evapourates due to the advantages of the new paradigm.
Re:not the answer - you got that right! (Score:2, Insightful)
Re:Be wary of 'trusted' protocols (Score:5, Insightful)
First of all, they're applying a common practice used elsewhere (i.e. the use of PKI and trust metrics to control authentication and non-repudiation) to email. It's not like they've invented the special Microsoft Email System which is radically different from everything that's happened before.
Second of all, PGP and its web of trust are designed explicitly to avoid CA issues like you're describing. If the system is based on X509V3 certs and your web MUA controls your trusted roots, then yeah, they'd be in charge of what you'd be able to see (but presumably you'd have the ability to at least specify that you trust particular certificates).
Third of all, even if they then "sell the ability to send spam," it'd be pretty easy to tell that they've done it, tell who sent the spam, and take your business elsewhere! The whole point of authenticated, non-repudiatable email is that you actually CAN determine WHO sent the email in the first place, so that you can then track said person down and tell them (politely of course) not to do that anymore. Spam becomes much less of an issue if everybody has to legitimately say who sent every email.
So stop trying to bring about some type of scare tactic about what is probably the only real way to combat spam anyway.
Re:Check out Internet Mail 2000 (Score:3, Insightful)
For example, you see one of your own message IDs in the References: header of an incoming message. That tells you that it's a solicited response to one of your own messages. Unless you're a troll, presumably each one of your messages merits a response from an interested party. Additionally, by the virtue of sending the original message you've implicitly solicited appropriate replies.
Now, I don't think anyone needs to keep track of every message they send. This is just a conceptual example of identifying solicited versus unsolicited E-mail. Once a method for doing so can be worked out, and put in widespread use, the spam problem will be gone.
Re:not the answer - you got that right! (Score:4, Insightful)
Sure, but for most of us the *time* involved in dealing with spam is a greater cost than the bandwidth involved in receiving it. Bayesian does a great job at solving the "waste of time" problem. And, as I said, if everyone used it I believe spam would disappear quite quickly because the response rate would fall too low for even spammers to have an interest.
Blacklisting servers and not accepting connections from them (and accepting the collateral damage) is the only practical option I have.
In the case of a few a spamhauses, sure. But as an effective spam-fighting measure that's a useless approach. You (or someone) has to keep up to date with the latest servers to blacklist (and then whitelist them when they become clean), or you have to deal with an annoying level of false positives that you don't even see. Sure, you can say that that's the price of users dealing with an ISP that is spam-tolerant. But some of us want to do business with those users even if they chose their ISP poorly--or if they don't have any local choice of ISP.
Re:Be wary of 'trusted' protocols (Score:1, Insightful)
Can you explain how the web of trust will stop people from generating thousands of phoney identities (with the assistance of random helpful people; the same people who keep spreading email viruses)? Can you explain how this same system will allow anyone to send email to anyone else?
Clear as mud. (Score:4, Insightful)
A combination of white lists/black lists, and Baysian filtering...
Stop yer bitchin', people, and implement the technologies that are already out there and work great.
This doesn't sound like a general solution for J. Random Homeowner.
It sounds alot like "Well, shoot! Quit yer bitchin' an just put in some tuned ports and performance cam! Hell, my grandmother could do that!"
Re:Check out Internet Mail 2000 (Score:4, Insightful)
I don't risk job loss (since I'm self-employed), but I do risk missed contracts. I use Bayesian. I've had an occasional false positive, mostly during the first couple months when I was building my corpus.
BUT: I'm currently receiving about 140 spams per day. Do you really think I'm going to do any better than Bayesian regarding false positives when I'm mass-deleting 140 messages? I think not. I'm almost positive I'm going to incorrectly delete more good messages than Bayesian's false positive rate when I'm dealing with such a huge number of emails manually.
Plus if your Bayesian system is half-decent you should be able to adjust its sensitivity. On mine 0-49% is almost always good email and 50%-99% is almost always spam. When I look for false positives I look for anything unusually low, such as 65% or so. If you want, just adjust your threshold to 65%. I've never had a valid email with a Bayesian score of, say, 90%.
Re:Check out Internet Mail 2000 (Score:2, Insightful)
It's not so much the storage, it's the bandwidth that costs, I think.
Sure, they can store one email and send out millions of notifications from "different" addresses.
But as soon as people start reading it, those networks are going to start getting really busy.
A self-slashdotting type effect.
- Muggins the Mad
Re:not the answer - you got that right! (Score:2, Insightful)
Recipient-based Pay-By-Email (Score:3, Insightful)
Implementing it on a recipient-controlled basis is not only politically correct anarcho-capitalism (as opposed to central-planner-based and doomed to failure), it matches the main problem with spam, which is that it wastes the recipient's time, and only the recipient knows what that time is worth., and the recipient of the mail is the one who gets the money, not some middleman whose time isn't being wasted. If I dislike spam more than you do and am more willing to reject mail from people who might have interesting things to say, I can charge more money to read email than you do. Yes, email-provider ISPs also have higher workloads because ~50% of email is spam, but that's a much smaller amount of money per recipient than the wasted time, and they can charge the users accordingly.
Of course, if the recipient of the email gets paid for receiving the email, there's an obvious product for spammers to sell "Get paid receiving email" ;-)
What's the power curve on that? (Score:3, Insightful)
Have you done the power-curve analysis on that? My mother works at a law firm, and they once tried to install a spam filter. It was state-of-the-art, with Bayesian filtering, and white/black lists, and additional whitefilters on top. It blocked most (not all) of their spam. But it also blocked some tiny fraction of legitimate messages.
Even if you have the (extremely impressive) power curves of Paul Graham's Plan for Spam -- and that was on a very well-trained Bayesian filter written by a coding genious -- it is Not Good Enough when missing a legit email could get you sued for millions. Either you risk blocking legit email, or else you have to wade through a pile of spam bigger than that legit email... either way, another protocol would be nice.
This is not the right group to do this (Score:3, Insightful)
Re:What's the power curve on that? (Score:5, Insightful)
Email is not a guaranteed service - no one is ever going to be sued for millions for not receiving an email. Things that *must* be delivered will continue to be put into a hard copy format and delivered by courier with a signature required.
Put your money where your mouth is! (Score:3, Insightful)
[...]
Stop yer bitchin', people, and implement the technologies that are already out there and work great. Plus use yer freakin' brains for a change, and don't spew out your real e-mail address to everybody who asks for it.
Don't make claims that the current technologies are "so close to 100%" if you're not willing to put your money where your mouth is. Post your email address far and wide, then see if you still don't get any spam. Otherwise, don't get up on your high horse and tell people they shouldn't worry about spam.
Security through obscurity sucks. We need a better solution for spam.
Re:What's the power curve on that? (Score:4, Insightful)
Email can fail to arrive, or be read, for any number of reasons. It passes through several servers, each of which could fail and lose mail. Same for snail mail -- you require assurance/proof snail mail was delivered, you use registered mail, and get a receipt. There are few if any circumstances you could claim in court someone was liable for not receiving an email that you had sent and not verified had been received.
If something is in the "millions" category, you fly there and do it in person.
Re:Check out Internet Mail 2000 (Score:3, Insightful)
You've never wanted to e-mail somebody who doesn't know you personally?