Replacing SMTP? 539
dousette asks: "In reading over one of the RFC's governing the SMTP protocol, and other RFC's as well, it's interesting to note that you see some big names and big companies from time to time. With all the loopholes in the current SMTP specification, is it possible for the Slashdot collective to come up with another one? Would it stand a chance in making it into a standard, or do they just listen to Cisco, AT&T, etc? I realize that a lot of people have a lot of ideas how things should be done (and they haven't been shy about posting them to Slashdot), but has anyone tried to write the RFC for a replacement protocol? As a side note (where I won't be shy about posting how things should be done), if there were a replacement trusted protocol, one could have mail received via that protocol bypass spam filtering, id checking, or whatever checks might be in place (saving processor cycles, etc). The regular checks could still be done on other mail received via the 'older' SMTP protocol. If more and more ISP's make use of this, SMTP could be gradually phased out... or if you are one for a sudden cut-over, just cut to the new one at the same time as the IPv6 upgrade!"
Jabber (Score:3, Interesting)
Costs (Score:5, Interesting)
A lot of research and ideas and papers have been thrown around to replace SMTP with a better protocol but the costs involved are a major discouraging factor and people don't want to install a system when there is no guarantee that all the recipients have it too.
Maybe servers using a new mail protocol should be designed such that they first attempt to use the new protocol and if connect fails, try the good old SMTP
SPF (Score:4, Interesting)
Re:Check out Internet Mail 2000 (Score:0, Interesting)
Where my main cause for concern is, is the use of email by corporations for notifications and account issues. Every time i make a purchase from amazon.com, i don't want to have a $0.30 or so in hidden costs factored into my shopping experience for the order confirmation and shipping notification messages i should get. When you factor the number of sales (and thus, the number of emails they send out) it can really eat into amazon's bottom line, which affects me.
I totally agree that the system needs to punish and prevent spammers though. I just think doing harm to a section of home and legitimate email users is not the answer.
Will receive email for work. (Score:5, Interesting)
This would make it too slow to send spam, by making it simply too processor intensive. Legitimate users would be unaffected.
Difficult Problem (Score:3, Interesting)
I agree that something ought to be done to cut down on the huge volume of spam that clogs most SMTP traffic.
On the surface of it, a white-listing system, perhaps based on public-key cryptography and endorsements might work.
But, as someone who values freedom and anonymity, I'd hate to have a system that closes off completely the opportunity for more anonymous communication via email.
Whistleblowers in the government and in the corporate sector, dissidents under a repressive political regime are some of the use cases for email that I'm not really inclined to sacrifice merely to eliminate spam.
Re:Check out Internet Mail 2000 (Score:3, Interesting)
Re:What loopholes in SMTP? (Score:5, Interesting)
At least if some one had to authenticate to send as joe@bar.com, some spammer would have to hack your password before they used your email address as the "From:" in a mailing...which just happened to me.
not the answer - you got that right! (Score:5, Interesting)
And the perfect example is regular junk snail mail. It costs them to send it, yet even in the Internet Age(tm), I still get a ton of it. Obviously that's NOT the answer, so "Don't Go There"(tm).
I think locking down SMTP servers and requiring verified & correct return addresses would go a long way toward curbing spam. Then when you disallow someone to send you mail, it could really work.
A combination of white lists/black lists, and Baysian filtering stops so close to 100% of spam that it's really silly for anyone to be bitching about spam these days. I don't GET any spam anymore - 0. Not 0.001%, 0 - the integer 0, as in none. If I ever get another piece of spam, then I'll change my email address (I can do that more easily than most as I have my own domain.), though this isn't the answer for everyone - lots of people have e-mail addresses printed up on lots of expensive cards & letterhead, etc. For them, the white list / black list / Baysian filtering solution should suffice way more than anyone should practically need.
Stop yer bitchin', people, and implement the technologies that are already out there and work great. Plus use yer freakin' brains for a change, and don't spew out your real e-mail address to everybody who asks for it. Use your friend's!
... at the same time as the IPv6 upgrade! ??? (Score:5, Interesting)
C'mon now; the IPv6 upgrade will be spread out over at least several decades. And both Microsoft systems and many US Government installations will still be using it a century from now, because it's "standard".
After all, it's now past the death of typewriters, and we're still using the typewriter keyboard from nearly two centuries ago. And we use a ridiculous rail gauge, because the standard was set centuries ago.
And here in the US, we're still using inches and feet, measurements based on the lengths of the thumb and foot of a long-dead king. And we call them "standard".
We will be stuck with IPv4 for long past the final download of anyone reading this.
SMTP will probably be around even longer. But that's OK; it's fun to impress friends by a "telnet 25", followed by typing in a message directly to the server. I like to use "MAIL From: dubya@whitehouse.gov", and ask them if they'd be interested in a nice job in the TIA program. Then I challenge them to prove from the message they get who actually sent it.
Re:Costs (Score:3, Interesting)
The advantage to this is that you can introduce new protocols completely painlessly. You pick a new name (after asking around on the newsgroup if anyone is using it), link your new protocol module into the protocol tables on the systems where you want to use it, and start using it. If you connect to a machine that doesn't have your protocol, it will simply tell you to use one of the others on your list. If your protocol is good, it will spread and will be early in the table for a lot of software. It can then slowly supplant the older protocols.
And you stay compatible with older systems by merely keeping the old protocol modules in your tables.
This is 1970's technology. So I suppose we'll soon read that Microsoft has just patented it.
Re:Check out Internet Mail 2000 (Score:5, Interesting)
Because the cost becomes built in to their business model. it won't stop, it will only hurt regular users to charge for email/services. Sure, their profits may be cut a little bit, but that's not going to stop them. if anything, they'll do it more, because if their profit margin is smaller, they'll have to spam harder... right?
Are you sure the problem is primarily with SMTP? (Score:5, Interesting)
Non-email messaging systems have been thinking about virtually the same problem quite a bit, and have come up with a set of solutions that try to solve what are fundamentally the same issues: message integrity, message non-repudiation, and message authentication. And the surprising part of this is that nobody really focused on the protocol, because it doesn't provide the path to a meaningful solution to the problem.
Case in point: web services. While initially the people who were playing iwth web services started out doing security at the transport level (i.e. with SSL and various derivatives thereof), but realized that something like WS-Security (where the security of a message is a part of the message itself) is the more optimal approach.
Why not just force the issue into the realm of S/MIME (and similar extensions to rfc822) and handle it at MUA space? You can cover virtually all the problems with SPAM by following the example of the reliable messaging systems and doing more with the contents of the message itself, rather than trying to say that messages have to transmit over a particular protocol. For example, depending on your trust environment, S/MIME signatures solve the authentication, non-repudiation, and integrity problems perfectly. What more do you need/want?
Re:not the answer - you got that right! (Score:3, Interesting)
In that case, who would define "correct" addresses, the ISP? And how would they be defined? I have at least 1-2 email accounts that I retrieve mail from with POP3, but send outgoing mail with the same domain through my ISPs mail server because there is currently no other way. I own (or, more correctly, lease) the domains myself, so no one can legally tell me tell me that I can't send email using those domains. The fact that I send outgoing mail through my ISPs mail server happens to be a necessary evil.
On the other hand, my mail server is definitely locked down. The failed open relay probe that someone tried last night proves that. That's the part that needs to (and can easily) be done, but the few that I've contacted about open relays won't respond or do anything with that information.
Re:Check out Internet Mail 2000 (Score:2, Interesting)
Whatever is implemented must be allowed to work with the current SMTP during a transition phase and must prioritize itself in some way (otherwise the spammers will just keep using SMTP).
Re:Will receive email for work. (Score:5, Interesting)
How about this? (Score:3, Interesting)
This doesn't break anything as it stands now, users/admin can choose how its handled, and should be fairly simple to implement.. There would be an overhead cost of keeping track of the MD5's... But it could be done...
Just an idea... Waiting to be shot down...
SMTP is not the problem. (Score:4, Interesting)
In the country where I live there is a general rule for farm animals, the farmer is not responsible for fencing them in, it is your job to fence them out. Mail is the same, its not my job to stop spam being sent, but to stop it being delivered (to my users) There are many ways to do this, a combination of a few can be very effective.
As for the home user, well stop buying into the "submit your e-mail and we will send you porn" forms on the sites you wife does not want you to look at.
Re:not the answer - you got that right! (Score:5, Interesting)
This has already been developed by the IETF anti-spam working group, well, kind of. They propose that an additional DNS record type (RMX IIRC) is added to your domain that lists all the trusted IPs that may originate email for that domain. That would include your own outbound mailserver IPs, and/or your ISPs depending on the situation, email that doesn't come from one of the listed IPs is highly likely to be spam.
The good points:
SMTP should have been replaced long ago (Score:5, Interesting)
There once was a very interesting competing standard from OSI, the X.400 standard. Most people now think of X.400 as an interconnect standard for bridging the various email systems out there. Yet, it actually is a specification for a very robust email system in and of itself. It is based on a self-describing data representation... no, not XML since XML wasn't even a twinkle in someone's eye at that time, but ASN.1. That standard has been somewhat successful as used in X.500, which has become somewhat popular through its exposure via LDAP.
SMTP has never been a particularly strong standard. First, it is not the specification for a complete email system. It mearly describes a protocol for exchanging messages between two processes via the network. This is not sufficient to build an email system. Thus we also get POP and IMAP, and any number of supplimental bits that are not necessarily standards. Even sticking to exchanging email between two processes, SMTP has always been rather loosely specified. Sendmail has served as the reference implementation. Supporting sendmail was more a matter of figuring out what it was doing than reading the SMTP specification since sendmail used a far richer protocol for exchanging email than described in the specification. Thus, the question of what comprised a compliant implementation was more like (does it interoperate fully with sendmail) than going through a specification and checking off each element it described.
Apollo started a project to produce a native X.400 email system. It had a very rich set of features that go far beyond what we see today in Unix and Windows email systems. The project was put on hold when I was reassigned to a higher priority task, I was a member of a strategic technology team given the task of determining what "everyone" meant by the term "CASE Integration" with the goal of producing a corporate strategy and piloting and/or prototyping some initial products. Given the state of the CASE community, it sure seems like pursuing the email strategy would have had better long term success. Of course the CASE Integration project died a painful and horrible death when HP bought the company. Surely "SoftBench" did everything and more...
Re:not the answer - you got that right! (Score:2, Interesting)
While it'd be great to not have to accept the entire message to determine if it is spam and I'd be the first one to jump on board if there's some way to do it, I just don't see it as being possible. You can't base a filtering decision when you don't have the message to analyze.
That said, Bayesian should be done on the server before the real client downloads it. Sure, it still gets to the server but at least the client doesn't have to waste further time and bandwidth downloading it. Client-side Bayesian is better than nothing, but I would personally never use it. You still have to download hundreds of spam per day. I'd rather the server do that for me and only give me the good stuff.
Lessening Spam: The True Hollywood Story (Score:5, Interesting)
And this is the thing - they really _shouldn't_ have to. Bad UI really ticks me off.
> I agree, however, that people are generally naive/dumb when it comes to common sense issues like sending out email addresses at will or even worse...
The thing is - I intended that for this particular audience. _SLASHDOT_ users, of all people, should know by now how to avoid getting spam. I mean _really_.
> clicking on the "Remove" links from spam! VERY DUMB!
Actually, I've proven to myself this is a myth.
Here's my story:
Last year, around, say, September or October, I was getting, on average, about 200-250 pieces of spam PER DAY. This, I realized, just Would Not Do(tm).
So, since it was obvious I was going to have to shut down all my existing e-mail addresses, generate new ones, and be ultra-selective about giving them out in the future, I realized it was time to test that "Don't click on the remove me links" piece of advice. I'd given it out myself many times, even in an article I once wrote. Time to put it to the test! So, for the period of one month, I followed all the instructions on each piece of spam, every day, to see what would happen to my flow of spam. I kept track of who was sending me spam (the company/product/service, not the 'return address'). I found out that you WILL get LESS spam if you actually follow the advice in the spam, in general. My spam reception went from the 200-250 per day to around 20 per day, in the span of about a month. Obviously, this was still way too much frigging spam, but let me say this: the spam I kept getting was almost entirely from the sources that didn't have a (working) removal method, not from the ones that did. Many of the ones that did have an (apparently) working method DO indeed take a few weeks to start working. But, surprise of surprises, it CAN indeed lessen your spam, when it's offered and is working. Bizarre, I know, but I swear it's true.
I'd still rather make spam technically impossible than rely on that, though.
I propose TMTP - the Trusted Mail Transfer Protocol.
Re:Check out Internet Mail 2000 (Score:2, Interesting)
I have a technical website where users can subscribe to a nightly mailing list that sends them a single email containing all the messages posted to the forum in the last 24 hours. The users subscribe by signing-up with their email address. A single email is then sent to that user who is then asked to click a link to confirm they want to receive the nightly mailing list.
Now, I've heard people propose that there should be a system whereby an "unknown" email is charged 10 cents (or whatever) unless the receiver subsequently tells the system "Yeah, I wanted that" at which point the 10 cents is effectively refunded and, presumably, all future emails from that source have the charge waived.
The problem is if someone starts signing up random email addresses on my mailing list. Each time the system sends out the single confirmation email I will be dinged 10 cents with the expectation that everyone will "refund" that back to me since they asked for it. But what if someone with a grudge and a lot of extra time goes to my site and starts signing up lots of people--people that have never heard of my site. The "confirmation email" is precisely to protect users from getting on my mailing list without asking for it, but I'm not at risk of having to pay to have that confirmation email delivered.
If such a system were imposed then how would a mailing list be able to send out confirmation emails without the risk of someone maliciously signing up random users just so that I get hit with a bunch of email charges?
Re:QWERTY speeds typing. QWERTY 4ever! (Score:2, Interesting)
If you do trust the applet (man, I should really verify the results to be sure
1. You stay on the home row more.
2. You alternate hands more.
3. You change fingers more.
4. Your fingers don't travel as far.
Well, whatever the reason, I feel more comfortable typing on a Dvorak layout
Re:not the answer - you got that right! (Score:2, Interesting)
Try contacting their upstream provider. If that doesn't work, try contacting the provider's upstream provider. I used to work at a teir two ISP. When we'd get a complaint of an open relay, we'd first test it ourselves, then hound the hell out of the responsible admin. You won't always get this result, but if you go high enough, they'll at least close the relay temporarily and open another in it's place. Either way it will add a black mark on the client's record, and it may even keep a dirty admin busy...
A simple as hell answer. (Score:5, Interesting)
Obviously ISPs will have to have the ability to store the messages of their users so they can deliver them while the user is offline, but that's no problem. If a user, or someone else, sends spam, once the ISP is notified, they can remove it from their servers, so that no further people who were sent the spam will actually recieve it upon reading their email.
Why I'm writing this I don't know. No one reads below score 3 anyway unless you're lucky and get one of the first 10 replies. Slashdot is useless. I'd shit myself if one person actually read this post. Hell, I can't even find posts after I make them, even after waiting several hours.
Re:Check out Internet Mail 2000 (Score:2, Interesting)
Ok, so that requires additional steps. Before the user signs up the user must be made aware of the address from which emails will be sent so that it can be whitelisted BEFORE the verification email is sent. A new business plan for spammers wold be to cruise or spider the net and gather the email addresses of common mailing lists and use that as the "From" when sending spam to increase the probability of using a whitelisted, toll-free address.
If you're talking about whitelisting an email address AND an email server that's fine for tech people but would probably a bit complicated for normal users... and for lazy technies.
Re:SMTP should have been replaced long ago (Score:2, Interesting)
Re:Check out Internet Mail 2000 (Score:2, Interesting)
Ahh, but if your E-mail address is out there on some web site -- together with some arbitrary content -- and someone chooses to send you E-mail after reading said contents, and wanting to respond to you for some reason; then, well, from where I'm standing it looks to me like you've solicited that response, didn't you? You certainly didn't solicit some spambot sucking it out of the HTML, and then feeding it to a Viagra spam engine, of course. But, by the virtue of signing your name+E-mail address to some article, I would argue that you've solicited individual readers to drop you a note after they've read the article, if they felt like it. If you didn't want, or solicit, people to reply to your Slashdot posts, then you simply leave out your address, that's it. What good would posting your E-mail address here if you don't want people replying to it?
I mean that's what an E-mail address is for, I think: for people to know how to get in touch with the author, regarding the written subject matter.
Similarly, by signing your name to your Slashdot posts, I would argue that you've solicited, or invited, E-mails from anyone who've read them and wanted to give you a piece of their mind, for some reason.
I frequent other web discussion forums, where I do NOT use my E-mail address. That's because I really don't care for any replies on the subject matter over there. Here, if someone wants to flame me away for something -- go right ahead. I'll bitbucket it, of course, but I wouldn't say that it was unsolicited.
So, I think, it all comes down to "solicited" vs. "unsolicited". It's rather impossible to give a precise definition of "bulk". What is "bulk"? If "bulk" means a certain amount of substantially similar messages, then there has to be some number X where X is bulk, but X-1 is not bulk.
So, what is X, then, and can you provide a valid, cogent argument why X is bulk, and X-1 is not bulk?
I don't think this is going to work. I don't think you can make a common-sense based argument for bulk vs non-bulk. But I think an argument on solicited vs. unsolicited can be made, based on a common-sense definition of "solicited."
It appears that your definition of "solicited" means something that's exclusively a response to some previous written screed of yours. I think that a more liberal definition of "solicited" works better: meaning "did you reasonably expect to receive this kind of a message." Obviously, from the fact that you've posted a message of your own to a discussion group, with a valid E-mail address, it can be reasonably inferred that you've asked -- hence solicited -- replies. But, at the same time, if you put up an average home page, where you wrote that you're a graduate of East Side Nerd High School, and if you've provided your E-mail address, then it can be reasonably said that if an old buddy of your from East Side Nerd High accidentally stumbled across it, his E-mail wasn't exactly unsolicited. He didn't just sent something, addressed to your E-mail address, by random. It was in direct response to what he read -- hence it was solicited.
On the other hand, if some spambot lifted your address, and started sending you Viagra spam, then it can't be reasonably argued that you've solicited Viagra spam simply by the virtue of describing your high school follies.
Now, you might think that this approach is not going to work because "reasonable" is in the eye of the beholder. Something may be reasonable to one person, but not reasonable to another person. Spammers will argue that using your E-mail address on slashdot can be inferred to reasonbly means that you want
Re:not the answer - you got that right! (Score:3, Interesting)
Re:not the answer - you got that right! (Score:1, Interesting)
False positives throws everything into the air. We don't allow a system with a low false positive rate, it has to be almost nil before we consider it.
Re:not the answer - you got that right! (Score:3, Interesting)
well, no. if the scanning it set up correctly on your server then you can receive the email and scan it before giving the okay. if you don't want the email then the server simply gives a reject message and refuses to accept responsibility. ie, the scanning is underway while the sending server is waiting for it's ACK that the email has been recieved correctly. if you reject it at this stage then the sending server is still responsible and it has to deal with the bounces.
dave
Re:Check out Internet Mail 2000 (Score:2, Interesting)
But wait. What if ISPs simply charge each other for traffic depending not on the direction of traffic but depending on which side initiated the TCP connection. That way the person downloading from a web site will be the one paying (because he made the HTTP connection) and not the web site host. And the person sending the email will be the one paying (because he made the SMTP connection) and not the recipient.
If only a few big ISPs agree to work like this others will follow and soon even small ISPs will start charging their customers for traffic based on this method.
Could this help to put an end to spam?
Two problems - Two solutions (Score:2, Interesting)
But the more important problem is the impact that the problem has on the overall infrastructure of the net. For example LINX, the (one of?) the major link points into the UK, recently reported some metric about how much spam was coming down the lines. (Sorry can't find the reference). It is when the concentration of this rubbish gets so higha s to affect this level of infrastructure that we all have a problem.
Now the tricks required by the spammers to try and get around the filters may preclude some of these ideas, but the idea that I should receive 'a copy' of Jane, or Sarahy or Ken's Viagra offer, ie a copy of the same mail that was sent to all the people at my mailhost (this is particularly important for big providers like AOL or BT where they have millions of customers) rather than an individualised one would certainly reduce the strain on the underlying infrastrucutre. How would one achieve that? Good question, I haven't really thought about it enough, but I am not suggesting an economic answer, I don't want to try and coerce the spammers, let's try and make the solutions in their own interests before we use the price tool.
However having rasied the price question, it is critical to remember that the reason why the junk snail mail, SPAM comparison falls down is that the marginal cost of increasing the distribution list of a SPAM is zero and the marginal cost of increasing the distribution list of JUNK is (in addition to postage) the cost of the JUNK. This provides a natural price imperative to the mass mailer. Can we introduce such a cost to the spammers. Er, well um no. Not that it is hard, ot is just impossible. Think as hard as you like and anything you come up with will be flawed in some way. So don't try. Just accept it.
So, we must return to the politics of SPAM. Clearly the sender of the spam must be identifiable. Perhaps it must be identifiable from the "International register of spammers" what ever that might be. SPAM not sent by one of these servers is automatically dropped. Then one must decide how one determines what is SPAM and there is probably the rub. I don't really know maybe a global bayesian filter?
Anyway. The point is that the first problem is to contain SPAM, make it in the spammers interest to identify their mail as SPAM. From then we can manage the SPAM bit by bit. Of course the obvious solution is _don't buy from spamvertisers_ but then it only takes a few idiots to make that strategy infeasible.
Re:not the answer - you got that right! (Score:3, Interesting)
OK, so imagine (in a perfect world) that everybody has 100% locked down SMTP servers, and there is an addition to SMTP that requires verified and correct return addresses on every email (regardless of the problems that such verification would cause.)
What's to stop a spammer from running his/her own mailserver (you know, like they do today), and providing 100% verified and correct email addresses on all the spam they send you?
The answer I've heard from people who've proposed the same thing as you is that you'd just start a blacklist..
So I ask: How is this any different from what we have today?
implement the technologies that are already out there
The problem is that spam is not a technological problem - it's a social one. And you can't solve a social problem with technology.
Forget fees and filters. Shoot the relays! (Score:3, Interesting)
The only really effective way I can think of is another fscking registry. ISPs and companies large enough to really need external relays pay the fee to register their mail-server there, and the new implementation of the SMTP-protocol only accepts external mail from other listed servers.
The downside? A fee comparable to the price of a domain name for ISPs, companies and stubborn individuals. Don't give me the old crap about "having to run your own relay", because you still could, by in turn having it relay through your ISPs server. Your ISP doesn't provide you with a relay? After this, they would have to.
The upside? It would be a lot easier to blacklist spammers. No more hijacked boxes on broadband-connections flooding us with spam.
Oh, I know, it will be shot down because there's a fee involved, but keep in mind that I would be one of the people that would have to pay that fee, and it would be a very small price to pay to protect myself and my users from spam.