Gmail, SPF, and Broken Email Forwarding? 300
alek writes "I recently stopped getting Email from a friend ... which turns out to be related to his use of SPF records and my forwarding to gmail. This 'lost Email problem' may get worse with
Google implementing Domain Keys." Alek is looking for a non-complicated solution to this non-trivial problem; read on below for more details.
"Background: Like many people, I have me@mydomain.com as my public facing Email address. When Email comes into my server, I forward it to me@gmail.com. But since my friend has published SPF (Sender Policy Framework) records that say only his server is allowed to send Emails for friend@frienddomain.com, gmail apparently rejects (silently buries actually!) the Email since it is forwarding through my server. Please note that this is exactly what SPF is designed to prevent — spammers from sending Emails with your address — but it breaks forwarding and has other problems.
What's *really* strange is that if I look at the raw sendmail logs on my server, the Email from friend@frienddomain.com comes in, and is forwarded to gmail ... with an "OK" as the response — i.e. the gmail MTA doesn't reject the message as it ideally should. However, the Email then disappears — it's not even in my gmail spam filter ... so there is no trace of it at all. If my friend sends directly to me@gmail.com, it shows up ... since his domain sends directly and the SPF test is passed. Note that on my gmail account, I associate me@mydomain.com with my me@gmail.com account ... so perhaps there should be a recipient test applied before SPF is tested on the sender ... although this arguably defeats the purpose of SPF.
The logical solution is to configure sendmail on my server to do Sender Rewriting — anyone have an easy FAQ to do this? But many people/domains aren't doing this ... and my Email forwarding to gmail is quite common, so I'm surprised that this issue hasn't gotten more attention. Is there another solution?"
Please adhere to RFC (Score:5, Informative)
The Internet Assigned Numbers Authority (IANA) also currently has the following second level domain names reserved which can be used as examples.
Is there another solution? (Score:5, Informative)
Easy answer (Score:1, Informative)
You need to implement sender-rewriting scheme in your mail server. Google it.
Next issue?
Pull instead of push? (Score:5, Informative)
Doesn't GMail offer the ability to fetch your email from POP accounts now? It would probably not be the ideal solution, but perhaps you should stop forwarding and instead start POPping.
Re:Is there another solution? (Score:5, Informative)
Our company forwards email to google (MX record in the DNS), where it runs through the spam filter and then a forwarding rule (an anything-but-spam rule) sends it on to our mailboxes.
For free...
Domain Keys doesn't have the same issue (Score:4, Informative)
Domain Keys authenticates that the message was generated by a server with access to the DK private key. Forwarding the message does not affect the originator of the message, so the Domain Key authentication still checks out.
SPF and DKs solve similar issues, but in a much different manner.
Re:Easy answer (Score:1, Informative)
See link in summary.
http://david.woodhou.se/why-not-spf.html
Solution is for your friend to use something OTHER than SPF
Re:Pull instead of push? (Score:5, Informative)
Or given the box of horrors that is POP, you could try IMAP, which google now also supports.
Easy -- sign up for Google Apps for your Domain (Score:4, Informative)
Sign up for Google Apps, and then you can have all mail sent to me@mydomain.com be handled by GMail. All you have to do is sign up at http://www.google.com/a/ [google.com] and link your domain. Then point your domain's MX records to aspmx.l.google.com.
In the future, all you have to do in order to get your mail is to go to http://mail.google.com/a/mydomain.com/ instead of http://www.gmail.com (and you can even set it up so that http://mail.mydomain.com CNAMES to your email login page)
FAQ (Score:5, Informative)
You seem to have answered the question already (Score:5, Informative)
This is also known as, "The Problem With SPF." SPF breaks forwarding. This is well known. People who use SPF need to be aware of the ramifications.
The SPF people have created SRS, as you are aware, to work around this problem. It is a complicated and unappealing workaround. I certainly won't do it.
You have three options as I see it:
1) Stop forwarding. It's really a terrible idea. Install webmail on your mailserver. Check out RoundCube, for instance.
2) Wait for people to figure out that strict SPF policies break SMTP too badly for most users.
3) Implement SRS. (this would probably be easier if you were using a modern MTA)
I guess you were hoping for an easy fix, but there simply isn't one.
Re:Please adhere to RFC (Score:5, Informative)
Um, no. If you actually read RFC 2606, it is for TESTING. If this guy were really sending test emails to me@mydomain.com, then he would be in violation. Simply posting it on Slashdot as an example is not prohibited.
Re:Pull instead of push? (Score:4, Informative)
gmail does let you pull via pop3 BUT the scheduler is not configurable. Gmail checks pop randomly when it feels like it. For me it's about every 30 minutes to 1 hour. YMMV
Comment removed (Score:5, Informative)
Re:Please adhere to RFC (Score:3, Informative)
"To reduce the likelihood of conflict and confusion, a few top level domain names are reserved for use in private testing, as examples in documentation, and the like. In addition, a few second level domain names reserved for use as examples are documented."
And no, it's not prohibited per se, but it is a good practice so as not to annoy those who own the domains the submitter used.
Re:Easy -- sign up for Google Apps for your Domain (Score:1, Informative)
Unfortunately, http://mail.example.com goes to http, not https.
And of course, you get cert warnings if you try https://mail.example.com.
And yes, I know in either case the authentication part is secure, but the post-auth part is not.
you want https://mail.google.com/a/example.com
There is an easy way to do e-mail forwarding... (Score:3, Informative)
There's an easy way to do e-mail forwarding, which unfortunately is wrong. We no longer live in a world where you can just create a .forward file with the destination address in it (unless it's on the same server).
If you're going to run your own mail server, there are things you need to do if you want it to run correctly. One of them is that if you are forwarding to a mail server that does SPF, you need to do SRS. Though you probably also need to be doing all the spam rejection on your mail server as well, because otherwise you may be allowing mail through that you wouldn't otherwise.
For example, say that your server doesn't check SPF, and you do SRS. Now you're basically bypassing the destination server's SPF checking.
How to do SRS? I would personally probably just change my .forward file from the destination address into a small script that re-injects the message with a different envelope sender, but I'm sure there are already scripts that do this and much more fancy....
Ideally, you probably just want to move your mail for your domain directly to google, as another repondant says. Don't have it shunting your your own server if at all possible. If you have mail that you want handled directly on your server, either forward it from gmail to your home machine, or use a different domain ("address@homebox.example.com").
Sean
Re:I knew .. (Score:4, Informative)
My e-mail goes through my domain, forwarded to Gmail, and then is downloaded to my computer via POP. Gmail is my offsite back-up (that is accessible from anywhere) and home is where I do most of my mail viewing/sending. All of those GB of space, local copies in case Gmail fails, remote copies in case my computer fails. And assuming Google is "not evil", then I should be ok.
Layne
The solution, Return Path Header (Score:4, Informative)
SPF will validate the Return-path header if there is one instead of the From: address.
Unfortunately, I don't know how to make either sendmail or postfix insert a return path when they forward an e-mail, but the easy work around is to install mail list software as your forwarder. You can create a mailing list as your incoming e-mail, with only 1 mail list member, (which is your g-mail account). Mail list software will automagically insert the appropriate return-path header that is needed in this case.
DomainKeys DOES NOT HAVE THIS PROBLEM (Score:3, Informative)
DKIM and DomainKeys work in a fundamentally different way. The message is SIGNED. Hosts are not indicated one way or the other. So any DKIM signed mail can transit any number of hosts provided they don't modify the signed sections.
SPF has no such luxury unless implemented in a much more advanced manner in terms of the senders publishing. And it's not GMail's fault for following the SPF records as published, they should do a better job of rejecting early rather than just /dev/null-ing the email though.
Re:maybe a silly question but.. (Score:5, Informative)
People over-generalize terms quite often, and "forwarding" has different meanings in different situations. Generally the difference boils down to if you're talking about a "server" implementation or a "mail client" implementation.
In this case, the SPF folks are addressing server admins, so by "forwarding" they mean sending the message to a new recipient without altering the headers. This use problably originates back to the old ".forward" files on unix machines, but may go back further. Most server-side implementations use this meaning for "forward".
However, forwarding by hitting the "forward" button most mail clients does something different. That creates a new message with new headers and preserves the old body text. sending with the same headers is called "redirect" in most mail clients.
Isn't it great how mail clients and mail servers use different meanings for the same word?
Even the client/server pair that go together from the same company have this problem. For example, Microsoft - exchange server has forwarding contacts, which forward without header changes, while Outlook clients do change the headers when you hit the "forward" button.
How to make it work (Score:5, Informative)
For this example, I'm assuming that your email is joe@example.com and your gmail address is joe-example@gmail.com.
Create an alias (/etc/mail/aliases) for the address that get's forwarded to gmail.
joe: joe-example@gmail.com
Also create an alias for <foo>-owner:
joe-owner: joe
Sendmail will look for this special <foo>-owner alias whenever sending mail to the <foo> alias, and use it as the envelope sender on the outgoing mail. So any mail that is sent to joe@example.com will be resent by sendmail with a sender address of joe@example.com. The header addresses will remain unchanged, so hitting reply will still go to the right person.
Is this the solution to all SPF forwarding brokeness? Of course not, but it's a surpisingly simple solution to a number of common forwarding situation. Note that you better be careful about spam filtering on your machine, or your mail server (your sender's address) will appear to Google as a source of spam, and might get filtered.
Re:I knew .. (Score:1, Informative)
That's funny - I use it the opposite way. Google apps receives my mail to user@example.com and forwards it to user@z.example.com, my zimbra server. That way Google apps does all my spam filtering and archiving, and I still use a better mail server.
Re:How to make it work (Score:5, Informative)
It's owner-<foo>, not the other way around. So the aliases example should read:
joe: joe-example@gmail.com
owner-joe: joe
See the aliases man page [freebsd.org] for further details.
Re:Is there another solution? (Score:3, Informative)
Our company forwards email to google (MX record in the DNS), where it runs through the spam filter and then a forwarding rule (an anything-but-spam rule) sends it on to our mailboxes.
Or you could just use Spamassassin, which properly configured is every bit as good as commercial offerings (and I have actually trialled them to do the comparison). If you put MAIA Mailguard [maiamailguard.com] on top of it, you have a solution that leaves the commercial offerings for dead - per user, server based sensitivity settings, quarantine, anti-virus and most importantly, no stupid bounces to the sender address of spam, since the sender address is almost always forged and if you are sending those stupid bounces you are the spammer.
Yes, I am sick of Messagelabs spamming me.
Re:Is there another solution? (Score:3, Informative)
SPF, Gmail, and SRS (Score:5, Informative)
Since you are running your own SMTP server, you signed on to be a sysadmin. I am replying to you as a fellow sysadmin and I'll give sysadmin-style answers. Please don't take my response to be negative in any way, as I'm trying to help.
The logical solution is to configure sendmail on my server to do Sender Rewriting [openspf.org] -- anyone have an easy FAQ to do this?
If you follow the link that you just gave for Sender Rewriting, it answers your question. "Implementation" links to modules, source, and configurations.
But many people/domains aren't doing this ... and my Email forwarding to gmail is quite common, so I'm surprised that this issue hasn't gotten more attention. Is there another solution?"
I say that you don't know how many people are implementing SRS, nor do you know how many forward e-mail to Gmail. Let's stick to the basics before giving up so readily. I take it that you absolutely do not want to give up carte blanche forwarding from your own SMTP server to Gmail; so I'll tailor my reply to that.
But since my friend has published SPF (Sender Policy Framework) records that say only his server is allowed to send Emails for friend@frienddomain.com, gmail apparently rejects (silently buries actually!) the Email since it is forwarding through my server.
Your friend has published an SPF record because he doesn't want people forging his domain in the envelope-sender field. This is a common spam tactic that ruins the reputation of someone's domain, either through spammer apathy or sometimes pure malice. Your e-mail forwarding (especially since you run your own SMTP) to Gmail is out of pure convenience to you and is unnecessary, so don't ask your friend to drop his SPF record.
There are two ways to solve this:
1) Have your friend add your SMTP server to his SPF record.
2) Implement SRS if you want to solve it once and for all. If you follow your own links, there are explanations, examples, and actual code. You haven't said which SMTP server you're running, so you've limited the responses people can give you for your situation.
I publish SPF records for my domains. There isn't anything "broken" about wanting to protect my domains' reputations from forgery. Very few people have a problem with forwarding that they didn't create themselves. This exception I'm talking about is people who have old university accounts (or similar) which only allow e-mail checking through a shell account and forwarding purely through a ".forward" file (or similar), with no POP, IMAP, or administrative access. This is not you. But for anyone who this describes, because of the draconian service policies, they shouldn't be giving out that e-mail address to new contacts, publish on papers, etc.
My SMTP server checks SPF, but not DK. With SPF, the forged domains are instantly rejected, requiring minimal overhead. DK requires reception of the entire message (because the headers are in the DATA phase) in order to validate the message, on every message -- this uses unnecessary network bandwidth, and it places an extra load on my system since it would have to calculate and verify signatures for every single message. Maybe that's not an issue for you if you only receive a handful a day, but I receive thousands. Spammers know that including fake DK info in a message and then sending millions of these is effectively a Denial of Service attack on the servers that indiscriminately check DK signatures.
I also use backup relays. For the relays that are not under my control and don't implement SRS, I simply bypass SPF checks from those IP addresses.
About Google silently dropping your e-mail: Keep in mind that with your carte blanche forwarding, you're also forwarding spam. You are essentially spamming Gmail, even though it is you simply forwarding e-mail to your own account. It is difficult for Google to know this without human intervention or implementing some co
It is in the SPF spec (Score:2, Informative)
Did whoever owns the domain even read how to implement SPF?
You could easily have added
+a:otherpermittedmailserver
in the TXT record...
See here: http://www.openspf.org/SPF_Record_Syntax
Re:silently dropping is not unexpected (Score:5, Informative)
No, fuck the spammer.
Following the RFC fucks the innocent bystander, not the spammer. Is following the RFC worth fucking innocent bystanders over?
Either respect the RFC, or come up with a solution with at least as much attention as the RFCs were given.
In the meantime, while you come up with a solution, I'll disregard the RFC for this situation, because fucking innocent bystanders over while the world figures out a 'real solution' isn't acceptable.
Re:SPF, Gmail, and SRS (Score:3, Informative)
Replying to myself because I just spotted the article submitter did mention "sendmail" as his solution. There are plenty of solutions readily available for sendmail. Like I said above, he can follow his own links for that information, and many others here have helpfully posted sendmail solutions also.
I don't know why my eyes filtered out sendmail. Odd.
Re:Forwarded messages will be fine (Score:3, Informative)
Actually, the term "forwarding" applies both to client-forwarding (remailing the mail with all of the headers) and server-forwarding (what you call bouncing.) It's the difference between clicking forward and using a .forward file (hey, why do you think they called it that?)
Does business take gmail addresses seriously? (Score:5, Informative)
One very good reason not to have your email address @gmail.com, if you are using it for your business, is that a LOT of businesses, wholesale vendors, even the federal government will not accept an @gmail.com address because of the large number of frauds associated with free email accounts (not just gmail, but also hotmail, yahoo mail, etc.) For example, this last tax season the federal govenment would not accept a gmail account for notification of your tax return status when filing electronically.
It is much better from a business standpoint to have your own domain and email sent to your domain. If your MX points at gmail, that's okay. Just don't make your email address me@gmail.com if you want to be taken seriously.
Re:Dump SPF (Score:4, Informative)
Of course it won't stop spam. It wasn't designed to. Its purpose is to stop joe jobs [wikipedia.org].
Re:silently dropping is not unexpected (Score:3, Informative)
If somebody has a problem with back scatter then they obviously don't have their SPF records set up correctly. They aren't so innocent. I'm getting spam traffic from their domain.
I'm sorry, Mr Holier Than Thou Standards Guru, but could you please point me to the standard that requires e-mail systems to support SPF?
You'll be there a while, because there is no such standard. Moreover, there probably never will be, because SPF is fundamentally broken in several ways. If you use SPF, either setting it up for your own domains or filtering on it, then you are not part of the solution, you are part of the problem. And it's is a lousy way to filter e-mail anyway, since it's statistically beyond hope of anything close to acceptable reliability, while any decent multi-pronged approach can easily get high-90s accuracy with negligible false positive rates.
Fail/HardFail vs SoftFail (Score:3, Informative)
Re:maybe a silly question but.. (Score:2, Informative)
That's because the "server" implementation you described really ought to be called "redirecting". As you said, there's *still* a lot of confusion about SPF because of the unfortunate ambiguity of this term. Blanket statements like "SPF breaks forwarding" don't help either.
So the title of this article really ought to be "Gmail, SPF, and Broken Email Redirecting" since most people's concept of "email forwarding" involves hitting the Forward button on their MUA client, or setting forwarding rules therein (which doesn't break under SPF).
It's 2008, folks. I can't believe we're still mired in confusion over terms like "forwarding" and "bouncing" (which could either mean generating a backscatter-prone bounce message, or rejecting the message during the SMTP transaction, which all MTAs really really really really really need to get on the ball with ...)
SimpleR answer: have Gmail POP your mail elsewhere (Score:2, Informative)
1.) What are you talking about. Wrong, wrong. I do precisely what you say I should not do and Gmail filters UCE nicely.
I own a domain, on which I have one public email alias, that is 8 years old. It had/gets spam/UCE. Gmail POPs that account/alias for me. Gmail filters the spam quite nicely!
It is not perfect, occasionally I have to "report spam", train Gmail, but nothing overwhelming. I can understand most times why, as I have an alias that "forwards" to the above alias. And things come in that are "infrequent", e.g., yearly seminar newsletters and the like. Consequently I have to train Gmail to know about it. I am saying that it is understandable.
Gmail understands one's email aliases, or relationships if you have Gmail POP, OR, IMAP your non Gmail accounts.
And or use Gmail Domains / Google Apps?
2.) A different problem I have is that my domain registrar butchered my name during a transfer recently. They have suggested "it would be easier to do a change of ownership to fix the problem." I have asked around and it seems Netsol, Godaddy (as an example of fruity ass registrars, i.e., inept =) or dumb registrars will sometimes change the creation date of one's domain at a whim-- I haven't been able to find where in ICANN regs, RFC or elsewhere creation date guidelines, rules are spelled out. Anyone???
Friends tell me that a change in domain creation date matters for folks such as Google/Gmail as one data point in determining the spaminess of an email. IOW, I might look like a newborn spammer. Which is especially important to me considering that I use Gmail to send email that is not from @gmail.com, as I discussed above.
I wonder if SPF records for my domain can be created using the free Gmail Domains / Google Apps?
Comment removed (Score:3, Informative)
Re:Is there another solution? ... Maybe? (Score:2, Informative)
I just got an email with this redacted SPF header. It was sent from example.net to my domain, example.com, and forwarded to my Gmail account (not gafyd):
Received-SPF: fail (google.com: domain of friend@example.net does not designate 111.111.111.111 as permitted sender) client-ip=111.111.111.111;Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of friend@example.net does not designate 111.111.111.111 as permitted sender) smtp.mail=friend@example.net
Incidentally, prior to that, my server had passed the SPF record from the original host.
Received-SPF: pass (smtp.example.com: SPF record at example.net designates 123.123.123.123 as permitted sender)
So why did I get the email, the header clearly says "fail" and "hardfail"? My only guess...
In my Gmail account, I have my an account at my (forwarding) domain setup as an authorized sender. I'm allowed to send as joe@example.com, having previously proved ownership by receiving an email from Google at that account.
To me, this makes sense. I would think that Google could make the leap of faith that if you receive email at a domain, they might as well relay all email from that domain to your Gmail account, and ignore mis-matched SPF.
And if this is not actually the case... well, it should be. So, my simple answer to the poster's question: add your domain (email addy) to your account.
Add another email address [google.com].