Gmail, SPF, and Broken Email Forwarding?

alek writes "I recently stopped getting Email from a friend ... which turns out to be related to his use of SPF records and my forwarding to gmail. This 'lost Email problem' may get worse with Google implementing Domain Keys." Alek is looking for a non-complicated solution to this non-trivial problem; read on below for more details.

"Background: Like many people, I have [e-mail] as my public facing Email address. When Email comes into my server, I forward it to [e-mail]. But since my friend has published SPF (Sender Policy Framework) records that say only his server is allowed to send Emails for [e-mail], gmail apparently rejects (silently buries actually!) the Email since it is forwarding through my server. Please note that this is exactly what SPF is designed to prevent — spammers from sending Emails with your address — but it breaks forwarding and has other problems.

What's *really* strange is that if I look at the raw sendmail logs on my server, the Email from [e-mail] comes in, and is forwarded to gmail ... with an "OK" as the response — i.e. the gmail MTA doesn't reject the message as it ideally should. However, the Email then disappears — it's not even in my gmail spam filter ... so there is no trace of it at all. If my friend sends directly to [e-mail], it shows up ... since his domain sends directly and the SPF test is passed. Note that on my gmail account, I associate [e-mail] with my [e-mail] account ... so perhaps there should be a recipient test applied before SPF is tested on the sender ... although this arguably defeats the purpose of SPF.

The logical solution is to configure sendmail on my server to do Sender Rewriting — anyone have an easy FAQ to do this? But many people/domains aren't doing this ... and my Email forwarding to gmail is quite common, so I'm surprised that this issue hasn't gotten more attention. Is there another solution?"

Please adhere to RFC

[DNS-and-BIND (461968)]

Please stop using mydomain.com and other such nonsense. Example.com is reserved by RFC 2606 [ietf.org] for use as a...wait for it...example domain name. Please make a habit of using it instead of whatever name strikes your fancy, as it is probably in use by real people.

The Internet Assigned Numbers Authority (IANA) also currently has the following second level domain names reserved which can be used as examples.

• example.com
• example.net
• example.org

Re:Please adhere to RFC

[TheRealMindChild (743925)]

Ironic you bring this up when thedailywtf.com posted this little bit [thedailywtf.com] today.

Re:Please adhere to RFC

[MyLongNickName (822545)]

Um, no. If you actually read RFC 2606, it is for TESTING. If this guy were really sending test emails to [e-mail], then he would be in violation. Simply posting it on Slashdot as an example is not prohibited.

Re:Sorry, Swoosh belongs to Nike.

[_ivy_ivy_ (1081273)]

RFC 9835 specifically calls for a "whoosh." The use of "swoosh" has been depreciated.

Re:Sorry, Swoosh belongs to Nike.

[Sciros (986030)]

deprecated

Re:Sorry, Swoosh belongs to Nike.

[neltana (795825)]

Actually, RFC 0444 has been reserved by the IETF for use as an example RFC number. Your joke should have used that.

Come on, people!

Is there another solution?

[jeffmeden (135043)]

Yes, of course. Have all your email sent to Google in the first place! You don't have to switch everything over to the Google app tool, you can just set MX records for your domain pointing to them, and collect it all (or forward it inside or outside Google.) It's free (with a paid version available.) Check it out here http://www.google.com/a/help/intl/en/index.html [google.com]

Re:Is there another solution?

[dch24 (904899)]

It really works! (ob. disclaimer: satisfied customer)

Our company forwards email to google (MX record in the DNS), where it runs through the spam filter and then a forwarding rule (an anything-but-spam rule) sends it on to our mailboxes.

For free... :-)

Simple answer: stop forwarding

[mattbee (17533)]

Effective spam filtering for forwarded email is pretty much impossible, as you lose vital information in the forwarding. Either get rid of your forwarding address, or have it hosted at Google as well. Probably the largest single reduction in spam I've ever made was the week that I got rid of years-old forwarding addresses. If the forwarding address is more important, just get it hosted at Google directly, or tell people to stop using it!

silently dropping is not unexpected

[Ungrounded Lightning (62228)]

What's *really* strange is that if I look at the raw sendmail logs on my server, the Email from [e-mail] comes in, and is forwarded to gmail ... with an "OK" as the response -- i.e. the gmail MTA doesn't reject the message as it ideally should. However, the Email then disappears -- it's not even in my gmail spam filter ... so there is no trace of it at all.

While the RFCs specify that an MTA that is dropping should notify the sender in various ways, modern MTAs often violate these parts of the spec, pretending to accept and then dropping the mail and/or failing to send bounce notifications.

This is deliberate. Not sending bounce messages reduces the load on the servers and net (now that most mail traffic bounces). Pretending to accept mail which is actually dropped is a defense against guessing email addresses and probing filters to see what gets past them.

Re:silently dropping is not unexpected

[liquidpele (663430)]

People violate the RFC because spammers spoof the sender as the people they are spamming, so the bounce goes back to that person and they get the spam. The RFC does not account for this, so fuck it.

Re:silently dropping is not unexpected

[AVee (557523)]

That would be true is google would actually first accept the email and then send a bounce message because it doesn't like it after all.
What they should do is reject the email immediately, in which case they don't have to send a bouce email but the mail is properly logged as being rejected. Ofcourse this does mean google will have to do all of their checks before accepting the message which is a bit harder to do but it is the only correct solution for the bounce problem.

Pull instead of push?

[Robotech_Master (14247)]

Doesn't GMail offer the ability to fetch your email from POP accounts now? It would probably not be the ideal solution, but perhaps you should stop forwarding and instead start POPping.

Re:Pull instead of push?

[i kan reed (749298)]

Or given the box of horrors that is POP, you could try IMAP, which google now also supports.

FAQ

[RzTen1 (1323533)]

There's actually a fairly simple procmail fix right on the spf site: http://www.openspf.org/FAQ/Forwarding [openspf.org]

You seem to have answered the question already

[RevDigger (4288)]

This is also known as, "The Problem With SPF." SPF breaks forwarding. This is well known. People who use SPF need to be aware of the ramifications.

The SPF people have created SRS, as you are aware, to work around this problem. It is a complicated and unappealing workaround. I certainly won't do it.

You have three options as I see it:

1) Stop forwarding. It's really a terrible idea. Install webmail on your mailserver. Check out RoundCube, for instance.
2) Wait for people to figure out that strict SPF policies break SMTP too badly for most users.
3) Implement SRS. (this would probably be easier if you were using a modern MTA)

I guess you were hoping for an easy fix, but there simply isn't one.

Re:maybe a silly question but..

[mkettler (6309)]

People over-generalize terms quite often, and "forwarding" has different meanings in different situations. Generally the difference boils down to if you're talking about a "server" implementation or a "mail client" implementation.

In this case, the SPF folks are addressing server admins, so by "forwarding" they mean sending the message to a new recipient without altering the headers. This use problably originates back to the old ".forward" files on unix machines, but may go back further. Most server-side implementations use this meaning for "forward".

However, forwarding by hitting the "forward" button most mail clients does something different. That creates a new message with new headers and preserves the old body text. sending with the same headers is called "redirect" in most mail clients.

Isn't it great how mail clients and mail servers use different meanings for the same word?

Even the client/server pair that go together from the same company have this problem. For example, Microsoft - exchange server has forwarding contacts, which forward without header changes, while Outlook clients do change the headers when you hit the "forward" button.

How to make it work

[stefanb (21140)]

Amazing what a bunch of unhelpful whiners take the time to *not* answer the actual question, and get modded up for it.

For this example, I'm assuming that your email is [e-mail] and your gmail address is [e-mail].

Create an alias (/etc/mail/aliases) for the address that get's forwarded to gmail.

joe: [e-mail]

Also create an alias for <foo>-owner:

joe-owner: joe

Sendmail will look for this special <foo>-owner alias whenever sending mail to the <foo> alias, and use it as the envelope sender on the outgoing mail. So any mail that is sent to [e-mail] will be resent by sendmail with a sender address of [e-mail]. The header addresses will remain unchanged, so hitting reply will still go to the right person.

Is this the solution to all SPF forwarding brokeness? Of course not, but it's a surpisingly simple solution to a number of common forwarding situation. Note that you better be careful about spam filtering on your machine, or your mail server (your sender's address) will appear to Google as a source of spam, and might get filtered.

Re:How to make it work

[stefanb (21140)]

Argh, RTFM really helps.

It's owner-<foo>, not the other way around. So the aliases example should read:

joe: [e-mail]
owner-joe: joe

See the aliases man page [freebsd.org] for further details.

Re:Easy answer

[SatanicPuppy (611928)]

That's outstandingly unhelpful. How about attaching a link to a decent SRS implementation [srs-socketmap.info]? Or sending them to OpenSPF [openspf.org]?

Randomly throwing down on people legitimately asking for some technical help is a big problem in the OSS community. Whether or not /. is the appropriate place to ask this question is debatable, but since it made the front page and there is no helpful SRS faq on this site, might as well direct them somewhere.

Re:I knew ..

[cayenne8 (626475)]

"I use my gmail account for catching all the junk mail you get for signing up for a mailing list. I guess i need to have my email server just send me a message stating that i have new mail waiting."

At first I was wondering why they hell someone that had a working email server would shuttle it through Gmail, but then I read about using the spam filters, etc.

While that sounds good on the surface, is anyone out there not a little apprehensive about having all your email, particularly if you're a business, going through and being stored on their servers? I mean, someday Google will bend completely for govt. wanting to search all emails for 'terrorists' activities, and God knows who else will too.

I guess I'd want a bit more privacy on my emails, especially if they contained sensitive or proprietary information. I know...they're in plain text and could be intercepted if not encrypted, but, this is altogether different. It is stored on google's servers and there for easy data mining.

I'm getting ready to dig out my old email server post Katrina...can you not use procmail and spamassassin to filter spam as effectively as Gmail does?

Re:Sunblock

[by Anonymous Coward]

I prefer SPF 60. It allows me to keep the pasty white, computer nerd complexion that drives the women away.

There, fixed that for ya.

Re:Sunblock

[Spy der Mann (805235)]

I prefer SPF 60. It allows me to keep the pasty white, computer nerd complexion that drives the women away.

There, fixed that for ya.

. o <-- joke
.
. </sarcasm> tag
. o <-- you
./|\
./ \

Re:Sunblock

[by Anonymous Coward]

company that makes servers.

Re:Sunblock

[onkelonkel (560274)]

Beware! The elders sometimes speak in hushed tones of "The Daystar". Its evil blinding rays will singe you to the finest ash and "The Wind" (like moving air from a cooling fan, but hideously amplified) will blow the ash away as if it had never existed.