Forgot your password?
typodupeerror
Communications Businesses Google Spam The Internet IT

Gmail, SPF, and Broken Email Forwarding? 300

Posted by timothy
from the rejected-mail-for-too-many-ellipses dept.
alek writes "I recently stopped getting Email from a friend ... which turns out to be related to his use of SPF records and my forwarding to gmail. This 'lost Email problem' may get worse with Google implementing Domain Keys." Alek is looking for a non-complicated solution to this non-trivial problem; read on below for more details.
"Background: Like many people, I have me@mydomain.com as my public facing Email address. When Email comes into my server, I forward it to me@gmail.com. But since my friend has published SPF (Sender Policy Framework) records that say only his server is allowed to send Emails for friend@frienddomain.com, gmail apparently rejects (silently buries actually!) the Email since it is forwarding through my server. Please note that this is exactly what SPF is designed to prevent — spammers from sending Emails with your address — but it breaks forwarding and has other problems.

What's *really* strange is that if I look at the raw sendmail logs on my server, the Email from friend@frienddomain.com comes in, and is forwarded to gmail ... with an "OK" as the response — i.e. the gmail MTA doesn't reject the message as it ideally should. However, the Email then disappears — it's not even in my gmail spam filter ... so there is no trace of it at all. If my friend sends directly to me@gmail.com, it shows up ... since his domain sends directly and the SPF test is passed. Note that on my gmail account, I associate me@mydomain.com with my me@gmail.com account ... so perhaps there should be a recipient test applied before SPF is tested on the sender ... although this arguably defeats the purpose of SPF.

The logical solution is to configure sendmail on my server to do Sender Rewriting — anyone have an easy FAQ to do this? But many people/domains aren't doing this ... and my Email forwarding to gmail is quite common, so I'm surprised that this issue hasn't gotten more attention. Is there another solution?"
This discussion has been archived. No new comments can be posted.

Gmail, SPF, and Broken Email Forwarding?

Comments Filter:
  • Sunblock (Score:4, Funny)

    by MyLongNickName (822545) on Thursday July 10, 2008 @03:25PM (#24140891) Journal

    I prefer SPF 60. It allows me to keep the pasty white, computer nerd complexion that drives the women wild.

  • Please adhere to RFC (Score:5, Informative)

    by DNS-and-BIND (461968) on Thursday July 10, 2008 @03:28PM (#24140927) Homepage
    Please stop using mydomain.com and other such nonsense. Example.com is reserved by RFC 2606 [ietf.org] for use as a...wait for it...example domain name. Please make a habit of using it instead of whatever name strikes your fancy, as it is probably in use by real people.

    The Internet Assigned Numbers Authority (IANA) also currently has the following second level domain names reserved which can be used as examples.
    • example.com
    • example.net
    • example.org
  • by jeffmeden (135043) on Thursday July 10, 2008 @03:29PM (#24140967) Homepage Journal
    Yes, of course. Have all your email sent to Google in the first place! You don't have to switch everything over to the Google app tool, you can just set MX records for your domain pointing to them, and collect it all (or forward it inside or outside Google.) It's free (with a paid version available.) Check it out here http://www.google.com/a/help/intl/en/index.html [google.com]
    • by dch24 (904899) on Thursday July 10, 2008 @03:33PM (#24141075) Journal
      It really works! (ob. disclaimer: satisfied customer)

      Our company forwards email to google (MX record in the DNS), where it runs through the spam filter and then a forwarding rule (an anything-but-spam rule) sends it on to our mailboxes.

      For free... :-)
      • Re: (Score:3, Informative)

        by TekPolitik (147802)

        Our company forwards email to google (MX record in the DNS), where it runs through the spam filter and then a forwarding rule (an anything-but-spam rule) sends it on to our mailboxes.

        Or you could just use Spamassassin, which properly configured is every bit as good as commercial offerings (and I have actually trialled them to do the comparison). If you put MAIA Mailguard [maiamailguard.com] on top of it, you have a solution that leaves the commercial offerings for dead - per user, server based sensitivity settings, quarantin

    • by klocwerk (48514) on Thursday July 10, 2008 @04:00PM (#24141677) Homepage

      Another satisfied google hosted apps customer chiming in. I have a reseller webhosting account that I keep about 10-15 domains on for myself/friends/family which does acceptable e-mail, but I advise everyone to just shove their e-mail over to gmail/a instead.

      You get your own hosted mail/webmail service with (currently) 7gb of storage per/account, no preset account limit, POP and IMAP, as well as great spam-filtering.
      All free.

      And for $50/acct/year you can have 25gb/acct storage, API access to customize it for single-signon and/or gateways, a full Postini implementation, and 99.9% uptime guarantee.

      Hate to sound like a shill, but it's a fantastic service and I don't mind pimping it.

    • Re: (Score:3, Informative)

      by jj00 (599158)
      I'll start by saying that I also use this service, and for the most part I like it. However, there are some downsides:
      • If you use Google Apps, you do not have as easy access to: Reader, Photos, and other Google utilities that are provided with a general GMail account.
      • You could use a regular GMail account, using POP/IMAP and have it send mail as if it was coming from that server. However, if you send an email to someone who uses Outlook - a message is tagged to the email (ex: from person@example.com on be
  • by mattbee (17533) <matthew@bytemark.co.uk> on Thursday July 10, 2008 @03:30PM (#24140991) Homepage

    Effective spam filtering for forwarded email is pretty much impossible, as you lose vital information in the forwarding. Either get rid of your forwarding address, or have it hosted at Google as well. Probably the largest single reduction in spam I've ever made was the week that I got rid of years-old forwarding addresses. If the forwarding address is more important, just get it hosted at Google directly, or tell people to stop using it!

  • If you are having problems with forwarded messages, then none of the emails from your server would make it in to gmail.
    Forwarded messages will have all the headers and information to indicate they came from your server.
    Bounced messages, where none of the headers are rewritten but it seems to come from your server, is the issue you are describing and it isn't one that I have an easy answer for.
    The only solution that I can think of would use greasemonkey and special rules on your server to make it easy
    • Re: (Score:3, Informative)

      by Sancho (17056) *

      Actually, the term "forwarding" applies both to client-forwarding (remailing the mail with all of the headers) and server-forwarding (what you call bouncing.) It's the difference between clicking forward and using a .forward file (hey, why do you think they called it that?)

  • by Ungrounded Lightning (62228) on Thursday July 10, 2008 @03:31PM (#24141025) Journal

    What's *really* strange is that if I look at the raw sendmail logs on my server, the Email from friend@frienddomain.com comes in, and is forwarded to gmail ... with an "OK" as the response -- i.e. the gmail MTA doesn't reject the message as it ideally should. However, the Email then disappears -- it's not even in my gmail spam filter ... so there is no trace of it at all.

    While the RFCs specify that an MTA that is dropping should notify the sender in various ways, modern MTAs often violate these parts of the spec, pretending to accept and then dropping the mail and/or failing to send bounce notifications.

    This is deliberate. Not sending bounce messages reduces the load on the servers and net (now that most mail traffic bounces). Pretending to accept mail which is actually dropped is a defense against guessing email addresses and probing filters to see what gets past them.

    • by X0563511 (793323) on Thursday July 10, 2008 @03:49PM (#24141409) Homepage Journal

      It violates RFCs and causes problems like we are reading about now. It needs to stop.

      • by liquidpele (663430) on Thursday July 10, 2008 @04:01PM (#24141697) Journal
        People violate the RFC because spammers spoof the sender as the people they are spamming, so the bounce goes back to that person and they get the spam. The RFC does not account for this, so fuck it.
        • by X0563511 (793323)

          No, fuck the spammer. Either respect the RFC, or come up with a solution with at least as much attention as the RFCs were given.

          Or, give up and come up with a proper solution from the start, and let traditional email rot.

          • by vux984 (928602) on Thursday July 10, 2008 @06:06PM (#24144191)

            No, fuck the spammer.

            Following the RFC fucks the innocent bystander, not the spammer. Is following the RFC worth fucking innocent bystanders over?

            Either respect the RFC, or come up with a solution with at least as much attention as the RFCs were given.

            In the meantime, while you come up with a solution, I'll disregard the RFC for this situation, because fucking innocent bystanders over while the world figures out a 'real solution' isn't acceptable.

          • Re: (Score:3, Insightful)

            by statusbar (314703)

            The rfc is broken, as it assumes no one would lie in their 'MAIL FROM' field.

            Will you fix it for us?

            --jeffk++

        • by AVee (557523) <slashdot.avee@org> on Thursday July 10, 2008 @04:27PM (#24142233) Homepage
          That would be true is google would actually first accept the email and then send a bounce message because it doesn't like it after all.
          What they should do is reject the email immediately, in which case they don't have to send a bouce email but the mail is properly logged as being rejected. Ofcourse this does mean google will have to do all of their checks before accepting the message which is a bit harder to do but it is the only correct solution for the bounce problem.
        • Re: (Score:3, Interesting)

          by geminidomino (614729) *

          People violate the RFC because spammers spoof the sender as the people they are spamming, so the bounce goes back to that person and they get the spam. The RFC does not account for this, so fuck it.

          Only if the mail admin is incompetent. This comes up every time there's a story about something in mail that's been screwed by spammers.

          Receiving mail server should not be sending ANYTHING to the sender's mail address, faked or not. The Receiving server's responsibility is to generate a 5xx error on a permanent error and send that back to the SENDING MTA. The Sending MTA has the responsibility to generate the appropriate mailer-daemon message

      • That means you Hotmail!

      • by Anonymous Brave Guy (457657) on Thursday July 10, 2008 @08:16PM (#24145773)

        It violates RFCs

        I'm giving up mods to post this, but it really needs to be said.

        People need to stop blaming things on services who pragmatically choose to violate selected aspects of decades-old standards that don't address today's realities. The problem with modern e-mail is that the standard is hopelessly out of touch with modern demands. There should long ago have been a consistent standard that covered things like sender authentication, encryption and signing, formatted messages ("HTML e-mails"), smart handling of errors without treating them all as e-mails in their own right, and numerous other fundamentally broken parts of the original e-mail specs. But there isn't, so people try to do reasonable things and stay as true to the standard as they can without being dogmatic about it when it's obviously a stupid thing to do.

        So no, I don't think silent dropping needs to stop under all circumstances. E-mail has never had useful reliability of delivery (another thing a replacement standard should deal with) so you can't count on it anyway. On the other hand, I'm sick and tired of getting a deluge of hundreds of unwanted e-mails in ten minutes because someone sent out a mail with webmaster@my.domain as the sender, and loads of people who were confident enough that the message was spam to block it still sent back a bounce message to an address that is 99.99% likely to have been faked as well in that case. I'm sorry, but that's just antisocial behaviour, and responsible sysadmins should take steps to avoid it: if you're confident enough to refuse delivery, why aren't you confident enough not to reverse-spam the innocent bystander? If you're running a sensible service where a user can whitelist specific senders or switch off spam filtering altogether for specific receiving addresses if they want to guarantee receiving everything, and they've opted in to your spam filtering, this shouldn't be a problem.

    • Re: (Score:3, Funny)

      by Klaus_1250 (987230)

      Hotmail has been doing the same for years... And it is bad bad bad. There is a reason for those RFC's you know. I've had several complaints from people that I was loosing their mail. Checked the server logs and the mails were sent to Hotmail and it replied with a nice message received and accepted. Yet it dropped them afterwards even though it was 100% Ham. Fantastic. I get complaint about their mistakes, it takes me time and effort, and best of all, you can't contact them about it.

  • by Robotech_Master (14247) on Thursday July 10, 2008 @03:32PM (#24141039) Homepage Journal

    Doesn't GMail offer the ability to fetch your email from POP accounts now? It would probably not be the ideal solution, but perhaps you should stop forwarding and instead start POPping.

    • and wtf would one want to enable pop on a server that is already doing IMAP just fine? Maybe google should implement IMAP checking, then I wouldn't have to forward (it's temporary until my own web server is back online, but it certainly is convenient).

    • by i kan reed (749298) on Thursday July 10, 2008 @03:44PM (#24141317) Homepage Journal

      Or given the box of horrors that is POP, you could try IMAP, which google now also supports.

      • Re: (Score:3, Interesting)

        by ceka (1092107)

        ... IMAP, which google now also supports.

        Gmail claims to support IMAP, but if you try really using it, its awful.

        Eg deleting an email from my mail client inbox only removes it from the Inbox label, it still stays in AllMail. And deleting from AllMail is impossible, the email reappears in thunderbird in a minute or two. Deleting attachments doesnt work. Etc.

        I understand they want to keep my data as long as possible and also that they want to make IMAP work with their Labels, but I don't care I just want an IMAP compliant email account...

    • by Loether (769074) on Thursday July 10, 2008 @03:55PM (#24141557) Homepage

      gmail does let you pull via pop3 BUT the scheduler is not configurable. Gmail checks pop randomly when it feels like it. For me it's about every 30 minutes to 1 hour. YMMV

    • by tgd (2822)

      Or better yet don't do either, just have the e-mail go to gmail. Google Apps for Domains is free, and less clunky than forwarding.

    • by ady1 (873490) *

      I'm using that for an old account and trust me, that is extremely slow. As one poster already pointed out, you can just host the email on google instead of forwarding it which is not efficient in the first place.

    • That means you have to implement an additional mail server program that does POP, as forwarding only requires SMTP.

  • by thadman08 (732965) on Thursday July 10, 2008 @03:34PM (#24141087) Homepage

    Domain Keys authenticates that the message was generated by a server with access to the DK private key. Forwarding the message does not affect the originator of the message, so the Domain Key authentication still checks out.

    SPF and DKs solve similar issues, but in a much different manner.

  • by ahecht (567934) on Thursday July 10, 2008 @03:45PM (#24141333) Homepage

    Sign up for Google Apps, and then you can have all mail sent to me@mydomain.com be handled by GMail. All you have to do is sign up at http://www.google.com/a/ [google.com] and link your domain. Then point your domain's MX records to aspmx.l.google.com.

    In the future, all you have to do in order to get your mail is to go to http://mail.google.com/a/mydomain.com/ instead of http://www.gmail.com (and you can even set it up so that http://mail.mydomain.com CNAMES to your email login page)

  • FAQ (Score:5, Informative)

    by RzTen1 (1323533) on Thursday July 10, 2008 @03:47PM (#24141355)
    There's actually a fairly simple procmail fix right on the spf site: http://www.openspf.org/FAQ/Forwarding [openspf.org]
  • This is also known as, "The Problem With SPF." SPF breaks forwarding. This is well known. People who use SPF need to be aware of the ramifications.

    The SPF people have created SRS, as you are aware, to work around this problem. It is a complicated and unappealing workaround. I certainly won't do it.

    You have three options as I see it:

    1) Stop forwarding. It's really a terrible idea. Install webmail on your mailserver. Check out RoundCube, for instance.
    2) Wait for people to figure out that strict SPF policies break SMTP too badly for most users.
    3) Implement SRS. (this would probably be easier if you were using a modern MTA)

    I guess you were hoping for an easy fix, but there simply isn't one.

    • by BobMcD (601576)

      Or...

      1) Stop forwarding...

      ...and use Gmail to fetch stored mail instead.

  • Proper forwarding should rewrite the SMTP envelope sender (leaving the "From" header intact). There's just no other way to do it that doesn't break with SPF and other things these days.

    Yes, that means the new sender address will have to be valid. Yes, that means it'll look like spam is coming from your domain if your forwarding service is easy to abuse. You might also want to preserve what's happened in headers for future reference and debugging uses, and rewrite the SMTP envelope sender to something that m

  • by jafo (11982) on Thursday July 10, 2008 @04:18PM (#24142031) Homepage

    There's an easy way to do e-mail forwarding, which unfortunately is wrong. We no longer live in a world where you can just create a .forward file with the destination address in it (unless it's on the same server).

    If you're going to run your own mail server, there are things you need to do if you want it to run correctly. One of them is that if you are forwarding to a mail server that does SPF, you need to do SRS. Though you probably also need to be doing all the spam rejection on your mail server as well, because otherwise you may be allowing mail through that you wouldn't otherwise.

    For example, say that your server doesn't check SPF, and you do SRS. Now you're basically bypassing the destination server's SPF checking.

    How to do SRS? I would personally probably just change my .forward file from the destination address into a small script that re-injects the message with a different envelope sender, but I'm sure there are already scripts that do this and much more fancy....

    Ideally, you probably just want to move your mail for your domain directly to google, as another repondant says. Don't have it shunting your your own server if at all possible. If you have mail that you want handled directly on your server, either forward it from gmail to your home machine, or use a different domain ("address@homebox.example.com").

    Sean

    • Re: (Score:3, Interesting)

      by Sancho (17056) *

      How do you deal with the problem of being blacklisted as a spammer if you end up forwarding lots of spam mail off of your domain? Remember, SPF itself doesn't address the problem of spam, so the fact that you're checking SPF doesn't matter a lot in this regard.

  • Gmail Is Broken (Score:2, Flamebait)

    by RAMMS+EIN (578166)

    Gmail has been silently dropping emails for as long as I remember. It's broken, and that's yet another reason I don't use it.

  • I am not rereading the specification, so I might be wrong.

    SPF probably checks the Envelope Address and not the From: address which are not the same.

    The envelope address is the address that the SMTP client says to the server who is the sender,
    the From: address is what is in the message header.

    Simply altering the Envelope Address to a valid mail from your server and google wont complain anymore.

  • by Rashkae (59673) on Thursday July 10, 2008 @04:39PM (#24142543) Homepage

    SPF will validate the Return-path header if there is one instead of the From: address.

    Unfortunately, I don't know how to make either sendmail or postfix insert a return path when they forward an e-mail, but the easy work around is to install mail list software as your forwarder. You can create a mailing list as your incoming e-mail, with only 1 mail list member, (which is your g-mail account). Mail list software will automagically insert the appropriate return-path header that is needed in this case.

  • by ZOP (240653) * on Thursday July 10, 2008 @04:50PM (#24142781) Homepage

    DKIM and DomainKeys work in a fundamentally different way. The message is SIGNED. Hosts are not indicated one way or the other. So any DKIM signed mail can transit any number of hosts provided they don't modify the signed sections.

    SPF has no such luxury unless implemented in a much more advanced manner in terms of the senders publishing. And it's not GMail's fault for following the SPF records as published, they should do a better job of rejecting early rather than just /dev/null-ing the email though.

  • How to make it work (Score:5, Informative)

    by stefanb (21140) * on Thursday July 10, 2008 @05:04PM (#24143067) Homepage
    Amazing what a bunch of unhelpful whiners take the time to *not* answer the actual question, and get modded up for it.

    For this example, I'm assuming that your email is joe@example.com and your gmail address is joe-example@gmail.com.

    Create an alias (/etc/mail/aliases) for the address that get's forwarded to gmail.

    joe: joe-example@gmail.com

    Also create an alias for <foo>-owner:

    joe-owner: joe

    Sendmail will look for this special <foo>-owner alias whenever sending mail to the <foo> alias, and use it as the envelope sender on the outgoing mail. So any mail that is sent to joe@example.com will be resent by sendmail with a sender address of joe@example.com. The header addresses will remain unchanged, so hitting reply will still go to the right person.

    Is this the solution to all SPF forwarding brokeness? Of course not, but it's a surpisingly simple solution to a number of common forwarding situation. Note that you better be careful about spam filtering on your machine, or your mail server (your sender's address) will appear to Google as a source of spam, and might get filtered.

  • SPF, Gmail, and SRS (Score:5, Informative)

    by statemachine (840641) on Thursday July 10, 2008 @05:40PM (#24143755)

    Since you are running your own SMTP server, you signed on to be a sysadmin. I am replying to you as a fellow sysadmin and I'll give sysadmin-style answers. Please don't take my response to be negative in any way, as I'm trying to help.

    The logical solution is to configure sendmail on my server to do Sender Rewriting [openspf.org] -- anyone have an easy FAQ to do this?

    If you follow the link that you just gave for Sender Rewriting, it answers your question. "Implementation" links to modules, source, and configurations.

    But many people/domains aren't doing this ... and my Email forwarding to gmail is quite common, so I'm surprised that this issue hasn't gotten more attention. Is there another solution?"

    I say that you don't know how many people are implementing SRS, nor do you know how many forward e-mail to Gmail. Let's stick to the basics before giving up so readily. I take it that you absolutely do not want to give up carte blanche forwarding from your own SMTP server to Gmail; so I'll tailor my reply to that.

    But since my friend has published SPF (Sender Policy Framework) records that say only his server is allowed to send Emails for friend@frienddomain.com, gmail apparently rejects (silently buries actually!) the Email since it is forwarding through my server.

    Your friend has published an SPF record because he doesn't want people forging his domain in the envelope-sender field. This is a common spam tactic that ruins the reputation of someone's domain, either through spammer apathy or sometimes pure malice. Your e-mail forwarding (especially since you run your own SMTP) to Gmail is out of pure convenience to you and is unnecessary, so don't ask your friend to drop his SPF record.

    There are two ways to solve this:
    1) Have your friend add your SMTP server to his SPF record.
    2) Implement SRS if you want to solve it once and for all. If you follow your own links, there are explanations, examples, and actual code. You haven't said which SMTP server you're running, so you've limited the responses people can give you for your situation.

    I publish SPF records for my domains. There isn't anything "broken" about wanting to protect my domains' reputations from forgery. Very few people have a problem with forwarding that they didn't create themselves. This exception I'm talking about is people who have old university accounts (or similar) which only allow e-mail checking through a shell account and forwarding purely through a ".forward" file (or similar), with no POP, IMAP, or administrative access. This is not you. But for anyone who this describes, because of the draconian service policies, they shouldn't be giving out that e-mail address to new contacts, publish on papers, etc.

    My SMTP server checks SPF, but not DK. With SPF, the forged domains are instantly rejected, requiring minimal overhead. DK requires reception of the entire message (because the headers are in the DATA phase) in order to validate the message, on every message -- this uses unnecessary network bandwidth, and it places an extra load on my system since it would have to calculate and verify signatures for every single message. Maybe that's not an issue for you if you only receive a handful a day, but I receive thousands. Spammers know that including fake DK info in a message and then sending millions of these is effectively a Denial of Service attack on the servers that indiscriminately check DK signatures.

    I also use backup relays. For the relays that are not under my control and don't implement SRS, I simply bypass SPF checks from those IP addresses.

    About Google silently dropping your e-mail: Keep in mind that with your carte blanche forwarding, you're also forwarding spam. You are essentially spamming Gmail, even though it is you simply forwarding e-mail to your own account. It is difficult for Google to know this without human intervention or implementing some co

    • Re: (Score:3, Informative)

      by statemachine (840641)

      Replying to myself because I just spotted the article submitter did mention "sendmail" as his solution. There are plenty of solutions readily available for sendmail. Like I said above, he can follow his own links for that information, and many others here have helpfully posted sendmail solutions also.

      I don't know why my eyes filtered out sendmail. Odd.

  • by tonyray (215820) on Thursday July 10, 2008 @07:33PM (#24145351)

    One very good reason not to have your email address @gmail.com, if you are using it for your business, is that a LOT of businesses, wholesale vendors, even the federal government will not accept an @gmail.com address because of the large number of frauds associated with free email accounts (not just gmail, but also hotmail, yahoo mail, etc.) For example, this last tax season the federal govenment would not accept a gmail account for notification of your tax return status when filing electronically.

    It is much better from a business standpoint to have your own domain and email sent to your domain. If your MX points at gmail, that's okay. Just don't make your email address me@gmail.com if you want to be taken seriously.

  • by n.e.watson (835126) on Thursday July 10, 2008 @08:43PM (#24146057) Journal
    Have your friend look up the SPF records for a bunch of big domains. He'll notice that most of them use "~all" - a SoftFail - which is accepted by Gmail. He's probably using "-all," which makes the message just drop. The only examples I've seen of SPF hardfails in the wild are from banks. However, loads of domains are using softfail - Facebook, Google, Microsoft, eBay, MIT, UC Berkeley - to name a few.

Never make anything simple and efficient when a way can be found to make it complex and wonderful.

Working...