Providers Ignoring DNS TTL? 445
cluge asks: "It seems that several large providers give their users DNS servers that simply ignore DNS time to live (TTL). Over the past decade I've seen this from time to time. Recently it seems to be a pandemic, affecting very large cable/broadband and dial up networks. Performing a few tests against our broadband cable provider has shown that only one of the three provided DNS servers picked up a change in seven days or less. After turning in a trouble ticket with that provider - two of the three provided DNS servers were responding correct - while the third was still providing bad information more than two weeks after that specific change. What DNS caches ignore TTL by default? Is there a valid technical reason to ignore TTL?"
"This struck me as odd, and I decided to run a few tests using my own domain. Lowering the TTL to twenty four hours, and making changes and then checking to see when a change was picked up. I queried twelve outside DNS servers/caches that I had access to (Thanks to my friends and relatives with dial ups and DSL who put up with me and my requests to reboot their machine daily!). Checks performed against these outside DNS servers indicate that it may take as much as four to five weeks before a DNS change is picked up! Most DNS servers picked up the change within 48 hours. A small number did not (three out of twelve - that's a quarter of them!)
This merits more study, and prompts a few questions. So, before I begin with a more serious broad study, I'd like to get some feedback on the problem as I've seen it. I know the tin foil hat crowd will see the failure to propagate DNS correctly as censorship, and the OS/bind/djb/whatever zealots will simply see this as an argument for their particular religion.
Based on the responses I get, I will then setup and test a couple of domains with different DNS servers for 6 weeks and report back the findings. [volunteers welcome!]"
Dumb question (Score:5, Interesting)
Re:Dumb question (Score:5, Informative)
For example
dig @your-isps-nameserver.net -t A www.example.com
For example:Note the TTL of 7184 seconds (this is how long the nameserver at 192.168.0.1 will continue to use the cached record for before fetching it again from slashdot.org's authoratative nameservers).
Re:Dumb question (Score:5, Informative)
TTL's (Score:5, Funny)
Let me guess... (Score:4, Interesting)
They can't run a DNS server properly to save their lives.
TTL is ignored, SpamAssassin is a "trojan DDOSing our network"
Re:Let me guess... (Score:3, Informative)
People running MailWasher on Windows also got the same warning from RR. All this was probably about a year ago.
I Noticed Too (Score:4, Insightful)
Re:I Noticed Too (Score:5, Insightful)
They've actually had to go to extra effort to break it on purpose.
24 hours ? (Score:4, Informative)
Re:24 hours ? (Score:4, Informative)
(For those that haven't messed with Akamai, they're intentionally setting the TTL insanely low to force clients to re-request often...Akamai uses the response they give as a way of doing path optimization to clients. It's ugly, but it kinda works.)
Re:24 hours ? (Score:3, Interesting)
If other people do this as well ISPs might not have much of a choice but to ignore TTL and maybe there isn't any easy way to force the machine to use a minimum TTL but respect TTL above that limit.
DNS practices (Score:5, Informative)
But I don't think they're setting a TTL longer than 24 hours, that would be kind of insane, isn't? At least from my own experience when I did a big DNS servers change (changed all the serials) the delay was less than 24 hours for almost all of them.
Re:DNS practices (Score:2, Informative)
Re:DNS practices --- CHANGE THE !@#$%^& serial (Score:3, Interesting)
Waaa.. Waaa.. somebody ignored my TTL.
Listen -- We are SOA for around 11,000 domains. Both myself and the other uber-admins get tickets like this "escalated" when some clueless newbie wet behind the ears freaking junior admin DOESN'T RTFM and doesn't realize that if the serial #'s don't change then TTL is ignored.
We interface with nearly every major ISP -- I assure you, their name servers are configured just fine --- It's yours that is broke.
For those of you who just aren't DNS afficiandos -- so how ret
Re:DNS practices --- CHANGE THE !@#$%^& serial (Score:3, Insightful)
But I have seen once when we had changed the serial, we had lowered the TTL for the week preceeding, and yet there were DNS servers out there that just refused to update. (AOL being one of them).
After we hit two weeks, and the IP still hadn't propagated, I did some digging -- somehow, 4 of the root name servers were forwarding queries to two development DNS servers that someone had set up, which
AOL: wayback machine for DNS (Score:3, Funny)
On the plus side, I've used AOL to find out what the IP of names *used* to be while researching problems. Kind of handy that way.
Re:DNS practices --- CHANGE THE !@#$%^& serial (Score:3, Informative)
Ok, can you point me to anywhere in RFC1034/RFC1035/RFC2308/etc that says that the SOA record has anything to do with the TTL? The nTTL, yes, but not the TTL. Yeah, if they don't change the serial number, their secondary name serve
Re:DNS practices --- CHANGE THE !@#$%^& serial (Score:5, Informative)
Serials are primarily for the two servers do get the same data (primary/secondary), so when the secondary is done waiting it goes to look at the serial on the primary and grabs the new zone transfer if the serial is higher.
TTL on an A record is just a recomendation (a specific setting that over-rides the default TTL for the zone up near the SOA).
IF a server has cached an A record with a TTL of 6000 seconds (just under 2 hours) it should hold and server data for only a maximum of 6000 seconds, and after that time dump the data and go get new data from the authoritative name servers.
If you do a DIG against them, they'll tell you how much time is left on a cached record.
Serial doesnt come into the "when to drop cached data" transaction at all.
Sure, not incrementing the serial can cause all sorts of problems. But that's not what the article is on about.
AOL et. al are ignoring specific A record TTL and putting their OWN TTL on cached information that over-rides mine. (I know this because the tool I use makes it so I CANT forget to incriment the serial, and I still run into TTL problems. What about that smartypants?) So when I set a domain from default to 3600 seconds a day before an MX record (email server) change and they ignore it, email migration from one server to another stays messed up for days rather than the hour my TTL would do. A good admin doesnt abuse TTL (like yahoo apparently does...) and sets it back up higher when finished moving stuff, most of the time I am prefectly happy with the nice long standard cache time. But sometimes you NEED a low TTL.
I got the O'Reilly Grasshopper book right here in front of me and none of the TTL sections mentions SOA needing increment for TTL caching. If someone wants to point out a page number that says I am wrong I'd be happy to shut up. But self-righteous indignation better be fact checked... seriously.
Nothing new here (Score:4, Interesting)
Hence whenever someone tells me that they're going to move their web hosting, I always advise them to allow for a couple of weeks of overlap, so that they don't lose a ton of traffic. Often they ignore my advice, because of course they have set their TTL low so their changes will take effect immediately, or so they think.
And then they wonder why they're getting hundreds of e-mails from people telling them that the site is down. And I forward them a copy of the e-mail I sent them beforehand warning them of the problem.
My gut feeling is that screaming about other people's DNS servers refusing to observe your TTLs is going to get you about as far as screaming about other people's SMTP servers refusing to deliver your spam. It's their server, they can do what they like. For instance, any TTL under 24 hours will be ignored by my caching DNS server. (RFCs say TTLs should be at least a day, more like 1-2 weeks.)
Re:DNS practices --- CHANGE THE !@#$%^& serial (Score:3, Insightful)
I don't know why you assume everyone should know what you meant. The rest of your hateful post made you look uninformed so folks probably generally presumed you were just a newbie admin with an inflated ego.
And why would they bump the TTL on their nameserver,
Re:DNS practices (Score:3, Interesting)
Why?
nscd (Score:4, Informative)
We use nscd quite a bit, as im sure many other providers do. We only cache positives for 30 minutes, so we dont end up ignoring it for too long.
Re:nscd (Score:3, Informative)
Yeah but nscd just caches for your local server box, it doesn't re-serve the cached results to your remote clients. He's describing actual dns cache/forward servers ignoring TTL and handing outdated/bad data to client machines.
maybe a consequence of the cache poisoning attacks (Score:2, Interesting)
Maybe some ISPs have altered their DNS servers to provide better protection, and in the process caused the 'ttl' problem ? (improbable imo, but who knows...)
It'd be interesting to know if this is recent or if it's already an old problem.
Knowing if it appeared suddenly or progressively would help as well.
Too bad there isn't such a thing as the wayback machine for dns and other servic
Spammer zombies too (Score:2, Interesting)
You can use TTL to keep customers from leaving! (Score:5, Funny)
Re:You can use TTL to keep customers from leaving! (Score:4, Interesting)
Re:You can use TTL to keep customers from leaving! (Score:3, Insightful)
old TTL? (Score:3, Insightful)
If you want to help then (Score:5, Informative)
dns-subscribe@angrypeoplerule.com
This is a moderated list, and is only for letting people who are interested know when the study will begin, how to participate and the final results.
Re:If you want to help then (Score:3, Insightful)
Subscribe to the list and find out.
So you're developing the methodology for something you've already done?
a troll, or just an ignoramus.
Mu. If you ask people for help, then insult them when they respond with legitimate concerns, you're going to have a tough time getting recruits. My statement was based entirely on the text you wrote, which was liberally peppered with statements that show you do not have a strong grasp of DNS concepts. Specifically:
You appeared to i
Re:If you want to help then (Score:3, Insightful)
If you don't want to volunteer, Fine.
If you think that the poster doesn't know how to plan the test; go make your own test.
He specifically said the test methodology will be improved on suggestions on the mailing list. How about contributing there instead of making an uppity jackass of yourself here.
(Would like to post my rant AC, but I don't want you think the GP is responsible for this reply)
old data (Score:5, Informative)
And then there's the times when I just plain forgot to bump the serial number field. Works great on my master server after I restart it, but nothing else (especially my secondary) notices the change.
Efficiency reasons ? (Score:2, Interesting)
I don't really know in-depth DNS mechanisms, but maybe ISPs are keeping a minimum TTL according to the average time between two updates of a given DNS entry ?
What's the point of not updating anyway... (Score:5, Interesting)
So, what's it really save you, even if you're a massive ISP to not obey the TTL's?
The only thing it's going to save you is from having to go out to the root servers and pull it again when the TTL expires. And, I speculate that this really is a very, *very* small amount of traffic compared to the other traffic to those servers.
I'd expect the highest bandwidth/resource users, by a very large margin, to be standard "in-TTL" answers to DNS clients.
So, these cranks, for lack of a better term, simply bork the systems they manage for no appreciable gain from doing so. Reducing spam by 0.0001% would have vastly more impact on the servers they maintain than ignoring TTL's.
Has anyone done any measurement stats on DNS queries. How much of the total traffic is DNS. I can't imagine it's even 0.5% of an average ISP.
Cheers,
Greg
Re:What's the point of not updating anyway... (Score:4, Informative)
According to my DNS hosting company's FAQ [zoneedit.com]:
"...or 200MB of usage is used (1 million DNS queries)"
Re:What's the point of not updating anyway... (Score:4, Interesting)
It's an underestimate, if anything.
I know AOL used to be an offender, likely still is (Score:5, Interesting)
Re:I know AOL used to be an offender, likely still (Score:5, Insightful)
I'd be curious who's the audience for the site(s) you're talking about. I'm pretty uncomfortable calling tens of millions of users unimportant, especially when it comes to e-commerce. Different "additude", I guess. Or attitude, even.
I maintain an ancient AOL account specifically so I can see things the way that some of my customers' customers see them. But it has one other advantage: if I've just made DNS changes to domain I care about, I set up a temporary new A record (like X.whatever.com) and then surf to it through AOL's proxies. This seems to get their name servers to notice that the SOA record is new, and it flushes out the rest of their cache. This seems to work on all sorts of servers, most of the time.
Is there a valid technical reason to ignore TTL? (Score:2)
But then, if you are a huge ISP, you probably don't care much about such things. As always, money trumps common courtesy.
reboot? (Score:5, Informative)
ipconfig
BW (Score:2, Insightful)
The other problem is lazy or incompetant sysadmins...
Classical Economics (Score:2)
Test case (Score:4, Insightful)
The TTL should stay the same for a while and then try simply making a change without modifying any other configuration to avoid any other problems with this test.
Why would you reboot? (Score:5, Insightful)
If you're rebooting client machines to check DNS records, then I'm forced to view your entire study with caution.
Re:Why would you reboot? (Score:3, Insightful)
I'd imagine rebooting was easier.
Re:Why would you reboot? (Score:4, Funny)
"Ok, grandma, open the start menu, now select run. Ok, now type c-m-d. No, grandma, m. MMMMM. M as in Mike. Ok. No, grandma, D. DEEEEE. Not g. D. Ok, now did a big black box open up? No? Oh, you're on Windows 95/98, you'll need to reboot."
comcast (Score:2, Informative)
quick fix (Score:3, Informative)
How to check your DNS (Score:4, Informative)
Why did you need to contact your friends/relatives to check whether or not your domain gets propagated?
Couldn't you just query DNS servers directly using nslookup and/or dig?
Querying them directly would eliminate you from wondering if the machine you are checking from has the DNS cached and you wouln't need to flush it (why would you need your friends/relatives to reboot their machines?). Not to mention the amount of time you would spend in having to coordinate this type of testing.
Even if you don't want to use nslookup and/or dig from your Windows/Linux/Mac/whatever, there are tools available via the web that can help as well.
This certainly is not a list of all the tools, or even the best ones... they're just ones that I have used in the past:
dig [kloth.net] Web-based "dig" tool
nslookup [kloth.net] Web-based "nslookup" tool
DNS Report [dnsreport.com] Checks for DNS errors and provides nicely formatted information on a given domain
DNS Stuff [dnsstuff.com] Various web-based DNS tools
direcway.com SUCKS (Score:4, Interesting)
One of my customers was behind their network and we moved his email to our server. They couldn't access their domain name of course since it didn't exist on the server direcway's dns pointed to.
So I called them. Huge mistake. I spent hours on the phone escalating through foreign phone monkies until I made it to someone in management. Her attitude was that I was in the wrong regardless of what I had to say. Finally she lowered her defense just long enough to see that I was right but told me there was nothing they could do and that I wasn't allowed to talk to the people that run the DNS servers.
So I wrote a nasty little letter to corporate. 4 days later it was fixed. Not sure if the letter helped or not.
Spam still coming to old IP address (Score:2)
As an aside, whitelisting our valid email accounts and filtering mail "received from" our own IP addy knocks out about 20-30% of our spam. It appears spammers try to sneek one past by spoofing the header to look like it came from inside. If so, spammers use DNS to get IP addys and then use the I
People abusing it on the other end... (Score:4, Interesting)
Re:People abusing it on the other end... (Score:2, Insightful)
Re:People abusing it on the other end... (Score:4, Insightful)
If DNS bandwidth were a non-issue, you wouldn't even be able to set TTL -- it would just be hardwired to a small value. But it is. Don't lapse into spammer logic: "I'm only wasting a tiny amount of resources, so it doesn't matter." But it does matter, because lots of other people are thinking the same thing.
not uncommon (Score:2)
Annoying, but true. (Score:2)
Once it's in DNS, it's going to stay there for a long time. Any transition plan has to monitor traffic on the old IP over the course of days to see when 99% of the traffic has tailed off. It sucks, but there it is. He who owns the DNS server, 0wnz the net.
You did what? (Score:3, Insightful)
ipconfig
ipconfig
But wouldn't an easier way be just using dig to directly query the name servers?
Re:You did what? (Score:3, Insightful)
Article is right on the money (Score:5, Informative)
In all there were 3 large US isps that were major offenders...
We have had this issue for years .. (Score:3, Informative)
A solution to this problem would be a law, that would create a set of standard services that a comunications company may give, with well defined names and categorys, and it should be MANDATORY for companys to market their services using this names, in their comercials too. So, for example, we would have categorys such as "Full Duplex Simetric DSL Conection", or "ADSL, With Proxy, Blocked Ports".
save money - set your ttl to 2147483647 (Score:3, Funny)
been a problem for a long time (Score:2)
The problem was not consistent, some of their servers displayed it and others (lesser number) did not - I believe tha
Spam, spam, spam (Score:5, Interesting)
I would expect, now almost two months later, to be getting -0- traffic to the
Technically -- even running your own DNS servers there is nothing you can do if you move/add/delete something and others out there decide not to honor it. Everybody loses.
AOL used to do this (and probably still does) (Score:3, Insightful)
Caching provides a response much more quickly (albeit not always right), and for a large scale ISP, DNS lookups consume not-insignificant amounts of bandwidth. This used to cost much more than it does today, and I'll bet much of this continues out of intertia.
Where to begin... (Score:3, Insightful)
Asking friends to REBOOT? Why not just ipconfig
I also have a really hard time taking someone seriously that, in the opening question, mentions something like "well, zealots will argue, and tin foil hats will bitch" or whatever. Yea, he's really unbiased..
TTL affects the time you should cache the records, at least he seems to get this. So, he can't think of one reason why a large ISP might want to ignore TTL's?
I'll name a couple and leave it to this guy to fill in the rest:
A) Because a lot of really terrible DNS admins set the value way too low and leave it there?
B) Because ISP's might have a need to keep their cache database activity to a resonable level?
GO on with your study! The results will probably prove to be very uninteresting.
New TTL won't take effect until old TTL runs out (Score:4, Insightful)
In fact, servers that are obeying TTL won't see the new record until the old record's TTL expires.
The querent doesn't say whether or not there was any wait for the old TTL to expire. They don't even mention what the old TTL was!
Ignoring TTLs not necessarily intentional (Score:5, Insightful)
1. Change in the hosting of a domain to new DNS servers without properly removing the domain from the old hosting DNS servers.
When this happens, a DNS server caching a domain's info will continue to check the old servers until the old server stops answering.
2. A change in the TTL of a domain to a lesser value.
If you change the TTL of a domain from 7 days to 1 hour, DNS servers currently caching that domain's information will hold onto it for 7 days before discovering the new TTL.
3. A bug in BIND 8 that prevents it from pulling updated information from the primary DNS server for a domain.
We see this rarely, but it requires a restart of an affected DNS server. We have not diagnosed the specific cause yet since we're moving servers to BIND 9.
Scientific article about this (Score:5, Informative)
Re:Faulty system (Score:5, Insightful)
Re:Faulty system (Score:2, Funny)
Bypass their DNS (Score:5, Interesting)
I know it's not a solution for everyone, but it does let me avoid stupidity. And having my own, reliable DNS server sure came in handy recently since Comcast has been having bad DNS problems the last couple of weeks.
Re:Bypass their DNS (Score:3, Interesting)
Even if you run your own DNS, your server still has to pull its data from some root server, and if that server isn't reliable, then your server isn't reliable either. You could avoid this by pulling from THE root DNS servers, but if everyone did that it would
Re:Bypass their DNS (Score:5, Informative)
Re:Bypass their DNS (Score:5, Informative)
That's not how DNS works.
The root servers simply point you into the direction of the authorative DNS server for a given domain name. That is why you have to register who is going to be the DNS server for any given domain so the root servers can point people to it. Your own DNS then caches the response from the DNS server (not the root) locally, only updating it after the TTL is expired (which isn't always happening with the provider's DNS, hence the problem).
The root servers are reliable... they have to be. Sure there have been DoS attacks and the like on them before, but they only need to update themselves for new domain name server registrations (which last I heard is every 5 minutes? So that's a much better "ttl").
Re:Bypass their DNS (Score:5, Informative)
Specifics per implementation might be off, but either way it ends in the same result:
Recursive -> Root Server: "ANY? www.google.com"
Root Server -> Recursive: "com NS a.gtld-servers.net
Recursive -> a.gtld-servers.net: "ANY? www.google.com"
a.gtld-servers.net -> Recursive: "google.com NS ns1.google.com
Recursive -> ns1.google.com: "ANY? www.google.com"
ns1.google.com -> Recursive: "www.google.com A 1.2.3.4
As you can see, the root server only provides information for the top level domains. Those being com, org, us, uk, au, etc.
It's commonly thought that they handle things like 'google.com' which isn't true. google.com, in thise case, would be known by {a,b,c,d,etc}.gtld-servers.net. Each TLD has its own nameservers, obviously. But com and net use those.
As for the TTL issue. I do offer Dynamic DNS which has a default TTL of 180 seconds, however I have not run into this personally. Or myself and my users just haven't noticed it.
Regards,
-JD-
Re:Bypass their DNS (Score:3, Informative)
That is to say, if i query a.gtld-servers.net for www.bob.com IN A, and bob.com is delegated to ns1.bob.com and ns2.bob.com, the registry will know the IP addresses of bob.com's two nameservers and return them in the 'additional' section (otherwise nobody would ever be able to find anything under bob.com, including ns1 and ns2.bob.com).
Re:Bypass their DNS (Score:3, Informative)
Re:Bypass their DNS (Score:3, Informative)
Re:Bypass their DNS (Score:3, Informative)
Re:Bypass their DNS (Score:3, Funny)
You're not banking in the clear on http: are you? On an unpatched Win box? With IE?
Of course not. That's what telnet's for.
Re:Bypass their DNS (Score:3, Insightful)
Re:Bypass their DNS (Score:3, Informative)
Setting up a proper DNS server isn't too hard (as indicated by the number of posters that have done just that). However, it does take a bit of knowledge about how DNS really works. To that end I suggest you read some books about Networking, and DNS in particular.
I've found the O'Reilly books to be fairly easy to read while providing a great starting point for those that have a broad, basic understanding of how networks (and
ELOGICFAULT (Score:5, Insightful)
Besides, this behavior blows up all sorts of geographical load balancing, datacenter failover, etc. type solutions (google for a F5 3DNS device sometime).
Bad stuff, mucking about with the TTL that someone has assigned to a record. It's not arbitrary information. To those fucking with TTLs, how about we arbitrarily alter the numbers in your paycheck? Oh? What's that? That doesn't seem like a good idea? Gee. Go Fig. HANDS OFF MY TTL, ASSHAT.
-AC
Re:ELOGICFAULT (Score:3, Interesting)
Case in point. A former high-traffic client (in the million hits per day range) did an 'inverse outsource' and reclaimed their site and corresponding domains. To prep for the move, the client migrated their DNS service to their new servers and changed the NS records with NetSol, several weeks in advance. They dropped their TTL down to two hours around that time. Early on the day of the cutover, changed the DNS records to point to
Re:Faulty system (Score:5, Insightful)
It's irresponsible tampering, it's that simple.
Re:Faulty system (Score:3, Informative)
Its completely within the spec, and as a fundemental principle I can do whatever I want with my server. So get with the program and understand there are other ways of dealing with the issue. Two weeks before the change, set the new IP address of the mail server as a lower priority (higher number) server, so if the info is cached, it will fall back to the new number when the old one fails. When you make the change, you can purge the old address entirely.
This
Re:Faulty system (Score:3, Interesting)
Anyone care to clarify how DNS cache works with MX records?
Re:For non geeks (Score:5, Informative)
Re:For non geeks (Score:3, Informative)
So close... (Score:5, Informative)
"TTLs also occur in the Domain Name System (DNS), where they are set by an authoritative nameserver for a particular Resource Record. When a Caching (recursive) nameserver queries the authoritative nameserver for a Resource Record, it will cache that record for the time specified by the TTL."
http://en.wikipedia.org/wiki/Time_to_live
Re:For non geeks (Score:2)
I hope that didn't help you, since it refers to the wrong kind of TTL.
TTL for DNS servers means (informally) how long they will consider a given resolved IP address as still-good. After that, the DNS server itself will do a DNS lookup to try to get a fresher address.
Re:For non geeks (Score:3, Informative)
Actually, the "TTL" in an IP header is different from the "TTL" in a DNS response (though in both cases the acronym means "time to live" and is intended as a limit on how long data hangs around).
IP header TTL is basically a hop-count, to stop IP packets going round in circles indefinately in the event of routing loops in the network.
Typically, when you look up a name like "www.example.com" your workstation consults a caching DNS server (on the local LAN, or offered by your ISP, or something). This DNS s
Re:For non geeks (Score:3, Informative)
Background
----------
Domain Name Servers (DNS) are usually configured in a heirarchy, such that each server has a parent. This fact will be important below.
Every domain (i.e. slashdot.org) has one or more "authoritative" name servers. These name servers know what web host slashdot.org is hosted on and how to get there.
Oth
Not laziness (Score:2)
Re:It's a strange pandemic... (Score:4, Funny)
Re:TTL is useless, so who cares? (Score:2, Flamebait)
It allows caching.
If you don't understand why that's an excellent reason or why it's useful, then you have an even poorer understanding of how DNS works and why it's around than I do, and I admit my understanding is rudimentary.
Re:TTL is useless, so who cares? (Score:2)
Re:Yes there is (Score:5, Insightful)
Re:Submitter is Correct, it's happening (Score:3, Informative)
What's the easiest way to check to see if your machine does indeed fetch records from another server? dig?