kube-dns never resolves if a domain returns NOERROR with 0 answer records once about dns HOT 22 CLOSED

Looks like - --no-negcache added to the dnsmasq args ought to do it.

Credit to https://rsmitty.github.io/KubeDNS-Tweaks/

from dns.

I will try playing with dnsmasq flags and see if we can change its negative caching behavior.

from dns.

Why did we disable neg-caching as default instead of setting reasonable TTL value with --neg-ttl=600?
With huge amount of queries in kubernetes related to ndots settings this would have negative impact.

from dns.

RCODE=0 with no response is the NODATA pseudo-rcode. For the purpose of caching, it shouldn't be treated differently from NXDOMAIN with one exception - it doesn't say anything about non-existence of names below the requested name. See https://tools.ietf.org/html/rfc2308#section-2.2 for guideline. It's possibly related to miekg/dns#428

from dns.

@miekg I think we don't know what will this change break. However, unless changed, many software that rely on domains eventually resolving stays broken. I'm not sure if we have enough tools to answer this question properly.

from dns.

@ahmetb I think it's legal in DNS to cache if we get a rcode == 0 response with 0 entries. This looks to be the behavior of the Cloudflare server (sending rcode 0 instead of NXDOMAIN). It looks like the TTL was around 30 minutes for the DNS record. If the record is going to be changing, it would be advisable to reduce the TTL to get faster cache updates.

from dns.

@bowei I think @viglesiasce reproduced this with Google Domains (or Cloud DNS) too.

In my experience, the cache was not invalidated even after 24 hours when I left it at that.

from dns.

Linking to #119 with respect to Cloudflare.

from dns.

I reported this to various folks at CloudFlare, still waiting a response. However, if anyone can help pinpoint where the caching happens, under what circumstances and why it lasts so long (i.e. >24h or in my experience, indefinitely), those would help fixing this problem, too.

from dns.

I work at Cloudflare, so I'm happy to answer any questions. It's not however specific to Cloudflare DNS; NODATA is a kind of answer you get from an authoritative server when the requested name exists, but the record type you're looking for doesn't, which is quite common. The RFC2038 I linked provides a guideline on how clients should implement negative caching for all cases of negative answers - hope that helps.

from dns.

Got an answer from the CloudFlare support:

We are aware of this behavior and it has been escalated previously to our DNS team. Their response is that at this moment we could not change/improve this behaviour. It's our design feature.

We are aware of this and will be working in improving the behavior in the future - but this will not happen earlier than 6 months.

Unfortunately, at this point, this is how our DNS is working.

We should look at fixing the caching behavior in kube-dns (or miekg/dns, or wherever it is) as a mitigation. Not caching 0-record answers sounds like it would yield a low-impact cache-miss rate to me. @bowei thoughts?

from dns.

@ahmetb caching is done with dnsmasq (http://www.thekelleys.org.uk/dnsmasq/doc.html) with no special tuneables. Maybe there is a flag that can disable caching that response? I'm surprised this does not impact more people, not just users of Kubernetes. dnsmasq is a popular piece of software, standard resolver on some Linux distros.

from dns.

I have this exact same issue (using kube-lego on GCE) but using google cloud DNS. Kube lego cannot resolve my domain when trying to request the token in order to issue a certificate. External to any kube pod the domain name resolves fine. When digging the domain within the pod it still gets ANSWER: 0 and status: NOERROR.

I tried restarting the kube-dns with

kubectl delete pods -n kube-system -l k8s-app=kube-dns

but to no avail.

Is there anything I can do to expedite invalidating the DNS cache or is it a matter of waiting it out? (It's been close to 24h for me)

from dns.

Can you post the output of dig for the entry? dnsmasq uses the TTL of the SOA record for negative replies, otherwise it will be 0.

from dns.

From the pod:

; <<>> DiG 9.10.4-P8 <<>> my.domain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 34318
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;my.domain.com			IN	A

;; Query time: 0 msec
;; SERVER: 10.39.240.10#53(10.39.240.10)
;; WHEN: Wed Aug 02 06:45:50 UTC 2017
;; MSG SIZE  rcvd: 43

but i will occasionally get this answer instead:

; <<>> DiG 9.10.4-P8 <<>> my.domain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 18668
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;my.domain.com.			IN	A

;; AUTHORITY SECTION:
my.domain.com		895	IN	SOA	ns-1660.awsdns-15.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 1 msec
;; SERVER: 10.39.240.10#53(10.39.240.10)
;; WHEN: Wed Aug 02 06:35:03 UTC 2017
;; MSG SIZE  rcvd: 130

This SOA is from AWS which was my prior DNS provider.

from dns.

@bowei Any luck?

from dns.

@coresolve whoa this is amazing. @bowei do you think it's sensible to incorporate this as a default in kube-dns distribution?

from dns.

yes, since we don't enable neg caching

from dns.

This would be just in https://github.com/kubernetes/kubernetes/blob/e557f184e2bf563c9d50719ca0942cc6128909d7/cluster/addons/dns/kube-dns.yaml.base#L158-L169, right?

from dns.

yes, that should be a one-line change to the yaml

from dns.

Opened kubernetes/kubernetes#53604 to add this

from dns.

Has anyone looked into the impact of removing negative caching on the volume of DNS requests that now need resolved again and again?

from dns.