Comments (22)
Looks like - --no-negcache
added to the dnsmasq args ought to do it.
Credit to https://rsmitty.github.io/KubeDNS-Tweaks/
from dns.
I will try playing with dnsmasq flags and see if we can change its negative caching behavior.
from dns.
Why did we disable neg-caching as default instead of setting reasonable TTL value with --neg-ttl=600
?
With huge amount of queries in kubernetes related to ndots settings this would have negative impact.
from dns.
RCODE=0 with no response is the NODATA pseudo-rcode. For the purpose of caching, it shouldn't be treated differently from NXDOMAIN with one exception - it doesn't say anything about non-existence of names below the requested name. See https://tools.ietf.org/html/rfc2308#section-2.2 for guideline. It's possibly related to miekg/dns#428
from dns.
@miekg I think we don't know what will this change break. However, unless changed, many software that rely on domains eventually resolving stays broken. I'm not sure if we have enough tools to answer this question properly.
from dns.
@ahmetb I think it's legal in DNS to cache if we get a rcode == 0 response with 0 entries. This looks to be the behavior of the Cloudflare server (sending rcode 0 instead of NXDOMAIN). It looks like the TTL was around 30 minutes for the DNS record. If the record is going to be changing, it would be advisable to reduce the TTL to get faster cache updates.
from dns.
@bowei I think @viglesiasce reproduced this with Google Domains (or Cloud DNS) too.
In my experience, the cache was not invalidated even after 24 hours when I left it at that.
from dns.
Linking to #119 with respect to Cloudflare.
from dns.
I reported this to various folks at CloudFlare, still waiting a response. However, if anyone can help pinpoint where the caching happens, under what circumstances and why it lasts so long (i.e. >24h or in my experience, indefinitely), those would help fixing this problem, too.
from dns.
I work at Cloudflare, so I'm happy to answer any questions. It's not however specific to Cloudflare DNS; NODATA is a kind of answer you get from an authoritative server when the requested name exists, but the record type you're looking for doesn't, which is quite common. The RFC2038 I linked provides a guideline on how clients should implement negative caching for all cases of negative answers - hope that helps.
from dns.
Got an answer from the CloudFlare support:
We are aware of this behavior and it has been escalated previously to our DNS team. Their response is that at this moment we could not change/improve this behaviour. It's our design feature.
We are aware of this and will be working in improving the behavior in the future - but this will not happen earlier than 6 months.
Unfortunately, at this point, this is how our DNS is working.
We should look at fixing the caching behavior in kube-dns (or miekg/dns, or wherever it is) as a mitigation. Not caching 0-record answers sounds like it would yield a low-impact cache-miss rate to me. @bowei thoughts?
from dns.
@ahmetb caching is done with dnsmasq (http://www.thekelleys.org.uk/dnsmasq/doc.html) with no special tuneables. Maybe there is a flag that can disable caching that response? I'm surprised this does not impact more people, not just users of Kubernetes. dnsmasq is a popular piece of software, standard resolver on some Linux distros.
from dns.
I have this exact same issue (using kube-lego on GCE) but using google cloud DNS. Kube lego cannot resolve my domain when trying to request the token in order to issue a certificate. External to any kube pod the domain name resolves fine. When digging the domain within the pod it still gets ANSWER: 0
and status: NOERROR
.
I tried restarting the kube-dns with
kubectl delete pods -n kube-system -l k8s-app=kube-dns
but to no avail.
Is there anything I can do to expedite invalidating the DNS cache or is it a matter of waiting it out? (It's been close to 24h for me)
from dns.
Can you post the output of dig
for the entry? dnsmasq uses the TTL of the SOA record for negative replies, otherwise it will be 0.
from dns.
From the pod:
; <<>> DiG 9.10.4-P8 <<>> my.domain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 34318
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;my.domain.com IN A
;; Query time: 0 msec
;; SERVER: 10.39.240.10#53(10.39.240.10)
;; WHEN: Wed Aug 02 06:45:50 UTC 2017
;; MSG SIZE rcvd: 43
but i will occasionally get this answer instead:
; <<>> DiG 9.10.4-P8 <<>> my.domain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 18668
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;my.domain.com. IN A
;; AUTHORITY SECTION:
my.domain.com 895 IN SOA ns-1660.awsdns-15.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
;; Query time: 1 msec
;; SERVER: 10.39.240.10#53(10.39.240.10)
;; WHEN: Wed Aug 02 06:35:03 UTC 2017
;; MSG SIZE rcvd: 130
This SOA is from AWS which was my prior DNS provider.
from dns.
@bowei Any luck?
from dns.
@coresolve whoa this is amazing. @bowei do you think it's sensible to incorporate this as a default in kube-dns distribution?
from dns.
yes, since we don't enable neg caching
from dns.
This would be just in https://github.com/kubernetes/kubernetes/blob/e557f184e2bf563c9d50719ca0942cc6128909d7/cluster/addons/dns/kube-dns.yaml.base#L158-L169, right?
from dns.
yes, that should be a one-line change to the yaml
from dns.
Opened kubernetes/kubernetes#53604 to add this
from dns.
Has anyone looked into the impact of removing negative caching on the volume of DNS requests that now need resolved again and again?
from dns.
Related Issues (20)
- [Improvement] Add value compatibility for -upstreamsvc HOT 5
- [node-local dns] DNS requests intermittently receive refused response errors HOT 4
- Intermittent timed out accessing nodelocaldns HOT 2
- [NodeLocal DNS Cache] DNS requests not directed to the local cache HOT 2
- pull-kubernetes-dns-test broken at head HOT 2
- Several old CVE's still present on the latest k8s-dns-node-cache versions HOT 4
- Image for 1.22.27 missing HOT 3
- [node-local-dns] Query loss HOT 5
- CVE-2023-5363 and CVE-2023-5528 in 1.22.28 HOT 24
- k8s node-local-dns high slab memory consumption leading to OOM HOT 1
- 1.23.0 is missing from registry.k8s.io/dns/k8s-dns-node-cache HOT 2
- node-local-dns-cache DNS i/o timeout errors HOT 2
- "make build-amd64" got an error HOT 2
- LocalDNS support for CoreDNS file plugin HOT 1
- NodeLocal DNS Cache Intercepts all dns queiris HOT 7
- kube-dns doesn't expose service scoped dns names for pod IPs HOT 1
- Getting UnkownHostException from SpringBoot Microservice running on top of kubernetes HOT 2
- Latency/timeout from Kube DNS
- `node-local-dns` should support graceful shutdown with the `ready` plugin
- IP table lock issues HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dns.