kubedns dnsmasq pod fails with blank IP for the DNS host about dns HOT 11 CLOSED

kube-dns ConfgiMap won't affect /etc/hosts file. Instead, kubelet is responsible to setting the hosts file up (https://github.com/kubernetes/kubernetes/blob/c3d7ee11d176d94108831bed2e76e94039338fd2/pkg/kubelet/kubelet_pods.go#L186-L207).

And seems like you've got a empty pod IP for this kube-dns pod. Can you also check the output of $ kubectl get pods KUBE_DNS_POD_NAME --namespace=kube-system -o yaml? Are the other pods having the same issue?

It would be helpful if you could also check kubelet's logs on the node where kube-dns pod is on.

from dns.

I see. Other pods don't seem to have this problem. Looking on kube-proxy for example has a complete hosts file.

The kube-dns pod's yaml is attached.
kube-dns.txt

from dns.

These lines from syslog on the master (the only node so far) are interesting:

 Write status for kube-dns-1630391940-bnl7d/kube-system: &container.PodStatus{ID:"43e3338c-042a-11e7-a063-02ead988c3bd", Name:"kube-dns-1630391940-bnl7d", Namespace:"kube-system", IP:"", ContainerStatuses:[]*container.ContainerStatus{(*container.ContainerStatus)(0xc421696000), (*container.ContainerStatus)(0xc42118c7e0), (*container.ContainerStatus)(0xc421696380), (*container.ContainerStatus)(0xc421696620), (*container.ContainerStatus)(0xc42118c9a0)}, SandboxStatuses:[]*runtime.PodSandboxStatus{(*runtime.PodSandboxStatus)(0xc42137fb80)}} (err: <nil>)

 NetworkPlugin cni failed on the status hook for pod "kube-dns-1630391940-bnl7d_kube-system": Unexpected command output Device "eth0" does not exist.

from dns.

And indeed, ifconfig in the kubedns container has only "lo":

# kubectl exec kube-dns-1630391940-bnl7d -c kubedns ifconfig
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:732 errors:0 dropped:0 overruns:0 frame:0
          TX packets:732 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:64918 (63.3 KiB)  TX bytes:64918 (63.3 KiB)

from dns.

Hmm, could you check other pods that are not using hostNetwork (kube-proxy set hostNetwork=true)? From your yaml file it seems like it does not have podIP allocated. I wonder is the underlying network broken?

Here is a similar issue: https://github.com/projectcalico/calicoctl/issues/1407

from dns.

Close. Due to a docker bug, the dns pod couldn't be spun up (couldn't write memory limits properly). Fixing that, I get a little closer, but it can't seem to reach what I'm guessing is the API server. kubeadm sets this up on port 6443, but this isn't what kube-dns is trying to reach:

I0308 19:02:15.998889       1 dns.go:174] DNS server not ready, retry in 500 milliseconds
E0308 19:02:16.017504       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0308 19:02:16.017504       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable

from dns.

Yeah, it was trying to reach the apiserver through the default kubernetes service but failed.

Not sure how kubeadm set this up, but as I know kube-dns either use the kubecfg file specified by --kubecfg-file and kube-master-url flags or use the env vars (KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, which should be set up automatically when pod is created) as the configurations when creating its k8s client.

from dns.

Its the env vars, and they're wrong. What controls setting those?

from dns.

These values are piped into kubelet through --kubeconfig or --api-servers flags.

I'm not pretty sure if the real issue is about "incorrect kubernetes service IP/port" or not. Feel like there may be some mechanism in kubeadm I misunderstood. You may want to open an issue against kubeadm to seek for the real root cause.

from dns.

Oops, looks like I am wrong. Those env vars are generated from services themselves but not from flags:
https://github.com/kubernetes/kubernetes/blob/c3d7ee11d176d94108831bed2e76e94039338fd2/pkg/kubelet/kubelet_pods.go#L389
https://github.com/kubernetes/kubernetes/blob/c3d7ee11d176d94108831bed2e76e94039338fd2/pkg/kubelet/envvars/envvars.go#L27-L60

from dns.

I agree, I don't think kube-dns has anything to do with this, closing.

from dns.