Pod's access to the internet suddenly stopped working, DNS resolution fails about microk8s HOT 8 OPEN

After two days of troubleshooting, I found a solution by disabling DNS and re-enabling it with my routers IP address as DNS:

microk8s disable dns
microk8s enable dns:192.168.178.1

I found the IP in /run/systemd/resolve/resolv.conf which, according to the Kubernetes docs, should be the correct resolve file when the host machine is using systemd-resolved.

What did not help was to set a different DNS (I've tried Cloudflare 1.1.1.1). Also, digworked the whole time, so I really do not understand what is going on.

from microk8s.

Hi @McToel

Which microk8s version are you using? Starting from MicroK8s 1.26, MicroK8s will attempt to pick the upstream nameservers from /run/systemd/resolve/resolv.conf by default.

1. How do you (and did you) enable dns?

2. What's the output of snap run --shell microk8s -c '$SNAP/scripts/find-resolv-conf.py'?

3. What's in your /run/systemd/resolve/resolv.conf?

from microk8s.

I'm running MicroK8s v1.29.2 revision 6641.

It could be, that as I first enabled DNS I was running a version prior to 1.26. But while investigating the problem, I have disabled and enabled dns a few times. I enabled dns with microk8s enable dns in the beginning.

Here is the output for 2. and 3.:

➜  ~ snap run --shell microk8s -c '$SNAP/scripts/find-resolv-conf.py'
/run/systemd/resolve/resolv.conf
➜  ~ cat /run/systemd/resolve/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.178.1
search fritz.box

from microk8s.

I have exactly the same problem... I can't resolve it by pointing it to my pi hole either. It works for a second and then the problems start again.

from microk8s.

I had a similar issue and fixed it with conditional forwarding on my pi hole

Maybe someone with more network knowledge can explain why this only happens to pods but not on other devices in my network.

Hoping this can help you out

from microk8s.

I recreated the dns service with the following settings, which worked for me.

First remove it:
microk8s disable dns

Then after I've recreated it:
microk8s enable dns:<pi-hole address>

from microk8s.

root@dnsutils:/# nslookup deb.debian.org
Server:		10.152.183.10
Address:	10.152.183.10#53

Non-authoritative answer:
Name:	deb.debian.org.fritz.box
Address: 45.76.93.104

It looks that you are suffering DNS hijacking, please check this https://crapts.org/2024/04/21/all-fritz-box-modems-have-been-hijacked/

from microk8s.

Hi ,
Applied below steps, pods started to communicate each-other.

1. microk8s disable dns
2. microk8s enable dns:<ip address from /run/systemd/resolve/resolv.conf>

But after 1 hour , pod suddenly stopped communicating with each other.

Please help me to resolve this issue.

from microk8s.