[support] guardian inside docker cannot access docker-local DNS about bin HOT 18 CLOSED

Just for info, found this Gist helpful: https://gist.github.com/colthreepv/6b818cfcf296dc1b5c2cf15eb76a140e
I guess the ENV did the trick for me.

 concourse-worker:
...
    environment:
      CONCOURSE_TSA_HOST: concourse-web
      CONCOURSE_GARDEN_ADDRESS: concourse-worker
      CONCOURSE_BAGGAGECLAIM_ADDRESS: concourse-worker
      CONCOURSE_GARDEN_FORWARD_ADDRESS: concourse-worker
      CONCOURSE_BAGGAGECLAIM_FORWARD_ADDRESS: concourse-worker
      CONCOURSE_GARDEN_DNS_SERVER: 8.8.8.8
    dns:
      - 8.8.8.8
      - 8.8.4.4

from bin.

Setting iptables -I FORWARD -j ACCEPT on worker instance does the trick.

from bin.

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #119571925 [support] garden/guardian inside docker cannot access DNS
• #126454419 reopened: [support] guardian inside docker cannot access docker-local DNS

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

from bin.

You could try tacking --garden-dns-server flags to the concourse worker command. Make sure you're not using the 127 address as that'll always try to resolve in the container's namespace - is there some other address you can reference? Maybe the host container's IP?

from bin.

Thanks, this should be what I need on the concourse side to get things moving again. I am still having issues on the docker side; docker seems to redirect 127.0.0.11 to some mystery socket, and simply pointing --garden-dns-server to the host container's ip doesn't seem to work. Time to brush up on my networking skills...

from bin.

Any word on this @RoboPhred? I'm not sure there's much left to do on our end, now that you at least know the flag that you'd be setting. May close this soon.

from bin.

I have not had time to figure out how to get the docker dns to the worker, but with the flag I should have everything I need once I revisit this issue. Closing for now, will open a new issue if I hit any other snags.

from bin.

Hi! I'm currently facing this issue. In order to solve it i endup adding socat to the worker's container, forwarding 127.0.0.11:53 (where the embedded dns lives):

socat UDP4-RECVFROM:53,fork UDP4-SENDTO:127.0.0.11:53 &

but i can't get the worker's runc to benefit from that server (the forwarding works as expected, see https://github.com/cirocosta/expose-edns). The inner containers are capable of pinging the worker but i'm not sure if that's just some iptables trick, or something like that, as i can't resolve any host by simply nslookup <worker_ip> <host_i_want_to_resolve>.

Any ideas @vito ?

from bin.

Just to add more info, the use case is:

• web and worker run on user-defined network n1
• worker has a dns server on 0.0.0.0:53/udp forwarded from 127.0.0.11:53 (docker's embedded dns server)
• worker has --garden-dns-server set to its hostname -i
• registry is also on user-defined network n1
• worker docker container is capable of resolving registry name as it has the dns server on 127.0.0.11:53working well
• inner runc worker containers should be able to resolve registry as their nameserver in /etc/resolv.conf is properly set to worker ip

Thx!

from bin.

What ends up being in /etc/resolv.conf in the runc worker container? As long as the address can resolve to the outer container it should work. But if it falls within a private local network range (127.x.x.x, or an IP conflicting with the Guardian container's network range) it won't work.

from bin.

in resolv.conf ends up the expected address: 192.168.48.3 (the user-define network is in 192.168.48.0/20). Guardian's ip range that you mean are those ip's that runc containers are being assigned right? So that's actually not the case (they're going to 10.254.0.13/30 as i understood) hmmm

Could you help me with understanding these rules?

root@3ca010dd57e0:/# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
w--input   all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
w--forward  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain w--default (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED

Chain w--forward (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
w--instance-ofgd99so74q  all  --  10.254.0.14          anywhere            [goto] 
DROP       all  --  anywhere             anywhere            

Chain w--input (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            

Chain w--instance-ofgd99so74q (1 references)
target     prot opt source               destination         
ACCEPT     all  --  10.254.0.12/30       10.254.0.12/30      
w--default  all  --  anywhere             anywhere            [goto] 

Chain w--instance-ofgd99so74q-log (0 references)
target     prot opt source               destination         
LOG        tcp  --  anywhere             anywhere             ctstate INVALID,NEW,UNTRACKED LOG level warning prefix "10cffbf5-f484-4637-573a-c5171"
RETURN     all  --  anywhere             anywhere    

Am i correct that packets from 10.254.0.14 are being dropped? If so, then the inner container wouldn't be capable of talking with worker, right?

from bin.

Yeah, come to think of it it's probably defaulting to rejecting traffic to the host's network. Try passing the --garden-allow-host-access flag.

from bin.

hmmm still not able to resolve, but the table changed to now:

root@102aa617e5ff:/# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
w--input   all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
w--forward  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain w--default (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED

Chain w--forward (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
w--instance-ofi078afcmn  all  --  10.254.0.10          anywhere            [goto] 
DROP       all  --  anywhere             anywhere            

Chain w--input (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            

Chain w--instance-ofi078afcmn (1 references)
target     prot opt source               destination         
ACCEPT     all  --  10.254.0.8/30        10.254.0.8/30       
w--default  all  --  anywhere             anywhere            [goto] 

Chain w--instance-ofi078afcmn-log (0 references)
target     prot opt source               destination         
LOG        tcp  --  anywhere             anywhere             ctstate INVALID,NEW,UNTRACKED LOG level warning prefix "6fd74f69-5a73-4002-77ab-032d3"
RETURN     all  --  anywhere             anywhere  

update: had other containers still around, updated with the correct table

from bin.

paging dr. @julz

from bin.

actually, inner worker is capable of accessing the worker as a request from the inner container to a python server in the worker has had the request served successfully; strange that dns lookups are still not working, maybe it has something to do with udp, need to check that

from bin.

Little update on this: got it working by explicitly binding to the desired interface in socat 👯 :

socat UDP4-RECVFROM:53,fork,bind=$(hostname -i) UDP4-SENDTO:127.0.0.11:53 &

Is there a way of doing that w/out socat? As we already have iptables in the worker container, could we do that forwarding using it? Or garden would blindly remove the rule when updating the table whenever new containers are created?

Thx

from bin.

Any ideas on this? I'd really like to remove socat in such setup

from bin.

Consolidating this into cloudfoundry/guardian#42 - if you actually want Docker's DNS working it may be more productive to talk about it there, as it's primarily a question of how to configure Guardian in this way. I don't think it's possible to day.

Thanks for the info and updates, though - sorry I couldn't be of more help.

from bin.