Container network collides with link-local 169.254.169.253 DNS resolver about hcsshim HOT 15 OPEN

@pradipd PTAL.

from hcsshim.

Investigating.

from hcsshim.

I just ran into this too, GCE uses a link-local address for their metadata server.

from hcsshim.

@jhowardmsft @pradipd any word on this? We have a slightly different set of steps to reproduce the issue:

• Create an 1803 machine on GCP
• You should be able to reach http://metadata.google.internal
• Create an HNS Network (https://github.com/Microsoft/SDN/blob/af7847857a4223f176b3fb69e288799af9fd8e34/Kubernetes/flannel/overlay/start.ps1#L75)
• See that you can no longer reach the metadata server

The metadata server resolves to a link-local IP, which is what makes us think it's the same underlying issue.

from hcsshim.

We discovered a workaround for this. If we recreate the route that GCP adds to the original network interface to the new interface that the HNS network creates, the metadata server is reachable again.

route /p add 169.254.169.254 mask 255.255.255.255 0.0.0.0

This still seems like a bug though: the new interface shouldn't be losing these routes.

from hcsshim.

This problem is also affecting me and I agree with everything @benmoss has said. I'm using the suggested workaround of adding the route back after configuring HNS, the annoying part is that the removal of the route doesn't seem to happen immediately after configuring HNS, there's some delay. The dance that I'm doing can be seen at https://github.com/pjh/kubernetes/blob/c6b518afa564ef167b2490de271d1f7a4b801edb/cluster/gce/win1803/k8s-node-setup.psm1#L708.

from hcsshim.

Adding @daschott as an FYI.

from hcsshim.

Ack, this issue of routes not being migrated after HNS network creation is on our consolidated bug list for root-causing.

from hcsshim.

Hi @daschott, is this going to be looked at any time soon? This is still pretty painful for us and waiting for the route to be removed so that we can re-add it (see my comment above) is a significant source of delay when we're joining Windows nodes to Kubernetes clusters.

cc @PatrickLang @yujuhong @dineshgovindasamy

from hcsshim.

Adding @mkostersitz as an FYI

from hcsshim.

thanks for the add @pradipd @pjh let me catch up on this and get back to you while @daschott is out of the office for a while.

from hcsshim.

In AWS and GCE, is there a route added to reach 169.254.169.253? If so, is the route added manually or through DHCP.

from hcsshim.

@ankeesler: Where is this configuration?
"In AWS there is a configuration option to add this address to the list of DNS resolvers for the public ethernet interface."

from hcsshim.

At least on GCE this is the script that adds the route: https://github.com/GoogleCloudPlatform/compute-image-windows/blob/22bd7e2701da9dc6bd931711cbb39c36a1d81df1/sysprep/instance_setup.ps1#L126

from hcsshim.

Is there a better workaround than simply waiting for 45 seconds? Is there a way to deterministically check if the data transfer has completed?

from hcsshim.