macOS & iOS doesn't use DNS set in the admin panel about tailscale HOT 51 CLOSED

Thanks for the feedback here, everybody.

I left a comment in a now-closed duplicate bug #281 (comment) but we'll keep using this bug for tracking & debugging.

@crawshaw, looks like one solution if all else fails is implementing NEDNSProxyProvider
(https://developer.apple.com/documentation/networkextension/nednsproxyprovider) ... "A DNS proxy allows your app to intercept all DNS traffic generated on a device."

from tailscale.

@crawshaw fixed this at HEAD in our iOS app and I've just confirmed it's fixed. The iOS app with a version greater than or equal to v0.98-12 will have the fix.

from tailscale.

I should clarify that the fix should equally apply to macOS (they share the same code). I just only tested it on iOS.

from tailscale.

0.98.12 for iOS and macOS have been released. May take a few hours to reach everyone's devices.

from tailscale.

Okay, I have a fix that makes search domains work. For reference, the trick was:
dnsSettings.searchDomains = domains
dnsSettings.matchDomains = [""] + domains
One would think matching against "" would work, and it does work for activating our DNS server everywhere, but it doesn't work for activating our searchDomains, which apparently don't kick in unless they appear in both sections.

/etc/resolv.conf is indeed entirely bogus on macOS, and there isn't much we can do about it. Both go's internal resolver and commands like host are apparently befuddled by this. If they parsed the output of scutil --dns instead they would do better, but the rules used by the macOS resolver are a lot more complicated than /etc/resolv.conf.

The good news is that, (unlike older macOS releases I remember?), most command line tools do use the proper resolver. So ping and python and ssh for example, all properly use the configured DNS server and search path.

These new settings cause macOS/iOS to always use the configured DNS server, not just for corporate DNS purposes, because there is no way yet for us to make the split-DNS configuration work on our other platforms. However, we've also added an option "Use corporate DNS" to the menu on macOS, to match the one on Windows-staging, which lets you disable Tailscale's DNS configuration entirely so you can use a local DNS server when you prefer to do so.

When MagicDNS (#416) is further along, we can consider addressing it there in a cross-platform way.

This fix should show up in the next macOS and iOS releases >= v0.99.1-5.

from tailscale.

@sa1, yes. I tested it on iOS with wifi off.

from tailscale.

Confirmed on the latest macOS 10.5.4 and iOS 13.4.

I set up a dnsmasq server on one of my Linux nodes running Tailscale I have connectivity:dig <fqdn> @<Tailscale-dns-ip> returns the expected result. However, using the default dns query does not work.

If I put the Tailscale DNS IP at the top of the nameserver list in advanced settings on a macOS interface, name resolution works (apart from the search domain). So it seems there’s something not correct with the way Tailscale is handling DNS on macOS and iOS. I couldn’t see the DNS entry when running macOS command networksetup -getdnsservers Wi-Fi , so they don’t seem to be added, but I don’t know if that’s the expected behaviour? I logged out and back in on my macOS Tailscale client, but this made no difference

from tailscale.

+1 for having the capability for forcing all DNS queries through the VPN, which is our primary use case

from tailscale.

+1

Both on iOS & macOS this behaviour persists

from tailscale.

Same:
MacOS: 10.15.3
Tailscale: 0.97.78

scutil output:

/etc ❯❯❯ scutil --dns
DNS configuration

resolver #1
  search domain[0] : attlocal.net
  nameserver[0] : 8.8.8.8
  flags    : Request A records, Request AAAA records
  reach    : 0x00000002 (Reachable)

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : attlocal.net
  nameserver[0] : 8.8.8.8
  if_index : 6 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00000002 (Reachable)

resolver #2
  search domain[0] : myinternal.domain
  nameserver[0] : 10.255.0.1
  nameserver[1] : 10.255.0.2
  if_index : 19 (utun4)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

from tailscale.

Also on 10.15.4 and 0.97.78

from tailscale.

It seems like this might be a general wireguard issue, as I've found a couple of reddit posts mentioning similar issue in the MacOS wireguard client (though I haven't tried it). I did come across this:

scutil example

that walks through creating a resolver for a specific domain. In short if I do this

sudo scutil
d.init
d.add ServerAddresses * 1.2.3.4
d.add SupplementalMatchDomains * mydomain.com
set State:/Network/Service/<my uuid>/DNS
d.show
quit

where 1.2.3.4 is part of the subnet my relay advertises and mydomain.com is the domain I want to use that DNS host for. I can then do DNS lookups over the tunnel.

I've only tested it for 5 minutes, but it also seems to not break things (in my case where mydomain.com is unavailable w/o the tunnel) when I shutdown Tailscale.

I'm sure there is a way to track it down, but might be worth remembering what you set <my uuid> to be because if you want to undo this:

sudo scutil
remove State:/Network/Service/<my uuid>/DNS

Looking at the scutil --dns output above (mine looks similar) and the documents for SupplementalMatchDomains in the OS X profile reference, maybe what is going wrong is that they are setting "search domain" rather than SupplementalMatchDomains? That is a pure guess though.

from tailscale.

Hi @bradfitz happy to help test on iOS, macOS once a fix is ready

from tailscale.

Awesome, looking forward to trying it out.

from tailscale.

Does it work with cellular networks too?

(If you're just setting system DNS, and not proxying DNS, I think that iOS doesn't allow DNS to be set for cellular networks.)

from tailscale.

Users report that the DNS server IPs are set correctly now, but not the search path.

from tailscale.

We are setting searchDomains (just confirmed with some logging). Preliminary testing with 8.8.8.8 and a search domain shows it isn't working though. I suspect we are running into some limitation of the NetworkExtension: https://forums.developer.apple.com/thread/35027

@bradfitz, next time you're connected to your home domain with your own DNS server, could you try setting a search domain (e.g. ts.tailscale.com) and see if ping derp1 successfully appends the search domain? I'm curious if the DNS server being an IP that is routed by the NetworkExtension changes macOS's behavior.

from tailscale.

scutil --dns shows the Tailscale-set search domain in the DNS configuration (for scoped queries) section (fitzpat.com below), which seems like it means it's only used for lookups bound to that interface, which in practice is approximately never.

DNS configuration

resolver #1
  nameserver[0] : 10.0.0.1
  flags    : Supplemental, Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
  order    : 102000

resolver #2
  nameserver[0] : 8.8.8.8
  flags    : Request A records
  reach    : 0x00000002 (Reachable)
  order    : 200000

...

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 8.8.8.8
  if_index : 4 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)

resolver #2
  search domain[0] : fitzpat.com
  nameserver[0] : 10.0.0.1
  if_index : 17 (utun2)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

from tailscale.

Disconnecting doesn't back out the changes.

For example if i connect the first time i can see changes made (at least scutil -dns) says so:

bash-3.2$ diff /tmp/tailscale-{running,first-connected}
3a4,10
>   nameserver[0] : 172.31.66.11
>   if_index : 18 (utun4)
>   flags    : Supplemental, Request A records, Request AAAA records
>   reach    : 0x00000003 (Reachable,Transient Connection)
>   order    : 102200
> 
> resolver #2
8a16
>   order    : 200000
10c18
< resolver #2
---
> resolver #3
18c26
< resolver #3
---
> resolver #4
26c34
< resolver #4
---
> resolver #5
34c42
< resolver #5
---
> resolver #6
42c50
< resolver #6
---
> resolver #7
50c58
< resolver #7
---
> resolver #8
65a74,80
> 
> resolver #2
>   search domain[0] : tervela.com
>   nameserver[0] : 172.31.66.11
>   if_index : 18 (utun4)
>   flags    : Scoped, Request A records
>   reach    : 0x00000003 (Reachable,Transient Connection)
bash-3.2$

But if i then disconnect some changes remain in the output of scutil -dns:

It removes the search domain

bash-3.2$ diff /tmp/tailscale-{first-connected,disconnected}
5d4
<   if_index : 18 (utun4)
7c6
<   reach    : 0x00000003 (Reachable,Transient Connection)
---
>   reach    : 0x00000002 (Reachable)
74,80d72
< 
< resolver #2
<   search domain[0] : tervela.com
<   nameserver[0] : 172.31.66.11
<   if_index : 18 (utun4)
<   flags    : Scoped, Request A records
<   reach    : 0x00000003 (Reachable,Transient Connection)

But it leaves the in house DNS server's entry:

bash-3.2$ head /tmp/tailscale-disconnected 
DNS configuration

resolver #1
  nameserver[0] : 172.31.66.11
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000002 (Reachable)
  order    : 102200

resolver #2
  nameserver[0] : 172.22.22.1
bash-3.2$ 

from tailscale.

@bhyde, the Tailscale app has two components: the GUI you interact with, and the Network Extension (process name "IPNExtension"). We have some bugs around lifetimes between them. (e.g. #334)

I've even found cases where closing the GUI kept the IPNExtension around and then I had to stop it in Activity Monitor, and then routes/DNS settings would go away.

from tailscale.

So if I'm following correctly IPNExtension is responsible for dutifully following thru on any changes. And in this case somebody stumbled since when the Tailscale application quit that ought to have cascaded into IPNExtension tearing down everything and then a bit of apoptosis.

I assume the cascade failed. Or, maybe I should reproduce this and wait a bit longer?

from tailscale.

This might be useful. This the output of scutil --dns when I'm using our current (hopefully soon to be legacy) VPN solution.

DNS configuration

resolver #1
  search domain[0] : example.com
  nameserver[0] : 172.22.22.1
  nameserver[1] : 2002:d106:7acc:0:deef:9ff:fea8:39e
  if_index : 5 (en0)
  flags    : Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  domain   : example.com
  nameserver[0] : 172.31.66.13
  nameserver[1] : 172.31.66.14
  if_index : 18 (utun4)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000002 (Reachable)
  order    : 102200

resolver #3
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #4
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #5
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #6
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #7
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #8
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 172.22.22.1
  nameserver[1] : 2002:d106:7acc:0:deef:9ff:fea8:39e
  if_index : 5 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  search domain[0] : example.com
  nameserver[0] : 172.31.66.13
  nameserver[1] : 172.31.66.14
  if_index : 18 (utun4)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)

from tailscale.

Based on reading some tea leaves in the Apple forums, I modified our NetworkExtension to set the ipv4 default route. That moves our DNS configuration out of scoped queries:

$ scutil --dns
DNS configuration

resolver #1
  search domain[0] : ts.tailscale.com
  nameserver[0] : 8.8.8.8
  if_index : 25 (utun2)
  flags    : Supplemental, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103800

resolver #2
  nameserver[0] : 8.8.8.8
  if_index : 25 (utun2)
  flags    : Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 200000

resolver #3
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000
...

So that raises the possibility: can we set the default route in NEIPv4Settings.includedRoutes, and then add every IP range except CGNAT to NEIPv4Settings.excludedRoutes?

from tailscale.

from tailscale.

According to the Apple forums, the NetworkExtension matchDomains is used to determine whether the DNS server set by the NetworkExtension should be used for resolution. We are now setting matchDomains to "", the wildcard that matches everything, so our limited-scope DNS server covers all queries.

I tried a default route on the theory that they aren't using matchDomains to determine the searchDomains used for suffixes, because, well that's mostly impossible, matchDomains is a list of suffixes.

I'd really prefer to avoid stealing the default route if we can. (Unless it also solves our local IP problem, but my quick testing didn't show my utun address as pingable.)

from tailscale.

Well, if the domain doesn't have a DNS search domain (the common case?), we can default to not being the default route.

If the Tailscale domain does have a search domain set, then we still could if we detect a simple network config (for some heuristic) on the mac. Otherwise worst case we could make it a client option, which I'm sure @apenwarr loves.

from tailscale.

With version 0.98.12 on OS X we are seeing:

1. Tailscale is not the default route (traceroute to a non-advertised route does not go through the tunnel)
2. The default domain set in the Tailscale admin interface is not appended to hostname-only queries (ping www fails when the Tailscale provided DNS host has an entry for www.tailscaledomain.com)
3. The Tailscale provided DNS servers are used for all queries...when I am on a network with its own local DNS and connected to tailscale this works:

ping host.tailscaledomain.com

but this fails:

ping host.localdomain.com

when that localdomain.com is an internal domain that is hosted only on the local DNS. If I quit tailscale the ping to the localdomain.com host works.

From our perspective we would very much like at least the ability to maintain #1 (not be the default route), and gain the ability to disable #3 (use local DNS for domains other than those specified in Tailscale). We are going to run into users in locations that have local domains that will lose access to them with Tailscale running I think. Thinking about it, #3 is probably more important.

from tailscale.

from tailscale.

@apenwarr I think for #4 we'd want the DHCP-provided DNS to get the query.

For #3 the real issue is when one of our users visits e.g. the office of another company and needs to access services in a domain that is only on their local DNS. If all queries went to our DNS via Tailscale they would be unable to get there.

#4 would basically be the same issue in the case where the service had a domain only on the DHCP DNS but was not the search domain provided by the DNS.

Thanks.

from tailscale.

from tailscale.

Not quite, 3 and (presumably) 4 from above don't work.

I'm currently on a network like we are discussing here. I have a local DNS handed out by my DHCP server and it is definitive for an internal domain.

When I am connected to Tailscale:

ping host1.tailscaledomain.com
works

ping host2.dhcp-domain.com
ping: cannot resolve host2.dhcp-domain.com: Unknown host

Then I quite the Tailscale client:

ping host2.dhcp-domain.com
works

This is a non-issue right now when all of our users are home (and no one has this setup), so no one is really noticing it. It would be somewhat of an issue once they get out of the house.

Thanks.

from tailscale.

There is an iOS/macOS feature that would let us implement that, matchDomains. But to achieve those semantics on other platforms we will have to implement a local resolver. I would like to do that, but we should probably have a separate issue for it.

from tailscale.

My biggest concern right now: we can not abbrevate domain names inspite of
the search path we have declared.

We have links and code that use names like consul-1, mongo-3,
etc. etc. rather than consul-1.example.com. Fixing all those isn't
likely. So we're stuck.

Some of the discussions here has made me curious. Is there Apple
doc and RFCs that spell out what seems like a tedious set of
choices around what goes where, what search really implies, etc.
etc.? DNS config used to seem so simple, LOL.

from tailscale.

Someone else just told me that some apps are working with the redirected DNS server, but specifically nslookup, dig, and kubectl do not. kubectl prints an especially interesting error:

Unable to connect to the server: dial tcp: lookup <apiserver-url> <...local DNS...>: server misbehaving

A quick googling suggests that Go's resolver is probably doing something suspicious to cause that last part: golang/go#12712

from tailscale.

@apenwarr This is true. It comes down to whether the app uses the OS X system lookup tools or lower level settings. e.g.:

host host.tailscaledomain.com

ping host.tailscaledomain.com

when connected to Tailscale.

from tailscale.

I read elsewhere that nslookup and dig have their own resolvers and just read /etc/resolv.conf (the synthetic file) on mac and use that, not using Apple's libc.

And kubectl if cross-compiled is likewise not using Apple's libc, falling back to /etc/resolv.conf without cgo.

And when we're a scoped resolver we don't get put in /etc/resolv.conf.

from tailscale.

@bradfitz I believe all of that is correct. It makes troubleshooting less than obvious, particularly on the iPhone.

from tailscale.

I "read elsewhere" that dns-sd is the perfered tool for CLI name lookups. SD? - service discovery. It's a very frustrating tool, as you'll learn. But it's manual advises " You can also query for a unicast name like www.apple.com and monitor its status. dns-sd -q www.apple.com"

Mostly i use ping -c 1 consul-1.example.com.

from tailscale.

from tailscale.

@apenwarr this seems right to me (even as someone who needs the scoped setting in some cases).

Frankly both DNS and default route seem like there isn't a single answer here. Sending everything over Tailscale is best for privacy, but it could cause issues in some cases for some users.

from tailscale.

from tailscale.

from tailscale.

Ping. I'm sad that we can't use abbreviated domain names since apparently the search setting in my admin DNS setting isn't getting any traction. As a consequence my eager beta users have begun to rework random things to use fully qualified names, and that is making some of us cranky. The phrase "stop this madness," has been heard.

from tailscale.

from tailscale.

Running app version on MacOS 0.100.94 but still things do not work properly.

Set up in the Admin Panel the DNS 172.20.0.2 corresponding to the AWS VPC default DNS.

➜ scutil --dns                                                                                                  13:12:19
DNS configuration

resolver #1
  search domain[0] : skynet.local
  nameserver[0] : 172.20.0.2
  if_index : 22 (utun4)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 102400

resolver #2
  nameserver[0] : 192.168.10.1
  if_index : 6 (en0)
  flags    : Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
  order    : 200000

resolver #3
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #4
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #5
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #6
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #7
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #8
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

resolver #9
  domain   : test
  nameserver[0] : 127.0.0.1
  flags    : Request A records, Request AAAA records
  reach    : 0x00030002 (Reachable,Local Address,Directly Reachable Address)

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : skynet.local
  nameserver[0] : 192.168.10.1
  if_index : 6 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  nameserver[0] : 172.20.0.2
  if_index : 22 (utun4)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

but only forcing for instance dig to use the DNS server things work

➜ dig +short ip-172-20-90-205.ec2.internal @172.20.0.2                                                          
172.20.90.205

otherwise

dig +short ip-172-20-90-205.ec2.internal

Of course I ticked the option Use Corporate DNS

from tailscale.

This might be just a characteristic of macOS. We have found that tools like dig and host do not respect the scoped query configurations, but higher level tools like ping, ssh or Safari do.

from tailscale.

yeah but in my case it does not work neither ssh.

from tailscale.

Ok, must be something different then.

from tailscale.

@thehilll I confirm that it works. as just a local misconfiguration

from tailscale.

from tailscale.

hey @apenwarr it was just a local misconfig of my ssh config file. nothing related to Tailscale.

from tailscale.