massmesh / autoygg Goto Github PK

Struggled for a bit over why gateway wasn;t working on a fresh 20.04 droplet, for some reason this wasn't getting done:

root@mm-gw-stephen-mullvad:~/autoygg/internal# iptables -A FORWARD -i eth0 -j ACCEPT
root@mm-gw-stephen-mullvad:~/autoygg/internal# iptables -A FORWARD -o eth0 -j ACCEPT
root@mm-gw-stephen-mullvad:~/autoygg/internal# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

eth0 should probably be configurable, since I'll have it as vpn0.

for ./internal/shared.go we should be passing netmask-cidr and interface-name
to more of the adaptable %%template-cmd-exampes%% in server/client configs.

_, err = exec.Command("ip", "addr", action, IPAddress+"/"+strconv.Itoa(NetMask), "dev", viper.GetString("YggdrasilInterface")).Output()

if some commands like the add default route command 'ip route add default xxx' fail, but the exit code doesn't get passed back as an error and result is 'OK' not 'FAIL'

Afaict, in the default configuration, a connection to a vpn will cause peering to go through, as well as system updates.

I've found this setup to work quite well in isolating mesh traffic to a tun interface:

.ovpn: comment out up/down directives

dev vpn0
dev-type tun
route-nopull

/etc/iproute2/rt_tables

42      mesh

$ sudo ip rule add from 10.42.0.0/16 table mesh
$ ip ro add default dev vpn0 table mesh

$ iptables -A FORWARD -i vpn0 -j ACCEPT
$ iptables -A FORWARD -o vpn0 -j ACCEPT
$ iptables -t nat -A POSTROUTING -o vpn0 -j MASQUERADE

If some of this could be automated, that would vastly simplify setup with a vpn. These steps should work fine applied to the regular wan interface too.

The peers collection function is: yggdrasilPeers() (collects peers and interfaces)

The error is in clientSetupRoutes(), where an ip6 destination (tcp://[fe80::109a:683d:a72:c4f5%wlan0]:45279) fails to add a static route. I'd turn around and make executeYggdrasilCtl() get called with a -json flag but... grr

yggdrasilctl getpeers (with -json)

    "201:9d5d:ff2:51f2:2cc9:5e1e:3659:89d0": {
      "box_pub_key": "56ebd7c92c9185df290eeec670....",
      "bytes_recvd": 787107,
      "bytes_sent": 4004369,
      "endpoint": "tcp://[fe80::7917:4e09:3aa1:3923%wlp0s20f3]:59571",
      "port": 4,
      "proto": "tcp",
      "uptime": 6485.19996654
    }
.....

Adding Yggdrasil remote subnet 0.0.0.0/0                              [ ok ]
Getting Yggdrasil peers                                               [ ok ]
Adding Yggdrasil peer route for 184.30.190.174 via 192.168.26.1       [ ok ]
Adding Yggdrasil peer route for [fe80 via 192.168.26.1                [ FAIL ]
-> Unable to run `ip ro list [fe80 via 192.168.26.1 dev wlp0s20f3`: exit status 1
2020/01/26 00:24:02 Error: Unable to run `ip ro list [fe80 via 192.168.26.1 dev wlp0s20f3`: exit status 1

A long running client thought it was still connected but the server reported "registration not found".

This shouldn't happen in normal operation (problem 1), but if it does, the client should try to re-register automatically (problem 2)

That way we can test if the interfacepeers workaround is good enough to avoid peering over the ygg interface

Sounds like the way to go is having autoygg wait. Just opening this so we don't forget

when there is no gateway, this doesn't apply. but when a gateway exists we cant add a default gateway

used this to fix it during demo

in file /etc/config/autoygg

option adddefaultgatewaycommand 'ip ro replace default via %%ClientGateway%%'

we have a bug when the OpenWrt WAN network goes down for a minute, then is again reconnected to the same WAN defined zone.
so it's called the isp-interface right so,

once it down:

• we loose the default route 0.0.0.0/0 on device isp-gateway
• yggdrasil continues to work on LAN and Mesh interfaces
• peers over isp-interface and coords update

once up:

• the isp-interface still has null route entries for IP peers via isp-interface
• yggdrasil cannot reach these autoygg-servers directly

null routes prevent 1 form of causing a double peering with multiple meshnode-autoygg-servers in the mesh

Nov 18 01:16:11 ubuntu-s-1vcpu-1gb-nyc3-01 autoygg-client[1457]: Adding Yggdrasil local subnet 0.0.0.0/0 [ ok ]
Nov 18 01:16:11 ubuntu-s-1vcpu-1gb-nyc3-01 autoygg-client[1457]: Adding tunnel IP 10.42.42.5/16 [ ok ]
Nov 18 01:16:11 ubuntu-s-1vcpu-1gb-nyc3-01 autoygg-client[1457]: Adding Yggdrasil remote subnet 0.0.0.0/0 [ ok ]
Nov 18 01:16:12 ubuntu-s-1vcpu-1gb-nyc3-01 autoygg-client[1457]: Getting Yggdrasil peers [ FAIL ]
Nov 18 01:16:12 ubuntu-s-1vcpu-1gb-nyc3-01 autoygg-client[1457]: Error: Unable to run command which ygguci: exit status 1

The failure doesn't seem to affect functionality. Obviously, it shouldn't be running ygguci on this machine.

Nov 18 01:58:31 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Adding default gateway pointing at 10.42.0.1 [ ok ]
Nov 18 01:58:31 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Set up cron job to renew lease every 30 minutes [ ok ]
Nov 18 01:59:56 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: Stopping autoygg-client...
Nov 18 01:59:56 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: [79B blob data]
Nov 18 01:59:56 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Error: Unable to run /bin/sh -c ip ro replace default via 165.227.16.1: signal: terminated
Nov 18 01:59:56 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Getting Yggdrasil peers from state file [ ok ]
Nov 18 01:59:56 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Removing Yggdrasil peer route for 104.131.107.105 [ ok ]
Nov 18 01:59:56 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Removing Yggdrasil remote subnet 0.0.0.0/0 [ ok ]
Nov 18 01:59:56 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Removing tunnel IP 10.42.42.5/16 [ ok ]
Nov 18 01:59:56 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Removing Yggdrasil local subnet 0.0.0.0/0 [ ok ]
Nov 18 01:59:56 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Disabling Yggdrasil tunnel routing [ ok ]
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: State 'stop-sigterm' timed out. Killing.
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: Killing process 13953 (autoygg-client) with signal SIGKILL.
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: Killing process 13964 (autoygg-client) with signal SIGKILL.
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: Killing process 13965 (autoygg-client) with signal SIGKILL.
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: Killing process 13966 (autoygg-client) with signal SIGKILL.
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: Killing process 13967 (autoygg-client) with signal SIGKILL.
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: Killing process 14029 (autoygg-client) with signal SIGKILL.
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: Killing process 14275 (autoygg-client) with signal SIGKILL.
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 autoygg-client[13953]: Sending release request to autoygg Sending renew request to autoygg
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: Main process exited, code=killed, status=9/KILL
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: autoygg-client.service: Failed with result 'timeout'.
Nov 18 02:00:01 ubuntu-s-1vcpu-1gb-sfo2-01 systemd[1]: Stopped autoygg-client.

• iptables -t nat -A POSTROUTING -o ${autoygg_defaultgatewaydev} -j MASQUERADE
• replace default route when one exists (using 'add' also returns OK when exit code is 2
  • for openwrt /etc/config/autoygg i added:
  • option adddefaultgatewaycommand 'ip ro replace default via %%ClientGateway%%'
• utilize null routes (am i on the right version?)

I think I finally figured out why my gateway needs a reboot every once in a while, the default route in table 42 gets lost at some point. Replacing it immediately fixes the issue. We could periodically check the route, maybe the vpn interface periodically gets refreshed (maybe something mullvad does involving refreshing connections). Or if we can hook into interface down/up events we could see if that fixes it too.

ip ro add default dev vpn0 table 42

I'm using v0.2.2