Use private network where possible in fully meshed network about kilo HOT 4 OPEN

Hi @boedy this configuration is not possible today but is absolutely something that could technically be implemented in Kilo. It would be a slight twist on code-path for the full mesh, where Kilo checks if other nodes are in the same topological location to determine which link to use. Currently, Kilo's implementation of the full-mesh is just a simple take on the location mesh, where every node is treated like it's in its own location. All of this is to say that this is very feasible work IMO!

Do you feel up to contributing? I'd be able to point you at the codepaths that need attention. Otherwise, stay tuned and we can track my progress on the topic in this issue.

from kilo.

Thanks for the prompt reply @squat! I'm definitely willing to give it a shot, but haven't really contributed to a k8s Golang project before. If it's just a few code paths that need modifying I might be able to pull if off tho 🤞

I could use some guidance in setting up a development environment. What has been your workflow whist working on the project?

from kilo.

I'm guessing the change could possibly effect this line:
https://github.com/squat/kilo/blob/37f4ea52dc54563b3ec8ff69b4d322022ef9ba12/pkg/mesh/topology.go#L186C4-L186C4

With a basic check for when a node is trying to communicate with another node in the same topological location, it should use the private IP as the WireGuard endpoint instead of the leader's public endpoint. This should also only be relevant when the granularity full is used.

from kilo.

Whoo I actually got it working! 🥳 here's the comparison of my changes with the main branch.

I mentioned in my previous message. I was looking for a way to determine if two IP addresses belong to the same subnet. During my tests, I initially hoped to determine the CIDR directly from the private IP address of nodes. However, I discovered that this isn't always set precisely. For instance, on Hetzner, where my private network has a subnet mask of 10.10.0.0/16, the CIDR couldn't be derived directly when I accessed a node. Using ip a, the output was inet 10.10.0.4/32 brd 10.10.0.4 scope global dynamic enp7s0.

To overcome this challenge, I leveraged the kilo.squat.ai/force-internal-ip annotation, which allowed me to determine the appropriate subnet mask. This was also necessary for my DigitalOcean nodes since the incorrect network interface was identified for the Private IP.

Moving forward, I have a few questions regarding how we should handle this feature:

1. Activation Mechanism: How should we enable this functionality? Via specific annotations, or maybe a configuration setting?
2. Default Behavior: Should this feature be active by default or be an opt-in?
3. Feedback and Suggestions: The changes for now are minimal. Are there any modifications or refinements you'd recommend based on the implementation?

from kilo.