Code Monkey home page Code Monkey logo

Comments (13)

aserdean avatar aserdean commented on August 23, 2024 1

A bit of more clarification on the subject. It is a bit long read but at least we won't forget it.

On Windows we require that you add the physical NIC under a bridge and set the internal IP of the bridge to that of the physical NIC. It is an unfortunate mandatory requirement for using OVS on Windows (it is a design decision of the Hyper-V vSwitch implementation, if we go one level down, we could bypass this in the future). For Linux users it is similar on what we do when setting up the gateway node: eg. ovn-k8s-util nics-to-bridge eth0 .
Thus, when playing around with the Hyper-V vSwitch and OVS, please be careful because you can nuke your connectivity (i.e. the script https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster/blob/master/worker/windows/install_ovn.ps1 is intended to have valid configurations and be run as a script, not individually line by line; https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster/blob/master/worker/windows/install_ovn.ps1#L14-L31 can break your network connectivity).

Regarding the environment on AWS using a private network.
Since we added the physical NIC under a bridge all packets that came through that interface were tagged with a specific VLAN (eg. 2301). Packets going out were untagged (at the host level) and reaching the other hosts (i.e. kube-system_kube-controller-manager-ip-10-5-35-142; 10.244.1.2 ) and even getting the reply but unfortunately they were tagged when receiving it. The outcome was to strip all tags for an easy fix.
This is due most probably because not all configuration from the physical NIC(Ethernet) were carried over by the internal NIC (the port of the bridge, HNSTransparent).
DHCP client on Windows also does not take into account the MTU sent by the DHCP server i.e., so that needs to be set also to the value of the physical NIC ( please see line: https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster/blob/master/worker/windows/install_ovn.ps1#L39-L40).

At the end of the day the lesson learned is to at least log all the properties that were on the physical NIC and match them with the internal port created via the bridge. As a nice to have is to have either a powershell script or cmd script that clones the values.

from ovn-kubernetes.

owain-je avatar owain-je commented on August 23, 2024

Windows box details

PS C:\Program Files\Cloudbase Solutions\Open vSwitch\logs> ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : EC2AMAZ-3S8E4H7
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : eu-west-1.ec2-utilities.amazonaws.com
                                       us-east-1.ec2-utilities.amazonaws.com
                                       eu-west-1.compute.internal

Ethernet adapter HNSTransparent:

   Connection-specific DNS Suffix  . : eu-west-1.compute.internal
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter
   Physical Address. . . . . . . . . : 06-F0-28-E3-D5-78
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::fd2a:62c8:d46:719f%10(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.5.54.78(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.224.0
   Lease Obtained. . . . . . . . . . : Thursday, August 17, 2017 10:15:00 PM
   Lease Expires . . . . . . . . . . : Monday, August 21, 2017 11:57:08 AM
   Default Gateway . . . . . . . . . : 10.5.32.1
   DHCP Server . . . . . . . . . . . : 10.5.32.1
   DHCPv6 IAID . . . . . . . . . . . : 168226856
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-22-F4-94-0A-2D-23-44-95-52
   DNS Servers . . . . . . . . . . . : 10.5.0.2
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Ethernet 3:

   Connection-specific DNS Suffix  . : eu-west-1.compute.internal
   Description . . . . . . . . . . . : Intel(R) 82599 Virtual Function #2
   Physical Address. . . . . . . . . : 06-B2-F1-31-7C-4A
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::405e:9a46:799e:3db8%14(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.5.43.51(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.224.0
   Lease Obtained. . . . . . . . . . : Monday, August 21, 2017 9:35:12 AM
   Lease Expires . . . . . . . . . . : Monday, August 21, 2017 12:05:11 PM
   Default Gateway . . . . . . . . . : 10.5.32.1
   DHCP Server . . . . . . . . . . . : 10.5.32.1
   DHCPv6 IAID . . . . . . . . . . . : 235306032
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-22-F4-94-0A-2D-23-44-95-52
   DNS Servers . . . . . . . . . . . : 10.5.0.2
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter k8s-EC2AMAZ-3S8:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #4
   Physical Address. . . . . . . . . : 00-15-5D-00-CA-03
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::148d:4f00:93ec:13ee%23(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.244.39.2(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DHCPv6 IAID . . . . . . . . . . . : 385881437
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-22-F4-94-0A-2D-23-44-95-52
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

from ovn-kubernetes.

owain-je avatar owain-je commented on August 23, 2024

On the windows node

PS C:\kubernetes> ovs-vsctl show
cd568120-44a6-46e4-a55a-664cc11351a9
    Bridge br-ex
        Port HNSTransparent
            Interface HNSTransparent
                type: internal
        Port br-ex
            Interface br-ex
                type: internal
        Port Ethernet
            Interface Ethernet
    Bridge br-int
        Port "Container NIC 0c2dabcb"
            Interface "Container NIC 0c2dabcb"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port "ovn-239a97-0"
            Interface "ovn-239a97-0"
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.5.38.109"}
        Port "Container NIC e63fb2b0"
            Interface "Container NIC e63fb2b0"
                type: internal
        Port "ovn-f06c10-0"
            Interface "ovn-f06c10-0"
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.5.35.142"}
        Port "ovn-a312f8-0"
            Interface "ovn-a312f8-0"
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.5.62.104"}
                error: "could not add network device ovn-a312f8-0 to ofproto (File exists)"
        Port "k8s-EC2AMAZ-3S8"
            Interface "k8s-EC2AMAZ-3S8"
                type: internal
        Port "ovn-ba7e1d-0"
            Interface "ovn-ba7e1d-0"
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.5.46.236"}
        Port "Container NIC ee8f7986"
            Interface "Container NIC ee8f7986"
                type: internal
        Port "ovn-95ef86-0"
            Interface "ovn-95ef86-0"
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.5.53.47"}
        Port "ovn-b037bf-0"
            Interface "ovn-b037bf-0"
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.5.62.104"}

from ovn-kubernetes.

shettyg avatar shettyg commented on August 23, 2024

@alinbalutoiu

Any hints?

from ovn-kubernetes.

rajatchopra avatar rajatchopra commented on August 23, 2024

error: "could not add network device ovn-a312f8-0 to ofproto (File exists)"

The openvswitch logs should tell more about that.

from ovn-kubernetes.

aserdean avatar aserdean commented on August 23, 2024

@owain-je
'EC2AMAZ-04IFR3J' is defined twice in SB
ovn-sbctl chassis-del a312f894-4a8a-4776-b3b7-3880e2b999a4
ovn-sbctl chassis-del b037bfa2-bd40-413b-b2f9-8c6384f581e9

The ovn-controller should pick it up from there.

I don't think our init script handles duplicate initialization.

Can you please verify the connectivity from the windows host to the remote_ip's:
i.e ping 10.5.53.47

Also can you please check if you can ping the following:
ping 10.244.39.1
ping 10.244.1.2

from ovn-kubernetes.

owain-je avatar owain-je commented on August 23, 2024

Sorry in all day workshops this week.
So I should have said this : I have 2 windows nodes , the first attempt is the one with duplicate entries
The node I am trying to get to work has the name: EC2AMAZ-3S8E4H7
So this node has 2 network cards , because when I make the primary nic transparent with hyper V it seems to lose local connectivity on the 10.5 range? is this expected? The command that causes this is

docker network create -d transparent --gateway $GATEWAY_IP --subnet $SUBNET -o com.docker.network.windowsshim.interface=$INTERFACE_ALIAS external

from these instructions: https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster

So pinging 10.5 addresses is fine everywhere (with 2nd nic in place, which I need to RDP on to the server)
ping 10.244 addresses from linux to linux addresses and from pod to pod is fine and from master host to pod.
but pinging 10.244 addresses from and to windows pods doesn't work, nor from the master

Any help really welcome.

from ovn-kubernetes.

owain-je avatar owain-je commented on August 23, 2024

From the windows box.

PS C:\Program Files\Cloudbase Solutions\Open vSwitch\bin> ping 10.244.39.2

Pinging 10.244.39.2 with 32 bytes of data:
Reply from 10.244.39.2: bytes=32 time<1ms TTL=128
Reply from 10.244.39.2: bytes=32 time<1ms TTL=128

PS C:\Program Files\Cloudbase Solutions\Open vSwitch\bin> ping 10.244.39.1

Pinging 10.244.39.1 with 32 bytes of data:
Reply from 10.244.39.1: bytes=32 time<1ms TTL=254

Ping statistics for 10.244.39.1:
    Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
PS C:\Program Files\Cloudbase Solutions\Open vSwitch\bin>
PS C:\Program Files\Cloudbase Solutions\Open vSwitch\bin> ping 10.244.1.2

Pinging 10.244.1.2 with 32 bytes of data:
Request timed out.

Ping statistics for 10.244.1.2:
    Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
Control-C

from ovn-kubernetes.

owain-je avatar owain-je commented on August 23, 2024

From the master node:

ubuntu@ip-10-5-35-142:~$ sudo su -
root@ip-10-5-35-142:~# ping 10.244.1.2
PING 10.244.1.2 (10.244.1.2) 56(84) bytes of data.
64 bytes from 10.244.1.2: icmp_seq=1 ttl=64 time=0.060 ms
^C
--- 10.244.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.060/0.060/0.060/0.000 ms
root@ip-10-5-35-142:~# ping 10.244.39.2
PING 10.244.39.2 (10.244.39.2) 56(84) bytes of data.
^C
--- 10.244.39.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4031ms

root@ip-10-5-35-142:~# ping 10.244.39.1
PING 10.244.39.1 (10.244.39.1) 56(84) bytes of data.
64 bytes from 10.244.39.1: icmp_seq=1 ttl=254 time=0.445 ms
^C
--- 10.244.39.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.445/0.445/0.445/0.000 ms

from ovn-kubernetes.

owain-je avatar owain-je commented on August 23, 2024

Nudge? Can anyone give me any hints?

from ovn-kubernetes.

hlrichardson avatar hlrichardson commented on August 23, 2024

I see nothing obviously wrong in the ovn configuration. It would be interesting to see the output of "ovs-dpctl dump-flows br-int | grep 10.244.39.2" while attempting the failing ping to 10.244.39.2.

from ovn-kubernetes.

owain-je avatar owain-je commented on August 23, 2024

So the issue appears to be due to using AWS internal networking vs public ip and stray vlans
running

ovs-ofctl.exe add-flow br-ex priority=1,action=strip_vlan,NORMAL

is the fix (thanks to cloudbase guys) abalutoiu & Alin Serdean

from ovn-kubernetes.

shettyg avatar shettyg commented on August 23, 2024

I assume this is fixed now.

from ovn-kubernetes.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.