Comments (13)
A bit of more clarification on the subject. It is a bit long read but at least we won't forget it.
On Windows we require that you add the physical NIC under a bridge and set the internal IP of the bridge to that of the physical NIC. It is an unfortunate mandatory requirement for using OVS on Windows (it is a design decision of the Hyper-V vSwitch implementation, if we go one level down, we could bypass this in the future). For Linux users it is similar on what we do when setting up the gateway node: eg. ovn-k8s-util nics-to-bridge eth0
.
Thus, when playing around with the Hyper-V vSwitch and OVS, please be careful because you can nuke your connectivity (i.e. the script https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster/blob/master/worker/windows/install_ovn.ps1 is intended to have valid configurations and be run as a script, not individually line by line; https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster/blob/master/worker/windows/install_ovn.ps1#L14-L31 can break your network connectivity).
Regarding the environment on AWS using a private network.
Since we added the physical NIC under a bridge all packets that came through that interface were tagged with a specific VLAN (eg. 2301). Packets going out were untagged (at the host level) and reaching the other hosts (i.e. kube-system_kube-controller-manager-ip-10-5-35-142
; 10.244.1.2
) and even getting the reply but unfortunately they were tagged when receiving it. The outcome was to strip all tags for an easy fix.
This is due most probably because not all configuration from the physical NIC(Ethernet
) were carried over by the internal NIC (the port of the bridge, HNSTransparent
).
DHCP client on Windows also does not take into account the MTU sent by the DHCP server i.e., so that needs to be set also to the value of the physical NIC ( please see line: https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster/blob/master/worker/windows/install_ovn.ps1#L39-L40).
At the end of the day the lesson learned is to at least log all the properties that were on the physical NIC and match them with the internal port created via the bridge. As a nice to have is to have either a powershell script or cmd script that clones the values.
from ovn-kubernetes.
Windows box details
PS C:\Program Files\Cloudbase Solutions\Open vSwitch\logs> ipconfig /all
Windows IP Configuration
Host Name . . . . . . . . . . . . : EC2AMAZ-3S8E4H7
Primary Dns Suffix . . . . . . . :
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : eu-west-1.ec2-utilities.amazonaws.com
us-east-1.ec2-utilities.amazonaws.com
eu-west-1.compute.internal
Ethernet adapter HNSTransparent:
Connection-specific DNS Suffix . : eu-west-1.compute.internal
Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter
Physical Address. . . . . . . . . : 06-F0-28-E3-D5-78
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::fd2a:62c8:d46:719f%10(Preferred)
IPv4 Address. . . . . . . . . . . : 10.5.54.78(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.224.0
Lease Obtained. . . . . . . . . . : Thursday, August 17, 2017 10:15:00 PM
Lease Expires . . . . . . . . . . : Monday, August 21, 2017 11:57:08 AM
Default Gateway . . . . . . . . . : 10.5.32.1
DHCP Server . . . . . . . . . . . : 10.5.32.1
DHCPv6 IAID . . . . . . . . . . . : 168226856
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-22-F4-94-0A-2D-23-44-95-52
DNS Servers . . . . . . . . . . . : 10.5.0.2
NetBIOS over Tcpip. . . . . . . . : Enabled
Ethernet adapter Ethernet 3:
Connection-specific DNS Suffix . : eu-west-1.compute.internal
Description . . . . . . . . . . . : Intel(R) 82599 Virtual Function #2
Physical Address. . . . . . . . . : 06-B2-F1-31-7C-4A
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::405e:9a46:799e:3db8%14(Preferred)
IPv4 Address. . . . . . . . . . . : 10.5.43.51(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.224.0
Lease Obtained. . . . . . . . . . : Monday, August 21, 2017 9:35:12 AM
Lease Expires . . . . . . . . . . : Monday, August 21, 2017 12:05:11 PM
Default Gateway . . . . . . . . . : 10.5.32.1
DHCP Server . . . . . . . . . . . : 10.5.32.1
DHCPv6 IAID . . . . . . . . . . . : 235306032
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-22-F4-94-0A-2D-23-44-95-52
DNS Servers . . . . . . . . . . . : 10.5.0.2
NetBIOS over Tcpip. . . . . . . . : Enabled
Ethernet adapter k8s-EC2AMAZ-3S8:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #4
Physical Address. . . . . . . . . : 00-15-5D-00-CA-03
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::148d:4f00:93ec:13ee%23(Preferred)
IPv4 Address. . . . . . . . . . . : 10.244.39.2(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . :
DHCPv6 IAID . . . . . . . . . . . : 385881437
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-22-F4-94-0A-2D-23-44-95-52
DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
fec0:0:0:ffff::2%1
fec0:0:0:ffff::3%1
NetBIOS over Tcpip. . . . . . . . : Enabled
from ovn-kubernetes.
On the windows node
PS C:\kubernetes> ovs-vsctl show
cd568120-44a6-46e4-a55a-664cc11351a9
Bridge br-ex
Port HNSTransparent
Interface HNSTransparent
type: internal
Port br-ex
Interface br-ex
type: internal
Port Ethernet
Interface Ethernet
Bridge br-int
Port "Container NIC 0c2dabcb"
Interface "Container NIC 0c2dabcb"
type: internal
Port br-int
Interface br-int
type: internal
Port "ovn-239a97-0"
Interface "ovn-239a97-0"
type: geneve
options: {csum="true", key=flow, remote_ip="10.5.38.109"}
Port "Container NIC e63fb2b0"
Interface "Container NIC e63fb2b0"
type: internal
Port "ovn-f06c10-0"
Interface "ovn-f06c10-0"
type: geneve
options: {csum="true", key=flow, remote_ip="10.5.35.142"}
Port "ovn-a312f8-0"
Interface "ovn-a312f8-0"
type: geneve
options: {csum="true", key=flow, remote_ip="10.5.62.104"}
error: "could not add network device ovn-a312f8-0 to ofproto (File exists)"
Port "k8s-EC2AMAZ-3S8"
Interface "k8s-EC2AMAZ-3S8"
type: internal
Port "ovn-ba7e1d-0"
Interface "ovn-ba7e1d-0"
type: geneve
options: {csum="true", key=flow, remote_ip="10.5.46.236"}
Port "Container NIC ee8f7986"
Interface "Container NIC ee8f7986"
type: internal
Port "ovn-95ef86-0"
Interface "ovn-95ef86-0"
type: geneve
options: {csum="true", key=flow, remote_ip="10.5.53.47"}
Port "ovn-b037bf-0"
Interface "ovn-b037bf-0"
type: geneve
options: {csum="true", key=flow, remote_ip="10.5.62.104"}
from ovn-kubernetes.
Any hints?
from ovn-kubernetes.
error: "could not add network device ovn-a312f8-0 to ofproto (File exists)"
The openvswitch logs should tell more about that.
from ovn-kubernetes.
@owain-je
'EC2AMAZ-04IFR3J' is defined twice in SB
ovn-sbctl chassis-del a312f894-4a8a-4776-b3b7-3880e2b999a4
ovn-sbctl chassis-del b037bfa2-bd40-413b-b2f9-8c6384f581e9
The ovn-controller should pick it up from there.
I don't think our init script handles duplicate initialization.
Can you please verify the connectivity from the windows host to the remote_ip's:
i.e ping 10.5.53.47
Also can you please check if you can ping the following:
ping 10.244.39.1
ping 10.244.1.2
from ovn-kubernetes.
Sorry in all day workshops this week.
So I should have said this : I have 2 windows nodes , the first attempt is the one with duplicate entries
The node I am trying to get to work has the name: EC2AMAZ-3S8E4H7
So this node has 2 network cards , because when I make the primary nic transparent with hyper V it seems to lose local connectivity on the 10.5 range? is this expected? The command that causes this is
docker network create -d transparent --gateway $GATEWAY_IP --subnet $SUBNET -o com.docker.network.windowsshim.interface=$INTERFACE_ALIAS external
from these instructions: https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster
So pinging 10.5 addresses is fine everywhere (with 2nd nic in place, which I need to RDP on to the server)
ping 10.244 addresses from linux to linux addresses and from pod to pod is fine and from master host to pod.
but pinging 10.244 addresses from and to windows pods doesn't work, nor from the master
Any help really welcome.
from ovn-kubernetes.
From the windows box.
PS C:\Program Files\Cloudbase Solutions\Open vSwitch\bin> ping 10.244.39.2
Pinging 10.244.39.2 with 32 bytes of data:
Reply from 10.244.39.2: bytes=32 time<1ms TTL=128
Reply from 10.244.39.2: bytes=32 time<1ms TTL=128
PS C:\Program Files\Cloudbase Solutions\Open vSwitch\bin> ping 10.244.39.1
Pinging 10.244.39.1 with 32 bytes of data:
Reply from 10.244.39.1: bytes=32 time<1ms TTL=254
Ping statistics for 10.244.39.1:
Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
PS C:\Program Files\Cloudbase Solutions\Open vSwitch\bin>
PS C:\Program Files\Cloudbase Solutions\Open vSwitch\bin> ping 10.244.1.2
Pinging 10.244.1.2 with 32 bytes of data:
Request timed out.
Ping statistics for 10.244.1.2:
Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
Control-C
from ovn-kubernetes.
From the master node:
ubuntu@ip-10-5-35-142:~$ sudo su -
root@ip-10-5-35-142:~# ping 10.244.1.2
PING 10.244.1.2 (10.244.1.2) 56(84) bytes of data.
64 bytes from 10.244.1.2: icmp_seq=1 ttl=64 time=0.060 ms
^C
--- 10.244.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.060/0.060/0.060/0.000 ms
root@ip-10-5-35-142:~# ping 10.244.39.2
PING 10.244.39.2 (10.244.39.2) 56(84) bytes of data.
^C
--- 10.244.39.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4031ms
root@ip-10-5-35-142:~# ping 10.244.39.1
PING 10.244.39.1 (10.244.39.1) 56(84) bytes of data.
64 bytes from 10.244.39.1: icmp_seq=1 ttl=254 time=0.445 ms
^C
--- 10.244.39.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.445/0.445/0.445/0.000 ms
from ovn-kubernetes.
Nudge? Can anyone give me any hints?
from ovn-kubernetes.
I see nothing obviously wrong in the ovn configuration. It would be interesting to see the output of "ovs-dpctl dump-flows br-int | grep 10.244.39.2" while attempting the failing ping to 10.244.39.2.
from ovn-kubernetes.
So the issue appears to be due to using AWS internal networking vs public ip and stray vlans
running
ovs-ofctl.exe add-flow br-ex priority=1,action=strip_vlan,NORMAL
is the fix (thanks to cloudbase guys) abalutoiu & Alin Serdean
from ovn-kubernetes.
I assume this is fixed now.
from ovn-kubernetes.
Related Issues (20)
- Support EgressIP for user defined networks
- Flake: [FAIL] External Gateway With Admin Policy Based External Route CRs e2e non-vxlan external gateway through a gateway pod Should validate ICMP connectivity to an external gateway's loopback address via a gateway pod [It] ipv4 HOT 9
- [FAIL] e2e egress firewall policy validation with external containers [It] Should validate the egress firewall policy functionality for allowed IP HOT 9
- Upgrades tests operation cancelled
- e2e EgressQoS validation -- account for single stack cluster
- Transit switch subnet overlap check is missing HOT 1
- UDN, non-IC: network isolation e2e tests fail on non-IC deployments HOT 1
- kind helm no longer works when deployed with NAD listener HOT 2
- ovn-kubernetes in a vm-s with undrecloud scenario HOT 5
- Node IP Handler event tests panics HOT 1
- [Race] Delete SNAT per pod on gateway route
- Routes created by hybrid-overlay in Windows are incompatible with AWS EC2LaunchV2 agent
- DATA RACE during egress firewall unit tests with config.PrepareTestConfig() HOT 3
- DATA RACE while running network attach def controller tests
- UDN: Convert pod2Egress e2e to also include CRD in addition to using NADs
- UDN API validations: Use standard CEL CIDR verbs to validate CIDR values
- [FAIL] Network Segmentation a user defined primary network created using NetworkAttachmentDefinitions is isolated from the default network [It] with L3 dualstack primary UDN HOT 2
- Ensure unexpected UDN traffic is dropped in the external OVS bridge.
- checkPorts should not use os.Exit
- FLAKE: Load Balancer Service Tests with MetalLB [It] Should ensure load balancer service works when ETP=local and session affinity is set
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ovn-kubernetes.