openshift / ovn-kubernetes Goto Github PK

vSphere:
ESXi version: 7.0 U3e
Seperate vDS (on version 6.5) for Front End and iSCSI

Hardware:
UCS B200-M4 Blade
	BIOS - B200M4.4.1.2a.0.0202211902
	Xeon(R) CPU E5-2667
	2 x 20Gb Cisco UCS VIC 1340 network adapter for front end connectivity (Firmware 4.5(1a))
	2 x 20Gb Cisco UCS VIC 1340 network adapter for iSCSI connectivity (Firmware 4.5(1a))
	
Storage:
Compellent SC4020 over iSCSI
	2 controller array with dual iSCSI IP connectivity (2 paths per LUN)
All cluster nodes on same Datastore

Benchmark Results

Name : knb-2672
Date : 2023-03-29 15:26:01 UTC
Generator : knb
Version : 1.5.0
Server : k8s-se-internal-01-582st-worker-n2wtp
Client : k8s-se-internal-01-582st-worker-cv7cd
UDP Socket size : auto

Discovered CPU : Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
Discovered Kernel : 5.18.5-100.fc35.x86_64
Discovered k8s version : v1.23.5-rc.0.2076+8cfebb1ce4a59f-dirty
Discovered MTU : 1400
Idle :
bandwidth = 0 Mbit/s
client cpu = total 12.31% (user 9.41%, nice 0.00%, system 2.83%, iowait 0.07%, steal 0.00%)
server cpu = total 9.04% (user 6.28%, nice 0.00%, system 2.74%, iowait 0.02%, steal 0.00%)
client ram = 4440 MB
server ram = 3828 MB
Pod to pod :
TCP :
bandwidth = 6306 Mbit/s
client cpu = total 26.15% (user 5.19%, nice 0.00%, system 20.96%, iowait 0.00%, steal 0.00%)
server cpu = total 29.39% (user 8.13%, nice 0.00%, system 21.26%, iowait 0.00%, steal 0.00%)
client ram = 4460 MB
server ram = 3820 MB
UDP :
bandwidth = 1424 Mbit/s
client cpu = total 26.08% (user 7.21%, nice 0.00%, system 18.82%, iowait 0.05%, steal 0.00%)
server cpu = total 24.82% (user 6.72%, nice 0.00%, system 18.05%, iowait 0.05%, steal 0.00%)
client ram = 4444 MB
server ram = 3824 MB
Pod to Service :
TCP :
bandwidth = 6227 Mbit/s
client cpu = total 27.90% (user 5.12%, nice 0.00%, system 22.73%, iowait 0.05%, steal 0.00%)
server cpu = total 29.85% (user 5.86%, nice 0.00%, system 23.99%, iowait 0.00%, steal 0.00%)
client ram = 4439 MB
server ram = 3811 MB
UDP :
bandwidth = 1576 Mbit/s
client cpu = total 32.31% (user 6.41%, nice 0.00%, system 25.90%, iowait 0.00%, steal 0.00%)
server cpu = total 26.12% (user 5.68%, nice 0.00%, system 20.39%, iowait 0.05%, steal 0.00%)
client ram = 4449 MB
server ram = 3818 MB

sh-4.4# time curl http://10.129.2.44:8080
Hello, 10.128.2.2. You have reached 10.129.2.44 on k8s-se-internal-01-582st-worker-cv7cd
real    0m26.496s
We have been able to reproduce this issue consistently on multiple different clusters.

POD=$(oc -n openshift-ovn-kubernetes get pod -o custom-columns=POD:.metadata.name --no-headers  --selector='app==ovnkube-master' | head -n1)
oc -n openshift-ovn-kubernetes exec $POD -c ovnkube-master -it -- ovsdb-client --private-key=/ovn-cert/tls.key --certificate=/ovn-cert/tls.crt --ca-cert=/ovn-ca/ca-bundle.crt -f csv --no-headings dump ssl:localhost:9641 OVN_Northbound NAT | grep "name=$EGRESS_NAME" | tr -d '"' | cut -d ',' -f5,9 | sort -u

10.1.1.129,k8s-okd4-xxx
10.1.1.129,k8s-okd4-yyy
10.1.1.129,k8s-okd4-zzz
10.1.1.129,k8s-okd4-aaa
10.1.1.130,k8s-okd4-bbb
10.1.1.130,k8s-okd4-ccc
10.1.1.130,k8s-okd4-ddd
10.1.1.130,k8s-okd4-eee

...
Step 10/32 : RUN yum install -y      selinux-policy &&     yum clean all
 ---> Running in 01784f0dd84f
Loaded plugins: ovl, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
http://base-4-0.ocp.svc/rhel-fast-datapath/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: base-4-0.ocp.svc; Unknown error"
Trying other mirror.


 One of the configured repositories failed (rhel-fast-datapath),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the