Logical port is down after creation about ovn-kubernetes HOT 6 CLOSED

@shettyg
Actually running core daemons is not a big issue. They are running with hostNetwork: true, so they are getting host IP address and not OVN private IP addresses.
On the other hand, kube-dns and kube-dashboard are getting OVN private IP addresses.

Basically, what should be done is the following:

1. Create manifests for core services (kube-apiserver.yml, kube-controller-manager.yml, kube-proxy.yml, kube-scheduler.yml) and place them in /etc/kubernetes/manifests folder
2. Create a custom image on top of hyperkube one with openvswitch, cni and ovn-kubernetes installed
3. Run kubelet container with this configuration

/usr/bin/docker run -d \
    --net=host \
    --pid=host \
    --privileged \
    --restart=unless-stopped \
    --name kubelet \
    --volume /etc/cni/net.d:/etc/cni/net.d \
    --volume /etc/kubernetes:/etc/kubernetes \
    --volume /sys:/sys:rw \
    --volume /var/run:/var/run:rw \
    --volume /run:/run:rw \
    --volume /var/lib/docker:/var/lib/docker:rw \
    --volume /var/lib/kubelet:/var/lib/kubelet:shared \
    --volume /var/log/containers:/var/log/containers:rw \
    <custom_hyperkube_image>:v1.3.6 \
    /hyperkube kubelet \
      --allow-privileged \
      --api-servers=http://127.0.0.1:8080 \
      --config=/etc/kubernetes/manifests \
      --cluster-dns=<kube_dns_service_ip> \
      --cluster-domain=certascale.local \
      --network-plugin=cni \
      --network-plugin-dir=/etc/cni/net.d \
      --hostname-override=<eth_ip_address> \
      --v=2

_
4. Run ovs commands inside container:

docker exec kubelet /usr/share/openvswitch/scripts/ovs-ctl start

...

docker exec kubelet ovn-k8s-watcher --overlay

One caveat is that /etc/cni/net.d should contain some configuration file. It won't be used but it should be present.

Another one is that it is not a CoreOS suggested way of running kubernetes. They suggest to use kubelet-wrapper, but it runs kubelet in rkt and you won't be able to manage OVS in it.

from ovn-kubernetes.

Which core daemons are you referring to?
I am not using coreos but on ubuntu I have no problem running the kubelet as well on the master node. I agree you probably should not be doing that in production, but in dev envs it gives you an extra node!

From what I recall the kubelet does not invoke network plugins for pods with special security context attributes such as hostnetwork.

from ovn-kubernetes.

@RostyslavFridman

The current documentation and code expects master to not run any containers. There are multiple reasons for it - one of them being containers mucking around with the north-bound database. So you likely ran just the master-init? That script does not create a CNI plugin and hence the containers started will get IP addresses from docker bridge.

Btwn, thanks for trying it out. We still are a little poor with documentation (and a couple of bugs). I just noticed that there is no mention of how to start kubelet with the OVN CNI plugin.
It should be something like:

./kubelet --api-servers=http://10.33.74.22:8080 --v=2 --address=0.0.0.0 --enable-server=true --network-plugin=cni

from ovn-kubernetes.

On that note, it should not be a big problem running pods on master as well.
I have indeed a 2 node setup where the kubelet is running also on the master node (mostly to save the hassle of using a 3rd vm to achieve a 2-node testbed).

To do so I think I ran both master-init and minion-init on the master, specifying the same subnet. If I recall correctly minion-init skipped the steps for creating the node ls and connecting it to the cluster lr as they were already performed by master-init, and simply configured the CNI plugin.

from ovn-kubernetes.

It finally works.
Moreover, it is OK to run master-init and minion-init on the same node in the same subnet.

I've struggled mostly with running this solution on CoreOS when everything is containerized.
If anyone would like to repeat this, dockerized kubelet should contain cni binaries, openvswitch binaries and ovn-kubernetes scripts.

Thank you for the help.

from ovn-kubernetes.

@RostyslavFridman
It looks like you are familiar with both k8s and OVN to have got this working with coreos. One challenge of having CNI plugin running in master would be that the core daemons running inside containers will also get OVN private IP addresses?

from ovn-kubernetes.