Multiple network support. about ovn-kubernetes HOT 20 CLOSED

@rkamudhan thanks, not sure the overlap between kubeconfig and multus.conf - seems to be some overlap - probably just my lack of knowledge about k8s

from ovn-kubernetes.

Annotations are a powerful (and dangerous!) tool. Basically they allow you to do everything you want with the API, including specifying additional networks!

As you say doing this is fairly trivial. Kubelet will know nothing about the additonal network interface, and this should not be a problem. We just need to make sure that when it retrieves container status nsenter always finds the interface kubelet knows about - otherwise the container status will have the IP address of the other interface.

Said that, I think it might be worth implementing this as an experimental feature, if nothing else to get a feeling of what it means to support multiple interfaces in kubernetes pods.

from ovn-kubernetes.

Kubernetes issue discussing multiple IPs: Issue kubernetes/kubernetes#27398

from ovn-kubernetes.

Latest multi-network discussion could found here.

from ovn-kubernetes.

I have a VNF that I am deploying as a DaemonSet and want to have two interfaces one for management and one for dataplane traffic. I want the data plane interface to be on OVN to enable traffic steering, the other can either be OVN or another CNI plugin.

I am trying to use Multus to set this up: https://github.com/Intel-Corp/multus-cni
I have Multus working for two interfaces using CNI bridge and that works fine. I have a single OVN interface coming up under Multus but when I try to bring up two interfaces either one bridge and one OVN or two OVN I am getting parsing errors in Multus.

Before I delve deeper into the code and try and isolate the issue a couple of questions:

1. Has anyone else got Multus (or CNI-Genie https://github.com/Huawei-PaaS/CNI-Genie) working with OVN ?
2. OVN-Kubernetes has the CNI version tied to 0.1.0, I did change this to 0.3.1 but no impact but slightly better error messages - does anyone know if I need to add some additional code to support Multus type networking?

Regards

John

from ovn-kubernetes.

@doonhammer I am Multus CNI developer, Can you say the parsing error details in Multus or log the issue in the https://github.com/Intel-Corp/multus-cni . I can help to solve the issues.

from ovn-kubernetes.

@rkamudhan thanks very much 👍

Included errors from k8s and OVN logs

from ./kubectl describe pod firewall-vhf-xxxxx
9s 9s 1 {kubelet k8sminion1} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "firewall-vnf-8hzkz_kube-system" with SetupNetworkError: "Failed to setup network for pod "firewall-vnf-8hzkz_kube-system(317efda3-a318-11e7-877a-5254007a695a)" using network plugins "cni": netplugin failed but error parsing its diagnostic message "{\n \"ip4\": {\n \"ip\": \"10.22.0.131/16\",\n \"gateway\": \"10.22.0.1\",\n \"routes\": [\n {\n \"dst\": \"0.0.0.0/0\",\n \"gw\": \"10.22.0.1\"\n }\n ]\n },\n \"dns\": {}\n}{\n \"code\": 100,\n \"msg\": \"Multus: error in invoke Delegate add - \\\"ovn-cni\\\": \"\n}": invalid character '{' after top-level value; Skipping pod"

and from : /var/log/openvswitch/ovn-k8s-cni-overlay.log

2017-09-27T00:12:31.931Z | 2 | ovn-k8s-cni-overlay | DBG | Creating veth pair for container d2c88d6ace883234cef730c973a6e64421f7f8da6a78397580b6ad861c36bd6e
2017-09-27T00:12:31.980Z | 3 | ovn-k8s-cni-overlay | DBG | Bringing up veth outer interface d2c88d6ace88323
2017-09-27T00:12:32.141Z | 4 | ovn-k8s-cni-overlay | DBG | Create a link for container namespace
2017-09-27T00:12:32.146Z | 5 | ovn-k8s-cni-overlay | DBG | Adding veth inner interface to namespace for container d2c88d6ace883234cef730c973a6e64421f7f8da6a78397580b6ad861c36bd6e
2017-09-27T00:12:32.167Z | 6 | ovn-k8s-cni-overlay | DBG | Configuring and bringing up veth inner interface d2c88d6ace883_c. New name:'net0',MAC address:'0a:00:00:00:00:02', MTU:'1400', IP:192.168.2.3/24
2017-09-27T00:12:32.244Z | 7 | ovn-k8s-cni-overlay | DBG | Setting gateway_ip 192.168.2.1 for container:d2c88d6ace883234cef730c973a6e64421f7f8da6a78397580b6ad861c36bd6e
2017-09-27T00:12:32.249Z | 8 | ovn-k8s-cni-overlay | WARN | Failed to setup veth pair for pod: (17, 'File exists')
2017-09-27T00:12:32.249Z | 9 | ovn-k8s-cni-overlay | ERR | {"cniVersion": "0.1.0", "code": 100, "message": "container interface setup failure"}
2017-09-27T00:12:33.999Z | 0 | ovn-k8s-cni-overlay | DBG | plugin invoked with cni_command = ADD cni_container_id = e3c60987b98e427413efd2dfd0ff600c9e973e3046a0a64fa65511117db6ce45 cni_ifname = net0 cni_netns = /proc/23900/ns/net cni_args = IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=firewall-vnf-8hzkz;K8S_POD_INFRA_CONTAINER_ID=e3c60987b98e427413efd2dfd0ff600c9e973e3046a0a64fa65511117db6ce45
2017-09-27T00:12:34.037Z | 1 | kubernetes | DBG | Annotations for pod firewall-vnf-8hzkz: {u'ovn': u"{'gateway_ip': '192.168.2.1', 'ip_address': '192.168.2.3/24', 'mac_address': '0a:00:00:00:00:02'}", u'networks': u'[ { "name": "cni0" }, { "name": "ovn-0" } ]', u'kubernetes.io/created-by': u'{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"DaemonSet","namespace":"kube-system","name":"firewall-vnf","uid":"317e419e-a318-11e7-877a-5254007a695a","apiVersion":"extensions","resourceVersion":"2224"}}

from ovn-kubernetes.

@doonhammer

I have only thought briefly on how OVN will support this without some native k8s network objects. The way I have thought about this is that, you will have a special annotation added to pod and if it exists, in OVN cni plugin, you just create an additional network interface and plug it.

from ovn-kubernetes.

@shettyg

So in cni_add I would check for a new annotation (would have ip/mac/interface name) and then add another interface by duplicating the existing code (setup_interface followed by adding a new port to br-int?

from ovn-kubernetes.

@doonhammer

I guess, it depends on whether this extra interface will be a OVN logical port? Is it a OVN logical port?

from ovn-kubernetes.

@shettyg

I would assume so as I would want the management server to run on a arbitrary pod and be able to talk to any VNF - does that make sense?

from ovn-kubernetes.

@doonhammer

IRC is a better place to continue the discussion. I need some clarification..

from ovn-kubernetes.

So I have it working with Multus a CNI-Bridge and CNI-OVN. I needed to delete a line in ovn-k8s-cni-overlay (line 115) where is adding default gateway was failing with File Exists (Route exists). Not sure why anyone got an idea?

I am still getting some of the networking configured to talk outside of the cluster through the cni-bridge, but east -west works within the cluster - it might be a combination of vagrant and kubernetes networking. However everything seems to hang together.

from ovn-kubernetes.

I have been hacking the oven-kubernetes code to experiment with multiple OVN/OVS interfaces. I think I have an approach and wanted to run it by the community.

When a pod gets created an event is generated and the ovn-k8s-watcher receives an add pod event. At that point a call is made to create_logical_port - here we need to create two logical ports. This requires annotations in the pod yaml file. Either specific ovn-kubernetes annotations or we could use the CRD/TPR Network objects proposed by Multus.

The issue I see is that the network definition in the CNI file is needed when the pod is created (by OVN) on the master while the CNI network creation is done on the minion and the parameters are on the minion. This is currently worked around by passing custom ovn annotations IP/MAC/Gateway for a single interface. So I could read the CRD/TPR annotations and extend them with the IP/MAC/Gateway by making two calls to create_logical_port with an additional parameter "ovn-'network-name'" derived from the CRD/TRP Network objects. Then the CNI add would use the "network-name " to create the veth pair etc.

This seems workable but a little complex and requires matching the cni conf information and the pod yaml. I am always leery of requiring matching information in two locations. Does anyone have a better idea/approach?

from ovn-kubernetes.

Sounds reasonable to me despite apparent complexity. Actually it is not complex, just the underlying assumption/protocol of looking up by network-name is like an unwritten secret. Let's document it well in the code and we should be good. Working on an API/parameter in CNI that standardizes this will take forever/never.

Not sure of this though:

Either specific ovn-kubernetes annotations or we could use the CRD/TPR Network objects proposed by Multus.

from ovn-kubernetes.

The issue I see is that the network definition in the CNI file is needed when the pod is created (by OVN) on the master while the CNI network creation is done on the minion and the parameters are on the minion. This is currently worked around by passing custom ovn annotations IP/MAC/Gateway for a single interface. So I could read the CRD/TPR annotations and extend them with the IP/MAC/Gateway by making two calls to create_logical_port with an additional parameter "ovn-'network-name'" derived from the CRD/TRP Network objects. Then the CNI add would use the "network-name " to create the veth pair etc.

IMO, ignore the CNI network config file. Get the network annotation from pod (the string can be changed if there is consensus eventually). If the string matches in ovn watcher code, just create a new logical port and annotate the pod. The current CNI plugin should just read the additional annotation and set the second interface.

from ovn-kubernetes.

Thanks for the feedback @shettyg and @rajachopra. I have a version working I used the multus network format in the pod yaml file

annotations:
networks: '[
{ "name": "ovn-data" },
{ "name": "ovn-mgmt" }
]'

On a pod creation event I check to see if the annotations exist then I call create_logical_port twice passing in the name field - as an optional parameter (defaults to ovn). So if the annotations do not exist the code works as is.

I use the multus conf file to define the two networks and it calls chi-add twice in the multis configs file i name the networks own-data and ovn-mgmt the code the grabs the annotations created by the create_logical_port function. The IPAM fields are ignored in the multus conf

{
"name": "multus-demo-network",
"type": "multus",
"delegates": [
{
"type": "ovn_cni",
"name": "ovn-data",
"bridge": "br-int",
"ipMasq": false,
"ipam": {
"subnet": "192.168.2.0/24",
"type": "host-local"
},
"isGateway": true
},
{
"type": "ovn_cni",
"name": "ovn-mgmt",
"bridge": "br-int;",
"masterplugin": true,
"isGateway": true,
"ipMasq": false,
"ipam": {
"type": "host-local",
"subnet": "10.4.33.0/24",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}
]
}

[vagrant@k8smaster bin]$ ./kubectl --namespace kube-system exec -it firewall-vnf-1mt05 -- /bin/bash
bash-4.2# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP qlen 1000
link/ether 0a:00:00:00:00:06 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.2.8/24 scope global eth0
valid_lft forever preferred_lft forever
17: net0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP qlen 1000
link/ether 0a:00:00:00:00:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.2.7/24 scope global net0
valid_lft forever preferred_lft forever

The result is two network interfaces in a container as above.

Any suggestions comments?

from ovn-kubernetes.

@doonhammer Mulltus is picking the delegates from the conf file, for the annotation network, you must add the Kubeconfig field, Please refer this subsection regarding using both kubeconfig and delegates: https://github.com/Intel-Corp/multus-cni/#configuring-multus-to-use-the-kubeconfig-and-also-default-networks

from ovn-kubernetes.

also @shettyg how do you seen OVN integrating with this?

from ovn-kubernetes.

Not relevant anymore.

from ovn-kubernetes.