Comments (19)
We probably don't even log a warning when someone uses "Type: LoadBalancer". I will send in a patch to do proper logging.
from ovn-kubernetes.
Ouch, I just spend a couple of hours trying to debug my broken services that I had setup for ingress, until I just found this issue. Is there any plan to fix this soon or is this a larger issue. What is the recommended way to setup ingress with OVN-Kubernetes?
from ovn-kubernetes.
So, I know that ingress works with ovn-kubernetes because the guys at openshift and vmware have previously confirmed it.
I have never set it up. So, I don't really understand why service type load-balancer is important for ingress to work. My understanding is that
- Your ingress POD will run with host network.
- Packet arrives to that host IP + port
- Ingress POD will NAT it to a Service IP.
- There is a route in host which says that the service IP is reachable from a OVS interface (the one we setup on each node)
- Linux pushes the packet to that interface
- From here, it should just work like a regular service IP that OVN supports.
@dcbw can add more color or correct me.
from ovn-kubernetes.
@shettyg There are several ways to setup ingress. With a service of type load balancer Kubernetes will automatically deply an AWS Classic Load Balancer that forwards traffic to a service port. On the service port you could then have an ingress controller listening, such as traefik. Once that is setup, you can create Ingress
resources in Kubernetes that will cause the ingress controller to route traffic to the respective application services.
Another way that I tried was to use the aws-alb-ingress-controller, it also registers a classic load balancer but it will forward traffic to a service of type node port and automatically updates the load balancers target groups if nodes change.
I have not yet attempted to run a pod with host network enabled, that would be essentially the same as a service of type node port, but without the indirection that services provide. So I am essentially tying my ingress to a specific node (which could be a problem if that node dies or the pod gets scheduled to a different node). Please, correct me if that statement is wrong.
from ovn-kubernetes.
There are several ways to setup ingress. With a service of type load balancer Kubernetes will automatically deply an AWS Classic Load Balancer that forwards traffic to a service port. On the service port you could then have an ingress controller listening, such as traefik. Once that is setup, you can create
Ingress
resources in Kubernetes that will cause the ingress controller to route traffic to the respective application services.
Another way that I tried was to use the aws-alb-ingress-controller, it also registers a classic load balancer but it will forward traffic to a service of type node port and automatically updates the load balancers target groups if nodes change.
I still do not understand how it works. What is a service port? An example? Is it an IP:port? Is traefik running in a pod or host? So the first step is for traffic to reach traefik. I would like to understand that first. And wondering, how is it that OVN prevents it.
I have not yet attempted to run a pod with host network enabled, that would be essentially the same as a service of type node port, but without the indirection that services provide. So I am essentially tying my ingress to a specific node (which could be a problem if that node dies or the pod gets scheduled to a different node). Please, correct me if that statement is wrong.
I think, there is usually 2 pods running in 2 nodes (with specific node selector added when deploying pod). Not sure how the HA is enforced.
from ovn-kubernetes.
@shettyg My understanding is that a LoadBalancer
service is basically just a service of type NodePort
(e.g its an IP:port opened on the host), with the additional benefit that a load balancer is created through the cloud provider API and the load balancer configuration is wired to target traffic to the correct nodes. Since NodePort services appear to be supported I am not sure why LoadBalancer
services are not.
Traefik is running in a pod. I configure it to expose its public listening port as a service of type load balancer. Kubernetes will then provision an AWS LoadBalancer that targets the node port of the traefik service.
The flow of traffic is then like this:
+-> Node with Traefik pod +---> Traefik Pod +---> Service Pod
| ^
| |
Inbound Traffic +-> AWS Load Balancer ++ |
| |
| +
+-> Node without Traefik pod
from ovn-kubernetes.
Here is some more example, if I deploy my traefik ingress controller then I get the following service (kubectl get svc --all-namespaces
):
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-system traefik-external LoadBalancer 10.96.0.204 af00174b939a011e9ba19022b2e3974d-2088150912.eu-central-1.elb.amazonaws.com 80:30466/TCP,443:32733/TCP,8080:32268/TCP 5m34s
[...]
Next I check on the Linux node if ovs knows about the node ports that the service opens (e.g. withovs-ofctl dump-flows brens5
), I am expecting to see ports 30466 and 32268 to appear in the output but they are not listed:
cookie=0x0, duration=1099559.417s, table=0, n_packets=9154376, n_bytes=1185501598, priority=100,ip,in_port="k8s-patch-brens" actions=ct(commit,zone=64000),output:ens5
cookie=0x0, duration=1099559.414s, table=0, n_packets=14101419, n_bytes=6936795811, priority=50,ip,in_port=ens5 actions=ct(table=1,zone=64000)
cookie=0x0, duration=1099559.362s, table=0, n_packets=14, n_bytes=859, priority=100,tcp,in_port=ens5,tp_dst=32114 actions=output:"k8s-patch-brens"
cookie=0x0, duration=1099559.359s, table=0, n_packets=2217452, n_bytes=168722795, priority=100,tcp,in_port=ens5,tp_dst=32306 actions=output:"k8s-patch-brens"
cookie=0x0, duration=1099561.029s, table=0, n_packets=5756529, n_bytes=659258214, priority=0 actions=NORMAL
cookie=0x0, duration=1099559.411s, table=1, n_packets=8400761, n_bytes=3935118050, priority=100,ct_state=+est+trk actions=output:"k8s-patch-brens"
cookie=0x0, duration=1099559.405s, table=1, n_packets=0, n_bytes=0, priority=100,ct_state=+rel+trk actions=output:"k8s-patch-brens"
cookie=0x0, duration=1099559.401s, table=1, n_packets=5701146, n_bytes=3001704113, priority=0 actions=LOCAL
from ovn-kubernetes.
@shettyg
I am not too familiar with the OVN code base, but while searching through the code, I found there are several place where logic is guarded explicitly against kapi.ServiceTypeNodePort
, such as:
https://github.com/openvswitch/ovn-kubernetes/blob/master/go-controller/pkg/ovn/service.go#L58
https://github.com/openvswitch/ovn-kubernetes/blob/master/go-controller/pkg/cluster/gateway_shared_intf.go#L18
https://github.com/openvswitch/ovn-kubernetes/blob/master/go-controller/pkg/ovn/endpoints.go#L58
https://github.com/openvswitch/ovn-kubernetes/blob/master/go-controller/pkg/ovn/endpoints.go#L90
Given that service type LoadBalancer is essentially just a service type NodePort, with some additional extra bits, I am wondering if it would just be enough to extend the logic to include kapi.ServiceTypeLoadBalancer
in those guards.
from ovn-kubernetes.
Since NodePort services appear to be supported I am not sure why
LoadBalancer
services are not.
That is because, I simply did not understand what "LoadBalancer" service is - in spite of trying to read about it multiple times.
Given that service type LoadBalancer is essentially just a service type NodePort, with some additional extra bits, I am wondering if it would just be enough to extend the logic to include
kapi.ServiceTypeLoadBalancer
in those guards.
With service type nodeport, we add an LB entry in OVN database for every OVN gateway (which could mean every node). Looks like for ServiceTypeLoadBalancer, we should only do it for specific nodes?
from ovn-kubernetes.
Reading, your comment, it looks like I misunderstood. Looks like what you are saying is that, it is the same as NodePort. You can simply expand the checks in code for ServiceTypeNodePort and see whether it solves your issue. If it does, we can look at adding it to code officially.
from ovn-kubernetes.
@lanoxx I think we already handle what you are looking for. We handle ExternalIPs for a service in handleExternalIPs(). This method adds a LoadBalancer entry (mapping External IP to NodePort IPs) on the first L3 gateway router (the one with IP address 100.64.1.2).
from ovn-kubernetes.
Here is some more example, if I deploy my traefik ingress controller then I get the following service (
kubectl get svc --all-namespaces
):NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-system traefik-external LoadBalancer 10.96.0.204 af00174b939a011e9ba19022b2e3974d-2088150912.eu-central-1.elb.amazonaws.com 80:30466/TCP,443:32733/TCP,8080:32268/TCP 5m34s [...]
Next I check on the Linux node if ovs knows about the node ports that the service opens (e.g. with
ovs-ofctl dump-flows brens5
), I am expecting to see ports 30466 and 32268 to appear in the output but they are not listed:
Does running following command on your setup return anything?
ovn-nbctl find logical_router options:lb_force_snat_ip=100.64.1.2
If it does return an entry, then on that L3 gateway router we should have created a load balance entry mapping ExternalIP to NodePort IPs.
Also, AFAIK OVS doesn't support adding DNS names for physical IP. In your case External-IP is af00174b939a011e9ba19022b2e3974d-2088150912.eu-central-1.elb.amazonaws.com.
from ovn-kubernetes.
You can simply expand the checks in code for ServiceTypeNodePort and see whether it solves your issue. If it does, we can look at adding it to code officially.
This did work to some degree and partially solved my problem. I searched through the code for all occurences of ServiceTypeNodePort
and expanded that with a logical OR
for ServiceTypeLoadBalancer
. Now I have NodePort
s exposed for services for type LoadBalancer
and I can access them on my Linux minion. The NodePorts also appear on the Windows machine when I run ovs-ofctl dump-flows "vEthernet (Ethernet 2)"
, but connecting to the ports only gives me timeouts.
@girishmg I don't quite understand the mechanism you are describing, but I think we are talking about two different things.
Kubernetes supports a special service type called LoadBalancer
that works in connection with the cloud provider option of Kubernetes. What it does is to invoke the cloud provider (in my case AWS) and create a native load balancer in the cloud. For AWS a classic load balancer is created. The DNS name for that load balancer is assigned by AWS automatically and Kubernetes only picks up DNS name and displays it in the output of kubectl get svc
or kubectl get ingress
.
The mechanism you are suggesting seems to be about something that OVN supports by itself. Where can I read more about that?
from ovn-kubernetes.
This did work to some degree and partially solved my problem. I searched through the code for all occurences of ServiceTypeNodePort and expanded that with a logical OR for ServiceTypeLoadBalancer. Now I have NodePorts exposed for services for type LoadBalancer and I can access them on my Linux minion. The NodePorts also appear on the Windows machine when I run ovs-ofctl dump-flows "vEthernet (Ethernet 2)", but connecting to the ports only gives me timeouts.
Can you access the pod from linux minion's nodeport?
from ovn-kubernetes.
Yes. The pod that I am trying to access is a traeffic-ingress-controller
. It runs on the Linux minion and I can access it via the Linux minion's NodePort.
Here is an example. The LoadBalancer service exposes a traefik reverse proxy which runs on port 80
via NodePort 30466
:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-system traefik-external LoadBalancer 10.96.0.204 af00174b939a011e9ba19022b2e3974d-2088150912.eu-central-1.elb.amazonaws.com 80:30466/TCP,443:32733/TCP,8080:32268/TCP 10d
The traefik reverse proxy forwards requests to the Kubernetes dashboard which runs on the Linux Minion (e.g. 172.33.69.225):
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
kube-system kubernetes-dashboard-6948557947-sfwdv 1/1 Running 1 22d 10.245.0.6 ip-172-33-69-225.eu-central-1.compute.internal <none>
When I access the NodePort of the Linux Minion (e.g. 172.33.69.225:30466
) from my master node (e.g. 172.33.75.8
), then I can get a response from the dashboard:
$ curl http://172.33.69.225:30466
<!doctype html> <html ng-app="kubernetesDashboard">
[...]
</html>
If I try the same from the Windows node (e.g. 172.33.66.23:30466
), I get a timeout.
from ovn-kubernetes.
Is this a problem just for LoadBalancer or does regular NodePort not work from Windows either?
from ovn-kubernetes.
I have a NodePort
service exposed on the Windows node which also does not work. It is visible in the output of ovs-ofctl dump-flows
but it does not return an traffic. Also see issue: #611
from ovn-kubernetes.
I think this was fixed by commit: 9b9d146
from ovn-kubernetes.
AWS-style LoadBalancers were fixed in #738. GCP-style LoadBalancers were fixed in #1126.
from ovn-kubernetes.
Related Issues (20)
- node deletion results stale lsps and IP leaking on layer2/localnet networks HOT 1
- UT Flake: `handles a HO node is switched to a OVN node` is flaking HOT 3
- Flake e2e: ACL Logging for NetworkPolicy when the namespace's ACL logging annotation is updated
- Load Balancer Service Tests with MetalLB [It] Should ensure load balancer service works with 0 node ports when ETP=local
- Cleanup Hardware Offload docs
- Cleanup DPU Support/Acceleration docs
- Cleanup Kubevirt Live Migration docs HOT 2
- Cleanup MultiNetworking Docs HOT 2
- Cleanup DNS name resolver docs HOT 1
- Add proper docs for observability, grafana dashboards, metrics
- Fix the PR labeler action
- FLAKE: External Gateway With Admin Policy Based External Route CRs e2e multiple external gateway validation Should validate ICMP connectivity to multiple external gateways for an ECMP scenario IPV4 HOT 2
- Flake: should work on secondary node interfaces for ETP=local and ETP=cluster when backend pods are also served by EgressIP HOT 2
- ovn-kube-f and ovn-kube-u image renaming was incomplete HOT 1
- Support EgressIP for user defined networks
- Flake: [FAIL] External Gateway With Admin Policy Based External Route CRs e2e non-vxlan external gateway through a gateway pod Should validate ICMP connectivity to an external gateway's loopback address via a gateway pod [It] ipv4 HOT 3
- [FAIL] e2e egress firewall policy validation with external containers [It] Should validate the egress firewall policy functionality for allowed IP HOT 7
- Upgrades tests operation cancelled
- e2e EgressQoS validation -- account for single stack cluster
- Transit switch subnet overlap check is missing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ovn-kubernetes.