helm chart: upgrade with the same value (ipFamilyPolicy=RequireDualStack) not possible about ingress-nginx HOT 13 CLOSED

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

from ingress-nginx.

• The project does not test upgrade dualstack on AWS so e could use more data
• The message in error is about releasing and recreating

Service "ingress-nginx-controller" is invalid: spec.ipFamilyPolicy: Invalid value: "RequireDualStack": must be 'SingleStack' to release the secondary IP family

This hints that a very clear targeted test is needed to be done by a developer to triage this

from ingress-nginx.

/remove-kind bug
/triage needs-information
/kind feature

cc @Gacko

from ingress-nginx.

That's not AWS, that's Oracle Cloud. I cannot test this, but I rather feel like that's an error produced by the according load balancer / cloud controller, so nothing caused by the chart or us.

from ingress-nginx.

Yup, that error is coming from Kubernetes itself: https://github.com/kubernetes/kubernetes/blob/dc3f5ec6ccb9855dfa99f4c1078625df5fdfab6a/pkg/registry/core/service/storage/alloc.go#L184

from ingress-nginx.

To make things clear, I do not use oracle load balancers but metallb ^^

from ingress-nginx.

According to the code, this error message happens when you're handing in an ipFamilyPolicy other than SingleStack (that's the case) and the updated service has fewer clusterIPs and/or ipFamilies than the old service. Can you try omitting the ipFamilyPolicy from your values? As the service already exists, I don't think the assigned IPs are getting dropped. Anyway, please do so in a testing environment.

from ingress-nginx.

What's the deal then if the error comes from Kubernetes? Shouldn't the helm template render exactly the same values for the yaml files that get send to the kubernetes api? In this case, kubernetes shouldn't do anything but rather reply with "ok", desired state already reached.

To verify, I dumped the current service as a yaml file and tried to push it straight as it is again with "kubectl apply -f":

kubectl -n ingress-nginx apply -f svc.yaml
Error from server (Conflict): error when applying patch:
{"metadata":{"resourceVersion":"10892"}}
to:
Resource: "/v1, Resource=services", GroupVersionKind: "/v1, Kind=Service"
Name: "ingress-nginx-controller", Namespace: "ingress-nginx"
for: "svc.yaml": error when patching "svc.yaml": Operation cannot be fulfilled on services "ingress-nginx-controller": the object has been modified; please apply your changes to the latest version and try again

Does the nginx helm-chart admission controller permanently update the resource? Because I needed maybe 5 seconds between getting the yaml output and sending it again.

Or is it a missing kubernetes feature/bug? If yes, I can close this issue and raise one on kubernetes side :)

from ingress-nginx.

The Service resource is not checked or manipulated by the Ingress NGINX chart. We do not deploy any webhooks for Service resources. Also you probably cannot kubectl apply those resources if they have been created by doing so. Again, that's nothing specific to Ingress NGINX, that's a Kubernetes in general topic.

from ingress-nginx.

Thanks! I will try to ask why this behaviour is desired on the kubernetes side :)

from ingress-nginx.

@Gacko I will reopen this issue, as I am pretty sure, that the error message is misleading, because I ran helm with --dry-run=server to see the actual patch templates:

# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.10.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: "1.10.0"
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  ipFamilyPolicy: RequireDualStack
  ipFamilies: 
    - IPv4
  ports:
  (...)

The problem might be, that the ipFamiles does not contain IPv6, therefore kuberentes detechts a change and runs into an issue that it does not allow RequireDualStack in combination with only one ipFamily set.
I am not 100% sure, but I also mentioned this on the kubernetes ticket itself: kubernetes/kubernetes#123761
And I am unsure, who's behaviour here is not working as expected. Might even be helm itself 🤔

from ingress-nginx.

The check in the code compares the old ipFamilies to the new ones and complains if you're reducing them, so from IPv4 & IPv6 to only IPv4 e.g., while still using RequireDualStack.

The Ingress NGINX chart has a property controller.service.ipFamilies which defaults to just IPv4. So if you change controller.service.ipFamilyPolicy while not changing controller.service.ipFamilies, you might run into that issue.

Unfortunately Kubernetes is maybe not complaining about that on creation and just adds IPv6 after creation to comply to your ipFamilyPolicy, but without changing ipFamilies to a matching value, it's quite clear why it is no longer working on updates.

Sorry that I didn't figure that earlier! I guess you're expecting to just set controller.service.ipFamilyPolicy and not care about controller.service.ipFamilies, right? I'm not sure if we can really have a good default behavior here as other users might eventually want to default to ipFamilyPolicy SingleStack with ipFamilies IPv6.

Of course setting ipFamilyPolicy to RequireDualStack might make one think "yeah, just set IPv4 and IPv6 in ipFamilies then, I don't wanna care", but actually order matters here. Kubernetes is acting different depending on what's mentioned first in the ipFamilies.

So in the end I don't think there is a one-fits-all default behavior we can implement and one always needs to configure both, ipFamilyPolicy and ipFamilies.

cc @strongjz @rikatz in case you are more into the whole "Kubernetes on Dual Stack" topic. 🙂

from ingress-nginx.

Just tested it. Indeed, setting the ipFamilies resolves my problem. I guess there is no need to catch this edge case, since it was a user error. But at least it is now documented here in an issue so others can find it 😄 Thanks a lot for your help!

from ingress-nginx.