sighupio / fury-kubernetes-ingress Goto Github PK

Hi everyone 👋 !
I am encountering problems trying to deploying multiple ingress in same cluster. This is mainly because there are several resources collapsed in one file on side (for example in rbac.yml there are both namespaced resources and cluster wide resources), and on the other side, because afaik there is no way to remove a resource with kustomize.
I tried a workaround trying to change resource name with a patchesJson6902 but was unsuccesful.
The best thing for this purpose can be to separate each resource in different file, this should allow to inherit them separately (in case you need) and avoid duplication of cluster-wide resources.
Let me know what you think!

Pasting and internal report by @beratio :
We experienced this problem on an EKS cluster v1.21 with Ingress Module v.1.11.0 : when Nginx is used in single mode, ingresses without annotation kubernetes.io/ingress.class: "nginx" are ignored by the controller. So, after a cluster upgrade, services behind those ingresses became unavailable.

Module has IngressClass object configured as default-backend and in theory it should cover ingresses without class specification, but it doesn't.

Searching for it, I found other people had same problem on different Kubernetes cluster versions/providers. It's necessary controlling and updating all ingress resources that lack the notation before upgrading the module.

Since this bug leads to an unexpected disservice, I suggest adding a warning in release docs. So whoever upgrades this module on a cluster using single nginx can be aware of the issue, can check ingress resources and fix them before the upgrade.

The JSONPatch specified in cert-manager README is using the outdated certmanager.k8s.io/v1alpha1 apiVersion and it fails to apply if used.

Hi!
while we were installing the module version 1.13.1 we had the following problem:

DaemonSet.apps "nginx-ingress-controller-internal" is invalid: spec.updateStrategy.rollingUpdate.maxUnavailable: Invalid value: intstr.IntOrString{Type:0, IntVal:0, StrVal:""}: cannot be 0

What we did was to add the following jsonPatch:

- op: replace
  path: /spec/updateStrategy/rollingUpdate/maxUnavailable
  value: 1

Kubernetes version : EKS 1.22

• Update NGINX Ingress Controller #85
• Update cert-manager #86
• Update forecastle #84
• #73
• #74
• #87

We could leverage kustomize's nameSuffix to avoid duplicating ingress-nginx deployment manifests.

We should split dual-nginx in nginx-internal and nginx-external while inheriting from ingress-nginx.

Due to the labels and selector we are using, the ServiceMonitor included with the module is picking up other pods that should not be scrapped for metrics:

Pods "CA Injector" and "Webhook" should not be scrapped for metrics.

Current e2e tests are a little flaky, we need to improve them for example testing that cert-manager creates a CA and certificates.

In order for the cert-manager webhook to work out-of-the-box on GKE cluster we should change the default secure-port from 6443 to 10250. This way we can use an unprivileged port without having to add a specific firewall rule.

Refs:

• https://cert-manager.io/docs/installation/compatibility/#gke
• https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules

• NGINX Ingress Controller #85
• cert-manager #86
• Forecastle #83

Testing the ugprade to ingress v1.13.0 we found that cert-manager-cainjector logs the following error:

leaderelection.go:330] error retrieving resource lock cert-manager/cert-manager-cainjector-leader-election: configmaps "cert-manager-cainjector-leader-election" is forbidden: User "system:serviceaccount:cert-manager:cert-manager-cainjector" cannot get resource "configmaps" in API group "" in the namespace "cert-manager"

indicating that there's some permission missing, most probably we'll need to add the following snippet to the cert-manager-cainjector:leaderelection role:

  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]  # get seems to be enough

We need to check if we missed it somehow or if it is missing also in upstream manifests.

Add E2E for K8s v1.24.0 in CI

Hi guys,
have you ever thought to let nginx directly log json instead of logging string and then parsing it with fluent?
You can do it with something like (names are just examples)

log_format json_combined escape=json
  '{'
    '"time_local":"$time_local",'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"request":"$request",'
    '"status": "$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"http_referrer":"$http_referer",'
    '"http_user_agent":"$http_user_agent"'
  '}';

This can simplify the fluentd/fluentbit log flow, and opens a lot of possibilities (i.e. parsing logs from command line)

Hi, thanks to this cert-manager/cert-manager#3828, very soon will be possible to pass annotations/labels to the secret generated by the Certificate resource, allowing https://github.com/kubeops/kubed to synchronize the secret along with the namespaces that might be used.

@lzecca78 spotted and issue while deploying the latest version of this module (cert-manager).

Seems like the serviceMonitor is trying to reach the cert-manager using the incorrect labels.

Current:

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: cert-manager
    app.kubernetes.io/component: cert-manager
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: cert-manager
    app.kubernetes.io/part-of: fury-kubernetes-ingress
    app.kubernetes.io/version: 1.1.0
  name: cert-manager
  namespace: cert-manager
spec:
  endpoints:
  - interval: 30s
    port: metrics
  jobLabel: app
  namespaceSelector:
    matchNames:
    - cert-manager
  selector:
    matchLabels:
      app: cert-manager
      app.kubernetes.io/version: 1.0.1
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: cert-manager
    app.kubernetes.io/component: cert-manager
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: cert-manager
    app.kubernetes.io/part-of: fury-kubernetes-ingress
    app.kubernetes.io/version: 1.1.0
  name: cert-manager
  namespace: cert-manager
spec:
  ports:
  - name: metrics
    port: 9402
    targetPort: metrics
  selector:
    app: cert-manager
    app.kubernetes.io/component: cert-manager
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: cert-manager
    app.kubernetes.io/part-of: fury-kubernetes-ingress

Service has the label app.kubernetes.io/version: 1.1.0 while ServiceMonitor points to: app.kubernetes.io/version: 1.0.1

It would be nice to have the external-dns project as a package of Ingress Module to provide better automatic management of external DNS providers for the created ingresses

Switch alerts annotations.message to annotations.description in order to make it work with the default Alertmanager template.

The current implementation of the nginx ingress controller we ship listens and binds to port 80 in the container, which can be a issue in clusters with podSecuirityPolicies deployed, we should move to an higher port number not requiring any privilege in order to avoid this kind of issues.

Hi team, i found a problem when using the ingressClass field instead of the annotation in an ingress object when using dual nginx.

What happens is that since it's not defined the name of the controller via --controller-class, each one of the controller picks up the ingress and serves it. See https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/

I'm opening a PR to address this and completely remove annotations in favor of ingressClass field.

in this new release every ClusterRoleBinding is missing the subject namespace and so we receive this error:
The ClusterRoleBinding "cert-manager-controller-certificates" is invalid: subjects[0].namespace: Required value

by simply adding the components namespace: cert-manager under the subject field the problem goes away and its possible to apply.
working example:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-manager-controller-challenges
  labels: {}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cert-manager-controller-challenges
subjects:
- kind: ServiceAccount
  name: cert-manager
  namespace: cert-manager

if its a confirmed bug i will open a PR later with the fix, just let me know.

Hi I've used fury-ingress to deploy cert-manager on IKS (IBM Kubernetes Service) a few notes:

1. The link to documentation (in your readme) is pointing to latest while you're using cert-manager 0.5. This created confusion for me when setting up the thing.
2. This is more of an hint. IKS's Ingress doesn't allow multiple ingress resources for a single hostname. In this case using the certmanager.k8s.io/acme-http01-edi-in-place annotation is the solution. Also ingress.bluemix.net/redirect-to-https must be false

The latest versions of cert-manager, include an ACME solver image reference within the cert-manager controller's arguments. This is evident in the deployment configuration:

containers:
  - name: cert-manager-controller
    image: "quay.io/jetstack/cert-manager-controller:v1.14.2"
    imagePullPolicy: IfNotPresent
    args:
    - --v=2
    - --cluster-resource-namespace=$(POD_NAMESPACE)
    - --leader-election-namespace=kube-system
    - --acme-http01-solver-image=quay.io/jetstack/cert-manager-acmesolver:v1.14.2
    - --max-concurrent-challenges=60

The --acme-http01-solver-image argument in the cert-manager controller points to an image on Quay, which may cause issues in environments with restricted external access, particularly during HTTP-01 challenges. It's essential to update this to use an image hosted in the SIGHUP container registry, aligning with distribution guidelines that advocate for a single, unified image registry.

Since both nginx-ovh and nginx-gke are just simple patches of simple nginx, we will handle both of them as patches when needed.