[bitnami/kafka] Can't configure internal & external access about charts HOT 5 CLOSED

Hi @Skull0ne,

Let me better understand your issue.

The Kafka chart provides 'two' different ways to configure external access:

• Using autoDiscovery.
• Manually providing the LoadBalancer IP/Port or nodePort port and IP/domain.

First of all, I would like to highlight one important detail, when using external access, each Kafka node needs to be individually addressed. So when external access is configured, it can not be domain:9094 for all nodes.

It may have been either a different port of each node (domain:9094 / domain:9095 / domain:9096) or a different domain for each or an individual domain for each (node-1.domain:9094 / node-2.domain:9094 / node-3.domain:9094).

How NodePort external access with autoDiscovery works, a NodePort service will be created for each Kafka node using externalAccess.controller.service.nodePorts[], in your case node-1 will use NodePort 30000, node-2 will use 30001,...

Then, during the pod initialization, it will execute curl -s https://ipinfo.io/ip unless externalAccess.service.domain or externalAccess.service.useHostIPs were provided.

You can find this and more information in the Kafka README: https://github.com/bitnami/charts/blob/main/bitnami/kafka/README.md#accessing-kafka-brokers-from-outside-the-cluster

For example:

#values.yaml
rbac:
  create: true
serviceAccount:
  create: true
controller:
  automountServiceAccountToken: true
externalAccess:
  enabled: true
  autoDiscovery:
    enabled: true
  controller:
    service:
      type: NodePort
      domain: "my-domain.com"
      nodePorts:
        - 30000
        - 30001
        - 30002

# Advertised listeners on each node:
kafka-controller-2   1/1     Running   0          3m22s
$ kubectl exec -it kafka-controller-0 cat /opt/bitnami/kafka/config/server.properties | grep advertised
advertised.listeners=CLIENT://kafka-controller-0.kafka-controller-headless.default.svc.cluster.local:9092,INTERNAL://kafka-controller-0.kafka-controller-headless.default.svc.cluster.local:9094,EXTERNAL://my-domain.com:30000
$ kubectl exec -it kafka-controller-1 cat /opt/bitnami/kafka/config/server.properties | grep advertised
advertised.listeners=CLIENT://kafka-controller-1.kafka-controller-headless.default.svc.cluster.local:9092,INTERNAL://kafka-controller-1.kafka-controller-headless.default.svc.cluster.local:9094,EXTERNAL://my-domain.com:30001
$ kubectl exec -it kafka-controller-2 cat /opt/bitnami/kafka/config/server.properties | grep advertised
advertised.listeners=CLIENT://kafka-controller-2.kafka-controller-headless.default.svc.cluster.local:9092,INTERNAL://kafka-controller-2.kafka-controller-headless.default.svc.cluster.local:9094,EXTERNAL://my-domain.com:30002

As you can see, each node was configured to expose at EXTERNAL listener its kafka-external service.

from charts.

Hi @migruiz4 and thanks for your answer.

First of all, I would like to highlight one important detail, when using external access, each Kafka node needs to be individually addressed. So when external access is configured, it can not be domain:9094 for all nodes.

I wasn't aware of this.

It may have been either a different port of each node (domain:9094 / domain:9095 / domain:9096) or a different domain for each or an individual domain for each (node-1.domain:9094 / node-2.domain:9094 / node-3.domain:9094).

Due to my LB setup the best option would be 3 domains :

• 3 pools each one targeting a different port
• create a rule to match each domain and to forward it to the right pool
• use the ClusterIP setup and expose each port with ingress (https://github.com/bitnami/charts/blob/main/bitnami/kafka/README.md#using-clusterip-services)

Does it make sense to you?

My issue with the NodePort setup is that I can't know which host will have run a kafka, so I have to add each node with each port and it generates a lot of noise as most of them don't expose the ports.

Regards

from charts.

I'm sorry but I'm not sure I understand what you meant here:

Due to my LB setup the best option would be 3 domains :

• 3 pools each one targeting a different port
• create a rule to match each domain and to forward it to the right pool
• use the ClusterIP setup and expose each port with ingress (https://github.com/bitnami/charts/blob/main/bitnami/kafka/README.md#using-clusterip-services)

What is important to understand is that, from the external client perspective, each node needs to be individually addressed.

Of course, meaning that each endpoint has a pool of 1 node, the node that has that address as advertised listener.

To do so, the bitnami/kafka chart provides several methods, and which method you should use depends on your use case, I'm sorry I can not help you on that matter.

I'm not sure if your question is regarding which method would result in which configuration, so I hope this helps:

By port (NodePort + domain):

• example.com:9092, example.com:9093, example.com:9094
  By domain (Cluster IP + Ingress or LoadBalancer using LoadBalancerNames):
• example-1.com:9092,example-1.com:9092, example-2.com:9092
• example.com/node-1:9092,example.com/node-2:9092, example.com/node-3:9092
  By IP (Load Balancer using loadBalancerIPs)
• 10.0.0.1:9092,10.0.0.2:9092, 10.0.0.3:9092

I may have missed some configuration, but I hope it helps you. Reached this point, deciding how to expose your service externally is a matter of kubernetes knowledge more than chart usage, we on our side try the chart to cover as many use cases as possible.

from charts.

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

from charts.

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

from charts.