Can not write or read data to SASL Kafka about envoy HOT 6 OPEN

• kafka-topics.sh can work well for example to list topics via envoy address.
• consumer client can not work (even consumer group can not found)
• write data also can not work

It would great to provide the log or the output of client, then that will help people to triage the issue, thanks!

from envoy.

cc @mattklein123 @adamkotwasinski

from envoy.

1. is this filter support Auth-kafka ?
2. is this a bug?
3. I notice: the kafka proxy is experimental and is currently under active development. , do we have plan to release this version?
4. any version now for envoy include stable kafka-proxy ?

1. The broker filter should be relatively transparent - all it does is decrypting payloads and updating metrics, in the most typical configuration it sends the data futher as-is, so even if the filter has bugs, you shouldn't be noticing it (unless the payloads cannot be parsed).
2. Need more info.
3. No concrete plans and right now there is not too much to talk about unless we identify proper issue and then we can decide whether it's 'working as intended' vs 'bug'.
4. Not feature-wise.

Are you using SASL_PLAINTEXT or SASL_SSL ?
If SASL_SSL then it won't work -> Kafka broker does not decrypt data == is not capable of processing your requests.

If that's not the case, please try to eliminate envoy.filters.network.kafka_broker from your test scenario - does it work then? If it's really caused by broker filter then check the logs for failed processing (run Envoy with debug logging) and check the request.unknown / request.failure metrics.

from envoy.

thanks team's feedback!

1. actually no client side log, but I keep watch the consumer group, always no active group, if works , the group should be rebalancing
2. we are using SASL_PLAINTEXT. will try to remove kafka_broker and then give you feedback~

from envoy.

hi adamkotwasinski

removed kafkaBroker, work now,

so one more question about this:

1. what's the best practice for kafkaBroker?
2. the kafkaBroker feature only for record some metrics such as how many datas we received?

thanks!

from envoy.

Hey @rainbowwiki I just realized you are running Kafka 2.8.1 & Envoy 1.18.2.
Envoy 1.18.2 supports protocol up to Kafka 2.4 - so you need to upgrade your Envoy instance to something newer.
Compare: 1.29 can understand Kafka up to 3.5.

1. what's the best practice for kafkaBroker?

Best way of debugging any issues is in order: check metrics (especially unknown ones to mitigate protocol incompatibility), then if everything looks normal removing of filter and/or running it in debug mode.

2. the kafkaBroker feature only for record some metrics such as how many datas we received?

Right now the filter doesn't track records/bytes, but only requests. PRODUCE request carries the records that are written by producers, but remember that one produce request can carry multiple records for many partitions. FETCH requests correspond to read (.poll) operations, but they can carry no return data (if there was nothing to read) so it's harder to monitor now.

I might work on that in the future though!

from envoy.