Allow for more control over consumer group used about kafka-webview HOT 6 OPEN

@Crim We are using SASL for authentication and unfortunately the Kafka cluster is operated by another team, so I'm making some assumptions here but I believe they have used this form of specifying the --group consumer group name included in the ACL together with the principal/username for specific topics: https://docs.confluent.io/current/kafka/authorization.html#adding-or-removing-a-principal-as-producer-or-consumer

bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 \
 --add --allow-principal User:Bob \
 --consumer --topic test-topic --group Group-1

Using the example above, I would like to configure Group-1 in Kafka Webview at a cluster level as the default consumer group to use for all Topics or if possible, specify a consumer group at a View level as well, which would override the cluster level consumer group for that view.

from kafka-webview.

I need to do some testing to see if we were to setup static consumer group ids for a view (or cluster), will Kafka-WebView run into issues when multiple users attempt to consume using the same view by only receiving a portion of the data back from Kafka. I feel like how we're currently explicitly subscribing to specific partitions during the consume, this wouldn't be a problem.

Along the same lines, I need to verify how consumer state gets stored. If it's tied to the consumer group, or consumer id specifically.

from kafka-webview.

So I've put some more thought into this and pretty sure what you've outlined makes perfect sense.

I like the idea of:

• Optionally set a default consumer group id at the cluster level.
• At the view level you can then optionally override it with something more specific.
• If no consumer group ids have been explicitly set at any level, fall back to using a generated consumer group id (similar to what is done today)

I believe doing the above, and still using consumer id's tied to the current user in Kafka-WebView allows things to work how one would expect in regards to consuming all data from a topic even if there are simultaneous subscribes/consumers in the same consumer group. I've started putting together a branch that incorporates these changes. 👍

from kafka-webview.

Ah that's interesting. So in your ideal scenario you would be able to define a fixed consumer group id per view?

I think the tricky part is figuring how that works when you have multiple users in kafka-webview all consuming the same topic using the same consumer group id. I feel like the consumers stored state would over write each other no?

I'll do some investigating tomorrow.

from kafka-webview.

@donovanmuller Which Authorizer are you using in your cluster? https://docs.confluent.io/current/kafka/authorization.html seems to indicate that the username for the ACL is generated from the client's certificate when using SSL. For SASL it's based on the supplied username.

I haven't found documentation that shows examples using the consumer group id as the username

from kafka-webview.

Awesome. Let me know if/when you'd like me to test on my side.

from kafka-webview.