Hey,

I have a feature request/suggestion. To get an overview of topics it would be nice to know how much messages to expect (e.g. per minute).
Of course the number could vary over the span of e.g. a day, but already an estimate for the magnitude of messages (thoundands per minutes vs less than one per minute) would also be a helpful information.

Cheers :)

In order to get more details about kafka partitions (such as their size) Kafka offers an admin request called "DescribeLogDirs". Sarama does not implement this request/response, but as it offers very valuable information we should add that request/response to Sarama.

The original Kafka documentation does not document the protocol but refers to the implementation:

Response package: https://github.com/apache/kafka/blob/05ba5aa00847b18b74369a821e972bbba9f155eb/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsResponse.java

Request package:
https://github.com/apache/kafka/blob/05ba5aa00847b18b74369a821e972bbba9f155eb/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsRequest.java

• Should be able to sort ConsumeGroups by "Lag (Sum)" column
• In ConsumerGroup detail view, add explanation to what the numbers mean behind the partitions
• In the list, show "Total Groups" more separate from the "count by state" entries (maybe group them in a box or something)
• Place quick-search in the name/ID header of each table as it consumes too much vertical space
• Add small popup to the refresh button to make it immediately clear

The copy button sucks, look at it:

Issues with it

• You don't really know what you're copying...
  There should be multiple options:

  • Key
  • Value (as nicely formatted JSON)
  • Value (raw data)
  • Timestamp (in UTC seconds)

• It takes up too much space!
  A whole column (even if its small) makes it more confusing to look at.

  • Maybe a context (rightclick) menu? Nahhh, that'd be the only use of a ctx menu, thus confusing UX
  • Maybe a button that only appears when the cursor is inside the row? Would make it visually less cluttered, but we'd still have to reserve the horizontal space for where the button will appear. (unless we want to re-layout, which is insanely bad lol)
  • Maybe make the copy button itself smaller (like the "expand message '+' icon"), and then put it inline as well (but where? into the 'value' column, together with the expand plus??)

I prefer dark theme. Could you please add it?

We'd like to be able to view all consumers of a topic in the corresponding topic detail view.

Similar to issue #35, viewing consumer-groups also does not work as intended when not all kafka brokers are available (for example when they are being restarted one-by-one by a rolling update)

• Backend should degrade gracefully (show as much info as possible, instead of returning a status:500)
• Frontend should be able to handle .lag == null and show an icon in place of the missing information
• HTTP error messages should be shown in the frontend. Currently the 'message' of statusCode: 500, message: "Could not describe consumer groups in the Kafka cluster" is not shown at all.

As mentioned in #8 :

• Avro data has to be transformed into Json in the backend.

• When sending messages to the frontend, the original format of the message must be sent along as a simple description. Something like sourceFormat:"avro".

• The frontend has to display the source format.

Use case

Kafka clusters which have grown over time and used by several teams might have topics which aren't used by anyone (anymore). In order to figure out what topics potentially could be deleted it would be helpful to know how many consumer groups consume each topic. An active consumer group for a topic could be defined as a group which has active group offsets for at least one partition for a given topic.

Topics which do not have any consumer groups consuming it are likely good candidates to look at.

• All tables should remember their page size (and have an option to change it in the first place)
• All columns should be sortable
• (Maybe) allow filtering in each column, just like in the antd demo?

There are two ways how we could implement the consumer group lags.

1. Implement the ListOffsetRequest in sarama (see: https://github.com/apache/kafka/blob/05ba5aa00847b18b74369a821e972bbba9f155eb/clients/src/main/java/org/apache/kafka/common/requests/ListOffsetRequest.java )

2. Setup a GRPC endpoint in Kafka Minion and fetch Lags etc. from Kafka Minion

Pro/Cons:

• Kafka Minion is an external dependency, which requires additional effort to run
• Kafka Minion can only report these stats after having completely consumed the __consumer_offsets topic
• Kafka Minion can offer more metrics (such as last commit activity), which we'll likely never get using the adminclient

Decision: We decided to opt in for the ListOffsetRequest to avoid external dependencies. We may add Kafka Minion integration at a later point which should definetely be optional.

TODO:

• Submit PR to Sarama which implements the ListOffsetRequest / ListOffsetResponse
• Once the PR has been merged, implement the current offset for each partition, as well as a "topic lag" in the /consumer-groups endpoint(s)

Just like in the topics list.

• Consumer Group List has a QuickSearch
• All QuickSearch bars should remember their last content
  • ...and have a clear button

A button that just refreshes the last/current message search.
Sometimes you just want to redo the search. Having to change some search parameter just to search again is a bad solution.

Create helm chart for easy deployment into Kubernetes clusters

We should create a pipeline which builds and potentially tests our containers. Upon creating a release we should also release that docker image into the package registry. Since we are activated for GitHub actions + package registry I suggest to use both.

The log dir size is an interesting metric to figure out how much space a partition/topic or broker requires. Therefore we should show this metric at all three aggregation levels (partition/topic/broker).

TODO:

• Implement log size on /topics and /brokers endpoints
• Add column on /topic and /brokers pages

This issue is dependent on: #15

For some requests Kafka Owl times out because the Kafka Cluster does not deliver the responses in a timely manner. The UI shows that it has aborted the request after 20s:

Backend API Error
Please check and modify the request before resubmitting.
Request '/api/topics/__consumer_offsets/messages?_offsetMode=-1&startOffset=-1&partitionID=94&pageSize=50&sortOrder=0&sortType=0' timed out after 20.0 sec

The backend logs:

{"level":"error","ts":"2019-12-27T09:50:50.433Z","msg":"Request was cancelled while waiting for messages from workers (probably timeout)","completedWorkers":0,"startedWorkers":1,"fetchedMessages":24}

It was expected that the backend returns the 24 fetched messages (all messages until the request context is cancelled) to the frontend, along with a flag that the request did not complete because it expired this timeout threshold.

The frontend should show the timeout properly as well (if it doesn't do so already).

Usecase:

As someone who works with Kafka I'd like to be able to find messages which have a specific key (which might have a semantic meaning - like a unique identifier for a customer). Equally I might want to filter/search for messages which have a specific value (either complete content or a subset of a JSON message like just some properties).

Kowl can do so by streaming all messages in a topic and evaluate each message against a set of user defined filters.

Examples:

• Search all messages in topic orders which have key xzy-3711
• Search all messages in topic orders where value customer.UUID = "abc123" && order.value > 6000
• Search all messages in topic orders where key and value filter from above bullets need to apply

Challenges:

• Value & Key can be plaintext, JSON, binary, avro, XML, ...
• In Kowl Business you want to apply additional limits what a user can do (e. g. max 1GB streaming / day is allowed) otherwise excessive usage can cause further troubles (traffic costs, resource usage of Kowl/Kafka, ...)
• Filters which do not just match a string but requiring the use of the correct type such as greater than equal, date comparisons, ..

Implementation hints:

Finding kafka messages by key: There are multiple partitioning strategies, but the default partitioner uses a 32-bit murmur2 to compute choose what PartitionID the message should go to. This could potentially be used to improve performance when searching a whole topic for a message by key.

As the title says

As of now the documentation is very rudimentary. The following things should be improved/added:

• Check Flags again (e. g. Telemetry Host / Port doesn't make sense as they are the same as the REST API)
• Add installation instructions
• Add badges (Container, Release version, License, Go Report)
• Feature list
• Supported Kafka versions

Messages with an empty value on a compact or compact,delete topic indicate a tombstone event. We should somehow mark them as Tombstone.

Either the backend or frontend could mark a message as Tombstone (this has yet to be discussed)

The estimated number of messages (watermark delta) should be added on the topic list for each topic, as well as on the topic detail (at the top), which should give users a better overview.

When listing messages Kafka Owl has to do a couple of requests sequentially. It fetches the partitionIDs for the requested topics, the low & high watermarks, starts a partition consumer for each partition etc. I have seen response times which take up to 4.3s for a topic with 50 partitions (regardless of the message size).

Therefore we should make sure that as many requests as possible will be run concurrently (or can be periodically prefetched so that we can potentially save a request upon invocation).

• Fetch watermarks concurrently
• Check what requests could be periodically prefetched so that they do not need to be fetched upon invocation (this is now tracked under #20)

We should make it possible to decode things like:
https://github.com/apache/kafka/blob/05ba5aa00847b18b74369a821e972bbba9f155eb/clients/src/main/java/org/apache/kafka/common/requests/DescribeLogDirsResponse.java

Either by starting a PR to integrate that functionality directly into sarama, or just as a small module in kafka-owl.

• Small tool that monitors specific files in github.com/apache/kafka for changes and alerts us on build. (so we never start a build with outdated struct information)

• Add a way for us to request 'DescribeLogDirs'; interpret the result; and show it in the frontend

• Add deleteRecords function

When using the value preview filters, JSON properties with a value of 0 will not be found.
They are treated as if the property is missing.

Usecase

Kowl is my first stop if I want to get insights (key/value structure, example messages, topic config, ...) about a specific Kafka topic. The producers of a topic may maintain additional documentation at different places (Git repositories, Confluence/SharePoint, ...). If I could embed this documentation into Kowl so that it appears in my topic detail view (for instance as separate tab) I'd get the following advantages:

• It's trivial to find the documentation for each Kafka topic
• Kowl can be considered as the one tool to get insights about your topic
• No context switching between tools (e. g. Kowl + Confluence) to get as much information as possible about a topic

The documentation could be a markdown file which will be commited via Git and can then be picked up and mapped to a Kafka Topic.

Examples:

The topic documentation could contain the following information:

• Team which produces the topic
• Semantic meaning of properties / messages
• Versioning/Historical changes
• GDPR / Data protection hints
• Producing service
• Schema (all properties along with descriptions)

Challenges:

• Mapping from Markdown file to Kafka topic (maybe via filename?)
• Schema could potentially be inferred automatically by analysing several messages in a topic. Investigate whether this can be used to enhance this documentation feature or can be neglected
• Hot reload new markdown files?
• How to keep config in sync for multiple stages (staging, pre, prod). Separate repo for docs?
• Do we want to offer the ability to enforce some schema (e. g. a topic documentation must specify whether it's GDPR sensitive or not)
• What information do we want to infer from Git (changelog, users who edited/created the files, ...)

When clicking 'Add Preview' it should actually show a dropdown/autocomplete with all the currently available property names.

Due to the amount of config options flags become cumbersome to maintain and it is not possible to have a well-aranged set of options. It is expected that the number of config options grows. Thus we'd like to migrate from flags to config files, except for sensitive inputs (e.g. passwords) where flags would still be accepted.

When connected to (for example) 7 out of 8 brokers, the backend will not respond to any requests, even though it could most likely handle them.

The backend should be able to handle this scenario and use the brokers it is connected to.

A page that shows all sorts of "admin information"

• list of kowl users (+ roles and permissions)

• find users by role, or by permission

• list of all role bindings, the "Google Group" or "GitHub Team" they resolve to; and maybe a button to trigger a refresh of those immediately

• new permission "actionManageKowl" element to allow/deny viewing this panel

• license overview: how many seats in use? by which users?

Things to keep in mind:

• What about users that only use direct entries (email addresses) instead of "Google Groups" or "GitHub Teams"

Beside a "messages" and "configuration" tab in the topics' details page we could have another tab "overview" which for now shows a table row for each partitionID along with it's low & high watermark. Something like this (ignore the "Group Offset" column):

• Preview fields
• QuickSearch (maybe?)
• Message search parameters
• Column sorting (a lot of work, pretty small feature, maybe later)

The consumer group overview lists all consumer groups along with it's group state. For users it might not be clear what each state means. One could improve this by adding a tooltip for all possible group states.

Existing group states are:

• Stable - Consumer group has members which have been assigned partitions
• Empty - Consumer group does not have any members
• Unknown - Group state is not known
• Dead - Consumer group does not have any members and it's metadata has been removed
• Preparing Rebalance - A reassignment of partitions is required. Members are asked to stop consuming so that rebalancing can take place
• Completing Rebalance - Kafka assigns partitions to group members

Additional information for reference: https://chrzaszcz.dev/2019/06/kafka-rebalancing/ (nice blogpost describing consumer group states)

Icons are matched here:

https://github.com/cloudhut/kowl/blob/master/frontend/src/components/pages/GroupDetails.tsx#L300-L305 (this list is not complete and requires more states to be added)

If a user is allowed to see a topic but is not allowed to inspect contents such as partitions, messages etc. we should list it in the topics overview and mark it as "locked" so that it is clear the user can not inspect it.

Right now the user would see it just as all the other topics and would run into an error message after trying to inspect the topic.

When the '/configuration' endpoint is called with a topic that doesn't exist it returns an empty result.

It should instead return a 404 http error code.

This concept applies to all routes though, so they should be checked out as well.

• When viewing a user, the roles/permissions are shown, but the page should also show how the user got that role! Specifically: what binding caused this role to be added to the user?

• Which groups is a given user part of (directly, and indirectly)?

• Who is in a given group?

• Bug: Clicking on the + in the bindings table expands all lines

• Bug: The expand icon is at the wrong position

• Issue: Users Tab: space should not be distributed that way. In every column there's only very little text, and when each column is 1/3 of the width, the rows are hard to read because the values are so far apart. There should be an empty column on the right that takes up all the remaining space.

Example scenario:
The user wants to see the last 100 messages, but the backend returns only one message, because the requested range (latest .. latest-100) has been compacted.

In that case the frontend should show some sort of message to explain what is going on; why only a single message is shown.

Similar to #25, which is also about handling of error-/edge-cases.

Current (really bad) workarounds:

• Delete tag, add it back again (new tags get added to the end)
• Open chrome dev tools; in local storage select the key uiSettings; inside the json navigate into topicList, then previewTags; then manually change stuff there; reload page

Fix is simple and likely won't take much time, just use something like:
https://github.com/clauderic/react-sortable-hoc/blob/master/examples/basic.js

Some people use Avro or XML as their message format. So of course kafka-owl has to support those too.

Since those formats are - just like Json - entirely self describing and also don't contain any structures that can't be expressed in Json, I think the best approach would be to convert them to Json in the backend.

Having a common data format like that will make things a lot easier (as opposed to literally copying all code 2 times, or even more if there is a new format...).

In order to be able to develop and test this, we need some sample data to work with.

• Our local dummy Kafka setup needs two new topics: avro-topic and XML-topic.

• Those topics need some sample data in them, so the already existing go-tool should be adapted to generate some random messages to populate the topics with.

• The messages should not all have the same schema. Some variance would be good to simulate real world scenarios. (objects that have new, renamed, and/or deleted fields)

In https://github.com/kafka-owl/kafka-owl/blob/master/backend/pkg/kafka/describe_consumer_groups.go#L152 we return a 500 error if a member assignment can not be decoded as desired. This should never happen as all consumers of protocolType == "consumer" should follow a given schema, but theoretically it's possible.

Instead of returning a server error we should log this at the error level and possibly dump the byte array / hex string so that we can decode the package and find the culprit.

Commit 8071a5d adds support for non consumer consumer groups (e. g. schema registry - "sr"). If the consumer group's protocolType is anything other than "consumer" we can not decode "memberAssignments".

I think we should make this visible to the frontend. For this purpose I added the "protocolType" property here https://github.com/kafka-owl/kafka-owl/blob/8071a5dc38635cdecbeeb5d59d6c9a9fa56d0cfd/backend/pkg/kafka/list_consumer_groups.go#L16 .

These are some improvements that probably will be good for collaborations.

1. Search query for the topics (in topic list)
2. Anchor for the offset number or offset numbers (in messages tab)
3. Search query for message search (message search tab)

TopicHighWaterMarks and TopicLowWaterMarks are way too similar and should be merged.

https://github.com/kafka-owl/kafka-owl/blob/7fca6e594215a2b13d597ba9108579f5ac0693af/backend/pkg/kafka/list_messages.go#L62-L120

• either taking oldest/newest as a new parameter,
• or, even better: returning a map of tuples (high, low)

Users should be able to:

• select what columns are visible, and in what order

• change the way the timestamp is displayed (default, only date, only time, unix seconds, relative (x days ago))

• 'No key' should have a tooltip to explain what the icon means

In the partitions overview we should add one or two clumns which show the Leader + Replica Broker IDs for each partition.

• Add leader and replica broker IDs in the /partitions response in the Backend
• Add the new information to the frontend

Update: Backend already returns the required information

Add a page which shows all ACL rules for each resource type and principal

There are no prometheus metrics yet. Interesting metrics can be:

• Request latency bucket (where request-type is a label)
• Number of incoming requests
• Bytes / IN / OUT - (helps to understand how busy the backend is - could be pretty excessive due to large messages or high throughput of messages)

We don't have any endpoints to check readiness & liveness yet.

see title

Use case

Most REST endpoints actually perform multiple (sequential) requests towards Kafka. There are various reasons why a cache is beneficial for Kafka Owl.:

• We can reduce those endpoints' response times by periodically prefetching some of these requests (e. g. topics list, partitionIDs and watermarks).
• By prefetching requests periodically the load towards Kafka becomes more predictable as we can share the responses across multiple concurrent requests (and endpoints) and distribute the requests equally over time
• A cache will potentially save requests towards Kafka

This feature request is prioritized low because we haven't had any performance issues with Kafka yet. The more users Kafka Owl serves, the more important this issue becomes. Even tens of developers who concurrently browse through Kafka Owl shouldn't be a problem for most Kafka clusters.

Implementation

We should create a storage/cache interface which allows a Redis / InMemory implementation. Additionally we'd need to add fetchedAt timestamps in the responses so that we can show the age of the provided data.

One of the initial ideas was to create one Cache instance for each type of data we want to cache (e. g. topics, partitions, consumer groups, ..) with the advantage that we can shift more responsibility (such as RWMutexes, MaxAge gets, generic set/get interface{}) towards the Cache struct.