abraham-leal / kafka-idle-topics Goto Github PK
View Code? Open in Web Editor NEWA tool to detect idle topics in your cluster
License: Apache License 2.0
A tool to detect idle topics in your cluster
License: Apache License 2.0
Hi Abraham
I am testing the tool and having issues which may be network related but need some way of confirming this
We have a private cluster which is connected to confluent via AWS private links and on-prem AWS Transit gateway and to laptop via VPN.
Command:
~/go/bin/kafka-idle-topics -bootstrap-servers ????.eu-west-1.aws.glb.confluent.cloud:9092 -username ??? -password ?? -kafkaSecurity plain_tls -idleMinutes 43800
Outputs - Attempt 1
2024/01/26 14:25:37 Loading Topics...
2024/01/26 14:25:37 Could not reach cluster within the last 30 seconds. Is the configuration correct? read tcp 172.31.193.152:59524->10.14.163.147:9092: read: connection reset by peer
Output - Attempt 2
2024/01/26 15:47:31 Loading Topics...
2024/01/26 15:47:34 Evaluating Topics that haven't been produced to since... 2023-12-27 05:47:34.83183 +0000 GMT m=-2627996.097033499
2024/01/26 15:47:35 Could not consume from topic: read tcp 172.31.193.152:49521->10.14.163.179:9092: read: connection reset by peer
From the same laptop I can confirm I can consume messages from a topic using the command
kafka-console-consumer --bootstrap-server??.eu-west-1.aws.glb.confluent.cloud:9092 --consumer.config command-nonprod.properties --topic ??????--from-beginning
Any suggests ?
For example: Describe a possible issue with credentials, etc.
We are currently testing the tool and have issues with running the tool as it appears to fail if the topic list being queried is greater that 100+ and is dependant on the topics an api key has.
For a client the tool would work well but for a support team it would mean creating and maintaining over 10 new service accounts as we have over 1000 topics which is not practical. We also would prefer more details in the output as we can not risk removing client topics with proper safeguards.
Is it possible to update the code
Resolve the issue with querying large number of topics (may not be possible)
Restrict queries to a list in a file or topic prefix rather that the access available to an api-key
update the output to include some verification or date last used as well as topic name so we can sort by least active
update the docker image to allow the use of the additional parameter (-idleMinutes) that can be used on the command line
Not really an issue but a Suggestion.
First of all, great tool.. Thanks Abraham.
I ran it for a cluster on CC. It listed dlq topics as well.
dlq topics are connector-created and are required.
I suggest to either flag this as a message, say at the beginning of text file, to carefully decide which topics to be deleted.
For eg; dlq topics, retry topics(created by devs/ applications), error topics (generally created by HTTP Sink Connector)
All these topics are generally empty but still required.
Hi,
I was trying to execute this utility against confluent Cloud Kafka to compile a list of idle topics but with whatever setting i run It fails with Broken Pipe Error. The said sever has more than 1500 topics and 600 Consumer groups.
./kafka-idle-topics -bootstrap-servers pkc-****.northeurope.azure.confluent.cloud:9092 -username REMOVED -password REMOVED -kafkaSecurity plain_tls -idleMinutes 43200 -productionAssessmentTimeMs 43200
2023/10/19 17:12:46 Loading Topics...
2023/10/19 17:12:49 Evaluating Topics that haven't been produced to since... 2023-09-19 17:12:49.563518 +0100 BST m=-2591995.340513124
2023/10/19 17:31:53 Evaluating Topics without active Consumer Groups...
2023/10/19 17:31:53 Could not obtain Consumer Groups from cluster: write tcp 192.168.0.48:56237->20.105.49.156:9092: write: broken pipe
Other options that I tried.
./kafka-idle-topics -bootstrap-servers pkc-****.northeurope.azure.confluent.cloud:9092 -username REMOVED -password REMOVED -kafkaSecurity plain_tls -idleMinutes 262800
Could you please help me to identify what is the issue here ?
Thank you
A
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.