abraham-leal / kafka-idle-topics Goto Github PK

Hi Abraham

I am testing the tool and having issues which may be network related but need some way of confirming this

We have a private cluster which is connected to confluent via AWS private links and on-prem AWS Transit gateway and to laptop via VPN.

Command:
~/go/bin/kafka-idle-topics -bootstrap-servers ????.eu-west-1.aws.glb.confluent.cloud:9092 -username ??? -password ?? -kafkaSecurity plain_tls -idleMinutes 43800

Outputs - Attempt 1
2024/01/26 14:25:37 Loading Topics...
2024/01/26 14:25:37 Could not reach cluster within the last 30 seconds. Is the configuration correct? read tcp 172.31.193.152:59524->10.14.163.147:9092: read: connection reset by peer

Output - Attempt 2
2024/01/26 15:47:31 Loading Topics...
2024/01/26 15:47:34 Evaluating Topics that haven't been produced to since... 2023-12-27 05:47:34.83183 +0000 GMT m=-2627996.097033499
2024/01/26 15:47:35 Could not consume from topic: read tcp 172.31.193.152:49521->10.14.163.179:9092: read: connection reset by peer

From the same laptop I can confirm I can consume messages from a topic using the command
kafka-console-consumer --bootstrap-server??.eu-west-1.aws.glb.confluent.cloud:9092 --consumer.config command-nonprod.properties --topic ??????--from-beginning

Any suggests ?

For example: Describe a possible issue with credentials, etc.

We are currently testing the tool and have issues with running the tool as it appears to fail if the topic list being queried is greater that 100+ and is dependant on the topics an api key has.
For a client the tool would work well but for a support team it would mean creating and maintaining over 10 new service accounts as we have over 1000 topics which is not practical. We also would prefer more details in the output as we can not risk removing client topics with proper safeguards.
Is it possible to update the code

• Resolve the issue with querying large number of topics (may not be possible)

• Restrict queries to a list in a file or topic prefix rather that the access available to an api-key

• update the output to include some verification or date last used as well as topic name so we can sort by least active

• update the docker image to allow the use of the additional parameter (-idleMinutes) that can be used on the command line

Not really an issue but a Suggestion.
First of all, great tool.. Thanks Abraham.

I ran it for a cluster on CC. It listed dlq topics as well.
dlq topics are connector-created and are required.
I suggest to either flag this as a message, say at the beginning of text file, to carefully decide which topics to be deleted.
For eg; dlq topics, retry topics(created by devs/ applications), error topics (generally created by HTTP Sink Connector)
All these topics are generally empty but still required.

Hi,
I was trying to execute this utility against confluent Cloud Kafka to compile a list of idle topics but with whatever setting i run It fails with Broken Pipe Error. The said sever has more than 1500 topics and 600 Consumer groups.

./kafka-idle-topics -bootstrap-servers pkc-****.northeurope.azure.confluent.cloud:9092 -username REMOVED -password REMOVED -kafkaSecurity plain_tls -idleMinutes 43200 -productionAssessmentTimeMs 43200
2023/10/19 17:12:46 Loading Topics...
2023/10/19 17:12:49 Evaluating Topics that haven't been produced to since... 2023-09-19 17:12:49.563518 +0100 BST m=-2591995.340513124
2023/10/19 17:31:53 Evaluating Topics without active Consumer Groups...
2023/10/19 17:31:53 Could not obtain Consumer Groups from cluster: write tcp 192.168.0.48:56237->20.105.49.156:9092: write: broken pipe

Other options that I tried.

./kafka-idle-topics -bootstrap-servers pkc-****.northeurope.azure.confluent.cloud:9092 -username REMOVED -password REMOVED -kafkaSecurity plain_tls -idleMinutes 262800

Could you please help me to identify what is the issue here ?
Thank you
A

and

This maybe an issue with parameters or the way we are issuing the command please advise