Kafka Monitor is a framework to implement and execute long-running kafka system tests in a real cluster. It complements Kafka’s existing system tests by capturing potential bugs or regressions that are only likely to occur after prolonged period of time or with low probability. Moreover, it allows you to monitor Kafka cluster using end-to-end pipelines to obtain a number of derived vital stats such as end-to-end latency, service availability and message loss rate. You can easily deploy Kafka Monitor to test and monitor your Kafka cluster without requiring any change to your application.

Kafka Monitor requires Gradle 2.0 or higher. Java 8 should be used for both compilation and execution.

$ git clone https://github.com/linkedin/kafka-monitor.git
$ cd kafka-monitor 
$ ./gradlew jar

$ ./bin/kafka-monitor-start.sh config/kafka-monitor.properties

$ ./bin/end-to-end-test.sh --topic test --broker-list localhost:9092 --zookeeper localhost:2181

Open localhost:8000/index.html in your web browser

You can edit webapp/index.html to easily add new metrics to be displayed.

curl localhost:8778/jolokia/read/kmf.services:type=produce-metrics,name=*/produce-availability-avg

You can query other JMX metric value as well by substituting object-name and attribute-name of the JMX metric in the query above.

./gradlew checkstyleMain checkstyleTest

./gradlew idea
./gradlew eclipse

• Motivation
• Design Overview
• Service Design
• Future Work