kafka producer and consumer example in scala and java
This is tested locally with
Gradle 2.0,
Java 1.8.0_65
Scala 2.11.7
Zookeeper 3.4.6
Kafka 2.10-0.9.0.0
Spark 1.6.0 : https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-building-from-sources.htmlif you have installed zookeeper, start it, or run the command:
bin/zkServer.sh start conf/zoo.cfg> bin/kafka-server-start.sh config/server.properties> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 10 --topic test_topic> git clone https://github.com/iamon3/as_spark_kafka_gradle.git
> cd as_spark_kafka_gradle
> gradle clean buildit will package compiled classes and its dependencies into a jar.
> cd build/lib/This example contains two consumers written in Java and in scala. You can run this for java:
> java -cp kafka_example-0.1.0-SNAPSHOT.jar com.colobu.kafka.ConsumerExample localhost:2181 group1 test_topic 10 0or this for scala:
> java -cp kafka_example-0.1.0-SNAPSHOT.jar com.colobu.kafka.ScalaConsumerExample localhost:2181 group1 test_topic 10 0Stop the consumer Ctrl+C
> cp kafka_example-0.1.0-SNAPSHOT.jar ${SPARK_INSTALLATION_DIR}/bin
> cd ${SPARK_INSTALLATION_DIR}/bin
> sh spark-submit --class com.colobu.kafka.ScalaConsumerExample \
> --master local[8] \
> kafka_example-0.1.0-SNAPSH.jar localhost:2181 group1 test_topic 10 0 group1 test_topic 10 0Push messages to Kafka
> ${KAFKA_INSTALLTION_DIR}/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test_topic
> Hi
> Hello
> What's Up.These messages will be consumed and will be displayed on spark terminal.
This example also contains two producers written in Java and in scala. you can run this for java:
> java -cp kafka_example-0.1.0-SNAPSHOT.jar com.colobu.kafka.ProducerExample 10000 colobu localhost:9092or this for scala
> java -cp kafka_example-0.1.0-SNAPSHOT.jar com.colobu.kafka.ScalaProducerExample 10000 colobu localhost:9092
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent