iot-streaming's Introduction

iot-streaming

IoT Streaming using Flink to connect Kafka and Cassandra, Elastic

Import in IntelliJ
- IntelliJ:
  - Select “File” -> “Import Project”
  - Select root folder of your project
  - Select “Import project from external model”, select “Maven”
  - Leave default options and finish the import
How to run this project

Using the kafka/Producer.java to simulate the IoT Data Producer
You can choose to use kafka/Consumer.java to test the simple process of IoT data.
You can also choose to use flink/Processor.java to process the IoT data by Flink, eventually store the data to Cassandra and Elastic.

Kafka 0.10

Maven dependency

       <dependency>
         <groupId>org.apache.kafka</groupId>
         <artifactId>kafka_2.11</artifactId>
         <version>0.10.2.0</version>
     </dependency>  
     
     <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-connector-kafka-0.10_2.11</artifactId>
         <version>1.2.0</version>
     </dependency>

Example Java code

     DataStream<String> stream = env.addSource(
             new FlinkKafkaConsumer10<>("test", new SimpleStringSchema(), properties));
     return stream;

Flink & Cassandra

Maven dependency

     <dependency>
         <groupId>com.datastax.cassandra</groupId>
         <artifactId>cassandra-driver-core</artifactId>
         <version>3.1.2</version>
     </dependency>

     <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-connector-cassandra_2.11</artifactId>
         <version>1.2.0</version>
     </dependency>

Flink & Elasticsearch Connector

Maven dependency

     <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-connector-elasticsearch2_2.11</artifactId>
         <version>1.2.0</version>
     </dependency>

Recommend Projects