Code Monkey home page Code Monkey logo

flank-airquality's Introduction

flank-airquality

Timothy Spann's demo for sending airquality data from Spring-Kafka Java to Kafka. And from NiFi to Kafka from REST.

FLaNK Diagram

1️⃣ Demo

Maven Build

Spring Run

AQ Produced Run

AQ Schema

SMM Kafka Records

2️⃣ Setup

  • Visual Code with Spring & Java 11-17
  • Set an environment variable with your api key code from airnow
  • Point to your Apache Pulsar cluster, if you are using StreamNative cloud I have SSL and configuration in the config class

3️⃣ Configuration


airnowapi.api.key=${API_KEY}
airnowapi.url=https://www.airnowapi.org/aq/observation/zipCode/current
airnowapi.uri=/?format=application/json&distance=250&zipCode=
airnowapi.key.name=&API_KEY=
zipcodes=08520,94027,02199,11962,94957,90402,94301,07070,90265,90272,10013,10007,94123,77449,11368,60629,79936,75034
topic.name=airquality
producer.name=airqualitykafka
send.timeout=60
security.mode=off
server.port=8999
#kafka
kafka.bootstrapAddress=kafka:9092
kafka.topic.name=airquality
logging.level.root=INFO
logging.level.dev.datainmotion.airquality=INFO


4️⃣ Build

mvn package

5️⃣ Run

mvn spring-boot:run

6️⃣ Environment Variables

export API_KEY=

7️⃣ Producing data


// send as json string

        UUID uuidKey = UUID.randomUUID();
        ProducerRecord<String, String> producerRecord = new ProducerRecord<>(topicName,
                uuidKey.toString(),
                DataUtility.serializeToJSON(message));
       kafkaTemplate.send(producerRecord);

8️⃣ Consume AirQuality Topic

See: consumeairquality.py

9️⃣ Consume OpenAQ Topic

See: consumeopenaq.py

1️⃣0️⃣ NiFi Flows

nifi

nifi

nifi

See: AirQuality.json

1️⃣1️⃣ Air Quality References

Let's use the new Spring to read a ton of Air Quality data for the area.
This is a REST feed, no extra equipment needed.

1️⃣2️⃣ Google Bart AI Step Suggestions

bart

nifieverywhere

© 2023 tim

flank-airquality's People

Contributors

tspannhw avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.