Code Monkey home page Code Monkey logo

streaming-data-pipeline's Introduction

Streaming-Data-Pipeline with Cloud Computing

This is the repository for CS441 final project.

S.No. First Name Last Name UIN Email
1 Ragavee Poosappagounder Kandavel 660425677 [email protected]
2 Smrithi Balki 668488598 [email protected]
3 Ramiya Shree Seshiah 660418618 [email protected]
4 Anandavignesh Velangadu Sivakumar 662139789 [email protected]
5 Lakshmanan Meiyappan 671997054 [email protected]

Introduction:

The goal of this course project is to create a streaming data by designing and implementing an actor-model service using Akka that ingests log file generated data in real time and delivers it via an event-based service called Kafka to Spark for further processing.

The project has been completed under the leadership by Ragavee Poosappagounder Kandavel. Ragavee Poosappagounder Kandavel, Ramiya Shree Seshiah, and Anandavignesh Velangadu Sivakumar worked on setting up Log Generator in the EC2 instances, Akka actors to monitor log changes and AWS MSK Kafka streams. Smrithi Balki and Lakshmanan Meiyappan worked on AWS MSK Kafka streams, processing and aggregating the log file changes in Spark and finally, email notification service after processing.

Documentation

File Watcher- Akka - Kafka

Spark - AWS Email notification

Video presentation

File Watcher - Akka - Kafka

Spark - AWS Email notification

streaming-data-pipeline's People

Contributors

gnzeleven avatar smrithi-balki avatar ragaveepk avatar ramiyashreeseshaiah avatar smrithibalki avatar laxmena avatar ramiyashree avatar vineet77 avatar

Watchers

0x1D0CD00D avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.