Code Monkey home page Code Monkey logo

cosmoschangefeed's Introduction

CosmosChangeFeed

This demo application showcases how to implement Change data capture using Cosmos Change Feed.

This is a 'Streaming ETL' application with Cosmos SQL API DB being source with Cosmos Mongo DB and Azure cache for Redis as Targets.

Azure event hub acts as a persistent stream to receive events from source and targets consumes events from event hub.

img.png

There are primarily 3 apps + 1 utility app,

  1. Reader App (ChangeFeedCDC) - Application which implements CDC to read changes from Cosmos SQL API DB in realtime using multiple threads (one per logical partition) and pushes to Event hub.
  2. Writer App (MongoReplicator) - Application reads the change events from Azure event hub using multiple threads (one per partition in Event hub) and writes it to Cosmos Mongo API DB
  3. Cache App (CacheRefresher) - Application reads the changes events from Azure event hub using multiple threads (one per partition in Event hub) and writes it to Azure cache for Redis
  4. Utility App (FlightUtil) - Console Application to insert one auto generated item into source Cosmos SQL API DB on hitting enter key

To compile individual application, cd to respective folder and do 'mvn clean install' and then run.

ps: Replace 'ToBeFilled' in main method of all 4 application with necessary host, connection url and keys before compiling.

cosmoschangefeed's People

Contributors

kpurusho avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.