Code Monkey home page Code Monkey logo

rdbms-to-geode's Introduction

Spring Cloud Stream ETL Demo

Setting up your environment

This demo requires PCF Dev and Apache Geode.

Starting PCF Dev

  1. Start CF
    cf dev start
    
  2. Login to CF
    cf login -a https://api.local.pcfdev.io --skip-ssl-validation
    
  3. Create MySQL and RabbitMQ services
    cf create-service p-mysql 512mb mysql-dev
    cf create-service p-rabbitmq standard rabbitmq-dev 
    
  4. Push jdbc-event-source-app, jdbc-event-processor-app, and geode-sink-app
    cf push -f geode-sink-app/manifest-dev.yml
    cf push -f jdbc-event-processor-app/manifest-dev.yml
    cf push -f jdbc-event-source-app/manifest-dev.yml
    

Starting Apache Geode

  1. Open gfsh terminal
  2. Start locator and server from gfsh
    start locator --name=locator
    configure pdx --auto-serializable-classes=com\.example\.springoneplatform\.scs\.demo\.model\.pdx\.* --read-serialized=true
    start server --name=server
    
  3. Create regions
    create region --name=customer --type=PARTITION --skip-if-exists
    create region --name=customerOrder --type=PARTITION --skip-if-exists
    create region --name=item --type=PARTITION --skip-if-exists
    

Running the demo

Insert initial data

  1. Run schema.sql to create tables
  2. Run data.sql to insert CUSTOMER, ITEM, CUSTOMER_ORDERS, and ORDER_ITEM rows
  3. Start a gfsh terminal and connect to the locator
    connect
    
  4. Query the regions to see that the data was extracted from the MySQL database, transformed to the proper types, and loaded into Geode
    query --query="select * from /customer"
    query --query="select * from /customerOrder"
    query --query="select * from /item"
    

Trigger a data change

  1. Run trigger.sql to create the triggers that will insert DB_EVENT rows when data in other tables is inserted, updated, or deleted
  2. Run delta-data.sql to do the following:
    1. DELETE an ORDER_ITEM from order1
    2. DELETE a CUSTOMER_ORDER from customer1 (will DELETE all ORDER_ITEM rows for order2 first)
    3. INSERT a ITEM (will add a new item (Stapler) that can be included in an order)
    4. INSERT a CUSTOMER_ORDER and `ORDER_ITEM (will create an order for the new item created above)
  3. Query the regions again to see that the data that was deleted from the MySQL database was also deleted from Geode
    query --query="select * from /customer"
    query --query="select * from /customerOrder"
    query --query="select * from /item"
    


Additional Information

Apache Geode Commands

In order to run gfsh commands you need to open a gfsh terminal. It's recommended that you create a working directory that you open gfsh from because starting locators and servers creates directories and files.

mkdir tmp && cd tmp
gfsh

Once in the gfsh terminal:

  • connect connects the gfsh session to the cluster
  • list members displays all members (locators, servers) that are part of the cluster
  • list regions displays all regions that have been created
  • query --query="<oql here>" executes a query (OQL help)

rdbms-to-geode's People

Watchers

James Cloos avatar Guxiangguo avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.