This demo is shown how to use the Glue Streaming feature to Manage continuous ingestion pipelines and processing data on-the-fly. The Glue Steaming Jobs is extending AWS Glue jobs, based on Apache Spark, to run continuously and consume data from streaming platforms such as Amazon Kinesis Data Streams and Apache Kafka (including the fully-managed Amazon MSK).

Glue can provision, manage, and scale the infrastructure to ingest data to data lakes on Amazon S3, data warehouses such as Amazon Redshift, or store streaming data in a DynamoDB table for quick lookups, or in Elasticsearch to look for specific operation patterns.

Glue Streaming is based on Spark Structured Streaming to implement data transformations, such as aggregating, partitioning, and formatting as well as joining with other data sets to enrich or cleanse the data for easier analysis.

Please find more details in Adding Streaming ETL Jobs in AWS Glue guide

IoT-Kafka-GlueStreaming-Demo

IoT-Kinesis-GlueStreaming-Demo

kinesis-kafka-connector-Demo

This demo is shown how to use the Kinesis Data Anlytics to Manage continuous ingestion pipelines and processing data on-the-fly. Kinesis Data Anlytics can help you run continuously and consume data from streaming platforms such as Amazon Kinesis Data Streams and Apache Kafka (including the fully-managed Amazon MSK).

IoT-Kinesis-KinesisDataAnlytics-Demo

IoT-Kafka-KinesisDataAnlytics-Demo

This demo is shown how to use the Glue to ingest data from RDS database.

Architeture

Glue ingest MySQL5.7 via Glue connector

Glue ingest MySQL8.0 via Glue connector

Connect the RDS which SSL connection enabled

Data On Boarding End2End Demo

Pyhton-Send-Data-Firefose Demo

Build a business intelligence capability for streaming IoT device data using AWS IoT Core, Amazon Firehose, Amazon S3, Amazon Athena and Amazon QuickSight

IoT-Athena-QuickSight