In this project, An End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka is done. I have used different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.
![](https://private-user-images.githubusercontent.com/93676625/258708509-8fe537c6-e05b-4c85-9e02-4e66fd6857d9.jpg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE3NDIzNDQsIm5iZiI6MTcyMTc0MjA0NCwicGF0aCI6Ii85MzY3NjYyNS8yNTg3MDg1MDktOGZlNTM3YzYtZTA1Yi00Yzg1LTllMDItNGU2NmZkNjg1N2Q5LmpwZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIzVDEzNDA0NFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWNmNjNmMjg4Nzc4Yzk1ODllZmEzZTY5OGI1MTkyZjRkMWRlY2MxNTQ0OGRkYTljMzBiNmVjZjNjOTdmYWRlMTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.XqBDMMlRm513TydP2fsVbdmyhVk_I8Qkc_NMCUYJ5rA)
- Programming Language - Python
- Amazon Web Service (AWS)
- S3 (Simple Storage Service)
- Athena
- Glue Crawler
- Glue Catalog
- EC2
- Apache Kafka
Here is the dataset used- https://github.com/darshilparmar/stock-market-kafka-data-engineering-project/blob/main/indexProcessed.csv