Topic: data-engineering-pipeline Goto Github

👇 Here are 113 public repositories matching this topic...

ahmetfurkandemir / data-engineering-project-with-hdfs-and-kafka
data-engineering-pipeline,Data Engineering Project with Hadoop HDFS and Kafka
User: ahmetfurkandemir
data data-engineer data-engineering data-engineering-pipeline docker docker-compose hadoop hadoop-filesystem hadoop-hdfs hdfs hdfs-client hdfs-dfs kafka kafka-consumer kafka-producer kafka-ui kafkaui pipline python python-hdfs-client

alanchn31 / movalytics-data-warehouse
data-engineering-pipeline, Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow
User: alanchn31
docker airflow spark sql python3 redshift data-engineering-pipeline data-engineer-nanodegree pyspark data-engineering

alero-awani / batch-data-engineering-project
data-engineering-pipeline,A batch Data Pipeline that retrieves data from a user purchase table and a movie review table and is transformed to form a user behaviour metric table.
User: alero-awani
airflow aws-s3 data-engineering-pipeline docker pipeline pyspark sql terraform

alphanaksoyoglu / tweeter-etl-pipeline
data-engineering-pipeline,A streaming ETL pipeline for Realtime Tweet Collection, Analysis and Reporting
User: alphanaksoyoglu
data-engineering-pipeline data-engineering tweeter tweeter-stream-api tweepy-library sentiment-analysis slackbot mongodb-database postgresql

anki-code / xontrib-pipeliner
data-engineering-pipeline,Let your pipe lines flow thru the Python code in xonsh.
User: anki-code
xonsh xontrib pipe pipeline pipelines python shell data-engineering data-engineering-pipeline

anna-geller / dataflow-ops
data-engineering-pipeline,Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate
User: anna-geller
automation aws dataflow orchestration prefect serverless analytics analytics-engineering cicd data

anna-geller / dataflow-ops-aws-eks
data-engineering-pipeline,Project demonstrating how to automate Prefect 2.0 deployments to AWS EKS
User: anna-geller
Home Page: https://discourse.prefect.io/t/how-to-build-a-prefect-2-0-poc-on-aws/1066
automation aws data eks eks-cluster eksctl karpenter kubernetes kubernetes-deployment kubernetes-setup

anna-geller / prefect-aws-lambda
data-engineering-pipeline,Deploy a Prefect flow to serverless AWS Lambda function
User: anna-geller
aws aws-lambda data-engineering data-engineering-infrastructure data-engineering-pipeline dataflow dataflow-ops event-driven event-driven-architecture lambda

anna-geller / prefect-deployment-patterns
data-engineering-pipeline,Code examples showing flow deployment to various types of infrastructure
User: anna-geller
automation aws data data-engineering data-engineering-pipeline data-engineering-team data-science dataflow orchestration pipeline prefect python serverless serverless-framework dataflow-ops data-products data-engineering-infrastructure

anna-geller / prefect-getting-started
data-engineering-pipeline,Get started with Prefect by scheduling your Prefect flows with GitHub Actions
User: anna-geller
Home Page: http://prefect.io/
analytics-engineering automation cicd data data-engineering data-engineering-infrastructure data-engineering-pipeline data-pipeline data-science dataflow dataflow-ops github-actions orchestration pipeline prefect python scheduling serverless

antimoz-om / antimoz
data-engineering-pipeline,A data engineering pipeline for digital marketers.
Organization: antimoz-om
data-analytics data-engineering-pipeline digital-marketing hadoop kafka

bayoadejare / lightning-containers
data-engineering-pipeline,Docker powered starter for geospatial analysis of lightning atmospheric data.
User: bayoadejare
Home Page: https://lightning-containers.streamlit.app/
clustering-analysis csv-files data-engineer data-engineering-pipeline data-warehouse databases docker jupyter machine-learning-algorithms noaa-weather

benedekrozemberczki / av_ultimate_student_hunt
data-engineering-pipeline,Solution for the Ultimate Student Hunt Challenge (1st place).
User: benedekrozemberczki
winning-entry machine-learning r analytics-vidhya-competition kaggle driven-data competition xgboost gradient-boosting extreme-gradient-boosting

billynewport / datasurface
data-engineering-pipeline,Data Ecosystem Platform allowing multiple teams to work together on a firm wide data ecosystem definition whilst guaranteeing consistency and controlling who can change what
User: billynewport
data-engineering data-engineering-pipeline data-pipeline data-warehouse software-architecture

bramvdklinkenberg / adf-airflow-data-project
data-engineering-pipeline,Data engineering project using Azure Data Factory and Apache Airflow
User: bramvdklinkenberg
apache-airflow azure-data-factory bicep data-engineering data-engineering-infrastructure data-engineering-pipeline pulumi-azure terraform-azurerm

brunocampos01 / predicting-retail-churn-with-azure-ml-studio
data-engineering-pipeline,Challenge to job: Data Scientist
User: brunocampos01
azure azure-pipelines azure-machine-learning-studio machine-learning-studio softplan challenge python3 pandas machine-learning deploy-machine-learning

charliesergeant / airflow-minio-postgres-fastapi
data-engineering-pipeline,Sample data store project to be hosted on a remote server or cluster. CICD using GitHub actions for SSH Deploy to remote server for docker compose.
User: charliesergeant
airflow data-engineering-pipeline datalake docker-compose fastapi jupyter-notebook minio postgresql python selenium-python

chrisammon3000 / aws-permits-pipeline
data-engineering-pipeline,ETL pipeline for construction permits data in Los Angeles built on AWS S3, Lambda and RDS PostgreSQL.
User: chrisammon3000
aws data-engineering-pipeline postgis postgresql python

datarootsio / notion-dbs-data-quality
data-engineering-pipeline,Using Great Expectations and Notion's API, this repo aims to provide data quality for our databases in Notion.
Organization: datarootsio
data-quality notion notion-api great-expectations data-engineering-pipeline notion-database

delelinus / hfr-data-warehousing
data-engineering-pipeline,End-to-end data engineering processes for the NIGERIA Health Facility Registry (HFR). The project leveraged Selenium, Pandas, PySpark, PostgreSQL and Airflow
User: delelinus
airflow airflow-dags apache-spark data-engineer data-engineering-pipeline data-ingestion data-mining data-science data-warehouse-architecture data-warehousing

dylanzenner / business_closures_de_pipeline
data-engineering-pipeline,Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DocumentDB as the database
User: dylanzenner
documentdb data-engineering aws-ecosystem data-engineering-pipeline aws aws-secretsmanager aws-ec2-bastion slack quicksight aws-ssm-document

dylanzenner / greenhouse_gas_emissions_de_pipeline
data-engineering-pipeline,Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DynamoDB as the database
User: dylanzenner
data-engineering data-engineering-pipeline aws dynamodb aws-ecosystem cloudformation aws-lambda aws-secretsmanager aws-sns aws-sqs

gg-delta / northwind-elt-pipeline
data-engineering-pipeline,Showcasing a high-performance data engineering workflow (ELT) using the classical "northwind" database
User: gg-delta
airbyte aws-rds dbt docker preset snowflake sql aws-s3 data-engineering-pipeline

hellosongi / spotify-data-pipeline
data-engineering-pipeline,Routinely collects trending songs world and stores them to a storage pool. Utilizes various Microsoft Azure services(ADLS, ADF, Synapse Analysis, Azure Functions, Logic Apps), Spotify API
User: hellosongi
data-engineering-pipeline python spotify-api apache-airflow azure

immu0001 / udacity-data-engineer-nanodegree
data-engineering-pipeline,Classwork projects and home works done through Udacity data engineering nano degree
User: immu0001
spark data-analysis big-data etl data-pipelines classwork airflow-dags data-science data-lake-analytics s3-bucket

inosrahul / f1-data-pipeline
data-engineering-pipeline,F1 Data Pipeline
User: inosrahul
Home Page: https://lookerstudio.google.com/reporting/9fd225dd-a9b8-45d9-87dc-7d7dbae0c841
bigquery data-engineering-pipeline dbt gcs looker prefect python terraform

kaoutaar / velib_v1
data-engineering-pipeline,end to end data engineering project
User: kaoutaar
data-engineering data-engineering-pipeline docker kafka pipeline python spark streamlit webapp spark-streaming

ketgo / marshmallow-pyspark
data-engineering-pipeline,Marshmallow serializer integration with pyspark
User: ketgo
marshmallow pyspark data-pipelines schema spark data-engineering data-engineering-pipeline data-cleaning data-schemas

kishlayjeet / stock-market-real-time-data-pipeline-with-apache-kafka-and-cassandra
data-engineering-pipeline,A end-to-end real-time stock market data pipeline with Python, AWS EC2, Apache Kafka, and Cassandra Data is processed on AWS EC2 with Apache Kafka and stored in a local Cassandra database.
User: kishlayjeet
apache-kafka cassandra kafka pipeline python stock-market stock-market-data-pipeline aws aws-ec2 data-engineering data-engineering-pipeline data-pipeline ec2 ec2-instance etl etl-pipeline kafka-streaming kafka-streams real-time real-time-data-pipeline

kishlayjeet / twitter-data-pipeline-using-airflow-and-aws-s3
data-engineering-pipeline,An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.
User: kishlayjeet
airflow airflow-dags apache-airflow boto3 data-engineering data-engineering-pipeline data-pipeline etl etl-job etl-pipeline python s3 scheduler tweepy twitter twitter-api twitter-data-pipeline

koksang / social-media-analysis
data-engineering-pipeline,Social Media Analysis, scalable solution, flexible deployment that analyses social media contents
User: koksang
bigquery python ray kafka apache-airflow social-media twitter python3 data-engineering data-engineering-pipeline

markditsworth / tweetanalyzer
data-engineering-pipeline,An environment for analyzing Twitter
User: markditsworth
twitter elasticsearch logstash kibana kafka docker python nlp-machine-learning data-engineering-pipeline

mikeroyal / apache-spark-guide
data-engineering-pipeline,Apache Spark Guide
User: mikeroyal
spark spark-streaming data-engineering pyspark apache-spark machine-learning big-data data-science data-engineering-pipeline awesome

minhky2185 / healthcare_data_pipeline
data-engineering-pipeline,An end-to-end data pipeline for building Data Lake and supporting report using Apache Spark.
User: minhky2185
analytics big-data data data-engineering data-engineering-pipeline data-lake emr-cluster mysql postgresql powerbi

nitindatta8 / realtime-data-streaming
data-engineering-pipeline,End-to-end data engineering pipeline with various technologies to ingest real time data.
User: nitindatta8
apache-airflow apache-kafka apache-spark apache-zookeeper big-data cassandra data-engineering data-engineering-pipeline data-processing docker

piyushsingariya / shift
data-engineering-pipeline,Shift is a high performance better alternative to Airbyte, Singer, Meltano
User: piyushsingariya
golang data data-collection data-engineering data-engineering-pipeline data-ingestion elt etl-framework reverse-etl airbyte

pyprogrammerblog / tiny-blocks
data-engineering-pipeline,Tiny Blocks to build large and complex data pipelines!
User: pyprogrammerblog
Home Page: https://tiny-blocks.readthedocs.io/en/latest/
data etl-automation etl-framework pipelines python3 data-engineering data-engineering-pipeline etl generator pandas

salimt / spotify-api-pipeline
data-engineering-pipeline,Spotify API, Airflow, Docker, AWS S3, Snowflake, dbt, localstack, Looker Studio
User: salimt
api aws-s3 data-engineering data-engineering-pipeline dbt docker elt localstack snowflake looker

san089 / goodreads_etl_pipeline
data-engineering-pipeline,An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
User: san089
etl-pipeline etl-framework spark apache-spark apache-airflow airflow redshift emr-cluster livy s3

san089 / udacity-data-engineering-projects
data-engineering-pipeline,Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
User: san089
data data-engineering data-engineering-pipeline etl-pipeline cassandra-database postgresql-database data-modeling data-warehouse data-lake airflow airflow-operators cluster cassandra infrastructure postgres aws aws-ec2 aws-sdk aws-s3 cloudformation

sanjeevai / disaster-response-pipeline
data-engineering-pipeline,ETL pipeline combined with supervised learning and grid search to classify text messages sent during a disaster event
User: sanjeevai
data-engineering-pipeline disaster-event etl-pipeline grid-search-hyperparameters sqlite-database supervised-learning

shihwen / tpe-mrt-traffic-etl
data-engineering-pipeline,A data pipeline from source to data warehouse using Taipei Metro Hourly Traffic data
User: shihwen
airflow python data-engineering data-engineering-pipeline data-warehouse etl-job etl-pipeline redshift s3 taipei-metro

siddharth271101 / covid-19-and-aviation-industry
data-engineering-pipeline,The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technologies such as Apache Airflow, Apache Spark, Tableau and couple of AWS services
User: siddharth271101
analytics athena aws-s3 covid-19 dashboard docker emr opensky tableau airflow

tmaferreira / dataengineeringzoomcampproject
data-engineering-pipeline,Data Engineering ZoomCamp Course Project
User: tmaferreira
bigquery cloud-storage data-engineering data-engineering-pipeline data-visualization dbt prefect terraform data-lake python3

umairthakur / uber-data-analysis-etl-pipeline-data-analysis_project
data-engineering-pipeline,Uber Data Analysis Project, an End-to-End Data Engineering Project from creating data pipelines to finally creating the dashboard.
User: umairthakur
Home Page: https://lookerstudio.google.com/s/nQI06ax2wMY
bigquery data-engineering-pipeline data-science data-visualization google-cloud-platform looker-studio mage-data-pipeline-tool python3 sql

verazab / nyc-stats
data-engineering-pipeline,Analysis of 311 Service Requests for the City of NYC (from 2010 to 2023) Tech: Prefect cloud, dbt core, BigQuery, Compute Engine, CloudRun, Artifact Registry, Terraform, Docker
User: verazab
Home Page: https://lookerstudio.google.com/reporting/65ee32b0-4626-4a39-8065-5d8c27380a1a
gcp prefect terraform dbt looker-studio cloudrun-jobs compute-engine data-engineering-pipeline ci-cd docker

vmware / versatile-data-kit
data-engineering-pipeline,One framework to develop, deploy and operate data workflows with Python and SQL.
Organization: vmware
data-science data-engineering sql trino data-lineage etl elt data-pipelines data-engineer data-warehouse

yan-luo-au / data_engineer_project_etl_bi
data-engineering-pipeline,This is an ETL project - extracting data from an ecommerce transactional database on RDS, transforming the data using AWS glue job, and loading it to a Redshift data warehouse, and connected it to Tableau for BI
User: yan-luo-au
data-engineering-pipeline redshift rds-mssql glue datapipeline secret-manager

yashksaini-coder / python-for-data-engineering
data-engineering-pipeline, Data Engineering 🛠️ is like the backbone of data processing 📊, managing data pipelines 🚀, warehouses 🏢, and lakes 🌊. It's the bridge 🌉 between raw data and actionable insights, powering businesses 🚀 with efficient data management and analytics 📈.
User: yashksaini-coder
aws data-engineer data-engineering data-engineering-pipeline data-science kafka python

zarexalvindaria / data-engineering
data-engineering-pipeline,This repo contains the Data Engineering exercises I took in Datacamp.
User: zarexalvindaria
data-engineering data-engineering-pipeline python spark singer airflow airflow-dags