Topic: data-pipeline Goto Github

👇 Here are 601 public repositories matching this topic...

adilkhash / data-engineering-howto
data-pipeline,A list of useful resources to learn Data Engineering from scratch
User: adilkhash
distributed-systems data-engineering data-pipeline cloud-providers scala

aeksco / aws-pdf-textract-pipeline
data-pipeline,:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript
User: aeksco
aws textract lambda dynamodb sns typescript pdf serverless webscraping s3

agnostiqhq / covalent
data-pipeline,Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
Organization: agnostiqhq
Home Page: https://www.covalent.xyz
hpc workflow workflow-automation workflow-management quantum-computing hpc-applications python machine-learning pipelines covalent

airbytehq / airbyte
data-pipeline,The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Organization: airbytehq
Home Page: https://airbyte.com
data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration

airscholar / e2e-data-engineering
data-pipeline,An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
User: airscholar
Home Page: https://www.youtube.com/watch?v=GqAcTrqKcrY
apache-airflow apache-kafka apache-spark apache-zookeeper big-data cassandra containerization data-engineering data-pipeline data-processing

alexcasalboni / serverless-data-pipeline-sam
data-pipeline,Serverless Data Pipeline powered by Kinesis Firehose, API Gateway, Lambda, S3, and Athena
User: alexcasalboni
aws aws-lambda aws-s3 cloudformation aws-sam data-pipeline amazon-web-services

apache / flink-cdc
data-pipeline,Flink CDC is a streaming data integration tool
Organization: apache
Home Page: https://nightlies.apache.org/flink/flink-cdc-docs-stable
batch cdc change-data-capture data-integration data-pipeline distributed elt etl flink kafka mysql paimon postgresql real-time schema-evolution

apache / seatunnel-web
data-pipeline,SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).
Organization: apache
Home Page: https://seatunnel.apache.org/
apache data-integration data-pipeline etl-framework high-performance offline real-time seatunnel sql-engine

bruin-data / ingestr
data-pipeline,ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
Organization: bruin-data
Home Page: https://bruin-data.github.io/ingestr/
bigquery copy-database data-ingestion data-integration data-pipeline duckdb ingestion-pipeline mssql postgresql snowflake

bytedance / bitsail
data-pipeline,BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
Organization: bytedance
Home Page: https://bytedance.github.io/bitsail/
flink big-data data-integration data-lake data-pipeline data-synchronization high-performance real-time

conduitio / conduit
data-pipeline,Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
Organization: conduitio
Home Page: https://conduit.io
go data-integration etl data-pipeline data-engineering data-stream conduit kafka kafkaconnect

confluentinc / learn-kafka-courses
data-pipeline,Learn the basics of Apache Kafka® from leaders in the Kafka community with these video courses covering the Kafka ecosystem and hands-on exercises.
Organization: confluentinc
Home Page: https://developer.confluent.io/
apache-kafka kafka kafka-streams data-pipelines data-pipeline ksqldb stream-processing

cuebook / cuelake
data-pipeline,Use SQL to build ELT pipelines on a data lakehouse.
Organization: cuebook
Home Page: https://cuelake.cuebook.ai
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion

damklis / dataengineeringproject
data-pipeline,Example end to end data engineering project.
User: damklis
big-data scraping mongodb elasticsearch data-engineering kafka kafka-connect debezium django-rest-framework redis

dataflint / spark
data-pipeline,Performance Observability for Apache Spark
Organization: dataflint
apache-spark big-data observability optimization data-pipeline data-pipelines databricks dataproc emr etl

digitalghost-dev / premier-league
data-pipeline,A Data Engineering project. Repository for backend infrastructure and Streamlit app files for a Premier League Dashboard.
User: digitalghost-dev
Home Page: https://streamlit.digitalghost.dev/
python go google-cloud streamlit data-visualization bigquery cloud-run docker data-engineer data-pipeline

elementary-data / elementary
data-pipeline,The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
Organization: elementary-data
Home Page: https://www.elementary-data.com/
data-lineage data-governance data-warehouse snowflake bigquery data-analysis data-pipelines data-pipeline lineage data-reliability

feldera / feldera
data-pipeline,Feldera Continuous Analytics Platform
Organization: feldera
Home Page: https://feldera.com
analytics continous data-analysis data-pipeline database etl kafka materialized-view rust sql

googlecloudplatform / data-science-on-gcp
data-pipeline,Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Organization: googlecloudplatform
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering

indexical-metrics-measure-advisory / watchmen-matryoshka-doll
data-pipeline,Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management
Organization: indexical-metrics-measure-advisory
Home Page: https://imma-watchmen.com/
visualization charts data-visualization data-pipeline etl-pipeline data-quality-monitoring pipeline

infoslack / awesome-kafka
data-pipeline,A list about Apache Kafka
User: infoslack
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing

infuseai / piperider
data-pipeline,Code review for data in dbt
Organization: infuseai
Home Page: https://www.piperider.io/
data-pipeline data-profiling data-quality data-science data-exploration eda exploratory-data-analysis data-testing python data-observability

jvalue / jayvee
data-pipeline,Jayvee is a domain-specific language and runtime for automated processing of data pipelines
Organization: jvalue
Home Page: https://jvalue.github.io/jayvee/
data-engineering data-pipeline data-science domain-specific-language etl-pipeline typescript

kestra-io / kestra
data-pipeline,Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
Organization: kestra-io
Home Page: https://kestra.io
workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration

marcosmarxm / airflow-testing-ci-workflow
data-pipeline,(project & tutorial) dag pipeline tests + ci/cd setup
User: marcosmarxm
Home Page: https://blog.magrathealabs.com/how-to-develop-data-pipeline-in-airflow-through-tdd-test-driven-development-c3333439f358
project airflow-testing airflow testing airflow-cicd tdd data-pipeline data-engineering

memphisdev / memphis
data-pipeline,Memphis.dev is a highly scalable and effortless data streaming platform
Organization: memphisdev
Home Page: https://memphis.dev
data data-stream-processing data-streaming kubernetes messaging-queue data-engineering data-pipeline golang enrichment message-broker

msamogh / nonechucks
data-pipeline,Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
User: msamogh
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch

multiwoven / multiwoven
data-pipeline,🔥🔥🔥 Open Source Hightouch, Census, and RudderStack Alternative
Organization: multiwoven
Home Page: https://multiwoven.com
data-engineering reverse-etl data-pipeline data-activation etl react ruby self-hosted open-source dbt

olirice / flupy
data-pipeline,Fluent data pipelines for python and your shell
User: olirice
python data-pipeline fluent collections

openbridge / ob_bulkstash
data-pipeline,Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.
Organization: openbridge
Home Page: https://www.openbridge.com
rclone data-pipeline amazon-web-services google-cloud sftp-synchronisation docker s3 oracle-cloud google-cloud-storage storage-service

patterns-app / patterns-devkit
data-pipeline,Data pipelines from re-usable components
Organization: patterns-app
data-science data-analysis pipelines etl etl-pipeline etl-framework functional-reactive-programming data-engineering sql immutability

pipeline-tools / gusty
data-pipeline,Making DAG construction easier
Organization: pipeline-tools
Home Page: https://pipeline-tools.github.io/gusty-docs/
airflow data-pipeline data-etl

pydoit / doit
data-pipeline,task management & automation tool
Organization: pydoit
Home Page: http://pydoit.org
python build-tool build-automation task-runner build-system workflow-management data-pipeline workflow workflow-automation data-science

reugn / go-streams
data-pipeline,A lightweight stream processing library for Go
User: reugn
Home Page: https://pkg.go.dev/github.com/reugn/go-streams
stream-processing pipeline etl kafka data-stream kafka-streams streams redis pulsar data-pipeline

rudderlabs / rudder-server
data-pipeline,Privacy and Security focused Segment-alternative, in Golang and React
Organization: rudderlabs
Home Page: https://www.rudderstack.com/
golang hybrid-cloud privacy security warehouse-management data-warehouse rudderstack customer-data customer-data-pipeline customer-data-platform

scicloj / scicloj.ml
data-pipeline,A Clojure machine learning library
Organization: scicloj
machine-learning data-science data-pipeline hyperparameter-optimization classification regression scicloj nlp clojure clustering

snowplow / snowplow
data-pipeline,The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP
Organization: snowplow
Home Page: http://snowplowanalytics.com
snowplow analytics data data-pipeline data-collection product-analytics marketing-analytics snowplow-pipeline snowplow-events

sparkfish / augraphy
data-pipeline,Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes
Organization: sparkfish
Home Page: https://github.com/sparkfish/augraphy
data-augmentation crappification deep-neural-networks training-data machine-learning data-pipeline image-processing augmentation-pipeline synthetic-data synthetic-dataset-generation

spotify / klio
data-pipeline,Smarter data pipelines for audio.
Organization: spotify
Home Page: https://docs.klio.io
audio-processing data-pipeline signal-processing media-processing

sspaeti-com / practical-data-engineering
data-pipeline,Practical Data Engineering: A Hands-On Real-Estate Project Guide
Organization: sspaeti-com
Home Page: https://ssp.sh/blog/data-engineering-project-in-twenty-minutes
dagster data-engineering data-pipeline

streamlet-dev / tributary
data-pipeline,Streaming reactive and dataflow graphs in Python
Organization: streamlet-dev
python python3 stream data-pipeline streaming asynchronous reactive-data-streams python-data-streams kafka websockets

taogeyt / pyetl
data-pipeline,python ETL framework
User: taogeyt
python etl es hive export csv excel mysql sqlserver db

thedataengineeringbook / thedataengineeringbook
data-pipeline,The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
Organization: thedataengineeringbook
Home Page: https://thedataengineeringbook.online
data-engineering data hacktoberfest book data-engineer data-pipeline data-integration data-ingestion data-infrastructure

tvdboom / atom
data-pipeline,Automated Tool for Optimized Modelling
User: tvdboom
Home Page: https://tvdboom.github.io/ATOM/
machine-learning data-pipeline modelling data-exploration data-science python visualization automl dagshub interactive-visualizations

ubisoft / mobydq
data-pipeline,:whale: Tool to automate data quality checks on data pipelines
Organization: ubisoft
Home Page: https://ubisoft.github.io/mobydq/
data-quality data-pipeline data-warehouse big-data data-quality-checks data-quality-monitoring

unnati-xyz / scalable-data-science-platform
data-pipeline,Content for architecting a data science platform for products using Luigi, Spark & Flask.
Organization: unnati-xyz
Home Page: http://www.unnati.xyz
data-engineer data-pipeline data-science luigi machine-learning rest-api spark

vincentclaes / datajob
data-pipeline,Build and deploy a serverless data pipeline on AWS with no effort.
User: vincentclaes
Home Page: https://pypi.org/project/datajob/
aws-cdk data-pipeline stepfunctions glue glue-job sagemaker aws pipeline serverless machine-learning

whylabs / whylogs
data-pipeline,An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈
Organization: whylabs
Home Page: https://whylogs.readthedocs.io/
ai-pipelines approximate-statistics statistical-properties data-quality calculate-statistics python logging mlops dataops ml-pipelines

wybiral / hookah
data-pipeline,A cross-platform tool for data pipelines.
User: wybiral
golang pipeline tcp http unix-domain-socket fan-out fan-in firehose websockets pipe

yc9701 / pansori
data-pipeline,Tools for ASR Corpus Generation from Online Video
User: yc9701
speech-recognition speech-corpus corpus dataset-generation data-pipeline online-video