Topic: etl-pipeline Goto Github

👇 Here are 1299 public repositories matching this topic...

airscholar / e2e-data-engineering
etl-pipeline,An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
User: airscholar
Home Page: https://www.youtube.com/watch?v=GqAcTrqKcrY
apache-airflow apache-kafka apache-spark apache-zookeeper big-data cassandra containerization data-engineering data-pipeline data-processing

alexioannides / pyspark-example-project
etl-pipeline,Implementing best practices for PySpark ETL jobs and applications.
User: alexioannides
pyspark etl-job python data-engineering spark data-science etl etl-pipeline

anmol12499 / reddit-analytics-integration-platform
etl-pipeline,Project was based on an interest in Data Engineering, ETL pipeline. It also provided a good opportunity to develop skills and experience in a range of tools. As such, project is more complex than required, utilising dbt, airflow, docker and cloud based storage.
User: anmol12499
airflow aws-redshift aws-s3 dbt docker etl-pipeline google-studio python terraform

apache / incubator-streampark
etl-pipeline,Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
Organization: apache
Home Page: https://streampark.apache.org/
streaming streampark apache development-framework easy-to-use etl-pipeline operation-platform

conductor-sdk / conductor-python
etl-pipeline,Conductor OSS SDK for Python programming language
Organization: conductor-sdk
python conductor workflow data-pipelines durable-computing durable-execution etl-pipeline

cyber-drop / ethereum_analytical_db
etl-pipeline, Ethereum Analytical Database - Ethereum data access solution that can be used for analytics and application development. The solution works on a fast DB - Clickhouse.
Organization: cyber-drop
Home Page: https://cyberdrop.ai
ethereum blockchain etl-pipeline etl clickhouse eth erc20 erc721 erc223 dex

dagworks-inc / hamilton
etl-pipeline,Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
Organization: dagworks-inc
Home Page: https://hamilton.dagworks.io/en/latest/
data-science python dag data-engineering dataframe etl etl-framework etl-pipeline feature-engineering featurization

danielsaban / data-scraping-sofascore
etl-pipeline,Data Engineering/Scraping Project. Creating a detailed Sports Relational Database for the Top European Soccer Leagues.
User: danielsaban
dataengineering datamining webscraping relational-databases parser etl-pipeline rest-api

dappboard / dappboard-etl
etl-pipeline,ETL pipeline for the Ethereum blockchain
Organization: dappboard
Home Page: http://dappboard.com
ethereum-blockchain blockchain etl etl-pipeline javascript

data-engineering-community / data-engineering-project-template
etl-pipeline,This is a template you can use for your next data engineering portfolio project.
Organization: data-engineering-community
Home Page: https://dataengineering.wiki
data data-engineering data-warehouse etl etl-pipeline python sql

flow-php / flow
etl-pipeline,Flow PHP - data processing framework
Organization: flow-php
Home Page: https://flow-php.com
etl etl-pipeline etl-framework

hifxit / dataligo
etl-pipeline,A library to accelerate ML and ETL pipeline by connecting all data sources
User: hifxit
database datalake datawarehouse etl-pipeline ml-pipeline nosql python

imsanjoykb / data-science-regular-bootcamp
etl-pipeline,Regular practice on Data Science, Machien Learning, Deep Learning, Solving ML Project problem, Analytical Issue. Regular boost up my knowledge. The goal is to help learner with learning resource on Data Science filed.
User: imsanjoykb
Home Page: https://imsanjoykb.github.io/
data-science machine-learning artificial-intelligence deep-learning neural-network data-science-projects data-visualization data-analysis data-science-notebook database-connection

indexical-metrics-measure-advisory / watchmen-matryoshka-doll
etl-pipeline,Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management
Organization: indexical-metrics-measure-advisory
Home Page: https://imma-watchmen.com/
visualization charts data-visualization data-pipeline etl-pipeline data-quality-monitoring pipeline

interestinglab / seatunnel-example
etl-pipeline,seatunnel plugin developing examples.
Organization: interestinglab
spark etl-framework etl-pipeline spark-streaming flink sql-engine waterdrop

jitsucom / bulker
etl-pipeline,Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)
Organization: jitsucom
Home Page: https://github.com/jitsucom/bulker
data-engineering datawarehouse etl etl-pipeline ingestion pipeline

josephmachado / bitcoinmonitor
etl-pipeline,Near real time ETL to populate a dashboard.
User: josephmachado
Home Page: https://www.startdataengineering.com/post/data-engineering-project-to-impress-hiring-managers/
python3 etl-pipeline metabase cron nearrealtime postgres docker docker-compose pytest

josephmachado / socialetl
etl-pipeline,Project for "Data pipeline design patterns" blog.
User: josephmachado
Home Page: https://www.startdataengineering.com/post/code-patterns/
dataengineering design-patterns etl-pipeline makefile python social-media-data sqllite3 reddit

jvalue / jayvee
etl-pipeline,Jayvee is a domain-specific language and runtime for automated processing of data pipelines
Organization: jvalue
Home Page: https://jvalue.github.io/jayvee/
data-engineering data-pipeline data-science domain-specific-language etl-pipeline typescript

martandsingh / apachespark
etl-pipeline,This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
User: martandsingh
apachespark data-analysis data-engineering database databricks datalake deltalake etl-pipeline hadoop hive

massstreetanalytics / etl-framework
etl-pipeline,A framework for moving data into a data warehouse.
Organization: massstreetanalytics
Home Page: http://www.datawarehouseetlframework.com/
etl python sqlserver sql etl-framework etl-pipeline etl-components data-warehouse

maxim2266 / csvplus
etl-pipeline,csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.
User: maxim2266
csv csv-format etl etl-framework etl-pipeline fluent-interface go go-csv stream-processing

mmodarre / azuredatafactoryhol
etl-pipeline,Azure Data Factory Hands On Lab - Step by Step - A Comprehensive Azure Data Factory and Mapping Data Flow step by step tutorial
User: mmodarre
azure-data-factory azure adf-pipeline filter-activity lookup-activity foreach-activity metadata-activity mapping-dataflows azure-key-vault etl-pipeline hands-on-lab hands-on-azure-data-factory azure-data-factory-tutorial azure-modern-data-warehous web-activity foreach-loop-activity

mobiletelesystems / onetl
etl-pipeline,One ETL tool to rule them all
Organization: mobiletelesystems
Home Page: https://onetl.readthedocs.io/
spark etl etl-components etl-pipeline hwm plugin-system pydantic

mohamedhmini / tweetsolaping
etl-pipeline,implementing an end-to-end tweets ETL/Analysis pipeline.
User: mohamedhmini
datawarehousing datawarehouse etl-pipeline tweets tweets-classification tweets-scraper twitter-api google-api-client api-client web-crawling

mycelial / mycelial
etl-pipeline,Move your data with ease.
Organization: mycelial
Home Page: https://mycelial.com
data-pipelines edge-computing etl etl-pipeline rust

orchest / orchest
etl-pipeline,Build data pipelines, the easy way 🛠️
Organization: orchest
Home Page: https://orchest.readthedocs.io/en/stable/
data-science machine-learning pipelines ide jupyter cloud self-hosted jupyterlab notebooks docker

patterns-app / patterns-devkit
etl-pipeline,Data pipelines from re-usable components
Organization: patterns-app
data-science data-analysis pipelines etl etl-pipeline etl-framework functional-reactive-programming data-engineering sql immutability

razevedo1994 / razv-data-engineering
etl-pipeline,Portfolio of projects and studies conducted in data engineering.
User: razevedo1994
etl-pipeline python3 etl data-warehouse data-lake data-modeling sql spark aws aws-lambda

redis-field-engineering / redis-connect-dist
etl-pipeline,Real-Time Event Streaming & Change Data Capture
Organization: redis-field-engineering
Home Page: https://redis-field-engineering.github.io/redis-connect-dist/
rediscdc cdc replication etl etl-pipeline etl-framework etl-automation csv redisconnect connect redis event-sourcing event-streaming

restarone / violet_rails
etl-pipeline,an app engine for your business. Seamlessly implement business logic with a powerful API. Out of the box CMS, blog, forum and email functionality. Developer friendly & easily extendable for your next SaaS/XaaS project. Built with Rails 6, Devise, Sidekiq & PostgreSQL
Organization: restarone
Home Page: https://violet.restarone.solutions
rubyonrails xaas saas template multitenancy violet-rails multi-tenancy forum blog saas-boilerplate

runprism / prism
etl-pipeline,Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.
Organization: runprism
Home Page: https://runprism.com
data data-engineering data-science etl etl-pipeline machine-learning orchestration pipeline python data-orc

san089 / goodreads_etl_pipeline
etl-pipeline,An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
User: san089
etl-pipeline etl-framework spark apache-spark apache-airflow airflow redshift emr-cluster livy s3

san089 / udacity-data-engineering-projects
etl-pipeline,Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
User: san089
data data-engineering data-engineering-pipeline etl-pipeline cassandra-database postgresql-database data-modeling data-warehouse data-lake airflow

setl-framework / setl
etl-pipeline,A simple Spark-powered ETL framework that just works 🍺
Organization: setl-framework
spark etl framework scala setl pipeline data-transformation data-science data-engineering data-analysis

souvik-databricks / dlt-with-debug
etl-pipeline,A lightweight helper utility which allows developers to do interactive pipeline development by having a unified source code for both DLT run and Non-DLT interactive notebook run.
User: souvik-databricks
Home Page: https://pypi.org/project/dlt-with-debug/
big-data big-data-processing databricks delta-live-tables dlt etl etl-pipeline python3 spark

stellar / stellar-etl
etl-pipeline,Stellar ETL will enable real-time analytics on the Stellar network
Organization: stellar
stellar stellar-network stellar-lumens etl-pipeline data-analysis etl-framework ethereum bitcoin blockchain

stellar / stellar-etl-airflow
etl-pipeline,Airflow DAGs for the Stellar ETL project
Organization: stellar
blockchain stellar-lumens etl-pipeline python stellar airflow stellar-network etl-framework data-analysis

stitchfix / hamilton
etl-pipeline,A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
Organization: stitchfix
Home Page: https://www.github.com/dagworks-inc/hamilton
python pandas dag data-science data-engineering numpy software-engineering etl-framework etl-pipeline etl

sundios / seo-dashboard
etl-pipeline,SEO dashboard from Search console Data using the Google Search API, Mysql database , NodeJS RESTAPI( ExpressJS) and reactJs Dashboard
User: sundios
Home Page: https://www.kwrds.ai
seo seotools etl-pipeline dashboard react seo-monitor rest-api google-search-console-python google-search-console mysql

techascent / tech.ml.dataset
etl-pipeline,A Clojure high performance data processing system
Organization: techascent
clojure dataframe csv xlsx datascience machine-learning dataset etl-pipeline java

tharwaninitin / etlflow
etl-pipeline,EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases, SFTP servers, On-Prem Systems and more.
User: tharwaninitin
aws bigquery dataproc etl etl-framework etl-pipeline gcp gcs redis s3 scala spark zio

toddbirchard / jira-database-etl
etl-pipeline,:mens: 💾 Script to import issues from a JIRA instance into a database.
User: toddbirchard
Home Page: https://hackersandslackers.com/jira-to-sql-etl/
jira-rest-api python3 pandas flask flask-sqlalchemy etl etl-pipeline

usc-isi-i2 / dig-etl-engine
etl-pipeline,Download DIG to run on your laptop or server.
Organization: usc-isi-i2
Home Page: http://usc-isi-i2.github.io/dig/
crawling information-extraction search-engine etl-framework etl-pipeline information-visualization

vim89 / datapipelines-essentials-python
etl-pipeline,Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
User: vim89
apache-spark spark spark-sql python python3 pyspark etl etl-pipeline etl-framework etl-components

visiologyofficial / vixtract
etl-pipeline,
Organization: visiologyofficial
Home Page: https://www.vixtract.ru
etl etl-pipeline etl-framework etl-automation etl-components etl-job

wathon / data_engineering_with_python-track-datacamp
etl-pipeline,Data Engineer with Python lecture notes from #datacamp.
User: wathon
data-engineering python datacamp-track shell sql scala aws-boto pyspark spark-sql mongodb

wittline / uber-expenses-tracking
etl-pipeline,The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.
User: wittline
Home Page: https://wittline.github.io/uber-expenses-tracking/
data-engineering python apache-airflow etl-pipeline aws-redshift power-bi data-modeling uber-eats uber-data expenses-dashboard

yotpoltd / metorikku
etl-pipeline,A simplified, lightweight ETL Framework based on Apache Spark
Organization: yotpoltd
Home Page: https://yotpoltd.github.io/metorikku/
big-data spark scala etl-framework distributed-computing sql etl etl-pipeline

zipstack / unstract
etl-pipeline,No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents
Organization: zipstack
Home Page: https://unstract.com
etl-pipeline llm-platform unstructured-data