orcascope Goto Github PK

7-llm-driven-data-engineering

This is a public repository to go over all the LLM-driven data engineering concepts.

99-problems

This is an adaptation of the Ninety-Nine Prolog Problems written by Werner Hett.

adf_demo_meteostat

airflow_scripts_repo

Some Airflow scripts that can be referred to and reused

amazon-s3-step-functions-ingestion-orchestration

Design pattern for orchestrating an incremental data ingestion pipeline using AWS Step Functions from an on premise location into an Amazon S3 datalake bucket

apacheds-docker

Dockerfile to build an ApacheDS container providing an LDAP and optionally a Kerberos service.

arrow-spark-publication

Implementation connecting Arrow to Spark, effectively making all code related to reading in Spark redundant.

astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

azure-aks-kubernetes-masterclass

Azure AKS Kubernetes Masterclass

big-data-projects

This project has customization likes custom data sources, plugins written for the distributed systems like Apache Spark, Apache Ignite etc

calcite-examples

Some examples on how to use Apache Calcite

calcite-notebooks

:notebook: A series of Jupyter notebooks to demonstrate the functionality of Apache Calcite

chispa

PySpark test helper methods with beautiful error messages

cicd-pipeline-train-schedule-canary

Train Schedule sample app for Jenkins Pipelines CD with Kubernetes Canary exercises

cicd-pipeline-train-schedule-dockerdeploy

Train Schedule sample app for Jenkins Pipelines CD with Docker exercises

confluent_kafka_multi_queuing

Implementation of multi queuing using confluent kafka

consul-dns-srv

Helper class for Consul DNS SRV records

data-engineering-dimensional-modeling

data-engineering-dimensional-modeling

data-engineering-zoomcamp

Free Data Engineering course!

data-modeling-with-postgres

A project to design a fact and dimension star schema for optimizing queries on a flight booking database using PostgreSQL, a relational database management system. This schema is well-suited for a flight booking database, as it allows for efficient querying of data such as booking dates, flight routes, and passenger information.

data_engineering_databricks_pyspark

ETL, data warehouse creation and advanced analytics, data modelling

dbt-tutorial

delight

A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.

deploying-machine-learning-models

Example Repo for the Udemy Course "Deployment of Machine Learning Models"

docker-hadoop-hive-metastore-db

Docker image for Apache Hive Metastore Postgres Database

docker-spark-cluster

A simple spark standalone cluster for your testing environment purposses

drf_api

Django API with Django Rest Framework

drill

Apache Drill

dvc_parquet

show dvc capabilities in versioning parquet files

flask-confluent-kafka

Flask confluent-Kafka-python consumer example. extended from https://github.com/NimzyMaina/flask_kafka