nnthanh101 / platform-engineering Goto Github PK

MLP 1. [Batch] Data Engineering End-to-End Project - PostgreSQL, Airflow, Pandas

1. Tech Stack

• Apache Airflow
• SQL & PostgreSQL
• Python & Pandas

2. 🚀 Deliverables:

• Reproducible Local Development with Dev Containers
• Architecture Diagram
• [Batch] Data Engineering End-to-End Project - PostgreSQL, Airflow, Pandas

References:

• https://medium.com/@dogukannulu/data-engineering-end-to-end-project-postgresql-airflow-docker-pandas-91c6aa529030
• https://github.com/dogukannulu/csv_extract_airflow_docker

• Cloud provisioning tool for building GitOps Kubernetes platforms

  • 1. Git provider: GitHub, GitLab
  • 2. Automate Infrastructure as Code: Terraform, Atlantis,Pulumi, CDK
  • 3. GitOps Driver: Argo workflow, OpenGitOps, Flux, GitLab
  • 4. Secrets Manager: Hashicorp Vault
  • 5. Artifact Repository: DockerHub, Github
  • 6. Continuous Integration Pipeline: Argo Workflows, GitHub Actions Runner Controller, GitLab Runner, Atlantis Terraform Automation
  • 7. Authentication and Authorization: principle of least privilege
  • 8. Observability: Prometheus for metrics, Grafana for dashboarding, Jaeger for tracing and application performance monitoring
  • 9. Local: Minikube, K3d

• 🚀 Reproducible Local Development with DevContainers: VSCode, K8s, TF, Python/R

References:

• https://github.com/vilaca/awesome-k8s-tools/blob/main/README.md

Docker Resources Allocation
The more resources you give Docker, the faster your cluster creation will go, but here are the minimum requirements:

• CPU: 6 Cores
• Memory (RAM): 6 GB
• Swap: 1 GB
• Virtual Disk limit (for Docker images & containers): 10 GB

MLP2. [Streaming] Data Engineering End-to-End Project - Spark, Kafka, Airflow, Cassandra, Python

1. Tech Stack

• Apache Cassandra
• Python & API
• Apache Airflow
• Apache Kafka
• Apache Spark

2. 🚀 Deliverables:

• Reproducible Local Development with Dev Containers
• Architecture Diagram
• [Streaming] Data Engineering End-to-End Project - Spark, Kafka, Airflow, Cassandra, Python

References:

• https://medium.com/@dogukannulu/data-engineering-end-to-end-project-1-7a7be2a3671
• https://github.com/dogukannulu/kafka_spark_structured_streaming

• https://github.com/awslabs/data-on-eks

• https://github.com/aws-samples/terraform-ssp-eks-patterns
• https://github.com/aws-samples/ssp-eks-patterns
• https://github.com/aws-samples/ssp-eks-workloads
• https://github.com/aws-samples/aws-eks-accelerator-for-terraform

• https://github.com/k8sgpt-ai/k8sgpt
• https://medium.com/@martin.hodges/how-to-create-a-full-featured-kubernetes-platform-for-running-micro-services-c145ab71a2eb
• [ ]

• https://github.com/aws-samples/aws-eks-accelerator-for-terraform

• https://github.com/aws-samples/terraform-ssp-eks-patterns
• https://github.com/aws-samples/ssp-eks-patterns
• https://github.com/aws-samples/ssp-eks-workloads