Python-Devops, gathers Python stack for DevOps, these are usually my basic templates use for my implementations, so, feel free to use it and evolve it! Everything is Docker!
Table of contents
- Basic Python
- Basic Backend
- Scaling-up Backend
- Big data piping
- Apache big data
- Unit test
- Stress test
- Miscellaneous
Basic Python
- AutoPEP8
- Graph function dependencies
Basic Backend
- Flask
- Flask with MongoDB
- REST API Flask
- Flask Redis PubSub
- Flask Mysql with REST API
- Flask Postgres with REST API
- Flask Elasticsearch
- Flask Logstash with Gunicorn
- MLFlow with Nginx reversed proxy
Scaling Backend
- Flask SocketIO with Redis
- Multiple Flask with Nginx Loadbalancer
- Multiple Flask SocketIO with Nginx Loadbalancer
- RabbitMQ and multiple Celery with Flask
- Flask + Gunicorn + HAproxy
Big data piping
- Streaming Tweepy to Elasticsearch
- Scheduled crawler using Luigi Spotify to Elasticsearch
- Airflow to Elasticsearch
Apache big data
- Flask with Hadoop
- Flask with Kafka
- Flask with Hadoop Hive
- PySpark with Jupyter and Hadoop
- Flink with Jupyter (not working)
- Apache Storm with Redis
Unit test
- Pytest
Stress test
- Locust
Miscellaneous (huseinhouse collection)
- Elasticsearch + Kibana
- Elasticsearch + Cerebro
- Jupyter notebook
- Jupyterhub
- Jupyterhub + Github Auth
- Hadoop cluster + Luigi + Jupyter Notebook
- Kafka cluster
- Apache Storm
How-to Docker
Every folders contain .yml for docker-compose. You need to install Docker-Compose first.
To run,
compose/build
To safely close
compose/down
To remove all images
docker rmi $(docker images -q)
To remove none
images
docker rmi $(docker images -f "dangling=true" -q)
To remove all containers
docker rm $(docker ps -aq)