You need to have AWS EC2 Linux instance, and set up Jupyter Lab and Airflow there. We also need AWS RDS to store data. It's stressful to work on command line and configuration, so I tried to describe in detail each process and what command you need to type in documentation. Please check "setup_document.pdf" in this repository.
Problem
Data analysis requires pipeline of getting data, cleaning data, training model, making prediction, visualizing data, reporting and repeat. We want to automate an entire process and be productive.
Airflow allows you to only use python to specify what tasks you want to automate, when to be executed and set dependency for each taks. The example is below.
Airflow also has a nice user interface, so it's easy to tell what automation is currently running and their status.
Even though you have multiple files to execute, the next task can wait for the previous task to complete its task in the automation like below.
Process
Set up EC2 for always running server and RDS to store data.
Jupyter Notebook for analysis and visualization in Jupyter Lab.
TensorFlow to train neural network to automate developing model and model update.
Papermill to execute all the Notebooks.
Airflow to set dependency between the Notebooks and set time schedule.
Email python package to report out.
Our goal is to train neural network regression every day, make forecast, and send emails to check whether our forecast is good.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent