Kedro Docker

Run a Kedro project in a Docker environment

Prerequisites

Docker
Kedro 0.16.6
Kedro-Docker 0.2.1
scikit-learn 0.23.0
pickle 0.0.11

Workflows

Read data from csv files and excel file as well, pre-process files and then save csv files
Split data and then save pickle files
Read pickle files, run train model and then save the regression model(pickle format)
Load the regression model and run Predict from the pickle model

Get data from S3

Set configs in the conf/*/credentials.yml

dev_s3:
    client_kwargs:
        aws_access_key_id: token
        aws_secret_access_key: key

Build

Setup Kedro Environment
Install Kedro Docker
```
pip install kedro-docker==0.2.1
```
Generate a Dockerfile
```
example$ kedro docker init
```
Build a Docker image
```
example$ kedro docker build
```
It will create a Docker image with example:latest name
Run a Kedro project in a Docker Environment
```
example$ kedro docker run
```

Usage (in Docker)with code from github(Don't install Kedro Environment)

Download code from github

Build a Docker image

$cd example
$docker build --tag=kedro-docker .

Run a docker image
```
$docker run -it kedro-docker bash
```
Run a Kedro project
```
$kedro run
```

Result

```
kedro@b792338c69b3:~$ kedro run
2020-12-28 07:27:43,250 - root - INFO - ** Kedro project kedro
#### Pipeline execution order ####
Inputs: companies, parameters, reviews, shuttles

preprocessing_companies
preprocessing_shuttles
master_table
split_data
train_model
predict

Outputs: None
##################################
fatal: not a git repository (or any of the parent directories): .git
2020-12-28 07:27:43,259 - kedro.versioning.journal -
WARNING - Unable to git describe /home/kedro
/usr/local/lib/python3.7/site-packages/fsspec/implementations/local.py:33: FutureWarning:
The default value of auto_mkdir=True has been deprecated and will be changed to
auto_mkdir=False by default in a future release.
FutureWarning,
2020-12-28 07:27:43,280 - kedro.io.data_catalog -
INFO - Loading data from `companies` (CSVDataSet)...
2020-12-28 07:27:43,338 - kedro.pipeline.node -
INFO - Running node: preprocessing_companies: preprocess_companies([companies]) -> [preprocessed_companies]
2020-12-28 07:27:43,406 - kedro.io.data_catalog -
INFO - Saving data to `preprocessed_companies` (CSVDataSet)...
2020-12-28 07:27:43,650 - kedro.runner.sequential_runner -
INFO - Completed 1 out of 6 tasks
2020-12-28 07:27:43,651 - kedro.io.data_catalog -
INFO - Loading data from `shuttles` (ExcelDataSet)...
2020-12-28 07:27:55,006 - kedro.pipeline.node -
INFO - Running node: preprocessing_shuttles: preprocess_shuttles([shuttles]) -> [preprocessed_shuttles]
2020-12-28 07:27:55,095 - kedro.io.data_catalog -
INFO - Saving data to `preprocessed_shuttles` (CSVDataSet)...
2020-12-28 07:27:55,487 - kedro.runner.sequential_runner -
INFO - Completed 2 out of 6 tasks
2020-12-28 07:27:55,487 - kedro.io.data_catalog -
INFO - Loading data from `preprocessed_shuttles` (CSVDataSet)...
2020-12-28 07:27:55,567 - kedro.io.data_catalog -
INFO - Loading data from `preprocessed_companies` (CSVDataSet)...
2020-12-28 07:27:55,603 - kedro.io.data_catalog -
INFO - Loading data from `reviews` (CSVDataSet)...
2020-12-28 07:27:55,682 - kedro.pipeline.node -
INFO - Running node: master_table:
create_master_table([preprocessed_companies,preprocessed_shuttles,reviews]) -> [master_table]
2020-12-28 07:27:59,069 - kedro.io.data_catalog -
INFO - Saving data to `master_table` (CSVDataSet)...
2020-12-28 07:28:07,982 - kedro.runner.sequential_runner - INFO - Completed 3 out of 6 tasks
2020-12-28 07:28:07,983 - kedro.io.data_catalog -
INFO - Loading data from `master_table` (CSVDataSet)...
2020-12-28 07:28:09,799 - kedro.io.data_catalog -
INFO - Loading data from `parameters` (MemoryDataSet)...
2020-12-28 07:28:09,800 - kedro.pipeline.node -
INFO - Running node: split_data: split_data([master_table,parameters]) -> [Xtest,Xtrain,Ytest,Ytrain]
2020-12-28 07:28:10,562 - kedro.io.data_catalog - INFO - Saving data to `Xtrain` (MemoryDataSet)...
2020-12-28 07:28:10,635 - kedro.io.data_catalog - INFO - Saving data to `Xtest` (MemoryDataSet)...
2020-12-28 07:28:10,645 - kedro.io.data_catalog - INFO - Saving data to `Ytrain` (MemoryDataSet)...
2020-12-28 07:28:10,646 - kedro.io.data_catalog - INFO - Saving data to `Ytest` (MemoryDataSet)...
2020-12-28 07:28:10,711 - kedro.runner.sequential_runner - INFO - Completed 4 out of 6 tasks
2020-12-28 07:28:10,711 - kedro.io.data_catalog -
INFO - Loading data from `Xtrain` (MemoryDataSet)...
2020-12-28 07:28:10,765 - kedro.io.data_catalog - INFO - Loading data from `Ytrain` (MemoryDataSet)...
2020-12-28 07:28:10,767 - kedro.pipeline.node -
INFO - Running node: train_model: train_model([Xtrain,Ytrain]) -> [regression_model]
2020-12-28 07:28:11,173 - kedro.io.data_catalog -
INFO - Saving data to `regression_model` (MemoryDataSet)...
2020-12-28 07:28:11,290 - kedro.runner.sequential_runner - INFO - Completed 5 out of 6 tasks
2020-12-28 07:28:11,290 - kedro.io.data_catalog - INFO - Loading data from `Xtest` (MemoryDataSet)...
2020-12-28 07:28:11,302 - kedro.io.data_catalog - INFO - Loading data from `Ytest` (MemoryDataSet)...
2020-12-28 07:28:11,303 - kedro.io.data_catalog -
INFO - Loading data from `regression_model` (MemoryDataSet)...
2020-12-28 07:28:11,304 - kedro.pipeline.node -
INFO - Running node: predict: predict([Xtest,Ytest,regression_model]) -> None
2020-12-28 07:28:11,367 - example.pipelines.data_science.nodes -
INFO - Model has a coefficient R^2 of 0.456.
2020-12-28 07:28:11,411 - kedro.runner.sequential_runner -
INFO - Completed 6 out of 6 tasks
```

Issues

Could not load Excel Data Set

kedro.io.core.DataSetError: Failed while loading data from data set ExcelDataSet
(filepath=/home/kedro/data/01_raw/shuttles.xlsx,
load_args={'engine': xlrd}, protocol=file, save_args={'index': False},
writer_args={'engine': xlsxwriter}).
Excel xlsx file; not supported

Fixed: xlrd==1.2.0

Load Data from AWS S3

server_1  |     ds_name, ds_config, load_versions.get(ds_name), save_version
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/kedro/io/core.py",
line 185, in from_config
webserver_1  |     ) from err
webserver_1  | kedro.io.core.DataSetError:
webserver_1  | get_session() got an unexpected keyword argument 'aws_access_key_id'.
webserver_1  | DataSet 'companies' must only contain arguments valid for the co

File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
File "/home/kedro/src/example/hooks.py", line 78, in register_catalog
    catalog, credentials, load_versions, save_version, journal
File "/usr/local/lib/python3.7/site-packages/kedro/io/data_catalog.py", line 328, in from_config
    ds_name, ds_config, load_versions.get(ds_name), save_version
File "/usr/local/lib/python3.7/site-packages/kedro/io/core.py", line 185, in from_config
    ) from err
kedro.io.core.DataSetError:
create_client() got multiple values for keyword argument 'aws_access_key_id'.
DataSet 'companies' must only contain arguments valid for
the constructor of `kedro.extras.datasets.pandas.csv_dataset.CSVDataSet`.

Fixed: install s3fs==0.4.0 and update credentials.yml https://discourse.kedro.community/t/how-do-i-pass-s3-credentials-to-my-datasets/156

dev_s3:
    client_kwargs:
        aws_access_key_id: access_key
        aws_secret_access_key: secret_key

nhatthaiquang-agilityio / kedro-docker Goto Github PK

kedro-docker's Introduction

Kedro Docker

Prerequisites

Workflows

Get data from S3

Build

Usage (in Docker)with code from github(Don't install Kedro Environment)

Result

Issues

Reference

kedro-docker's People

Contributors

Watchers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent