sachua / mlflow-docker-compose Goto Github PK

View Code? Open in Web Editor NEW

138.0 4.0 54.0 152 KB

MLflow deployment with 1 command

License: Apache License 2.0

Dockerfile 2.30% Shell 97.70%

mlflow-docker-compose's Introduction

MLflow On-Premise Deployment using Docker Compose

Easily deploy an MLflow tracking server with 1 command.

MinIO S3 is used as the artifact store and MySQL server is used as the backend store.

How to run

Clone (download) this repository

git clone https://github.com/sachua/mlflow-docker-compose.git

cd into the mlflow-docker-compose directory
Build and run the containers with docker-compose
```
docker-compose up -d --build
```
Access MLflow UI with http://localhost:5000
Access MinIO UI with http://localhost:9000

Containerization

The MLflow tracking server is composed of 4 docker containers:

MLflow server
MinIO object storage server
MySQL database server

Example

Install conda
Install MLflow with extra dependencies, including scikit-learn
```
pip install mlflow boto3
```

Set environmental variables

export MLFLOW_TRACKING_URI=http://localhost:5000
export MLFLOW_S3_ENDPOINT_URL=http://localhost:9000

Set MinIO credentials

cat <<EOF > ~/.aws/credentials
[default]
aws_access_key_id=minio
aws_secret_access_key=minio123
EOF

Train a sample MLflow model

mlflow run https://github.com/sachua/mlflow-example.git -P alpha=0.42

Serve the model (replace ${MODEL_ID} with your model's ID)

export MODEL_ID=0ced24069348417fbbcb2cd41a7d2f07 # Replace this with your model's ID
mlflow models serve -m runs:/${MODEL_ID}/model -p 1234 --env-manager conda

You can check the input with this command

curl -X POST -H "Content-Type:application/json" --data '{"dataframe_split":{"columns":["fixed acidity", "volatile acidity", "citric acid", "residual sugar", "chlorides", "free sulfur dioxide", "total sulfur dioxide", "density", "pH", "sulphates", "alcohol"],"data":[[6.2, 0.66, 0.48, 1.2, 0.029, 29, 75, 0.98, 3.33, 0.39, 12.8]]}}' http://127.0.0.1:1234/invocations

mlflow-docker-compose's People

Contributors

Stargazers

Watchers

Forkers

dennislwm mathematicalmichael chorus12 jr-1991 mchrys ml-starter-packs schanne-job ybz21 danysolism anhquan075 optible tingiskhan gage-russell timtam94 jschuurmans annihilater ltoscano bmwas phamthanhtu310702 werayootk hnt2601 siddhi47 gawdemahesh munirabobaker wkangei abeusher ytl-liva ooktsl mshajarrazip enate ganbayard niyazed diegoquintanav tryolabs callmemsl lheudron hiseulgi drejom myadryshnikova comnguy trieule0801 carlsvelz toggrcpbayindir mtaboun pierregode chuckconway williamtran29 nickosipov jlcan altamiracorp gerritgeeraerts jorisganne

mlflow-docker-compose's Issues

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Update minio/mc Docker digest to 3adae97
Update minio/minio Docker digest to 2c8ac2a
Update dependency boto3 to v1.34.129
Update dependency cryptography to v42.0.8
Update dependency mlflow to v2.14.0
Update python Docker tag to v3.11
Click on this checkbox to rebase all open PRs at once

Detected dependencies

docker-compose

docker-compose.yml

minio/minio sha256:7b53a7bc5ca0b19350c5b852ffc399f7ad3be5020abd0303617a69423f2d926b

minio/mc sha256:92b495576197e48e9c7f864aac6aaa64b7eddf17db4d2e7c305bd69ca526848e

mysql/mysql-server sha256:5b40d96b11333570143d98d3a74100fefadb9abb17b27a95dbc9ad33544ec142

dockerfile

mlflow/Dockerfile

python 3.10-slim-buster

pip_requirements

mlflow/requirements.txt

cryptography ==42.0.7

boto3 ==1.34.117

mlflow ==2.13.1

pymysql ==1.1.1

Check this box to trigger a request for Renovate to run again on this repository

Traceback (most recent call last):
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/boto3/s3/transfer.py", line 292, in upload_file
future.result()
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/s3transfer/futures.py", line 103, in result
return self._coordinator.result()
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/s3transfer/futures.py", line 266, in result
raise self._exception
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/s3transfer/tasks.py", line 139, in call
return self._execute_main(kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/s3transfer/tasks.py", line 162, in _execute_main
return_value = self._main(**kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/s3transfer/upload.py", line 758, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/botocore/client.py", line 964, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The Access Key Id you provided does not exist in our records.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 70, in
mlflow.sklearn.log_model(lr, "model")
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/mlflow/sklearn/init.py", line 436, in log_model
return Model.log(
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/mlflow/models/model.py", line 563, in log
mlflow.tracking.fluent.log_artifacts(local_path, mlflow_model.artifact_path)
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/mlflow/tracking/fluent.py", line 903, in log_artifacts
MlflowClient().log_artifacts(run_id, local_dir, artifact_path)
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/mlflow/tracking/client.py", line 1137, in log_artifacts
self._tracking_client.log_artifacts(run_id, local_dir, artifact_path)
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 465, in log_artifacts
self._get_artifact_repo(run_id).log_artifacts(local_dir, artifact_path)
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/mlflow/store/artifact/s3_artifact_repo.py", line 170, in log_artifacts
self._upload_file(
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/mlflow/store/artifact/s3_artifact_repo.py", line 146, in _upload_file
s3_client.upload_file(Filename=local_file, Bucket=bucket, Key=key, ExtraArgs=extra_args)
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/boto3/s3/inject.py", line 143, in upload_file
return transfer.upload_file(
File "/opt/homebrew/Caskroom/miniforge/base/envs/mlflow/lib/python3.8/site-packages/boto3/s3/transfer.py", line 298, in upload_file
raise S3UploadFailedError(
boto3.exceptions.S3UploadFailedError: Failed to upload /var/folders/c0/z_mclgbd4xv322t8wdpyptz80000gp/T/tmpkg7o2um3/model/python_env.yaml to mlflow/1/6e54103d2d9849669c5eaa88033390ab/artifacts/model/python_env.yaml: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The Access Key Id you provided does not exist in our records.

Cannot use mlflow.log_artifacts to upload to mlflow tracking server

I've deployed a mlflow tracking server by using this repo. This works like a charm but when I use mlflow.log_artifacts to upload to the tracking servers, I faced this issue. Can anyone fix this bug, please?

model deployment

Hello,

FIrst, of all I want to say that this a great repo. Thank you for making it very easy to use.

I wanted to say that themlflow models serve -m S3://mlflow/0/98bdf6ec158145908af39f86156c347f/artifacts/model -p 1234 didn't work for me unless I added --env-manager conda.

Also, I had to downgrade my python on the new env created by mlflow run https://github.com/mlflow/mlflow-example.git -P alpha=0.42.

Lastly, I couldn't get the curl request to work. It says that the scoring protocol has changed but I can't figure out what has it changed to. And I couldn't find a swagger or anything so not really sure what to do for the example one.

Thank you again for this. Great work

try creating the credentials file in ~/.aws

cat > ~/.aws/credentials <<EOF
[default]
aws_access_key_id=minio
aws_secret_access_key=minio123
EOF

you can see where boto3 tries to find the credentials here: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html

Originally posted by @sachua in #5 (comment)

the bucket is not created on docker-compose up command

Even if it is display that the bucket should be created there are nothing on the server

Question on MLflow setup

Hi SongAnn,

Thank you for the wonderful mlflow-docker-compose repo. It is exactly what I am looking for.

Question:

With reference to Example(4), it seems the minio server to s3 bucket on AWS. May I know how I should setup if I just wanted to store the artifacts on localhost file directory?

Look forward to your reply soon.

Regards,
James

bash command in the readme ot working

cat > /.aws/credentials <<EOF [default] aws_access_key_id=minio aws_secret_access_key=minio123 EOF

it is not even an executable command, We are staying stuck in editing mode

something is missing

wrong absolute path in docker-compose.yml

In order to start mysql, the volume "dbdata" needs to be linked to **./**dbdata and not only dbdata
The dot is important.

If not, we get this error:

mlflow_server  | 2023/01/16 12:12:33 WARNING mlflow.store.db.utils: SQLAlchemy engine could not be created. The following exception is caught.
mlflow_server  | (pymysql.err.OperationalError) (1045, "Access denied for user 'mlflow_user'@'172.30.0.5' (using password: YES)"

A working setting :

    db:
        restart: always
        image: mysql/mysql-server@sha256:fcbe88694872e88ae406bc69540211505eae922a182690d85be6af1a48e5ca0a
        container_name: mlflow_db
        ports:
            - "3306:3306"
        environment:
            - MYSQL_DATABASE=${MYSQL_DATABASE}
            - MYSQL_USER=${MYSQL_USER}
            - MYSQL_PASSWORD=${MYSQL_PASSWORD}
            - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
        volumes:
            - ./dbdata:/var/lib/mysql

Jupyter notebook fails storing artefacts

Hi Jpyter notebook fails on saving artefacts but script running the terminal works fine. The error says An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records. Do you know how to fix this please?
Thanks

version update request

Can we please get the latest versions of minio/mysql used in the docker compose file. Thanks.

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: Invalid JSON (parsing failed)
Message: Syntax error near }, ] }

latest commits recreated volumes?

what happened with the last few commits, exactly? I noticed you updated an image to address "command not found" but I think in doing so all the data in my volume was wiped out. i lost over an hour trying to find it and eventually couldn't (and noticed the disk shrank), had to get it from a daily snapshot (thank goodness for that).

is this an issue with my filesystem / docker somehow? or are there commands in the docker compose file that somehow could have removed the data?

I'm noticing the files were in docker/overlay2/l/ instead of docker/volumes/ ... that makes me suspect that my previous deployment was never writing to the minio_data volume at all but rather to the overlay filesystem inside the container, which is why creating a new container made the files disappear. that would explain why upgrading made it look like all my files disappeared...the container changed.