latest commits recreated volumes? about mlflow-docker-compose HOT 4 CLOSED

The reason why mc was required is to automate the creation the mlflow bucket that will be used as the artifact store for MLflow when the MinIO service is first initialised. The link you shared is to deploy a distributed MinIO stack and will not solve this issue.

I also like the idea to be able to just git clone this repo and immediately run a docker-compose up to get a fully working MLflow stack. If we remove MinIO from the services it would no longer be a one-command deployment, as we would need to separately deploy a MinIO and then set-up MLflow to use it as an object store.

However, I have changed the mc service to use a wait-for-it.sh script to check if MinIO is ready before creating the bucket, instead of relying on the nc which was removed in the newer versions of mc images, so it no longer depends on an older image!

Doing a search in GitHub I found a few repos such as https://github.com/ymym3412/mlflow-docker-compose and https://github.com/Toumash/mlflow-docker, but I have not used them personally. You can check them out if you're interested. Of course, I will still try to support this repo to the best of my ability.

from mlflow-docker-compose.

hi @mathematicalmichael, this project was originally created for a very simple proof of concept of a working MLFlow server using docker-compose. Therefore the mc command was supposed to recreate the mlfow bucket when the docker-compose service was restarted.

I noticed the mc service was not running as intended to automatically create the mlflow bucket in minio for mlflow to store its objects, and updated the image to a older version that was able to run the nc command properly. I suspect that your original issue of requiring a newer minio version was probably linked to the previous newer version of the mc image used, and that the mc service was never running properly in the first place. I have removed the command that removes the mlflow bucket in 629fe84

However, the files should have been stored in the minio_data volume. I have just tested that the minio_data volume was indeed created and the files were stored there. Can you run a docker volume ls to confirm that the volumes were created? If the files were not stored in the minio_data volume mount the files will not be moved to the new container when your container changes, which would seem like your files have "disappeared".

from mlflow-docker-compose.

now they are, but at the time that I originally set them up, it was storing everything in overlay storage, not in the volume.

I suppose the first time I spun them up, the command was missing, so when it finally started working, it deleted everything. that could be it. but there were non-mlflow buckets that disappeared as well... something very odd happened with the docker volume indeed.

the PR I submitted a while back was what I was running for months. Then I upgraded, restarted the service, and all my data was gone. Irreversibly. I restored from backup the night before.

I'll be more careful validating, but this time I could indeed SEE data created within the volume when writing to the buckets, so I hope this is "good to go" from here on out. I'll keep trying to update versions and submitting PRs here, because I don't see another more-popular repo that deploys this same stack. If you know of any, please point me to them.

from mlflow-docker-compose.

"mlflow docker compose" yields this repo really high up, most recent popular result.

It is strange that mc in the stack depends on an image that is so old, that feels wrong to have it frozen in time.

I suppose one option is to "extend" the official minio stack https://docs.min.io/docs/deploy-minio-on-docker-compose.html somehow, or just run a stack that interacts with it and only handles mlflow.

from mlflow-docker-compose.