Comments (17)
Im gonna see whats the problem tomorrow. In the meantime please post your docker-compose with caddyserver setup.
Maybe thats a opportunity to update official minio docs :)
from mlflow-docker.
Did you try using the official way of using minio + caddy from minio docs?
Whats the problem exactly? Which part does not work (artifacts upload from cli/python or download from mlflow ui?). Could you post some diagram which represents the problematic situation? How did you try to diagnose it?
About the idea of showing ready to launch service with authentication
Yeah, thats basically what the readme says (so the authentication should be baked into the project)... I need to think about adding the authentication, but in a way that will be super modular, and wont require big changes in the project to change the way its distributed.
Something like a environment variable switch in the dockercompose that will run the reverse proxy with basic auth by default/swtich - depends really how to approach it with the yt video (its the main way to discover this repository, so the demo should work)
from mlflow-docker.
Hi Toumash, thanks for answering!
The official way of using minio+caddy that you linked, is outdated, sadly. It is written for Caddyserver v.1.
When I'm using a Caddy-file with the following content:
my-url.com
reverse_proxy s3:9000 #"s3" is the MinIO-docker-service
opening my-url.com in the browser redirects me to the MinIO-console-UI at port 9001, but the connection is is marked "not secure" in the address-bar. When I replace 's3:9000 ' with 'mlflow:5000' for the MLflow-tracking, everything works fine and the connection is secure.
I'm wondering whether maybe the connection to port 9000 is actually secure and only the MinIO-redirect to 9001 opens a new, unsafe connection...
About the idea: I don't know if you are familiar with Caddyserver, but apart from MinIO it is very easy to get a secure basic authentication up and running! So if you think this would be well-received on youtube, you should go for it :)
from mlflow-docker.
Caddyfile (the 'basicauth' expression can be left out):
my-url.com {
redir /s3 /s3/
handle_path /s3/* {
reverse_proxy s3:9000
}
redir /mlflow /mlflow/
basicauth /mlflow/* {
<user_name> <encrypted_hash> # hid private information here
}
handle_path /mlflow/* {
reverse_proxy mlflow:5000
}
handle {
reverse_proxy ui:8051 # unrelated service from another docker project
}
}
docker-compose for Caddy:
version: "3"
services:
caddy:
image: caddy:2-alpine
container_name: caddy
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- /caddy/data:/data
- /caddy/config:/config
ports:
- 80:80
- 443:443
restart: unless-stopped
networks:
default:
name: proxy-net
docker-compose for MLflow (I use the one from this project, just added the "proxy-net"-network to make the services accessible to Caddy):
version: '3.2'
services:
s3:
restart: always
image: minio/minio:latest
container_name: aws-s3
ports:
- "9000:9000"
- "9001:9001"
environment:
- MINIO_ACCESS_KEY=${AWS_ACCESS_KEY_ID}
- MINIO_SECRET_KEY=${AWS_SECRET_ACCESS_KEY}
command:
server /date --console-address ":9001"
networks:
- default
- proxy-net
volumes:
- ./s3:/date
db:
restart: always
image: mysql/mysql-server:5.7.28
container_name: mlflow_db
expose:
- "3306"
environment:
- MYSQL_DATABASE=${MYSQL_DATABASE}
- MYSQL_USER=${MYSQL_USER}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
volumes:
- ./dbdata:/var/lib/mysql
networks:
- default
mlflow:
restart: always
container_name: tracker_mlflow
image: tracker_ml
build:
context: ./mlflow
dockerfile: Dockerfile
ports:
- "5000:5000"
environment:
- AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
- AWS_DEFAULT_REGION=${AWS_REGION}
- MLFLOW_S3_ENDPOINT_URL=http://s3:9000
networks:
- proxy-net
- default
entrypoint: ./wait-for-it.sh db:3306 -t 90 -- mlflow server --backend-store-uri mysql+pymysql://${MYSQL_USER}:${MYSQL_PASSWORD}@db:3306/${MYSQL_DATABASE} --default-artifact-root s3://${AWS_BUCKET_NAME}/ -h 0.0.0.0
networks:
proxy-net:
external: true
from mlflow-docker.
Hi, did you have a chance to look at the issue yet? I had no success in resolving it yet and am now considering to switch to a SFTP-artifact store :(
from mlflow-docker.
I dont get one thing: about what insecure connection you are talking about?
On my end everything is insecure, because i dont have any letsencrypt certificate and everything works just fine - see the PR #11
I can normaly access minio console, minio api + mlflow tracking server with a password from the browser.
If the green lock in the browser is the only issue then maybe look into the devtools console - maybe theres some unsafe resource on the page.
Basicall from what i've experienced the browser sometimes tells you whats the reason to not display the page as "save".
In my case its because of the badly configured ssl certificate.
I cannot run any code to connect to mlflow tracking, cause for some reason i cannot just trust my local caddy certificate (im on windows), but i dont see that something would block it
from mlflow-docker.
Sorry for the delayed answer, I was very busy lately.
I have a letsencrypt certificate and mlflow-tracking works fine for me, too. The problem is, that the minio-API does not respond when I try to access it through the reverse-proxy (at myurl.com/s3/). I tried both simply saving artifacts from the client and sending requests through postman (software).
Can you help me: what did you do to get a response there?
from mlflow-docker.
I've linked the PR #11
I'm not using path-based virtual host, but domain-based instead (so not example.org/s3, but s3.example.org)
Let me know if that helps
from mlflow-docker.
Thanks for your implementation @Toumash, that certainly helped! :)
The only remaining question for me now is, whether the communication with the MinIO-API can be sent over https (Port 443) instead of directly over port 9000. As far as I understand it, operations on MinIO need to be authenticated (AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY) - but the communication with the MinIO-API is not encrypted and therefore vulnerable to man-in-the-middle-attacks?
from mlflow-docker.
@Clemens123
That would be the job for a Caddy Server. I believe the only change you need to make is to change vHosts names to something other ther IP or *.localhost. Then the Caddy will automatically try to generate a letsencrypt certyficate for you. (Of course you will need a public domain to do that (so that the letsencrypt servers can authorize your cert)
Docs are clear about this: https://caddyserver.com/docs/automatic-https
Unfortunately i dont have access to any any machine where i could test this out. Can you please provice an example?
I would love to put it in the youtube tutorial, but we need to figure this stuff out to the end ;D
from mlflow-docker.
So you are suggesting that changing the Caddy-file to
[...]
# Minio API
my.url.com:9000 {
handle_path /* {
reverse_proxy s3:9000
}
}
would work? I.e. it would process an initial API-request at port 9000 by first establishing a connection over port 443 (https) and then forwarding the request to the s3-service?
from mlflow-docker.
@Clemens123 I mean if you have domain name routing, then you can just host the minio api to the mentioned minio.example.com:443
public address and the https would be added automatically, by running letsencrypt (used internally by caddyfile i believe)
If you already have ssl certificate you could use it without taking the api public https://caddy.community/t/custom-ssl-certificate/6981
from mlflow-docker.
Hi, I tried to run it today and it seems like it should work. Sadly, I got an error now even though the AWS-credentials on the client are 100% the same as in the .env-file.
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
Probably I changed something somewhere without resetting - so I cannot yet confirm whether it works :(
I probably won't find time the next weeks, so I can only look further into this afterwards.
Until then, I wish you a merry christmas and a happy new year :) Thank you for your help so far!
from mlflow-docker.
Do you have the same error on the master branch?
Have you tried to reset the docker like this?
Watch out, it removes data from containers
docker compose down;
docker compose rm;
docker compose --build
Merry Christmas and Happy New Year ^^
_ _
( >o< ) _
,%%/'^`\%%. (@) _
%%'V V`%% 6 |-|\____________________ (@)
. %%. ,%% ,~. | | \|-|
| *%%. ,%%* | | |-| Happy Holidays! | |
| *%%%%%* | | ____ |_|\___________________ |-|
=l-------------------l===l----(oooo) (@) `\|_|
_______________________________|--| ~ (@)
================,==============| | ~
============,,~'')=============| |
-------,,~''......)------------| |
('............) | | ,
(........,### | | I\,
(....,####### | __ ,_/__ \__A
``########### |/'o\, 1 @, `/ ,/ ___,
' ########### |I `-'' , /__,--'_,-\
########### |\>--.______/ /V /
` ########### | | `---._ \`\==. `-.
########## | |/\,--==,.) ,\\ \==. \
####### , | /__\ `\`-. \) _ \=.\
) ,***** | | (`=,`-.'/ `),`-)
/ \( / \ | | __(_( \=,`--.'
( {/ \ ,\ ) | | ,=--(__,\ i\=, `-.
\(\6/\\// | | `//'\__ ` I \=, `-.
/============\ | | ``(__ `-._,' __ \=, \
______________________________(elya) (__, ,' `-\=, .\
\ (___, ( )=, ,)
====================================I ,--`(_,--.`_.. /=, ,/
====================================I /w_,--/w_,--.__,(==__,'
Elya Arrasmith
------------------------------------------------
https://asciiart.website/index.php?art=holiday/christmas/other
from mlflow-docker.
@Clemens123 any update on this?
from mlflow-docker.
Hi @Toumash , sorry for the late answer: I'm extremely busy at the moment and don't know yet when I will find the time to look into it again, deeply sorry :( I hope maybe in 2 weeks?
from mlflow-docker.
Closing due to inactivity. The PR will be open for anyone to get in and develop further
from mlflow-docker.
Related Issues (12)
- Multiple docker-compose up runs not needed
- I think there is no need to expose the db HOT 2
- Jupyter notebook fails storing artifacts HOT 4
- Lets use docker compose depends healthcheck instead of wait-for-it.sh
- Wrong bucket name in mc mb command. HOT 1
- Cannot start mysql container on arm v8 HOT 3
- mysql update request HOT 1
- failed to install in macos m1 HOT 6
- Add optuna to the mix HOT 2
- [Need help] Minio Port 9000 is redirecting to http://localhost:32965/ and not able connect. HOT 6
- No module named 'minio' HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mlflow-docker.