[bitnami/mlflow] Artifacts Issue in UI - botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: The specified key does not exist. about charts HOT 8 OPEN

@CeliaGMqrz any update on this ?

from charts.

@iamhritik290799 I am doing an ugly hack to work around this for now. You can figure out what your tracking args are by ssh-ing into a running tracking pod and running ps aux | cat. Thanks for troubleshooting this.

tracking:
  command: [ "/bin/sh", "-c" ]
  args:
    - >
      unset MLFLOW_S3_ENDPOINT_URL;
      mlflow server --host=0.0.0.0 --port=5000 --app-name=basic-auth
      --serve-artifacts --artifacts-destination=s3://$YOUR_BUCKET
      --backend-store-uri=postgresql://postgres:$(MLFLOW_DATABASE_PASSWORD)@$YOUR_DB_HOST:5432/mlflow_db;

from charts.

Hi,

Looking at the issue, it is not clear to me that the issue is related to the Bitnami packaging of MLflow or some issue with S3 inside the mlflow code. Did you check with the upstream developers?

from charts.

checked but they were saying maybe there is some issue in bitnami mlflow image due to which you are getting this error when trying to access your Artifacts from S3 bucket.

btw we are these args in our mlflow container :

containers:
  - args:
    - server
    - --backend-store-uri=postgresql://admin:$(MLFLOW_DATABASE_PASSWORD)@rds-instance-endpoint.rds.amazonaws.com:5432/mlflow
    - --artifacts-destination=s3://mlflow-artifacts
    - --serve-artifacts
    - --host=0.0.0.0
    - --port=5000
    - --app-name=basic-auth

from charts.

The issue may not be directly related to the Bitnami container image or Helm chart, but rather to how the application is being utilized or configured in your specific environment.

Having said that, if you think that's not the case and are interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

If you have any questions about the application itself, customizing its content, or questions about technology and infrastructure usage, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

from charts.

Hi @carrodher

I have checked and after removing this ENV MLFLOW_S3_ENDPOINT_URL from the deployment, it's working fine and I'm able to load artifact in my mlflow experiments

I have checked on the mlflow documentation as well and they suggested to unset MLFLOW_S3_ENDPOINT_URL env on the client system but somehow after removing this env in deployment it worked.

but in the bitnami/mlflow helm chart tracking deployment template there is no parameter used to exclude this env if not required. PFBR

from charts.

Team, have you made any changes on the helm chart to customize env variable for the mlflow deployment ?

from charts.

Even before we make any changes in the Helm chart, I think we should clarify in which specific cases or scenarios setting the MLFLOW_S3_ENDPOINT_URL env-var for the tracking component is required. @iamhritik290799, @act-mreeves, can you help clarifying that?

BTW @iamhritik290799, just to confirm, the screenshot you shared that suggests to unset the env-var comes from this documentation page, right? https://mlflow.org/docs/2.10.2/tracking/artifacts-stores.html#setting-bucket-region

from charts.