azure / mlops-v2-ado-demo Goto Github PK

I found a trouble that occurs when executing the contents of this repository and how to solve it, and report it in this issue.

■ Details of the trouble

Using the model created by running "mlops/azureml/train/pipeline.yml", an endpoint was created by "mlops/azureml/deploy/online/online-endpoint.yml" and then deployed by "mlops/azureml/ deploy/online/online-deployment.yml", the deployment failed with the following error:

$ az ml online-deployment create --name --endpoint --file online-deployment.yml
Check: endpoint exists
Creating/updating online deployment .............................................................................................................(None) ResourceNotReady: User container has crashed or terminated: Liveness probe failed: HTTP probe failed with statuscode: 502. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready
Code: None
Message: ResourceNotReady: User container has crashed or terminated: Liveness probe failed: HTTP probe failed with statuscode: 502. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready
Exception Details: (None) ResourceNotReady: User container has crashed or terminated: Liveness probe failed: HTTP probe failed with statuscode: 502. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready
Code: None
Message: ResourceNotReady: User container has crashed or terminated: Liveness probe failed: HTTP probe failed with statuscode: 502. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready

The deployment logs contained the following entries:

WARNING: Package(s) not found: azureml-inference-server-http
2023-12-15T06:00:35,374820764+00:00 | gunicorn/run | Install azureml-inference-server-http version 1.0.0
ERROR: Ignored the following versions that require a different python version: 1.0.0 Requires-Python >=3.8
ERROR: Could not find a version that satisfies the requirement azureml-inference-server-http==1.0.0 (from versions: 0.1.5, 0.1.9, 0.1.10, 0.2.0, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.4.1, 0.4.2, 0.4.9, 0.4.10, 0.4.11, 0.4.13, 0.4.14, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.6.1, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.4.1, 0.8.4.2)
ERROR: No matching distribution found for azureml-inference-server-http==1.0.0
Error occurred. Sleeping to send error logs.
2023-12-15T06:01:20,934851673+00:00 - gunicorn/finish 95 0
2023-12-15T06:01:20,936341197+00:00 - Exit code 95 is not normal. Killing image.

■ Details of the solution

The problem was resolved by modifying the python version of "data-science/environment/train-conda.yml" used as the training environment from 3.7.5 to 3.8 as shown below.

channels:
  - defaults
  - anaconda
  - conda-forge
dependencies:
  - python=3.8
  - pip
  - pip:
      - azureml-mlflow==1.38.0
      - azureml-sdk==1.38.0
      - scikit-learn==0.24.1
      - pandas==1.2.1
      - joblib==1.0.0
      - matplotlib==3.3.3

Deployment logs indicate that Python 3.8 or higher is required to install "azureml-inference-server-http". To this end, I tried modifying the yml file as described above and were able to confirm that the model artifact with the updated python version was created and deployed successfully.

Thanks for developing such a nice project. I look forward to future updates.