I was deploying this and I ended up having to update the 00_prepare_data.ipynb notebook to rename a bunch of columns from the complaints.csv file that is downloaded and uploaded to ADLS because many of the column names had spaces in them. Not entirely sure what to rename them to, except by trial and error of running the script, erroring out, fixing, etc.

Can the notebooks in this accelerator please be updated taking the current version of complaints.csv that we need to download?

I am trying to deploy this accelerator code, and I am getting the follow error message when the script is attempting to create the Spark pools:

{
    "status": "Failed",
    "error": {
        "code": "ValidationFailed",
        "message": "Spark pool request validation failed.",
        "details": [
            {
                "code": "DynamicExecutorAllocationNotValid",
                "message": "Min and max executors properties are mandatory fields when dynamic executor allocation is enabled"
            }
        ]
    }
}

I haven't changed anything about the IP, so I would suspect that other users who are running this as is will experience similar issues. Can you please correct the issue and update the IP in GitHub?

Thanks

Steve

I have performed the pre-requisites of the lab and tested it manually till Step 4. While running the notebook 00_prepare_data in Synapse workspace in Step 5, I am facing below issue.

Referring to the failure reason, I tried increasing the Spark pool size from medium to large, still faced issue (Issue saying extra Vcores are needed).
Please find the screenshots below.

Please look into this issue and help us with solution.

So the notebook seems to run fine all the way through the very last step, where it is going to deploy an endpoint to my AML service.

No matter what I try, I cannot get the endpoint to successfully run.

The first time, it was complaining about not being able to find scipy. So I added that in the next-to-last cell where the env file is being prepared. (myenv.add_conda_package('scipy'))

After that, I no longer get that error, but the ACI will still not launch properly to create the endpoint.

Help would be appreciated as we are trying to figure out how to demo this successfully.

Thanks. Complete error message can be found below:

/tmp/ipykernel_29952/2293928557.py:13: FutureWarning: azureml.core.model:
To leverage new model deployment capabilities, AzureML recommends using CLI/SDK v2 to deploy models as online endpoint,
please refer to respective documentations
https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoints /
https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoint-sdk-v2 /
https://docs.microsoft.com/azure/machine-learning/how-to-attach-kubernetes-anywhere
For more information on migration, see https://aka.ms/acimoemigration.
To disable CLI/SDK v1 deprecation warning set AZUREML_LOG_DEPRECATION_WARNING_ENABLED to 'False'
service = Model.deploy(ws, service_name, [registered_model], inference_config, deployment_config)
Tips: You can try get_logs(): https://aka.ms/debugimage#dockerlog or local deployment: https://aka.ms/debugimage#debug-locally to debug if deployment takes longer than 10 minutes.
Running
2023-03-13 18:20:40+00:00 Creating Container Registry if not exists.
2023-03-13 18:20:40+00:00 Registering the environment.
2023-03-13 18:20:42+00:00 Use the existing image.
2023-03-13 18:20:43+00:00 Submitting deployment to compute.
2023-03-13 18:20:50+00:00 Checking the status of deployment ccm-service-3..
2023-03-13 18:22:44+00:00 Checking the status of inference endpoint ccm-service-3.
Failed
Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 58fdf70e-6ca8-4a29-ac81-4f175bc48a5f
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.
1. Please check the logs for your container instance: ccm-service-3. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.
2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
3. You can also try to run image 9b987bd5e01d463799dfd4bdbf7bba56.azurecr.io/azureml/azureml_54be50da6f33c7dc72e3ab0bcd05c6ed locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.
1. Please check the logs for your container instance: ccm-service-3. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.
2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
3. You can also try to run image 9b987bd5e01d463799dfd4bdbf7bba56.azurecr.io/azureml/azureml_54be50da6f33c7dc72e3ab0bcd05c6ed locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information."
},
{
"code": "AciDeploymentFailed",
"message": "Your container application crashed. Please follow the steps to debug:
1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.
2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.
3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
4. View the diagnostic events to check status of container, it may help you to debug the issue.
"RestartCount": 3
"CurrentState": {"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off restarting failed"}
"PreviousState": {"state":"Terminated","startTime":"2023-03-13T18:24:58.898Z","exitCode":111,"finishTime":"2023-03-13T18:25:43.341Z","detailStatus":"Error"}
"Events": null
"
}
]
}

WebserviceException Traceback (most recent call last)
Cell In [39], line 15
11 service_name = "ccm-service-3"
13 service = Model.deploy(ws, service_name, [registered_model], inference_config, deployment_config)
---> 15 service.wait_for_deployment(show_output =True)
16 print(service.state)

File ~/cluster-env/env/lib/python3.10/site-packages/azureml/core/webservice/webservice.py:918, in Webservice.wait_for_deployment(self, show_output, timeout_sec)
915 if not logs_response:
916 logs_response = 'Current sub-operation type not known, more logs unavailable.'
--> 918 raise WebserviceException('Service deployment polling reached non-successful terminal state, current '
919 'service state: {}\n'
920 'Operation ID: {}\n'
921 '{}\n'
922 'Error:\n'
923 '{}'.format(self.state, self._operation_endpoint.split('/')[-1],
924 logs_response, format_error_response), logger=module_logger)
925 print('{} service creation operation finished, operation "{}"'.format(self._webservice_type,
926 operation_state))
927 except WebserviceException as e:

WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 58fdf70e-6ca8-4a29-ac81-4f175bc48a5f
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.
1. Please check the logs for your container instance: ccm-service-3. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.
2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
3. You can also try to run image 9b987bd5e01d463799dfd4bdbf7bba56.azurecr.io/azureml/azureml_54be50da6f33c7dc72e3ab0bcd05c6ed locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.
1. Please check the logs for your container instance: ccm-service-3. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.
2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
3. You can also try to run image 9b987bd5e01d463799dfd4bdbf7bba56.azurecr.io/azureml/azureml_54be50da6f33c7dc72e3ab0bcd05c6ed locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information."
},
{
"code": "AciDeploymentFailed",
"message": "Your container application crashed. Please follow the steps to debug:
1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.
2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.
3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
4. View the diagnostic events to check status of container, it may help you to debug the issue.
"RestartCount": 3
"CurrentState": {"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off restarting failed"}
"PreviousState": {"state":"Terminated","startTime":"2023-03-13T18:24:58.898Z","exitCode":111,"finishTime":"2023-03-13T18:25:43.341Z","detailStatus":"Error"}
"Events": null
"
}
]
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 58fdf70e-6ca8-4a29-ac81-4f175bc48a5f\nMore information can be found using '.get_logs()'\nError:\n{\n "code": "AciDeploymentFailed",\n "statusCode": 400,\n "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.\n\t1. Please check the logs for your container instance: ccm-service-3. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.\n\t2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t3. You can also try to run image 9b987bd5e01d463799dfd4bdbf7bba56.azurecr.io/azureml/azureml_54be50da6f33c7dc72e3ab0bcd05c6ed locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.",\n "details": [\n {\n "code": "CrashLoopBackOff",\n "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.\n\t1. Please check the logs for your container instance: ccm-service-3. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.\n\t2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t3. You can also try to run image 9b987bd5e01d463799dfd4bdbf7bba56.azurecr.io/azureml/azureml_54be50da6f33c7dc72e3ab0bcd05c6ed locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information."\n },\n {\n "code": "AciDeploymentFailed",\n "message": "Your container application crashed. Please follow the steps to debug:\n\t1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n\t2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.\n\t3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t4. View the diagnostic events to check status of container, it may help you to debug the issue.\n"RestartCount": 3\n"CurrentState": {"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off restarting failed"}\n"PreviousState": {"state":"Terminated","startTime":"2023-03-13T18:24:58.898Z","exitCode":111,"finishTime":"2023-03-13T18:25:43.341Z","detailStatus":"Error"}\n"Events": null\n"\n }\n ]\n}"
}
}

I'm attempting to deploy this accelerator and came across an issue in Step 2 of the 00 notebook. Apparently in the complaints.csv file there are a number of columns that need to be reformatted during import.

I was able to get around the issue by including the following in step 2:

.withColumnRenamed("Date received", "datereceived")
.withColumnRenamed("Consumer complaint narrative","consumersomplaintnarrative")
.withColumnRenamed("Company public response","companypublicresponse")
.withColumnRenamed("ZIP code","zipcode")
.withColumnRenamed("Consumer consent provided?","consumerconsentprovided")
.withColumnRenamed("Submitted via","submittedvia")
.withColumnRenamed("Date sent to company","datesenttocompany")
.withColumnRenamed("Company response to consumer","companyresponsetoconsumer")
.withColumnRenamed("Timely response?","timelyresponse")
.withColumnRenamed("Consumer disputed?","consumerdisputed")
.withColumnRenamed("Complaint ID","complaintid")

However, notebook code should be updated in the repo to make sure any other users don't get tripped up by this.