azure / ai-toolkit-iot-edge Goto Github PK

AI Toolkit for Azure IoT Edge

License: MIT License

Python 17.89% C# 18.71% Shell 0.50% Jupyter Notebook 62.90%

ai-toolkit-iot-edge's Introduction

Welcome to the AI Toolkit for Azure IoT Edge

The integration of Azure Machine Learning and Azure IoT Edge enables organizations and developers to apply AI and ML to data that can’t make it to the cloud due to data sovereignty, privacy, and/or bandwidth issues. All models created using Azure Machine Learning can now be deployed to IoT gateways and devices with the Azure IoT Edge runtime. Models are operationalized as containers and can run on many types of hardware, from very small devices all the way to powerful servers.

We're releasing this toolkit to help get you started with AI and Azure IoT Edge. The toolkit will show you how to package deep learning models in Azure IoT Edge-compatible Docker containers and expose those models as REST APIs. We've included examples to help get you started, but the possibilities are endless. We'll be adding new examples and tools often. The models can be used as-is or and customized to better meet your specific needs and use cases.

Please ask any questions on our forum. We welcome your feedback and contributions and look forward to building together.

Concepts

Azure Machine Learning is designed for data scientists to build, deploy, manage, and monitor models at any scale
Azure IoT Edge moves cloud analytics and custom business logic to devices as an Internet of Things (IoT) service that builds on top of IoT Hub
AI Toolkit for Azure IoT Edge is an evolving set of scripts, sample code, and tutorials that enable you to easily set up a test environment and run AI and ML on an edge device

Quick start

AI on the edge

One use case for edge devices is image processing and object classification. For example, images taken by cameras of products on an assembly line in a factory may be analyzed for manufacturing defects without having to send the images to the cloud. To simplify this problem for the tutorial, we will create and deploy a model that will take in an image of a handwritten digit and predict what that number is. We will use the well-known MNIST data set and a pre-trained TensorFlow model.

Environment set up

If you don't have an Azure subscription, create a free account before you begin.

Install Azure Machine Learning
Create an IoT Hub and register an IoT Edge device
Create an IoT Edge device with the Data Science VM (DSVM)

You will need a connection string from the IoT Hub you created in the previous step
This DSVM doubles as an IoT Edge device and the machine you can use to operationalize models

Set up Model Management for Azure ML

If you are already an Azure ML user then skip to the next section.

If you are not using the DSVM from the previous section for Azure ML, then set up Model Management on your machine.

Otherwise, follow these steps (more details in the Model Management documentation):

Connect and log into the DSVM you created in the previous section
Open a command prompt (type az ml -h to see options)
Run the script below to configure Docker correctly (Docker is pre-installed on the DSVM). Remember to log out and log back in after running the script.

sudo /opt/microsoft/azureml/initial_setup.sh

Set up the environment (only needs to be done one time). Note when completing the environment setup:

You are prompted to sign in to Azure. To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the provided code to authenticate.
During the authentication process, you are prompted for an account to authenticate with. Important: Select an account that has a valid Azure subscription and sufficient permissions to create resources in the account.
When the log-in is complete, your subscription information is presented and you are prompted whether you wish to continue with the selected account.

az provider register -n Microsoft.MachineLearningCompute

Set up a local environment using the following command. The resource group name is optional.

az ml env setup -l [Azure Region, e.g. eastus2] -n [your environment name] [-g [existing resource group]]

The local environment setup command creates the following resources in your subscription:

A resource group (if not provided, or if the name provided does not exist)
A storage account
An Azure Container Registry (ACR)
An Application insights account

After setup completes successfully, set the environment to be used using the following command:

az ml env set -n [environment name] -g [resource group]

Note: For subsequent deployments, you only need to use the set command above to reuse it.

You are now ready to deploy your saved model as a web service.

Create container for Azure IoT Edge

Follow these steps to create the container for deployment to Azure IoT Edge running on your DSVM.

Next Steps

Check out our set of rich tutorials, where you can create, train, and deploy models for predictive maintenance, aerial image classification, energy demand time series forecasting, and more. Then create your own!

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

ai-toolkit-iot-edge's People

Contributors

Stargazers

Watchers

Forkers

jameslyleyoung efratshabtai yien-tp kannangrajan fengweijp myamama dalestone mandur ru003ar ramponna maneeshs shashank545 kalyanb9119 progamercode mganta nzigel freedragon bestmikelee uneidel firas-bfk daverendon bartekgraczyk nakarumanchi duangenquan jiongfu liortur ludsh2000 tmukande cyysu lujohn74 direkshan-digital victorvikram geo1277 maxkhlupnov takehitoyoshidanttcom nadialee manisnesan t1seungy kavap tony979 iotlibrary kgremban tikyau sasauz xzammy vrishni ram-msft agarza152 vasaini j-martens cclauss dsnoor michaelduan8 meperrywork emgerner-msft yaniv650 jbjak klemzi leehyeongwon babianisha narayanareddytalakola bozhong68 leestott lafranke saelizondor eloisamurillo abigailtorresc charismaticzone tanglijhu isabella232 stefannich terrychang1015 tommywu052 akmuzammil vasyllyashkevych nemilshah007 suqingy mastloui-msft leelakrishna45 habmat jennycoca algomanic oange6214 tebellox gautamabhi mukeshkbj jonnydiamond sushsb augustine-beilel isamaljawarneh jimmy-inl jdfit omar-nav nermeen12 bytecrew rase6 saintpepsicola xwdreamer bochuxt sourcecodecheck

ai-toolkit-iot-edge's Issues

00-anomaly-detection-tutorial: Container deployment failure Azure ML - AciDeploymentFailed

Recieve the following ERROR - Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 43f65bff-fbd4-434f-bdd9-2af56ece7041 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "message": "Aci Deployment failed with exception: Your container application crashed.

This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.", "details": [ { "code": "CrashLoopBackOff", "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information." }, { "code": "AciDeploymentFailed", "message": "Your container application crashed. Please follow the steps to debug:\n1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first.

Please refer to https://aka.ms/debugimage#debug-locally for more information.\n3. View the diagnostic events to check status of container, it may help you to debug the issue. {"restartCount":3,"currentState":{"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off 40s restarting failed"},"previousState":{"state":"Terminated","startTime":"2020-09-01T12:51:21Z","exitCode":111,"finishTime":"2020-09-01T12:51:26Z","detailStatus":"Error"},"events":[{"count":3,"firstTimestamp":"2020-09-01T12:48:47Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Pulling","message":"pulling image \"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\"","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:04Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Pulled","message":"Successfully pulled image \"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\"","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:14Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Created","message":"Created container","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:14Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Started","message":"Started container","type":"Normal"},{"count":1,"firstTimestamp":"2020-09-01T12:50:37Z","lastTimestamp":"2020-09-01T12:50:37Z","name":"BackOff","message":"Back-off restarting failed container","type":"Warning"}]}" } ] } ERROR - Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 43f65bff-fbd4-434f-bdd9-2af56ece7041 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.", "details": [ { "code": "CrashLoopBackOff", "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information." }, { "code": "AciDeploymentFailed", "message": "Your container application crashed. Please follow the steps to debug:\n1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.\n3. View the diagnostic events to check status of container, it may help you to debug the issue. {"restartCount":3,"currentState":{"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off 40s restarting failed"},"previousState":{"state":"Terminated","startTime":"2020-09-01T12:51:21Z","exitCode":111,"finishTime":"2020-09-01T12:51:26Z","detailStatus":"Error"},"events":[{"count":3,"firstTimestamp":"2020-09-01T12:48:47Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Pulling","message":"pulling image \"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\"","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:04Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Pulled","message":"Successfully pulled image \"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\"","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:14Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Created","message":"Created container","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:14Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Started","message":"Started container","type":"Normal"},{"count":1,"firstTimestamp":"2020-09-01T12:50:37Z","lastTimestamp":"2020-09-01T12:50:37Z","name":"BackOff","message":"Back-off restarting failed container","type":"Warning"}]}" } ] }
--------------------------------------------------------------------------- WebserviceException Traceback (most recent call last) /opt/conda/lib/python3.8/site-packages/azureml/core/webservice/webservice.py in wait_for_deployment(self, show_output) 667 --> 668 raise WebserviceException('Service deployment polling reached non-successful terminal state, current ' 669 'service state: {}\n' WebserviceException: WebserviceException: Message: Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 43f65bff-fbd4-434f-bdd9-2af56ece7041 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.", "details": [ { "code": "CrashLoopBackOff", "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information." }, { "code": "AciDeploymentFailed", "message": "Your container application crashed. Please follow the steps to debug:\n1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.\n3. View the diagnostic events to check status of container, it may help you to debug the issue. {"restartCount":3,"currentState":{"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off 40s restarting failed"},"previousState":{"state":"Terminated","startTime":"2020-09-01T12:51:21Z","exitCode":111,"finishTime":"2020-09-01T12:51:26Z","detailStatus":"Error"},"events":[{"count":3,"firstTimestamp":"2020-09-01T12:48:47Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Pulling","message":"pulling image \"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\"","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:04Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Pulled","message":"Successfully pulled image \"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\"","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:14Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Created","message":"Created container","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:14Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Started","message":"Started container","type":"Normal"},{"count":1,"firstTimestamp":"2020-09-01T12:50:37Z","lastTimestamp":"2020-09-01T12:50:37Z","name":"BackOff","message":"Back-off restarting failed container","type":"Warning"}]}" } ] } InnerException None ErrorResponse { "error": { "message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 43f65bff-fbd4-434f-bdd9-2af56ece7041\nMore information can be found using '.get_logs()'\nError:\n{\n "code": "AciDeploymentFailed",\n "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.",\n "details": [\n {\n "code": "CrashLoopBackOff",\n "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information."\n },\n {\n "code": "AciDeploymentFailed",\n "message": "Your container application crashed. Please follow the steps to debug:\n1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.\n3. View the diagnostic events to check status of container, it may help you to debug the issue. {\"restartCount\":3,\"currentState\":{\"state\":\"Waiting\",\"startTime\":null,\"exitCode\":null,\"finishTime\":null,\"detailStatus\":\"CrashLoopBackOff: Back-off 40s restarting failed\"},\"previousState\":{\"state\":\"Terminated\",\"startTime\":\"2020-09-01T12:51:21Z\",\"exitCode\":111,\"finishTime\":\"2020-09-01T12:51:26Z\",\"detailStatus\":\"Error\"},\"events\":[{\"count\":3,\"firstTimestamp\":\"2020-09-01T12:48:47Z\",\"lastTimestamp\":\"2020-09-01T12:50:50Z\",\"name\":\"Pulling\",\"message\":\"pulling image \\\"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\\\"\",\"type\":\"Normal\"},{\"count\":3,\"firstTimestamp\":\"2020-09-01T12:50:04Z\",\"lastTimestamp\":\"2020-09-01T12:50:50Z\",\"name\":\"Pulled\",\"message\":\"Successfully pulled image \\\"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\\\"\",\"type\":\"Normal\"},{\"count\":3,\"firstTimestamp\":\"2020-09-01T12:50:14Z\",\"lastTimestamp\":\"2020-09-01T12:50:50Z\",\"name\":\"Created\",\"message\":\"Created container\",\"type\":\"Normal\"},{\"count\":3,\"firstTimestamp\":\"2020-09-01T12:50:14Z\",\"lastTimestamp\":\"2020-09-01T12:50:50Z\",\"name\":\"Started\",\"message\":\"Started container\",\"type\":\"Normal\"},{\"count\":1,\"firstTimestamp\":\"2020-09-01T12:50:37Z\",\"lastTimestamp\":\"2020-09-01T12:50:37Z\",\"name\":\"BackOff\",\"message\":\"Back-off restarting failed container\",\"type\":\"Warning\"}]}"\n }\n ]\n}" } } During handling of the above exception, another exception occurred: WebserviceException Traceback (most recent call last) in 7 name = aci_service_name, 8 workspace = ws) ----> 9 aci_service.wait_for_deployment(True) 10 print(aci_service.state) /opt/conda/lib/python3.8/site-packages/azureml/core/webservice/webservice.py in wait_for_deployment(self, show_output) 681 'Current state is {}'.format(self.state), logger=module_logger) 682 else: --> 683 raise WebserviceException(e.message, logger=module_logger) 684 685 def _wait_for_operation_to_complete(self, show_output): WebserviceException: WebserviceException: Message: Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 43f65bff-fbd4-434f-bdd9-2af56ece7041 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.", "details": [ { "code": "CrashLoopBackOff", "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information." }, { "code": "AciDeploymentFailed", "message": "Your container application crashed. Please follow the steps to debug:\n1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.\n3. View the diagnostic events to check status of container, it may help you to debug the issue. {"restartCount":3,"currentState":{"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off 40s restarting failed"},"previousState":{"state":"Terminated","startTime":"2020-09-01T12:51:21Z","exitCode":111,"finishTime":"2020-09-01T12:51:26Z","detailStatus":"Error"},"events":[{"count":3,"firstTimestamp":"2020-09-01T12:48:47Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Pulling","message":"pulling image \"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\"","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:04Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Pulled","message":"Successfully pulled image \"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\"","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:14Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Created","message":"Created container","type":"Normal"},{"count":3,"firstTimestamp":"2020-09-01T12:50:14Z","lastTimestamp":"2020-09-01T12:50:50Z","name":"Started","message":"Started container","type":"Normal"},{"count":1,"firstTimestamp":"2020-09-01T12:50:37Z","lastTimestamp":"2020-09-01T12:50:37Z","name":"BackOff","message":"Back-off restarting failed container","type":"Warning"}]}" } ] } InnerException None ErrorResponse { "error": { "message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 43f65bff-fbd4-434f-bdd9-2af56ece7041\nMore information can be found using '.get_logs()'\nError:\n{\n "code": "AciDeploymentFailed",\n "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.",\n "details": [\n {\n "code": "CrashLoopBackOff",\n "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check the logs for your container instance: tempsensor-iotedge-ml-test1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. \nYou can also try to run image f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information."\n },\n {\n "code": "AciDeploymentFailed",\n "message": "Your container application crashed. Please follow the steps to debug:\n1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.\n3. View the diagnostic events to check status of container, it may help you to debug the issue. {\"restartCount\":3,\"currentState\":{\"state\":\"Waiting\",\"startTime\":null,\"exitCode\":null,\"finishTime\":null,\"detailStatus\":\"CrashLoopBackOff: Back-off 40s restarting failed\"},\"previousState\":{\"state\":\"Terminated\",\"startTime\":\"2020-09-01T12:51:21Z\",\"exitCode\":111,\"finishTime\":\"2020-09-01T12:51:26Z\",\"detailStatus\":\"Error\"},\"events\":[{\"count\":3,\"firstTimestamp\":\"2020-09-01T12:48:47Z\",\"lastTimestamp\":\"2020-09-01T12:50:50Z\",\"name\":\"Pulling\",\"message\":\"pulling image \\\"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\\\"\",\"type\":\"Normal\"},{\"count\":3,\"firstTimestamp\":\"2020-09-01T12:50:04Z\",\"lastTimestamp\":\"2020-09-01T12:50:50Z\",\"name\":\"Pulled\",\"message\":\"Successfully pulled image \\\"f0ff5444e0bf4b99a497466dc8471dcb.azurecr.io/tempanomalydetection@sha256:e02bb4d4224980e098e369219f213847a86e4ae0e9636a011c9cce3f7680fa68\\\"\",\"type\":\"Normal\"},{\"count\":3,\"firstTimestamp\":\"2020-09-01T12:50:14Z\",\"lastTimestamp\":\"2020-09-01T12:50:50Z\",\"name\":\"Created\",\"message\":\"Created container\",\"type\":\"Normal\"},{\"count\":3,\"firstTimestamp\":\"2020-09-01T12:50:14Z\",\"lastTimestamp\":\"2020-09-01T12:50:50Z\",\"name\":\"Started\",\"message\":\"Started container\",\"type\":\"Normal\"},{\"count\":1,\"firstTimestamp\":\"2020-09-01T12:50:37Z\",\"lastTimestamp\":\"2020-09-01T12:50:37Z\",\"name\":\"BackOff\",\"message\":\"Back-off restarting failed container\",\"type\":\"Warning\"}]}"\n }\n ]\n}" } }

Is it possible to change image to arm32/v7

Hey, as I am running the ML python script on a Raspberry Pi I would need to use an arm32/v7 image.

Sadly, the following change

image_config = ContainerImage.image_configuration(runtime= "python",
                                 execution_script="iot_score.py",
                                 conda_file="myenv.yml",
                                 tags = {'area': "iot", 'type': "classification"},
                                 description = "IOT Edge anomaly detection demo",
                                 base_image="docker.io/arm32v7/python:alpine" # added
                                 )

returns error messages like this:

Step 6/22 : RUN cp -r /var/licenses/* /
 ---> [Warning] The requested image's platform (linux/arm) does not match the detected host platform (linux/amd64) and no specific platform was requested
 ---> Running in 61389f03f636
Removing intermediate container 61389f03f636
 ---> 8477ab23a627
Step 7/22 : RUN /var/requirements/install_system_requirements.sh
 ---> [Warning] The requested image's platform (linux/arm) does not match the detected host platform (linux/amd64) and no specific platform was requested
 ---> Running in c82a90b42c80
�[91m/bin/sh: �[0m�[91m/var/requirements/install_system_requirements.sh: not found�[0m�[91m
�[0mThe command '/bin/sh -c /var/requirements/install_system_requirements.sh' returned a non-zero code: 127
2021/05/12 10:49:46 Container failed during run: acb_step_0. No retries remaining.
failed to run step ID: acb_step_0: exit status 127

Run ID: cbe failed after 20s. Error: failed during run, err: exit status 1

Any ideas?

Container Deployment Error

In the section titled "Deploy Container to Azure IoT Edge Device", I am getting the following errors:

KeyError Traceback (most recent call last)
in
1 # Getting your container details
----> 2 container_reg = ws.get_details()["containerRegistry"]
3 reg_name=container_reg.split("/")[-1]
4 container_url = """ + image.image_location + "","
5 subscription_id = ws.subscription_id

KeyError: 'containerRegistry'

Getting Azure Exception

I am trying to deploy MNIST handwritten digit classification using TensorFlow for getting started. I already set up my environment ad tested basic tempsensor module which is very very basic.

Now when I just run command:

$ az ml service create realtime -m my_ConvNet_MNIST_model.meta -d my_ConvNet_MNIST_model.data-00001-of-00000 -d my_ConvNet_MNIST_model.index -d checkpoint -f webservice_driver.py -n [your service name] -r python -c conda_dependencies.yml

with proper service name getting following error:

{'Error': AzureException(ProtocolError('Connection aborted.', timeout('The write operation timed out',)),), 'Azure-cli-ml Version': '0.1.0a27.post3'}

I think it is because data is big and it just times out. But I am new to this world and don't know how to fix it :(

Documentation 404

I try to get my model running on iot edge. Currently the whole documentation is useless, On the MS web page there is no example or s.th. like that. The notebook in this git is not available (404). Where can i get help for the whole topic?

Access to Application Properties from Input Messages

Is there a way (or a sample showing how) to access message application properties from the input messages into the AI Module?
cc/ @tedway

Different python version for the az commands

The DSVM has two python versions
One under /anaconda/envs/py35/bin/python which the standard command pip install relates to.
One under /opt/az/bin/python3 which the az command line related to.

so If users run pip install azure-cli-ml, this will install the ml command on the wrong python environment and users won't be able to run the az ml commands. This could currently by overcome by running sudo /opt/az/bin/python3 -m pip install azure-cli-ml but it would be less confusing to either provide an harmonized default python for both commands, or to describe this in the doc.

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Can anyone provide the model already trained?

Hi, I'm italian student who is developing a java app to classify the user skin lesion, but I' m having some problem with the training process.
Since I'm running out of time, could someone give me the model?
Thank you in advice.

"MNIST handwritten digit classification using TensorFlow" failed due to "tensorflow.python.framework.errors_impl.NotFoundError: /tmp/model"

Hi,

I tried to follow Quick Start to run a sample model service on my edge device, but failed as follows.

$ az ml service logs realtime -i my-mnist
2018-05-20 21:55:11,845 CRIT Supervisor running as root (no user in config file)
2018-05-20 21:55:11,847 INFO supervisord started with pid 1
2018-05-20 21:55:12,849 INFO spawned: 'rsyslog' with pid 9
2018-05-20 21:55:12,851 INFO spawned: 'program_exit' with pid 10
2018-05-20 21:55:12,853 INFO spawned: 'nginx' with pid 11
2018-05-20 21:55:12,854 INFO spawned: 'iot' with pid 12
2018-05-20 21:55:12,856 INFO spawned: 'gunicorn' with pid 13
2018-05-20 21:55:12,868 INFO success: iot entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-05-20 21:55:12,931 INFO exited: iot (exit status 1; expected)
2018-05-20 21:55:13,932 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-20 21:55:13,932 INFO success: program_exit entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-20 21:55:17,937 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2018-05-20T21:55:18.305553Z, INFO, 00000000-0000-0000-0000-000000000000, , Starting gunicorn 19.6.0
2018-05-20T21:55:18.307199Z, INFO, 00000000-0000-0000-0000-000000000000, , Listening at: http://127.0.0.1:9090 (13)
2018-05-20T21:55:18.307539Z, INFO, 00000000-0000-0000-0000-000000000000, , Using worker: sync
2018-05-20T21:55:18.308591Z, INFO, 00000000-0000-0000-0000-000000000000, , worker timeout is set to 300
2018-05-20T21:55:18.309772Z, INFO, 00000000-0000-0000-0000-000000000000, , Booting worker with pid: 31
Initializing logger
2018-05-20T21:55:18.773498Z, INFO, 00000000-0000-0000-0000-000000000000, , Starting up app insights client
2018-05-20T21:55:18.773791Z, INFO, 00000000-0000-0000-0000-000000000000, , Starting up request id generator
2018-05-20T21:55:18.773942Z, INFO, 00000000-0000-0000-0000-000000000000, , Starting up app insight hooks
2018-05-20T21:55:18.774278Z, INFO, 00000000-0000-0000-0000-000000000000, , Invoking user's init function
2018-05-20 21:55:22.053772: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20 21:55:22.053811: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20 21:55:22.053835: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20 21:55:22.053845: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20 21:55:22.053855: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20T21:55:22.976338Z, ERROR, 00000000-0000-0000-0000-000000000000, , User's init function failed
2018-05-20T21:55:22.977815Z, ERROR, 00000000-0000-0000-0000-000000000000, , Encountered Exception Traceback (most recent call last):
  File "/var/azureml-app/aml_blueprint.py", line 109, in register
    main.init()
  File "/var/azureml-app/main.py", line 41, in init
    user_init_func()
  File "/var/azureml-app/webservice_driver.py", line 9, in init
    saver.restore(sess,tf.train.latest_checkpoint('./'))
  File "/home/mmlspark/lib/conda/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1511, in latest_checkpoint
    if file_io.get_matching_files(v2_path) or file_io.get_matching_files(
  File "/home/mmlspark/lib/conda/lib/python3.5/site-packages/tensorflow/python/lib/io/file_io.py", line 282, in get_matching_files
    compat.as_bytes(filename), status)]
  File "/home/mmlspark/lib/conda/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/home/mmlspark/lib/conda/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: /tmp/model

2018-05-20T21:55:22.978319Z, INFO, 00000000-0000-0000-0000-000000000000, , Worker exiting (pid: 31)
2018-05-20T21:55:23.243101Z, INFO, 00000000-0000-0000-0000-000000000000, , Shutting down: Master
2018-05-20T21:55:23.243485Z, INFO, 00000000-0000-0000-0000-000000000000, , Reason: Worker failed to boot.
2018-05-20 21:55:23,271 INFO exited: gunicorn (exit status 3; not expected)
2018-05-20 21:55:24,274 INFO spawned: 'gunicorn' with pid 41
2018-05-20T21:55:29.717610Z, INFO, 00000000-0000-0000-0000-000000000000, , Starting gunicorn 19.6.0
2018-05-20T21:55:29.718823Z, INFO, 00000000-0000-0000-0000-000000000000, , Listening at: http://127.0.0.1:9090 (41)
2018-05-20T21:55:29.719061Z, INFO, 00000000-0000-0000-0000-000000000000, , Using worker: sync
2018-05-20T21:55:29.719820Z, INFO, 00000000-0000-0000-0000-000000000000, , worker timeout is set to 300
2018-05-20T21:55:29.720920Z, INFO, 00000000-0000-0000-0000-000000000000, , Booting worker with pid: 45
Initializing logger
2018-05-20T21:55:30.045352Z, INFO, 00000000-0000-0000-0000-000000000000, , Starting up app insights client
2018-05-20T21:55:30.045671Z, INFO, 00000000-0000-0000-0000-000000000000, , Starting up request id generator
2018-05-20T21:55:30.046003Z, INFO, 00000000-0000-0000-0000-000000000000, , Starting up app insight hooks
2018-05-20T21:55:30.046263Z, INFO, 00000000-0000-0000-0000-000000000000, , Invoking user's init function
2018-05-20 21:55:31.715341: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20 21:55:31.715390: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20 21:55:31.715402: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20 21:55:31.715412: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20 21:55:31.715422: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2018-05-20T21:55:32.620562Z, ERROR, 00000000-0000-0000-0000-000000000000, , User's init function failed
2018-05-20T21:55:32.621796Z, ERROR, 00000000-0000-0000-0000-000000000000, , Encountered Exception Traceback (most recent call last):
  File "/var/azureml-app/aml_blueprint.py", line 109, in register
    main.init()
  File "/var/azureml-app/main.py", line 41, in init
    user_init_func()
  File "/var/azureml-app/webservice_driver.py", line 9, in init
    saver.restore(sess,tf.train.latest_checkpoint('./'))
  File "/home/mmlspark/lib/conda/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1511, in latest_checkpoint
    if file_io.get_matching_files(v2_path) or file_io.get_matching_files(
  File "/home/mmlspark/lib/conda/lib/python3.5/site-packages/tensorflow/python/lib/io/file_io.py", line 282, in get_matching_files
    compat.as_bytes(filename), status)]
  File "/home/mmlspark/lib/conda/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/home/mmlspark/lib/conda/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: /tmp/model

2018-05-20T21:55:32.622017Z, INFO, 00000000-0000-0000-0000-000000000000, , Worker exiting (pid: 45)
2018-05-20T21:55:32.873411Z, INFO, 00000000-0000-0000-0000-000000000000, , Shutting down: Master
2018-05-20T21:55:32.873754Z, INFO, 00000000-0000-0000-0000-000000000000, , Reason: Worker failed to boot.
2018-05-20 21:55:32,905 INFO exited: gunicorn (exit status 3; not expected)
2018-05-20 21:55:33,906 INFO gave up: gunicorn entered FATAL state, too many start retries too quickly
2018-05-20 21:55:34,908 WARN program_exit: bad result line: 'Killing supervisor with this event: ver:3.0 server:supervisor serial:0 pool:program_exit poolserial:0 eventname:PROCESS_STATE_FATAL len:58'
2018-05-20 21:55:34,908 WARN program_exit: has entered the UNKNOWN state and will no longer receive events, this usually indicates the process violated the eventlistener protocol
2018-05-20 21:55:34,908 WARN received SIGQUIT indicating exit request
2018-05-20 21:55:34,908 INFO waiting for nginx, rsyslog, program_exit to die
2018-05-20 21:55:35,913 INFO stopped: nginx (exit status 0)
2018-05-20 21:55:35,914 INFO stopped: program_exit (terminated by SIGTERM)
2018-05-20 21:55:35,915 INFO stopped: rsyslog (exit status 0)

Received 64 lines of log

Is there anything missing that I should have done?

I am not sure if this has anything to do with this issue, but I choose the file name my_ConvNet_MNIST_model.data-00000-of-00001, as instructed in 2, which is different from my_ConvNet_MNIST_model.data-00001-of-00000 in the az ml service create command.

Thanks in advance.

Improvement suggestion - Validate fields like username and password

I created the DSVM from https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FDataScienceVM%2Fmaster%2FExtensions%2FIoTEdge%2Fazuredeploy.json a couple times but upon VM deployment, I could not authenticate my SSH session with the following error:
Permission denied (publickey,password)

Then I realised that there is a new username requirement for Linux VMs on Azure which requires that the string must be at least 8 characters long. I used the standard "azureuser" and it works.

Improvement suggestion - please validate fields in https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FDataScienceVM%2Fmaster%2FExtensions%2FIoTEdge%2Fazuredeploy.json

00-anomaly-detection-tutorial: Container fails to start on ACI

Tutorial: 00-anomaly-detection-tutorial
Step: 5: Test model on Azure Container Instance

Issue: Container failed to start on ACI. Error log:

2020-06-28T11:55:11,852765769+00:00 - rsyslog/run
2020-06-28T11:55:11,852562969+00:00 - iot-server/run
2020-06-28T11:55:11,864671678+00:00 - nginx/run
2020-06-28T11:55:11,865188479+00:00 - gunicorn/run
EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...
2020-06-28T11:55:12,130239579+00:00 - iot-server/finish 1 0
2020-06-28T11:55:12,135812284+00:00 - Exit code 1 is normal. Not restarting iot-server.
Starting gunicorn 19.9.0
Listening at: http://127.0.0.1:31311 (11)
Using worker: sync
worker timeout is set to 300
Booting worker with pid: 43
Exception in worker process
Traceback (most recent call last):
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/var/azureml-server/wsgi.py", line 1, in
import create_app
File "/var/azureml-server/create_app.py", line 3, in
from app import main
File "/var/azureml-server/app.py", line 31, in
import main as user_main
File "/var/azureml-app/main.py", line 18, in
driver_module_spec.loader.exec_module(driver_module)
File "iot_score.py", line 8, in
from sklearn.externals import joblib
ImportError: cannot import name 'joblib'
Worker exiting (pid: 43)
Shutting down: Master
Reason: Worker failed to boot.
2020-06-28T11:55:15,652639446+00:00 - gunicorn/finish 3 0
2020-06-28T11:55:15,653865147+00:00 - Exit code 3 is not normal. Killing image.

Support for Multiple Outputs

Do AML modules support multiple outputs to IoT Edge Hub that could be configured based on decision outcome? Particularly to support alerting and notification systems local to the edge runtime?

"Next Steps" section links are now redirect

All of the links in the 'Next Steps' section like predictive maintenance and aerial classification now redirect to the article: "What happened to Azure Machine Learning Workbench?" If these samples are still available, can the links be updated or can the links be removed / changed to other samples if no longer available?