azure / ace_azure_ml Goto Github PK

This repository contains training material related to Azure and Machine Learning

Home Page: https://azure.github.io/ACE_Azure_ML/

Jupyter Notebook 99.42% Python 0.56% Shell 0.01%

ace_azure_ml's Introduction

Disclaimer: This repository is not actively maintained anymore. For an up-to-date introduction to AzureML, please see the official product documentation and example notebooks.

Introduction

Welcome to the ACE-team training on Azure Machine Learning (AML) service.

The material presented here is a deep-dive which combine real-world data science scenarios with many different technologies including Azure Databricks (ADB), Azure Machine Learning (AML) Services and Azure DevOps, with the goal of creating, deploying, and maintaining end-to-end data science and AI solutions.

Anomaly Detection in structured data

The data scientist has been tasked to develop a predictive maintenance (PdM) solution for a large set of production machines on a manufacturing floor.
The data scientist was asked to create a PdM solution that is executed weekly, to develop a maintenance schedule for the next week.
Previous experience suggests that anomalies in the telemetry data collected on each machine are indicative of impending catastrophic failures. The data scientist was therefore asked to include anomaly detection in their solution.
The organization also asked for a real-time anomaly detection service, to enable immediate machine inspection before the beginning of the next maintenance cycle.

Note: Anomaly detection can also be performed on unstructured data. One example is to detect unusual behavior in videos, like a car driving on a sidewalk, or violation of safety protocols on a manufacturing floor. If you are interested in this use case, please go to this repo: https://github.com/Microsoft/MLOps_VideoAnomalyDetection

Agendas

Please go to this page to find alternative agendas around the above use-cases.

References

Pre-requisites

Knowledge/Skills

You will need this basic knowledge:

Basic data science and machine learning concepts.
Moderate skills in coding with Python and machine learning using Python.
Familiarity with Jupyter Notebooks and/or Databricks Notebooks.
Familiarity with Azure databricks.
Basic skills using Git version control.

If you do not have any of the above pre-requisites, please find below links:

To Watch: Data Science for Beginners
To Watch: Get Started with Azure Machine Learning
To Watch: Python for Data Science: Introduction
To Watch: Jupyter Notebook Tutorial: Introduction, Setup, and Walkthrough
To Do: Go to [https://notebooks.azure.com/] and create and run a Jupyter notebook with Python
To Watch: Azure Databricks: A brief introduction
To Read (10 mins): Git Handbook

Infrastructure

An Azure Subscription (unless provided to you).
If you are not provided with a managed lab environment (course invitation will specify), then follow these instructions for configuring your development environment prior to the course or if you do it on your own. You will need an Azure Subscription (unless one is provided to you). Pay particular attention to version numbers, such as the version of the Spark runtime.

Contribute

We invite everybody to contribute.

Email us with questions an comments: [email protected]
Submit issues and comments in our github repository: https://github.com/Azure/LearnAI-CustomAI-Airlift/issues

ace_azure_ml's People

Contributors

Stargazers

Watchers

Forkers

satonaoki varunoist chrismbenson gloveboxes chitratsr neohack22 pcoric jpchen89 cbailiss dem108 raina777 mithun-prasad arielcedola star1262 shivp950 tianlai shalevy1 aid91 julmoser himanir balakrishnapushparajan yuvasri09 rasmi20 canoas sxycwzwzq mail2lalith antoniosql ghbaek64 andrewkim brntbeer bryancross kmoussa soniaang lidagh abhi0751 je-barker fasalzaman jmservera sbrillo89 mdragt tomz sanjajo alfeuduran olufemig secopsgit flipwebapps digitalarche osama222 keshava yiyangabc ganeshvedula subhransusahoo ssheff tusharkalecam totoro-ds isabella232 daliatarek huyuezhi luoanabecher blaise442 mohantym sourcecodecheck sithea-nou

ace_azure_ml's Issues

presenter/notebooks/day_1/01_introduction.ipynb auth failure with srorage

the mnt notebook runs successfully, from teh intor notebook, however when you det to the "%fs ls /mnt/data/telemetry" command it fails with the following error.

shaded.databricks.org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

01_introduction (cmd 6) StorageException: Server failed to authenticate the request

I failed to access the storage that data was stored.
Does anyone please confirm that SAS of Azure Blob storage is still working?

%fs ls /mnt/data/telemetry
[error msg]
shaded.databricks.org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

This repo is missing a license file

This repository is currently missing a LICENSE.MD file outlining its license. A license helps users understand how to use your project in a compliant manner. You can find the standard MIT license text at the Microsoft repo templates LICENSE file: https://github.com/microsoft/repo-templates/blob/main/shared/LICENSE.
If you would like to learn more about open source licenses, please visit the document at https://aka.ms/license.

01_introduction (cmd 46) - error: col("volt") not recognized

this col() code depends on an import that was not available yet, it should be instead:

from pyspark.sql.functions import col
display(telemetryDF.filter(col("volt") > 250).limit(10))

alternatively, DF column can also be referenced without pyspark using:

display(telemetryDF.filter(telemetryDF.volt > 250).limit(10))

would prefer to submit a pull request, but these notebooks are somehow baked inside a binary file we imported at the beginning of the lab
/presenter/notebooks.dbc

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Data files missing

Pls guide me from where I can get the Data Files needed to complete the labs. Immediately looking for Day 1 data files.

Link broken to "Part 1: Azure Databricks and its integration with Azure Machine Learning Services"

Link on bottom of this page: https://azure.github.io/LearnAI-CustomAI-Airlift/

"Part 1: Azure Databricks and its integration with Azure Machine Learning Services" is broken.

illegal Argument exception

I am getting error while running cmd 15 in 05_structured_streaming Notebook

Unexpected error (os.listdir fails) on mnt_blob issue, refactor

existing check for existing mount not the best and fails first time, something like this?
(ignoring the Exception and not reporting also particularly nasty to debug)

if os.path.exists('/dbfs/mnt/data/'):
print("Already mounted.")
else:
dbutils.fs.mount(
source = source,
mount_point = mount_point,
extra_configs = extra_configs)
print("Mounted: %s at %s" % (source, mount_point))

Links under Proposed three-day agenda are broken

Most of the links under Proposed three-day agenda are broken and giving 404 error

day 1
Machine Learning on Azure - Solution Brief (PPTX Presentation)
Introduction to Azure Databricks (Hands On)
Feature Engineering (Hands On)
Sentiment Analysis (Hands On)
Hyperparameter Tuning w/ Azure Databricks (Hands On)
Streaming on Azure Databricks (Hands On)
day 2
Logistic Regression w/ Spark ML (Hands On)
Random Forests w/ Spark ML (Hands On)
Integration of ADB and AML (PPTX Presentation)
Getting Started w/ AML service (Hands On)
AML Model Management and ML Experimentation (Hands On)
Automated ML (Hands On)
Real-time scoring w/ AKS, CI/CD and DevOps (PPTX Presentation)
day 3
Real-time scoring w/ AKS, CI/CD and DevOps (Hands On)
CI/CD and DevOps (Hands On)
Then continue with 5h on video anomaly detection