• Simple copy ADF pipeline
• Git integrated
• AzDevOps pipeline which releases to a separate environment (ADF)

Initial basic build pipeline that

1. Restore nuget packages (if any)
2. Build SQL Project and produces DACPAC
3. Run tests (if any)
4. Publishes DACPAC as build artifacts

Simple Azure Function that will:

• Read from EventHub filteredDevices
• Split steam based on temperature
  • Temperature <100: Send output to Eventhub TemperatureOutput
  • Temperature >100: Send output to Eventhub TemperatureBadOutput

Currently, deploying the Azure Pipelines is a manual step. Ideally this is part of the deployment script.

Script that deploys the following:

1. AzureSQL DB
2. CI/CD pipeline

Add second stage as the Release pipeline that deploys DACPAC to AzureSQL.

Deployment script needs to be re-runable so that it can recover from 'half run' states.

• if AzDevOps service connection, pipelines already exist, then either drop recreate, or skip -- make this configurable

Initial basic build pipeline that

1. Restore nuget packages (if any)
2. Build SQL Project and produces DACPAC
3. Run tests (if any)
4. Publishes DACPAC as build artifacts

Use IoT Telemetry Simulator and create a script that will generate a mixture of good/bad data.

The testing mixture should allow the pipeline stages to be worked at different ratios.

Ratio:

• 50% filtered out deviceId >= 1,000
• 40% "good" data deviceId <1,000 AND temperature <100
• 10% "bad" data deviceId <1,000 AND temperature >=100

Include the following:

• Overview
• Prerequisites - any necessary software pre-requisites
• Setup - how to setup and deploy the sample
• Running the sample - after deploying, how to do a build and release

Add Unit test to demonstrate testing Stored Procedures and UDFs

Create a sample pipelines which stores secrets in KeyVault, ideally demonstrating deploying across environments.

Create an ARM template that deploys Azure Synapse Analytics instance with default values. ARM template should auto-generate names of the resource(s) based on the Resource Group and require minimum input parameters.

Pipeline:

1. Restore a snapshot of Production database to Staging Database
2. Deploy SQL DACPAC changes to Staging Database.
3. Run tests.
4. Create "roll-forward" script between Production and Staging.
5. Test "roll-forward" script on Staging.
6. Re-run tests.
7. Deploy changes to Production.

Description

Update existing projects to reflect agreed way of managing work.

Add second stage as the Release pipeline that deploys DACPAC to Azure Synapse Analytics Instance.

I am thinking of running in github action

• https://github.com/avto-dev/markdown-lint
• https://github.com/gaurav-nelson/github-action-markdown-link-check

Ensure the salesdb deployed has data

• Automatically deploys samples to a targe AzDevOps and Azure subscription

I want to use the devcontainer for the Parking Sensors example.
I also verified that I have to do that by opening the repo from the /e2e_samples/parking_sensors folder in order to get vscode to compile it.

Initially I wanted to update the documentation with that information but found out that since we are now in a subfolder of the git repo you can not use git because it relies on the info in .git folder.

I followed the discussion in the PR on reasons not to put it on root level.

Still I want to get the full dev experience for e.g. contributing and deploying the samples.

Ideas:

1. Keeping the individual devcontainer related code in the sample folder level and document to copy the relevant one to root folder of the repo in order to get the full developer experience for this project X sample.
2. This approach here Connecting to multiple containers at once looks most interesting to me, I will investigate if we can set it up in a way that only the devcontainer needed is started. Here the container will be able to see the repo root based .git folder. It looks to me the most user friendly way if we don't end up spinning up 10+ devcontainers on the dev's host.

CC @devlace @Azadehkhojandi

Hi,

I got an error in the deployment. Seems that the scope in following line is incorrect:

I was able to fix this by removing the slash after $DATAFACTORY_NAME.

Currently, Release Pipelines are built using the visual (classic) editor and much harder to include in an automated deployment script. Need to convert this to YAML.

Integration tests which tests data from test input API are correctly outputted to Azure Synapse (SQLDW). This would require an additional Test environment, preferably that can be spun up automatically as part of the pipeline.

Currently, only unit tests are demonstrated.

I followed the deployment steps in the README and I got a ADF Pipe with "https://australiaeast.azuredatabricks.net" as the Databricks Workspace URL. This is wrong... at least if you deploy everything with RESOURCE_GROUP_LOCATION=northeurope.

So all the databricks connections will fail in die ADF pipe due to the wrong workspace URL.

• Translate existing Azure DevOps Pipelines into Github Actions:
  https://github.com/Azure-Samples/modern-data-warehouse-dataops/tree/master/e2e_samples/parking_sensors/devops

• Document limitations if any

Currently pipelines are deployed manually. This can be automated with the az devops cli extension.

Include the following:

• Overview
• Prerequisites - any necessary software pre-requisites
• Setup - how to setup and deploy the sample
• Running the sample - after deploying, how to do a build and release

Currently, deployment script has the source branch for the AzDevOps Pipelines YAML definitions hardcoded to master. We need to change this so that this is configurable via Environment variable so that Developers can easily tests changes in other branches.

Create a terraform template that creates the following components:

• Application Insights
• EventHub: Ingress
• Azure Function: Include list filter
• EventHub: FilteredDevices
• Azure Function: Temperature filter
• EventHub: TemperatureOutput
• EventHub: TemperatureBadOutput

Script that deploys the following:

• Azure Synapse Analytics instance
• CI/CD pipeline

documentation for observability learnings

As blah, and to foo

• Each deployment should have an "id"/name. This can either be User Supplied or (if not supplied) generated random string. This id can then be appended as a suffix to azure resources which require unique names, such as sqlserver and keyvault
• This is so that resources can be easily identified to as to which deployment they belong to.
• All Resources should be tagged with this deployment id and "mdw-dataops".

A pipeline.yaml file, that can deploy the 2 Azure functions to an existing Azure subscription that has had infrastructure pre-created via the Terraform template

Azure functions:

• Device include-list
• temperature stream split

Options:

• GitHub actions pipeline yaml
• Azure Devops pipeline yaml.

A pipeline.yaml file, that can execute the terraform template.

Options:

• GitHub actions pipeline yaml
• Azure Devops pipeline yaml.

in the file mdw-dataops-clone/e2e_samples/parking_sensors/devops/azure-pipelines-cd-release.yml line 16, the repository name needs to be a variable to avoid an error deploying ADF

Document the steps in the Build and Release Pipeline, preferably with supplementary diagram.

I think there are two options for this.

1. github action to filter code in addition to default credscan in github
   Pros: It prevents commits to be merge from PR to main branch. And it can be applied to all the PR easily
   Cons: It does not prevent to commit to branch
2. Adding githook to check code before commiting anything
   Pros: It prevents to commit cred at all
   Cons: Each engineer needs to add this.

We could do both but need to prioritize which one to work first.
I am thinking to work on 1 first.

Currently Synapse is deployed separately (manually). This needs to be part of the larger deployment script.

Simple Azure Function that will:

• Read from EventHub ingress
• Filter out messages where sensorId > 1,000
• Send output to Eventhub filteredSensors

Create sample CI/CD Github Action pipelines, similar to the existing three Azure DevOps pipelines.

Include a diagram showing build and release pipeline and explanation of the process.

Looks like the linked-services reference is pointing to the incorrect folder.

Updating Data Factory LinkedService to point to newly deployed resources (KeyVault and DataLake). jq: error: Could not open file .tmp/adf/linkedService/Ls_KeyVault_01.json: No such file or directory jq: error: Could not open file .tmp/adf/linkedService/Ls_AdlsGen2_01.json: No such file or directory Deploying Data Factory artifacts. Creating ADF LinkedService: Ls_KeyVault_01 Unsupported Media Type({"message":"The request contains an entity body but no Content-Type header. The inferred media type 'application/octet-stream' is not supported for this resource."})

Fixes either:

1. Update the copy command to copy everything to .tmp/adf/

2. Update adfLsDir to point to .tmp/linkedService instead of .tmp/adf/linkedService

Data monitoring relies on logging to Application Insights. Currently, this is not deployed as part of the deployment script.

• Pipeline triggers on any PR to master.
• Builds DACPAC, runs tests (if any)
• Does not produce a build artifact.