Using Azurite to run blob storage tests in Azure DevOps Pipeline

Overview

This repo demonstrates the approach for writing automated tests with a short feedback loop (i.e. unit tests) against security considerations (private endpoints) for Azure Blob Storage functionality.

Once private endpoints are enabled for the Azure Storage accounts, tests will fail, when executed locally or as part of a pipeline, because this connection will be blocked.

This repo contains a sample for Azure Blob Storage that will upload images from a url and upload files from a local folder. To run, cp sample_env.txt .env file to load environment variables for running code on local or pipeline and using conftest.py to load environment variables for tests.

Getting Started

Folder structure

Here's the folder structure for the sample:

• build
  • azure-pipelines.yml - Azure Pipelines yaml file
• data - Local files examples to be uploaded to Blob Storage
• src
  • azure_blob_storage.py - Azure Blob Storage Sample Code
• tests
  • conftest.py - Configuration file for running tests
  • test_azure_blob_storage.py - Azure Blob Storage Test Code
• .env - Environment variable files for Blob Storage
• docker-compose.yml - Docker Compose yaml file to run Azurite Docker Image
• requirements.txt - Required pip packages to run python code on your local and pipeline
• requirements_dev.txt - Required pip packages to run test code on your local and pipeline

Prerequisites

• Azure Subscription
• Azure Resource Group
• Github Account
• Latest Stable Python Version. When installing python, please add the specific version you will be using to the path. This will ensure a consistent experience when using the python commands in the tutorial.
• Node JS
• Docker: if installing Azurite using docker-compose.
• Azurite
• Azure Storage Explorer.
• Azure Devops Account with an available project to house the deployment pipeline.

Install and Launch Azurite in local environment

There are several different ways to install and run Azurite on your local system as listed here. In this document, we will cover Install and run Azurite using NPM and Install and run the Azurite Docker image.

a. Using NPM

In order to run Azurite V3 you need Node.js >= 8.0 installed on your system. Azurite works cross-platform on Windows, Linux, and OS X.

After the Node.js installation, you can install Azurite simply with npm which is the Node.js package management tool included with every Node.js installation.

# Install Azurite
npm install -g azurite

# Create azurite directory
mkdir c:/azurite

# Launch Azurite for Windows
azurite --silent --location c:\azurite --debug c:\azurite\debug.log

#Launch Azurite for MacOs or Linux:
azurite -s -l /usr/local/lib/node_modules/azurite -d /usr/local/lib/node_modules/azurite/debug.log

b. Using docker image

Docker Compose will run the docker image using the docker-compose.yml file.

docker-compose up

Either option (a) Using NPM or (b) Using docker image, the output should be:

Azurite Blob service is starting at http://127.0.0.1:10000
Azurite Blob service is successfully listening at http://127.0.0.1:10000
Azurite Queue service is starting at http://127.0.0.1:10001
Azurite Queue service is successfully listening at http://127.0.0.1:10001

Run tests on local

Python 3.8 is used for this, but it should also work fine on other 3.6+ versions.

1. To test and see how these endpoints are running, you can attach your local blob storage to the Azure Storage Explorer. In Azure Storage Explorer:

• right click on Storage Accounts and select Connect to Azure Storage

• then select Attach to a local emulator
  

2. Provide a Display name and port number, then your connection will be ready and you can use Storage Explorer to manage your local blob storage.
   

Do not forget to start your emulator, Storage Explorer will not start it for you.

3. Create a virtual python environment from working directory:

   python3 -m venv env
source env/bin/activate [on Linux] or env/scripts/activate [on Windows]

4. Install the dependencies: python3 -m pip install -r requirements_dev.txt

5. Run tests:

   python3 -m pytest ./tests

After the tests run, you can see the files in your local blob storage.

Run tests on Azure DevOps Pipelines

After successfully running tests on local, run the azure-pipelines build yaml file using Azure DevOps Pipelines.

Note: You will need

• an existing Azure subscription
• an existing Azure DevOps organization and project.

1. Login to Azure DevOps

2. Under the Organization and Project, select the sample folder git Repo and Configure a Pipeline using the build/azure-pipelines.yml

3. Run the Pipeline

After a successful set up and running the pipeline in Azure DevOps Pipelines, the result should look like:

License:

See LICENSE.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Contributing

See CONTRIBUTING