Set Up a Minimal MLOps Stack on Azure
How to set up a minimal stack on Microsoft Azure
To get started using ZenML on the cloud, you need some basic infrastructure up and running which ZenML can use to run your pipelines. This step-by-step guide explains how to set up a basic cloud stack on Azure.
This guide represents one of many ways to create a cloud stack on Azure. You can customize this by adding additional components of replacing one of the components described in this guide.

What will the stack look like?

Prerequisites

  • Docker installed and running.
  • kubectl installed.
  • The az CLI installed and authenticated.
  • ZenML and the integrations for this tutorial stack installed:
    pip install zenml
    zenml integration install azure kubernetes

Setting up the Azure resources

All the Azure setup steps can either be done using the Azure UI or CLI. Simply select the tab for your preferred option and let's get started. First open up a terminal which we'll use to store some values along the way which we'll need to configure our ZenML stack later.

Create an account

If you don't have an Azure account yet, go to https://azure.microsoft.com/en-gb/free/ and create one.

Create a resource group

Resource groups are a concept in Azure that allows us to bundle different resources that share a similar lifecycle. We'll create a new resource group for this tutorial so we'll be able to differentiate them from other resources in our account and easily delete them at the end.
When creating a resource group, you need to provide a location for that resource group. You may be wondering, "Why does a resource group need a location? And, if the resources can have different locations than the resource group, why does the resource group location matter at all?" The resource group stores metadata about the resources. Therefore, when you specify a location for the resource group, you are specifying where that metadata is stored. For compliance reasons, you may need to ensure that your data is stored in a particular region. https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview
Azure UI
Azure CLI
  • Go to the Azure portal, click the hamburger button in the top left to open up the portal menu. Then, hover over the Resource groups section until a popup appears and click on the + Create button.
  • Select a region and enter a name for your resource group before clicking on Review + create.
  • Verify that all the information is correct and click on Create.
  • Set the following variables in your shell.
    RESOURCE_GROUP=<RESOURCE_GROUP_NAME>
    RG_LOCATION=<RG_LOCATION>
RESOURCE_GROUP=<RESOURCE_GROUP_NAME>
RG_LOCATION=<RG_LOCATION>
az group create --name $RESOURCE_GROUP --location $RG_LOCATION

Create a storage account

An Azure storage account is a grouping of Azure data storage objects which also provides a namespace and authentication options to access them. We'll need a storage account to hold the blob storage container we'll create in the next step.
Azure UI
Azure CLI
  • Open up the portal menu again, but this time hover over the Storage accounts section and click on the + Create button in the popup once it appears:
  • Select your previously created resource group, a region and a globally unique name and then click on Review + create:
  • Make sure that all the values are correct and click on Create:
  • Wait until the deployment is finished and click on Go to resource to open up your newly created storage account:
  • In the left menu, select Access keys:
  • Click on Show keys, and once the keys are visible, note down the storage account name and the value of the Key field of either key1 or key2. We're going to use them for the <STORAGE_ACCOUNT_NAME> and <STORAGE_ACCOUNT_KEY> placeholders later.
  • Set the following variables in your shell.
    REGION=<REGION>
    STORAGE_ACCOUNT_NAME=<STORAGE_ACCOUNT_NAME>
    STORAGE_ACCOUNT_KEY=<STORAGE_ACCOUNT_KEY>
# Set a name for your bucket and the Azure region for your resources
REGION=<REGION>
STORAGE_ACCOUNT_NAME=<STORAGE_ACCOUNT_NAME>
az storage account create -n $STORAGE_ACCOUNT_NAME -g $RESOURCE_GROUP -l $REGION
STORAGE_ACCOUNT_KEY=$(az storage account keys list -g $RESOURCE_GROUP -n $STORAGE_ACCOUNT_NAME --query [0].value -o tsv)

Create an Azure Blob Storage Container

Next, we're going to create an Azure Blob Storage Container. It will be used by ZenML to store the output artifacts of all our pipeline steps.
Azure UI
Azure CLI
  • To do so, select Containers in the Data storage section of the storage account:
  • Then click the + Container button on the top to create a new container:
  • Choose a name for the container and note it down. We're going to use it later for the <BLOB_STORAGE_CONTAINER_NAME> placeholder. Then create the container by clicking the Create button.
  • Set the following variables in your shell.
    BLOB_STORAGE_CONTAINER_NAME=<BLOB_STORAGE_CONTAINER_NAME>
BLOB_STORAGE_CONTAINER_NAME=<BLOB_STORAGE_CONTAINER_NAME>
az storage container create --$BLOB_STORAGE_CONTAINER_NAME

Set up a MySQL database

Now let's set up a managed MySQL database. This will act as ZenML's metadata store and store metadata regarding our pipeline runs which will enable features like caching and establish a consistent lineage between our pipeline steps.
Azure UI
Azure CLI
  • Open up the portal menu and click on + Create a resource.
  • Search for Azure Database for MySQL and once found click on Create.Make sure you select Flexible server and then continue by clicking the Create button.
  • Select a resource group and region and fill in values for the server name as well as admin username and password. Note down the username and password you chose as we're going to need them later for the <MYSQL_USERNAME> and <MYSQL_PASSWORD> placeholders. Then click on Next: Networking.
  • Now click on Add 0.0.0.0 - 255.255.255.255 to allow access from all public IPs. This is necessary so the machines running our GitHub Actions can access this database. It will still require username, password as well as a SSL certificate to authenticate.
  • In the opened up popup, click on Continue and click on Review + create.
  • Verify the configuration and click the Create button. Now we'll have to wait until the deployment is finished (this might take ~15 minutes).
Note: If the deployment fails for some reason, delete the resource and restart from the beginning of this section.
  • Once the deployment is finished, click on Go to resource.
  • On the overview page of your MySQL server resource, note down the server name in the top right. We'll use it later for the <MYSQL_SERVER_NAME> placeholder.
  • Then click on Networking in the left menu. Click on Download SSL Certificate on the top. Make sure to note down the path to the certificate file which we'll use for the <PATH_TO_SSL_CERTIFICATE> placeholder in a later step.
  • Set the following values in your shell.
    MYSQL_SERVER_NAME=<MYSQL_SERVER_NAME>
    PATH_TO_SSL_CERTIFICATE=<PATH_TO_SSL_CERTIFICATE>
    MYSQL_USERNAME=<MYSQL_USERNAME>
    MYSQL_PASSWORD=<MYSQL_PASSWORD>
MYSQL_SERVER_INSTANCE_NAME=<MYSQL_SERVER_INSTANCE_NAME>
MYSQL_USERNAME=<MYSQL_USERNAME>
MYSQL_PASSWORD=<MYSQL_PASSWORD>
az mysql flexible-server create -l $REGION -g $RESOURCE_GROUP \
-n $MYSQL_SERVER_INSTANCE_NAME -u $MYSQL_USERNAME -p $MYSQL_PASSWORD \
--public-access 0.0.0.0-255.255.255.255
MYSQL_SERVER_NAME="$MYSQL_SERVER_INSTANCE_NAME.mysql.database.azure.com"
PATH_TO_SSL_CERTIFICATE=<PATH_TO_SSL_CERTIFICATE> # should end with a filename like, /../../certificate.pem
wget --no-check-certificate https://dl.cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem -O $PATH_TO_SSL_CERTIFICATE

Container Registry

Azure UI
Azure CLI
  • Set up an Azure Container Registry (ACR).
  • Set the regsitry name in your shell.
    REGISTRY_NAME=<REGISTRY_NAME> # should be <some-name>.azurecr.io
REGISTRY_NAME=<REGISTRY_NAME> # should be <some-name>.azurecr.io
az acr create -n $REGISTRY_NAME -g $RESOURCE_GROUP

Orchestrator (Azure Kubernetes Service)

Azure UI
Azure CLI
  • On the Azure portal menu or from the Home page, select Create a resource. Select Containers > Kubernetes Service.
  • On the Basics page, configure the following options. Under Project details,
    • Select your Azure Subscription.
    • Select your Resource group.
  • Under Cluster details,
    • Ensure the the Preset configuration is Standard ($). For more details on preset configurations, see Cluster configuration presets in the Azure portal.
    • Enter a Kubernetes cluster name. We will use it later as <AKS_CLUSTER_NAME>
  • Enter your Region for the AKS cluster, and leave the default value selected for Kubernetes version.
  • Keep the default values for all other sections and click on Review + Create.
  • Set the following variable in your shell.
    AKS_CLUSTER_NAME=<AKS_CLUSTER_NAME>
AKS_CLUSTER_NAME=<AKS_CLUSTER_NAME>
az aks create -g $RESOURCE_GROUP -n $AKS_CLUSTER_NAME --node-count 1 \
--enable-cluster-autoscaler

Register the ZenML stack

  • Register the artifact store:
    zenml artifact-store register azure_store \
    --flavor=azure \
    --path=az://$BLOB_STORAGE_CONTAINER_NAME
  • Register the container registry and authenticate your local docker client
    zenml container-registry register acr_registry \
    --flavor=azure \
    --uri=$REGISTRY_NAME
    az acr login --name $REGISTRY_NAME
  • Register the metadata store:
    zenml metadata-store register azure_mysql \
    --flavor=mysql \
    --database=zenml \
    --secret=azure_authentication \
    --host=$MYSQL_SERVER_NAME
  • Register the secrets manager:
    zenml secrets-manager register azure_secrets_manager \
    --flavor=azure \
    --region_name=$REGION
  • Configure your kubectl client and register the orchestrator:
    az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME
    kubectl create namespace zenml
    zenml orchestrator register aks_kubernetes_orchestrator \
    --flavor=kubernetes \
    --kubernetes_context=$(kubectl config current-context)
  • Register the ZenML stack and activate it:
    zenml stack register kubernetes_stack \
    -o aks_kubernetes_orchestrator \
    -a azure_store \
    -m azure_mysql \
    -c acr_registry \
    -x azure_secrets_manager \
    --set
  • Register the secret for authenticating with your MySQL database:
    zenml secret register azure_authentication \
    --schema=mysql \
    --user=$MYSQL_USERNAME \
    --password=$MYSQL_PASSWORD
After all of this setup, you're now ready to run any ZenML pipeline on Azure!

Quick setup

If you're looking for a way to get started quickly, we've combined all the commands so you can copy-paste them and execute them in a single go. You'll only need to set values for the <REGION>, <RESOURCE_GROUP> and <MYSQL_PASSWORD> right at the beginning before executing the rest. If you're on Windows, also adjust the certificate path accordingly.
Quick setup commands