Run on Azure
A simple guide to create an Azure stack to run your ZenML pipelines
This page aims to quickly set up a minimal production stack on Azure. With just a few simple steps, you will set up a resource group, a service principal with correct permissions and the relevant ZenML stack and components.
Would you like to skip ahead and deploy a full Azure ZenML cloud stack already?
Check out the in-browser stack deployment wizard, the stack registration wizard, or the ZenML Azure Terraform module for a shortcut on how to deploy & register this stack.
To follow this guide, you need:
An active Azure account.
ZenML installed.
ZenML
azure
integration installed withzenml integration install azure
.
1. Set up proper credentials
You can start by creating a service principal by creating an app registration on Azure:
Go to the App Registrations on the Azure portal.
Click on
+ New registration
,Give it a name and click register.
Once you create the service principal, you will get an Application ID and Tenant ID as they will be needed later.
Next, go to your service principal and click on the Certificates & secrets
in the Manage
menu. Here, you have to create a client secret. Note down the secret value as it will be needed later.
2. Create a resource group and the AzureML instance
Now, you have to create a resource group on Azure. To do this, go to the Azure portal and go to the Resource Groups
page and click + Create
.
Once the resource group is created, go to the overview page of your new resource group and click + Create
. This will open up the marketplace where you can select a variety of resource to create. Look for Azure Machine Learning
.
Select it, and you will start the process of creating an AzureML workspace. As you can see from the Workspace details
, AzureML workspaces come equipped with a storage account, key vault, and application insights. It is highly recommended that you create a container registry as well.
3. Create the required role assignments
Now, that you have your app registration and the resources, you have to create the corresponding role assignments. In order to do this, go to your resource group, open up Access control (IAM)
on the left side and +Add
a new role assignment.
In the role assignment page, search for AzureML which will show you a list of roles defined with the scope of AzureML workspaces.
One by one, you have to select AzureML Compute Operator
, AzureML Data Scientist
, and AzureML Registry User
and click Next
.
Finally, click +Select Members
, search for your registered app by its ID, and assign the role accordingly.
4. Create a service connector
Now you have everything set up, you can go ahead and create a ZenML Azure Service Connector.
You will use this service connector later on to connect your components with proper authentication.
5. Create Stack Components
In order to run any workflows on Azure using ZenML, you need an artifact store, an orchestrator and a container registry.
Artifact Store (Azure Blob Storage)
For the artifact store, we will be using the storage account attached to our AzureML workspace. But before registering the component itself, you have to create a container for blob storage. To do this, go to the corresponding storage account in your workspace and create a new container:
Once you create the container, you can go ahead, register your artifact store using its path and connect it to your service connector:
For more information regarding Azure Blob Storage artifact stores, feel free to check the docs.
Orchestrator (AzureML)
As for the orchestrator, no additional setup is needed. Simply use the following command to register it and connect it your service connector:
For more information regarding AzureML orchestrator, feel free to check the docs.
Container Registry (Azure Container Registry)
Similar to the orchestrator, you can register and connect your container registry using the following command:
For more information regarding Azure container registries, feel free to check the docs.
6. Create a Stack
Now, you can use the registered components to create an Azure ZenML stack:
7. ...and you are done.
Just like that, you now have a fully working Azure stack ready to go. Feel free to take it for a spin by running a pipeline on it.
Define a ZenML pipeline:
Save this code to run.py and execute it. The pipeline will use Azure Blob Storage for artifact storage, AzureML for orchestration, and an Azure container registry.
Now that you have a functional Azure stack set up with ZenML, you can explore more advanced features and capabilities offered by ZenML. Some next steps to consider:
Dive deeper into ZenML's production guide to learn best practices for deploying and managing production-ready pipelines.
Explore ZenML's integrations with other popular tools and frameworks in the machine learning ecosystem.
Join the ZenML community to connect with other users, ask questions, and get support.
Last updated