Azure Blob Storage
How to store artifacts using Azure Blob Storage
The Azure Artifact Store is an Artifact Store flavor provided with the Azure ZenML integration that usesthe Azure Blob Storage managed object storage service to store ZenML artifacts in an Azure Blob Storage container.
Running ZenML pipelines with the local Artifact Store is usually sufficient if you just want to evaluate ZenML or get started quickly without incurring the trouble and the cost of employing cloud storage services in your stack. However, the local Artifact Store becomes insufficient or unsuitable if you have more elaborate needs for your project:
- if you want to share your pipeline run results with other team members or stakeholders inside or outside your organization
- if you have other components in your stack that are running remotely (e.g. a Kubeflow or Kubernetes Orchestrator running in public cloud).
- if you outgrow what your local machine can offer in terms of storage space and need to use some form of private or public storage service that is shared with others
- if you are running pipelines at scale and need an Artifact Store that can handle the demands of production grade MLOps
In all these cases, you need an Artifact Store that is backed by a form of public cloud or self-hosted shared object storage service.
You should use the Azure Artifact Store when you decide to keep your ZenML artifacts in a shared object storage and if you have access to the Azure Blob Storage managed service. You should consider one of the other Artifact Store flavors if you don't have access to the Azure Blob Storage service.
The Azure Artifact Store flavor is provided by the Azure ZenML integration, you need to install it on your local machine to be able to register an Azure Artifact Store and add it to your stack:
zenml integration install azure -y
The only configuration parameter mandatory for registering an Azure Artifact Store is the root path URI, which needs to point to an Azure Blog Storage container and takes the form
abfs://container-name. Please read the Azure Blob Storage documentation on how to configure an Azure Blob Storage container.
With the URI to your Azure Blob Storage container known, registering an Azure Artifact Store can be done as follows:
# Register the Azure artifact store
zenml artifact-store register az_store -f azure --path=az://container-name
# Register and set a stack with the new artifact store
zenml stack register custom_stack -a az_store ... --set
Integrating and using an Azure Artifact Store in your pipelines is not possible without employing some form of authentication. ZenML currently provides two options for configuring Azure credentials, the recommended one being to use a
Secrets Managerin your stack to store the sensitive information in a secure location.
You will need the following information to configure Azure credentials for ZenML, depending on which type of Azure credentials you want to use:
- an Azure connection string
- an Azure account key
- the client ID, client secret and tenant ID of the Azure service principal
Secrets Manager (Recommended)
This method uses the implicit Azure authentication available in the environment where the ZenML code is running. On your local machine, this is the quickest way to configure an Azure Artifact Store. You don't need to supply credentials explicitly when you register the Azure Artifact Store, instead you have to set one of the following sets of environment variables:
- to use an Azure connection string, set
AZURE_STORAGE_CONNECTION_STRINGto your Azure Storage Key connection string
- to use an Azure account key, set
AZURE_STORAGE_ACCOUNT_NAMEto your account name and one of
AZURE_STORAGE_SAS_TOKENto the Azure key value.
- to use Azure ServicePrincipal credentials, set
AZURE_STORAGE_ACCOUNT_NAMEto your account name and
The ZenML Azure secret schema supports a variety of credentials:
- use an Azure account name and access key or SAS token (the
sas_tokenattributes need to be configured in the ZenML secret)
- use an Azure connection string (configure the
connection_stringattribute in the ZenML secret)
- use Azure ServicePrincipal credentials (which requires
client_secretto be configured in the ZenML secret)
The Azure credentials are configured as a ZenML secret that is referenced in the Artifact Store configuration, e.g.:
# Register the Azure artifact store
zenml artifact-store register az_store -f azure \
# Register a secrets manager
zenml secrets-manager register secrets_manager \
# Register and set a stack with the new artifact store and secrets manager
zenml stack register custom_stack -a az_store -x secrets_manager ... --set
# Create the secret referenced in the artifact store
zenml secrets-manager secret register az_secret -s azure \