Configuring Azure Service Connectors to connect ZenML to Azure resources such as Blob storage buckets, AKS Kubernetes clusters, and ACR container registries.
The ZenML Azure Service Connector facilitates the authentication and access to managed Azure services and resources. These encompass a range of resources, including blob storage containers, ACR repositories, and AKS clusters.
This connector serves as a general means of accessing any Azure service by issuing credentials to clients. Additionally, the connector can handle specialized authentication for Azure blob storage, Docker and Kubernetes Python clients. It also allows for the configuration of local Docker and Kubernetes CLIs.
The Azure Service Connector is part of the Azure ZenML integration. You can either install the entire integration or use a pypi extra to install it independently of the integration:
pip install "zenml[connectors-azure]" installs only prerequisites for the Azure Service Connector Type
zenml integration install azure installs the entire Azure ZenML integration
It is not required to install and set up the Azure CLI on your local machine to use the Azure Service Connector to link Stack Components to Azure resources and services. However, it is recommended to do so if you are looking for a quick setup that includes using the auto-configuration Service Connector features.
The auto-configuration option is limited to using temporary access tokens that don't work with Azure blob storage resources. To unlock the full power of the Azure Service Connector it is therefore recommended that you configure and use an Azure service principal and its credentials.
Resource Types
Generic Azure resource
This resource type allows Stack Components to use the Azure Service Connector to connect to any Azure service or resource. When used by Stack Components, they are provided generic azure-identity credentials that can be used to create Azure python clients for any particular Azure service.
This generic Azure resource type is meant to be used with Stack Components that are not represented by other, more specific resource type, like Azure blob storage containers, Kubernetes clusters or Docker registries. It should be accompanied by a matching set of Azure permissions that allow access to the set of remote resources required by the Stack Components.
The resource name represents the name of the Azure subscription that the connector is authorized to access.
Azure blob storage container
Allows users to connect to Azure Blob containers. When used by Stack Components, they are provided a pre-configured Azure Blob Storage client.
The configured credentials must have at least the following Azure IAM permissions associated with the blob storage account or containers that the connector that the connector will be allowed to access:
allow read and write access to blobs (e.g. the Storage Blob Data Contributor role)
allow listing the storage accounts (e.g. the Reader and Data Access role). This is only required if a storage account is not configured in the connector.
allow listing the containers in a storage account (e.g. the Reader and Data Access role)
If set, the resource name must identify an Azure blob storage container using one of the following formats:
Azure blob container URI (canonical resource name): {az|abfs}://{container-name}
Azure blob container name: {container-name}
If a storage account is configured in the connector, only blob storage containers in that storage account will be accessible. Otherwise, if a resource group is configured in the connector, only blob storage containers in storage accounts in that resource group will be accessible. Finally, if neither a storage account nor a resource group is configured in the connector, all blob storage containers in all accessible storage accounts will be accessible.
The only Azure authentication method that works with Azure blob storage resources is the service principal authentication method.
AKS Kubernetes cluster
Allows Stack Components to access an AKS cluster as a standard Kubernetes cluster resource. When used by Stack Components, they are provided a pre-authenticated python-kubernetes client instance.
The configured credentials must have at least the following Azure IAM permissions associated with the AKS clusters that the connector will be allowed to access:
allow listing the AKS clusters and fetching their credentials (e.g. the Azure Kubernetes Service Cluster Admin Role role)
If set, the resource name must identify an EKS cluster using one of the following formats:
resource group scoped AKS cluster name (canonical): [{resource-group}/]{cluster-name}
AKS cluster name: {cluster-name}
Given that the AKS cluster name is unique within a resource group, the resource group name may be included in the resource name to avoid ambiguity. If a resource group is configured in the connector, the resource group name in the resource name must match the configured resource group. If no resource group is configured in the connector and a resource group name is not included in the resource name, the connector will attempt to find the AKS cluster in any resource group.
If a resource group is configured in the connector, only AKS clusters in that resource group will be accessible.
ACR container registry
Allows Stack Components to access one or more ACR registries as a standard Docker registry resource. When used by Stack Components, they are provided a pre-authenticated python-docker client instance.
The configured credentials must have at least the following Azure IAM permissions associated with the ACR registries that the connector will be allowed to access:
allow access to pull and push images (e.g. the AcrPull and AcrPush roles)
allow access to list registries (e.g. the Contributor role)
If set, the resource name must identify an ACR registry using one of the following formats:
ACR registry URI (canonical resource name): [https://]{registry-name}.azurecr.io
ACR registry name: {registry-name}
If a resource group is configured in the connector, only ACR registries in that resource group will be accessible.
If an authentication method other than the Azure service principal is used for authentication, the admin account must be enabled for the registry, otherwise, clients will not be able to authenticate to the registry. See the official Azure documentation on the admin account for more information.
Authentication Methods
Implicit authentication
Implicit authentication to Azure services using environment variables, local configuration files, workload or managed identities.
This method may constitute a security risk, because it can give users access to the same cloud resources and services that the ZenML Server itself is configured to access. For this reason, all implicit authentication methods are disabled by default and need to be explicitly enabled by setting the ZENML_ENABLE_IMPLICIT_AUTH_METHODS environment variable or the helm chart enableImplicitAuthMethods configuration option to true in the ZenML deployment.
This authentication method doesn't require any credentials to be explicitly configured. It automatically discovers and uses credentials from one of the following sources:
workload identity - if the application is deployed to an Azure Kubernetes Service with Managed Identity enabled. This option can only be used when running the ZenML server on an AKS cluster.
managed identity - if the application is deployed to an Azure host with Managed Identity enabled. This option can only be used when running the ZenML client or server on an Azure host.
This is the quickest and easiest way to authenticate to Azure services. However, the results depend on how ZenML is deployed and the environment where it is used and is thus not fully reproducible:
when used with the default local ZenML deployment or a local ZenML server, the credentials are the same as those used by the Azure CLI or extracted from local environment variables.
when connected to a ZenML server, this method only works if the ZenML server is deployed in Azure and will use the workload identity attached to the Azure resource where the ZenML server is running (e.g. an AKS cluster). The permissions of the managed identity may need to be adjusted to allows listing and accessing/describing the Azure resources that the connector is configured to access.
Note that the discovered credentials inherit the full set of permissions of the local Azure CLI configuration, environment variables or remote Azure managed identity. Depending on the extent of those permissions, this authentication method might not be recommended for production use, as it can lead to accidental privilege escalation. Instead, it is recommended to use the Azure service principal authentication method to limit the validity and/or permissions of the credentials being issued to connector clients.
Example configuration
The following assumes the local Azure CLI has already been configured with user account credentials by running the az login command:
⠙ Registering service connector 'azure-implicit'...
Successfully registered service connector `azure-implicit` with access to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠───────────────────────┼───────────────────────────────────────────────┨
┃ 🇦 azure-generic │ ZenML Subscription ┃
┠───────────────────────┼───────────────────────────────────────────────┨
┃ 📦 blob-container │ az://demo-zenmlartifactstore ┃
┠───────────────────────┼───────────────────────────────────────────────┨
┃ 🌀 kubernetes-cluster │ demo-zenml-demos/demo-zenml-terraform-cluster ┃
┠───────────────────────┼───────────────────────────────────────────────┨
┃ 🐳 docker-registry │ demozenmlcontainerregistry.azurecr.io ┃
┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
No credentials are stored with the Service Connector:
zenmlservice-connectordescribeazure-implicit
Example Command Output
Service connector 'azure-implicit' of type 'azure' with id 'ad645002-0cd4-4d4f-ae20-499ce888a00a' is owned by user 'default' and is 'private'.
'azure-implicit' azure Service Connector Details
┏━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ PROPERTY │ VALUE ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ ID │ ad645002-0cd4-4d4f-ae20-499ce888a00a ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ NAME │ azure-implicit ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ TYPE │ 🇦 azure ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ AUTH METHOD │ implicit ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ RESOURCE TYPES │ 🇦 azure-generic, 📦 blob-container, 🌀 kubernetes-cluster, 🐳 docker-registry ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ RESOURCE NAME │ <multiple> ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ SECRET ID │ ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ SESSION DURATION │ N/A ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ EXPIRES IN │ N/A ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ OWNER │ default ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ SHARED │ ➖ ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ CREATED_AT │ 2023-06-05 09:47:42.415949 ┃
┠──────────────────┼────────────────────────────────────────────────────────────────────────────────┨
┃ UPDATED_AT │ 2023-06-05 09:47:42.415954 ┃
┗━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Azure Service Principal
Azure service principal credentials consists of an Azure client ID and client secret. These credentials are used to authenticate clients to Azure services.
The following assumes an Azure service principal was configured with a client secret and has permissions to access an Azure blob storage container, an AKS Kubernetes cluster and an ACR container registry. The service principal client ID, tenant ID and client secret are then used to configure the Azure Service Connector.
This method has the major limitation that the user must regularly generate new tokens and update the connector configuration as API tokens expire. On the other hand, this method is ideal in cases where the connector only needs to be used for a short period of time, such as sharing access temporarily with someone else in your team.
This is the authentication method used during auto-configuration, if you have the local Azure CLI set up with credentials. The connector will generate an access token from the Azure CLI credentials and store it in the connector configuration.
Given that Azure access tokens are scoped to a particular Azure resource and the access token generated during auto-configuration is scoped to the Azure Management API, this method does not work with Azure blob storage resources. You should use the Azure service principal authentication method for blob storage resources instead.
Example auto-configuration
Fetching Azure session tokens from the local Azure CLI is possible if the Azure CLI is already configured with valid credentials (i.e. by running az login):
Note that the Azure local CLI can only be configured with credentials issued by the Azure Service Connector if the connector is configured with the service principal authentication method.
Local CLI configuration examples
The following shows an example of configuring the local Kubernetes CLI to access an AKS cluster reachable through an Azure Service Connector:
⠙ Verifying service connector 'azure-service-principal'...
Service connector 'azure-service-principal' is correctly configured with valid credentials and has access to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠───────────────────────┼───────────────────────────────────────────────┨
┃ 🌀 kubernetes-cluster │ demo-zenml-demos/demo-zenml-terraform-cluster ┃
┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
The login CLI command can be used to configure the local Kubernetes CLI to access a Kubernetes cluster reachable through an Azure Service Connector:
⠙ Attempting to configure local client using service connector 'azure-service-principal'...
Updated local kubeconfig with the cluster details. The current kubectl context was set to 'demo-zenml-terraform-cluster'.
The 'azure-service-principal' Kubernetes Service Connector connector was used to successfully configure the local Kubernetes cluster client/SDK.
The local Kubernetes CLI can now be used to interact with the Kubernetes cluster:
kubectlcluster-info
Example Command Output
Kubernetes control plane is running at https://demo-43c5776f7.hcp.westeurope.azmk8s.io:443
CoreDNS is running at https://demo-43c5776f7.hcp.westeurope.azmk8s.io:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://demo-43c5776f7.hcp.westeurope.azmk8s.io:443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
A similar process is possible with ACR container registries:
⠦ Verifying service connector 'azure-service-principal'...
Service connector 'azure-service-principal' is correctly configured with valid credentials and has access to the following resources:
┏━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠────────────────────┼───────────────────────────────────────┨
┃ 🐳 docker-registry │ demozenmlcontainerregistry.azurecr.io ┃
┗━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
⠹ Attempting to configure local client using service connector 'azure-service-principal'...
WARNING! Your password will be stored unencrypted in /home/stefan/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
The 'azure-service-principal' Docker Service Connector connector was used to successfully configure the local Docker/OCI container registry client/SDK.
The local Docker CLI can now be used to interact with the container registry:
Updated the local Azure CLI configuration with the connector's service principal credentials.
The 'azure-service-principal' Azure Service Connector connector was used to successfully configure the local Generic Azure resource client/SDK.
The Azure Service Connector can also be used with any Orchestrator or Model Deployer stack component flavor that relies on a Kubernetes clusters to manage workloads. This allows AKS Kubernetes container workloads to be managed without the need to configure and maintain explicit Azure or Kubernetes kubectl configuration contexts and credentials in the target environment or in the Stack Component itself.
Similarly, Container Registry Stack Components can be connected to a ACR Container Registry through an Azure Service Connector. This allows container images to be built and published to private ACR container registries without the need to configure explicit Azure credentials in the target environment or the Stack Component.
End-to-end examples
AKS Kubernetes Orchestrator, Azure Blob Storage Artifact Store and ACR Container Registry with a multi-type Azure Service Connector
This is an example of an end-to-end workflow involving Service Connectors that uses a single multi-type Azure Service Connector to give access to multiple resources for multiple Stack Components. A complete ZenML Stack is registered composed of the following Stack Components, all connected through the same Service Connector:
As a last step, a simple pipeline is run on the resulting Stack.
This example needs to use a remote ZenML Server that is reachable from Azure.
Configure an Azure service principal with a client secret and give it permissions to access an Azure blob storage container, an AKS Kubernetes cluster and an ACR container registry. Also make sure you have the Azure ZenML integration installed:
zenmlintegrationinstall-yazure
Make sure the Azure Service Connector Type is available
Register a multi-type Azure Service Connector using the Azure service principal credentials set up at the first step. Note the resources that it has access to:
```
Successfully connected container registry `acr-demo-registry` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼─────────────────────────┼────────────────┼────────────────────┼───────────────────────────────────────┨
┃ f2316191-d20b-4348-a68b-f5e347862196 │ azure-service-principal │ 🇦 azure │ 🐳 docker-registry │ demozenmlcontainerregistry.azurecr.io ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
Combine all Stack Components together into a Stack and set it as active (also throw in a local Image Builder for completion):
zenmlimage-builderregisterlocal--flavorlocal
Example Command Output
```
Running with active stack: 'default' (global)
Successfully registered image_builder `local`.
```
```sh
zenml stack register gcp-demo -a azure-demo -o aks-demo-cluster -c acr-demo-registry -i local --set
```
Example Command Output
```
Stack 'gcp-demo' successfully registered!
Active repository stack set to:'gcp-demo'
```
Finally, run a simple pipeline to prove that everything works as expected. We'll use the simplest pipelines possible for this example:
from zenml import pipeline, step@stepdefstep_1() ->str:"""Returns the `world` string."""return"world"@step(enable_cache=False)defstep_2(input_one:str,input_two:str) ->None:"""Combines the two strings at its input and prints them.""" combined_str =f"{input_one}{input_two}"print(combined_str)@pipelinedefmy_pipeline(): output_step_one =step_1()step_2(input_one="hello", input_two=output_step_one)if__name__=="__main__":my_pipeline()
Saving that to a run.py file and running it gives us:
Example Command Output
```
$ python run.py
Building Docker image(s) for pipeline simple_pipeline.
Building Docker image demozenmlcontainerregistry.azurecr.io/zenml:simple_pipeline-orchestrator.
- Including integration requirements: adlfs==2021.10.0, azure-identity==1.10.0, azure-keyvault-keys, azure-keyvault-secrets, azure-mgmt-containerservice>=20.0.0, azureml-core==1.48.0, kubernetes, kubernetes==18.20.0
No .dockerignore found, including all files inside build context.
Step 1/10 : FROM zenmldocker/zenml:0.40.0-py3.8
Step 2/10 : WORKDIR /app
Step 3/10 : COPY .zenml_user_requirements .
Step 4/10 : RUN pip install --default-timeout=60 --no-cache-dir -r .zenml_user_requirements
Step 5/10 : COPY .zenml_integration_requirements .
Step 6/10 : RUN pip install --default-timeout=60 --no-cache-dir -r .zenml_integration_requirements
Step 7/10 : ENV ZENML_ENABLE_REPO_INIT_WARNINGS=False
Step 8/10 : ENV ZENML_CONFIG_PATH=/app/.zenconfig
Step 9/10 : COPY . .
Step 10/10 : RUN chmod -R a+rw .
Pushing Docker image demozenmlcontainerregistry.azurecr.io/zenml:simple_pipeline-orchestrator.
Finished pushing Docker image.
Finished building Docker image(s).
Running pipeline simple_pipeline on stack gcp-demo (caching disabled)
Waiting for Kubernetes orchestrator pod...
Kubernetes orchestrator pod started.
Waiting for pod of step simple_step_one to start...
Step simple_step_one has started.
INFO:azure.identity._internal.get_token_mixin:ClientSecretCredential.get_token succeeded
INFO:azure.identity._internal.get_token_mixin:ClientSecretCredential.get_token succeeded
INFO:azure.identity._internal.get_token_mixin:ClientSecretCredential.get_token succeeded
INFO:azure.identity.aio._internal.get_token_mixin:ClientSecretCredential.get_token succeeded
Step simple_step_one has finished in 0.396s.
Pod of step simple_step_one completed.
Waiting for pod of step simple_step_two to start...
Step simple_step_two has started.
INFO:azure.identity._internal.get_token_mixin:ClientSecretCredential.get_token succeeded
INFO:azure.identity._internal.get_token_mixin:ClientSecretCredential.get_token succeeded
INFO:azure.identity.aio._internal.get_token_mixin:ClientSecretCredential.get_token succeeded
Hello World!
Step simple_step_two has finished in 3.203s.
Pod of step simple_step_two completed.
Orchestration pod completed.
Dashboard URL: https://zenml.stefan.20.23.46.143.nip.io/default/pipelines/98c41e2a-1ab0-4ec9-8375-6ea1ab473686/runs
```
