Azure Service Connector
Configuring Azure Service Connectors to connect ZenML to Azure resources such as Blob storage buckets, AKS Kubernetes clusters, and ACR container registries.
The ZenML Azure Service Connector facilitates the authentication and access to managed Azure services and resources. These encompass a range of resources, including blob storage containers, ACR repositories, and AKS clusters.
This connector also supports automatic configuration and detection of credentials locally configured through the Azure CLI.
This connector serves as a general means of accessing any Azure service by issuing credentials to clients. Additionally, the connector can handle specialized authentication for Azure blob storage, Docker and Kubernetes Python clients. It also allows for the configuration of local Docker and Kubernetes CLIs.
$ zenml service-connector list-types --type azure┏━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━┯━━━━━━━┯━━━━━━━━┓
┃ NAME │ TYPE │ RESOURCE TYPES │ AUTH METHODS │ LOCAL │ REMOTE ┃
┠─────────────────────────┼──────────┼───────────────────────┼───────────────────┼───────┼────────┨
┃ Azure Service Connector │ 🇦 azure │ 🇦 azure-generic │ implicit │ ✅ │ ✅ ┃
┃ │ │ 📦 blob-container │ service-principal │ │ ┃
┃ │ │ 🌀 kubernetes-cluster │ access-token │ │ ┃
┃ │ │ 🐳 docker-registry │ │ │ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━┷━━━━━━━┷━━━━━━━━┛Prerequisites
The Azure Service Connector is part of the Azure ZenML integration. You can either install the entire integration or use a pypi extra to install it independently of the integration:
pip install "zenml[connectors-azure]"installs only prerequisites for the Azure Service Connector Typezenml integration install azureinstalls the entire Azure ZenML integration
It is not required to install and set up the Azure CLI on your local machine to use the Azure Service Connector to link Stack Components to Azure resources and services. However, it is recommended to do so if you are looking for a quick setup that includes using the auto-configuration Service Connector features.
Resource Types
Generic Azure resource
This resource type allows Stack Components to use the Azure Service Connector to connect to any Azure service or resource. When used by Stack Components, they are provided generic azure-identity credentials that can be used to create Azure python clients for any particular Azure service.
This generic Azure resource type is meant to be used with Stack Components that are not represented by other, more specific resource type, like Azure blob storage containers, Kubernetes clusters or Docker registries. It should be accompanied by a matching set of Azure permissions that allow access to the set of remote resources required by the Stack Components.
The resource name represents the name of the Azure subscription that the connector is authorized to access.
Azure blob storage container
Allows users to connect to Azure Blob containers. When used by Stack Components, they are provided a pre-configured Azure Blob Storage client.
The configured credentials must have at least the following Azure IAM permissions associated with the blob storage account or containers that the connector that the connector will be allowed to access:
allow read and write access to blobs (e.g. the
Storage Blob Data Contributorrole)allow listing the storage accounts (e.g. the
Reader and Data Accessrole). This is only required if a storage account is not configured in the connector.allow listing the containers in a storage account (e.g. the
Reader and Data Accessrole)
If set, the resource name must identify an Azure blob storage container using one of the following formats:
Azure blob container URI (canonical resource name):
{az|abfs}://{container-name}Azure blob container name:
{container-name}
If a storage account is configured in the connector, only blob storage containers in that storage account will be accessible. Otherwise, if a resource group is configured in the connector, only blob storage containers in storage accounts in that resource group will be accessible. Finally, if neither a storage account nor a resource group is configured in the connector, all blob storage containers in all accessible storage accounts will be accessible.
The only Azure authentication methods that work with Azure blob storage resources are the implicit authentication and the service principal authentication method.
AKS Kubernetes cluster
Allows Stack Components to access an AKS cluster as a standard Kubernetes cluster resource. When used by Stack Components, they are provided a pre-authenticated python-kubernetes client instance.
The configured credentials must have at least the following Azure IAM permissions associated with the AKS clusters that the connector will be allowed to access:
allow listing the AKS clusters and fetching their credentials (e.g. the
Azure Kubernetes Service Cluster Admin Rolerole)
If set, the resource name must identify an AKS cluster using one of the following formats:
resource group scoped AKS cluster name (canonical):
[{resource-group}/]{cluster-name}AKS cluster name:
{cluster-name}
Given that the AKS cluster name is unique within a resource group, the resource group name may be included in the resource name to avoid ambiguity. If a resource group is configured in the connector, the resource group name in the resource name must match the configured resource group. If no resource group is configured in the connector and a resource group name is not included in the resource name, the connector will attempt to find the AKS cluster in any resource group.
If a resource group is configured in the connector, only AKS clusters in that resource group will be accessible.
ACR container registry
Allows Stack Components to access one or more ACR registries as a standard Docker registry resource. When used by Stack Components, they are provided a pre-authenticated python-docker client instance.
The configured credentials must have at least the following Azure IAM permissions associated with the ACR registries that the connector will be allowed to access:
allow access to pull and push images (e.g. the
AcrPullandAcrPushroles)allow access to list registries - instead of the broad
Contributorrole, use more specific permissions likeReaderrole or create a custom role with only theMicrosoft.ContainerRegistry/registries/readpermission
If set, the resource name must identify an ACR registry using one of the following formats:
ACR registry URI (canonical resource name):
[https://]{registry-name}.azurecr.ioACR registry name:
{registry-name}
If a resource group is configured in the connector, only ACR registries in that resource group will be accessible.
If an authentication method other than the Azure service principal is used, Entra ID authentication is used.
This requires the configured identity to have the AcrPush role to be configured.
If Entra ID authentication fails, admin account authentication is tried. For this the admin account must be enabled for the registry.
See the official Azuredocumentation on the admin account for more information.
Authentication Methods
Implicit authentication
Implicit authentication to Azure services using environment variables, local configuration files, workload or managed identities.
This method may constitute a security risk, because it can give users access to the same cloud resources and services that the ZenML Server itself is configured to access. For this reason, all implicit authentication methods are disabled by default and need to be explicitly enabled by setting the ZENML_ENABLE_IMPLICIT_AUTH_METHODS environment variable or the helm chart enableImplicitAuthMethods configuration option to true in the ZenML deployment.
This authentication method doesn't require any credentials to be explicitly configured. It automatically discovers and uses credentials from one of the following sources:
workload identity - if the application is deployed to an Azure Kubernetes Service with Managed Identity enabled. This option can only be used when running the ZenML server on an AKS cluster.
managed identity - if the application is deployed to an Azure host with Managed Identity enabled. This option can only be used when running the ZenML client or server on an Azure host.
Azure CLI - if a user has signed in via the Azure CLI
az logincommand.
This is the quickest and easiest way to authenticate to Azure services. However, the results depend on how ZenML is deployed and the environment where it is used and is thus not fully reproducible:
when used with the default local ZenML deployment or a local ZenML server, the credentials are the same as those used by the Azure CLI or extracted from local environment variables.
when connected to a ZenML server, this method only works if the ZenML server is deployed in Azure and will use the workload identity attached to the Azure resource where the ZenML server is running (e.g. an AKS cluster). The permissions of the managed identity may need to be adjusted to allows listing and accessing/describing the Azure resources that the connector is configured to access.
Note that the discovered credentials inherit the full set of permissions of the local Azure CLI configuration, environment variables or remote Azure managed identity. Depending on the extent of those permissions, this authentication method might not be recommended for production use, as it can lead to accidental privilege escalation. Instead, it is recommended to use the Azure service principal authentication method to limit the validity and/or permissions of the credentials being issued to connector clients.
Azure Service Principal
Azure service principal credentials consists of an Azure client ID and client secret. These credentials are used to authenticate clients to Azure services.
For this authentication method, the Azure Service Connector requires an Azure service principal to be created and a client secret to be generated.
Azure Access Token
Uses temporary Azure access tokens explicitly configured by the user or auto-configured from a local environment.
This method has the major limitation that the user must regularly generate new tokens and update the connector configuration as API tokens expire. On the other hand, this method is ideal in cases where the connector only needs to be used for a short period of time, such as sharing access temporarily with someone else in your team.
This is the authentication method used during auto-configuration, if you have the local Azure CLI set up with credentials. The connector will generate an access token from the Azure CLI credentials and store it in the connector configuration.
Given that Azure access tokens are scoped to a particular Azure resource and the access token generated during auto-configuration is scoped to the Azure Management API, this method does not work with Azure blob storage resources. You should use the Azure service principal authentication method for blob storage resources instead.
Auto-configuration
The Azure Service Connector allows auto-discovering and fetching credentials and configuration set up by the Azure CLI on your local host.
The Azure service connector auto-configuration comes with two limitations:
it can only pick up temporary Azure access tokens and therefore cannot be used for long-term authentication scenarios
it doesn't support authenticating to the Azure blob storage service. The Azure service principal authentication method can be used instead.
For an auto-configuration example, please refer to the section about Azure access tokens.
Local client provisioning
The local Azure CLI, Kubernetes kubectl CLI and the Docker CLI can be configured with credentials extracted from or generated by a compatible Azure Service Connector.
Stack Components use
The Azure Artifact Store Stack Component can be connected to a remote Azure blob storage container through an Azure Service Connector.
The Azure Service Connector can also be used with any Orchestrator or Model Deployer stack component flavor that relies on a Kubernetes clusters to manage workloads. This allows AKS Kubernetes container workloads to be managed without the need to configure and maintain explicit Azure or Kubernetes kubectl configuration contexts and credentials in the target environment or in the Stack Component itself.
Similarly, Container Registry Stack Components can be connected to a ACR Container Registry through an Azure Service Connector. This allows container images to be built and published to private ACR container registries without the need to configure explicit Azure credentials in the target environment or the Stack Component.
End-to-end examples
Last updated
Was this helpful?