Azure Blob Storage
How to store artifacts using Azure Blob Storage
When would you want to use it?
if you want to share your pipeline run results with other team members or stakeholders inside or outside your organization
if you have other components in your stack that are running remotely (e.g. a Kubeflow or Kubernetes Orchestrator running in public cloud).
if you outgrow what your local machine can offer in terms of storage space and need to use some form of private or public storage service that is shared with others
if you are running pipelines at scale and need an Artifact Store that can handle the demands of production grade MLOps
In all these cases, you need an Artifact Store that is backed by a form of public cloud or self-hosted shared object storage service.
How do you deploy it?
The Azure Artifact Store flavor is provided by the Azure ZenML integration, you need to install it on your local machine to be able to register an Azure Artifact Store and add it to your stack:
With the URI to your Azure Blob Storage container known, registering an Azure Artifact Store can be done as follows:
Authentication Methods
You will need the following information to configure Azure credentials for ZenML, depending on which type of Azure credentials you want to use:
an Azure connection string
an Azure account key
the client ID, client secret and tenant ID of the Azure service principal
This method uses the implicit Azure authentication available in the environment where the ZenML code is running. On your local machine, this is the quickest way to configure an Azure Artifact Store. You don't need to supply credentials explicitly when you register the Azure Artifact Store, instead you have to set one of the following sets of environment variables:
Certain dashboard functionality, such as visualizing or deleting artifacts, is not available when using an implicitly authenticated artifact store together with a deployed ZenML server because the ZenML server will not have permissions to access the filesystem.
The implicit authentication method also needs to be coordinated with other stack components that are highly dependent on the Artifact Store and need to interact with it directly to function. If these components are not running on your machine, they do not have access to the local environment variables and will encounter authentication failures while trying to access the Azure Artifact Store:
If you have remote stack components that are not running in Azure, or if you are unsure how to configure them to use Managed Identities, you should use one of the other authentication methods.
How do you use it?
Last updated