AzureML

How to execute individual steps in AzureML

This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.

The AzureML step operator is a step operator flavor provided with the ZenML azure integration that uses AzureML to execute individual steps of ZenML pipelines.

When to use it

You should use the AzureML step operator if:

  • one or more steps of your pipeline require computing resources (CPU, GPU, memory) that are not provided by your orchestrator.

  • you have access to AzureML. If you're using a different cloud provider, take a look at the SageMaker or Vertex step operators.

How to deploy it

  • Create a Machine learning resource on Azure.

  • Once your resource is created, you can head over to the Azure Machine Learning Studio and create a compute cluster to run your pipelines.

  • Create an environment for your pipelines. Follow this guide to set one up.

  • (Optional) Create a Service Principal for authentication. This is required if you intend to run your pipelines with a remote orchestrator.

How to use it

To use the AzureML step operator, we need:

  • The ZenML azure integration installed. If you haven't done so, run

    zenml integration install azure
  • An AzureML compute cluster and environment. See the deployment section for detailed instructions.

  • A remote artifact store as part of your stack. This is needed so that both your orchestration environment as well as AzureML can read and write step artifacts. Check out the documentation page of the artifact store you want to use for more information on how to set that up and configure authentication for it.

We can then register the step operator and use it in our active stack:

zenml step-operator register <NAME> \
    --flavor=azureml \
    --subscription_id=<AZURE_SUBSCRIPTION_ID> \
    --resource_group=<AZURE_RESOURCE_GROUP> \
    --workspace_name=<AZURE_WORKSPACE_NAME> \
    --compute_target_name=<AZURE_COMPUTE_TARGET_NAME> \
    --environment_name=<AZURE_ENVIRONMENT_NAME> \
# only pass these if using Service Principal Authentication
#   --tenant_id=<TENANT_ID> \
#   --service_principal_id=<SERVICE_PRINCIPAL_ID> \
#   --service_principal_password=<SERVICE_PRINCIPAL_PASSWORD> \

# Add the step operator to the active stack
zenml stack update -s <NAME>

Once you added the step operator to your active stack, you can use it to execute individual steps of your pipeline by specifying it in the @step decorator as follows:

from zenml.steps import step

@step(custom_step_operator=<NAME>)
def trainer(...) -> ...:
    """Train a model."""
    # This step will be executed in AzureML.

ZenML will build a Docker image called <CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME> which includes your code and use it to run your steps in AzureML. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.

A concrete example of using the AzureML step operator can be found here.

For more information and a full list of configurable attributes of the AzureML step operator, check out the API Docs.

Last updated