AzureML
Executing individual steps in AzureML.
This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.
AzureML
AzureML offers specialized compute instances to run your training jobs and has a comprehensive UI to track and manage your models and logs. ZenML's AzureML step operator allows you to submit individual steps to be run on AzureML compute instances.
When to use it
You should use the AzureML step operator if:
one or more steps of your pipeline require computing resources (CPU, GPU, memory) that are not provided by your orchestrator.
How to deploy it
Create a
Machine learning
resource on Azure .Once your resource is created, you can head over to the
Azure Machine Learning Studio
and create a compute cluster to run your pipelines.Create an
environment
for your pipelines. Follow this guide to set one up.(Optional) Create a Service Principal for authentication. This is required if you intend to run your pipelines with a remote orchestrator.
How to use it
To use the AzureML step operator, we need:
The ZenML
azure
integration installed. If you haven't done so, runAn AzureML compute cluster and environment. See the deployment section for detailed instructions.
A remote artifact store as part of your stack. This is needed so that both your orchestration environment and AzureML can read and write step artifacts. Check out the documentation page of the artifact store you want to use for more information on how to set that up and configure authentication for it.
We can then register the step operator and use it in our active stack:
Once you added the step operator to your active stack, you can use it to execute individual steps of your pipeline by specifying it in the @step
decorator as follows:
ZenML will build a Docker image called <CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME>
which includes your code and use it to run your steps in AzureML. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.
Additional configuration
For additional configuration of the AzureML step operator, you can pass AzureMLStepOperatorSettings
when defining or running your pipeline. Check out the SDK docs for a full list of available attributes and this docs page for more information on how to specify settings.
For more information and a full list of configurable attributes of the AzureML step operator, check out the API Docs .
Enabling CUDA for GPU-backed hardware
Note that if you wish to use this step operator to run steps on a GPU, you will need to follow the instructions on this page to ensure that it works. It requires adding some extra settings customization and is essential to enable CUDA for the GPU to give its full acceleration.
Last updated