How to execute individual steps in AzureML
You should use the AzureML step operator if:
- one or more steps of your pipeline require computing resources (CPU, GPU, memory) that are not provided by your orchestrator.
To use the AzureML step operator, we need:
- The ZenML
azureintegration installed. If you haven't done so, runzenml integration install azure
- A remote artifact store as part of your stack. This is needed so that both your orchestration environment and AzureML can read and write step artifacts. Check out the documentation page of the artifact store you want to use for more information on how to set that up and configure authentication for it.
We can then register the step operator and use it in our active stack:
zenml step-operator register <NAME> \
# only pass these if using Service Principal Authentication
# --tenant_id=<TENANT_ID> \
# --service_principal_id=<SERVICE_PRINCIPAL_ID> \
# --service_principal_password=<SERVICE_PRINCIPAL_PASSWORD> \
# Add the step operator to the active stack
zenml stack update -s <NAME>
Once you added the step operator to your active stack, you can use it to execute individual steps of your pipeline by specifying it in the
@stepdecorator as follows:
from zenml.steps import step
def trainer(...) -> ...:
"""Train a model."""
# This step will be executed in AzureML.
Note that if you wish to use this step operator to run steps on a GPU, you will need to follow the instructions on this page to ensure that it works. It requires adding some extra settings customization and is essential to enable CUDA for the GPU to give its full acceleration.