How to execute individual steps in specialized environments
The step operator enables the execution of individual pipeline steps in specialized runtime environments that are optimized for certain workloads. These specialized environments can give your steps access to resources like GPUs or distributed processing frameworks like Spark.
A step operator should be used if one or more steps of a pipeline require resources that are not available in the runtime environments provided by the orchestrator. An example would be a step that trains a computer vision model and requires a GPU to run in reasonable time, combined with a Kubeflow orchestrator running on a kubernetes cluster which does not contain any GPU nodes. In that case it makes sense to include a step operator like SageMaker, Vertex or AzureML to execute the training step with a GPU.
Step operators to execute steps on one of the big cloud providers are provided by the following ZenML integrations:
If you would like to see the available flavors of step operators, you can use the command:
zenml step-operator flavor list
from zenml.steps import step
def my_step(...) -> ...: