How to execute individual steps in SageMaker
The SageMaker step operator is a step operator flavor provided with the ZenML
awsintegration that uses SageMaker to execute individual steps of ZenML pipelines.
You should use the SageMaker step operator if:
- Create a role in the IAM console that you want the jobs running in SageMaker to assume. This role should at least have the
AmazonSageMakerFullAccesspolicies applied. Check here for a guide on how to set up this role.
To use the SageMaker step operator, we need:
- The ZenML
awsintegration installed. If you haven't done so, runzenml integration install aws
- A remote artifact store as part of your stack. This is needed so that both your orchestration environment and SageMaker can read and write step artifacts. Check out the documentation page of the artifact store you want to use for more information on how to set that up and configure authentication for it.
- If using a local orchestrator: The
awscli set up and authenticated. Make sure you have the permissions to create and manage SageMaker runs.
- If using a remote orchestrator: The environment in which the orchestrator runs its containers needs to be able to assume the IAM role specified when registering the SageMaker step operator.
- An instance type that we want to execute our steps on. See here for a list of available instance types.
- (Optional) An experiment which is used to group SageMaker runs. Check this guide to see how to create an experiment.
We can then register the step operator and use it in our active stack:
zenml step-operator register <NAME> \
# --experiment_name=<EXPERIMENT_NAME> # optionally specify an experiment to assign this run to
# Add the step operator to the active stack
zenml stack update -s <NAME>
Once you added the step operator to your active stack, you can use it to execute individual steps of your pipeline by specifying it in the
@stepdecorator as follows:
from zenml.steps import step
def trainer(...) -> ...:
"""Train a model."""
# This step will be executed in SageMaker.
ZenML will build a Docker image called
<CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME>which includes your code and use it to run your steps in SageMaker. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.
For additional configuration of the SageMaker step operator, you can pass
SagemakerStepOperatorSettingswhen defining or running your pipeline. Check out the API docs for a full list of available attributes and this docs page for more information on how to specify settings.
For more information and a full list of configurable attributes of the SageMaker step operator, check out the API Docs.
Note that if you wish to use this step operator to run steps on a GPU, you will need to follow the instructions on this page to ensure that it works. It requires adding some extra settings customization and is essential to enable CUDA for the GPU to give its full acceleration.