Advanced Usage

An overview of advanced ZenML use cases

This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.

The previous sections on Steps and Pipelines and Stacks, Profiles, Repositories already cover most of the concepts you will need for developing ML workflows with ZenML.

However, there are a few additional use cases that you might or might not encounter throughout your journey, about which you can learn more here.

List of Advanced Use Cases

• Writing Custom Stack Component Flavors can be useful when trying to use ZenML with tooling or infrastructure for which no official integration exists yet.

• Secret references allow you to securely configure sensitive information required in certain stack components.

• Managing Stack Component States is required for certain integrations with remote components and can also be used to configure custom setup behavior of custom stack component flavors.

• Passing Custom Data Types through Steps via Materializers is required if one of your steps outputs a custom class or other data types, for which materialization is not defined by ZenML itself.

• Accessing the Active Stack within Steps via Step Fixtures can, for instance, be used to load the best performing prior model to compare newly trained models against.

• Accessing Global Info within Steps via the Environment can be useful to get system information, the Python version, or the name of the current step, pipeline, and run.

• Managing External Services might be required for deploying custom models or for the UI's of some visualization tools like TensorBoard. These services are usually long-lived external processes that persist beyond the execution of your pipeline runs.

• Managing Docker Images is required for some remote orchestrators and step operators to run your pipeline code in an isolated and well-defined environment.

• Setting Stacks and Profiles with Environment Variables enables you to configure pipeline runs with environment variables instead of the repository.

• Migrating Legacy Stacks to ZenML Profiles contains additional information for returning users that want to port their ZenML stacks to the newest version.

• Specifying Hardware Resources for Steps explains how to specify hardware resources like memory or the amount of CPUs and GPUs that a step requires to execute.