Stacks: Configure MLOps Tooling and Infrastructure
How to configure MLOps tooling and infrastructure with stacks
This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.
In ZenML, a Stack represents a set of configurations for your MLOps tools and infrastructure. For instance, you might want to:
Orchestrate your ML workflows with Kubeflow,
Save ML artifacts in an Amazon S3 bucket,
Track ML metadata in a managed MySQL database,
Track your experiments with Weights & Biases,
Any such combination of tools and infrastructure can be registered as a separate stack in ZenML. Since ZenML code is tooling-independent, you can switch between stacks with a single command and then automatically execute your ML workflows on the desired stack without having to modify your code.
Stack Components
In ZenML, each MLOps tool is associated to a specific Stack Component, which is responsible for one specific task of your ML workflow.
For instance, each ZenML stack includes an Orchestrator which is responsible for the execution of the steps within your pipeline, an Artifact Store which is responsible for storing the artifacts generated by your pipelines, and a Metadata Store that tracks what artifact was produced or consumed by which pipeline steps.
Check out the Categories of MLOps Tools page for a detailed overview of available stack components in ZenML.
Orchestrator, Artifact Store, and Metadata Store
As mentioned above, orchestrators, artifact stores, and metadata stores are the three components that need to be in every ZenML stack. The interaction of these three components enables a lot of the magic of ZenML, such as data and model versioning, automated artifact lineage tracking, automated caching, and more.
Orchestrator
The Orchestrator is the component that defines how and where each pipeline step is executed when calling pipeline.run(). By default, all runs are executed locally, but by configuring a different orchestrator you can, e.g., automatically execute your ML workflows on Kubeflow instead.
Artifact Stores
Under the hood, all the artifacts in our ML pipeline are automatically stored in an Artifact Store. By default, this is simply a place in your local file system, but we could also configure ZenML to store this data in a cloud bucket like Amazon S3 or any other place instead.
Metadata Stores
In addition to the artifact itself, ZenML automatically stores metadata about each pipeline run in a Metadata Store. By default, this uses an SQLite database on your local machine, but we could again switch it out for another storage type, such as a MySQL database deployed in the cloud.
Stack Component Flavors
The specific tool you are using is called a Flavor of the stack component. E.g., Kubeflow is a flavor of the Orchestrator stack component.
Out-of-the-box, ZenML already comes with a wide variety of flavors, which are either built-in or enabled through the installation of specific Integrations.
The default Stack
By default, ZenML itself and every Repository that you create already come with an initial active default stack, which features:
If you followed the code examples in the Steps and Pipelines section, then you have already used this stack implicitly to run all of your pipelines.
Listing Stacks, Stack Components, and Flavors
You can see a list of all your registered stacks with the following command:
Similarly, you can see all registered stack components of a specific type using:
In order to see all the available flavors for a specific stack component, use:
Our CLI features a wide variety of commands that let you manage and use your stacks. If you would like to learn more, please run: "zenml stack --help" or visit our CLI docs.
Registering New Stacks
You can combine various MLOps tools into a ZenML stack as follows:
Register a stack component for each tool using
zenml <STACK_COMPONENT> register,Register a stack to bring all tools together using
zenml stack register,Activate the stack using
zenml stack set. Now all your code is automatically executed using the desired tools / infrastructure.
Registering Stack Components
First, you need to create a new instance of the respective stack component with the desired flavor using zenml <STACK_COMPONENT> register <NAME> --flavor=<FLAVOR>. Most flavors require further parameters that you can pass as additional arguments --param=value, similar to how we passed the flavor.
E.g., to register a local artifact store, we could use the following command:
In case you do not know all the available parameters, you can also use the interactive mode to register stack components. This will then walk you through each parameter (to skip just press ENTER):
Afterwards, you should be able to see the new artifact store in the list of registered artifact stores, which you can access using the following command:
Our CLI features a wide variety of commands that let you manage and use your stack components and flavors. If you would like to learn more, please run zenml <STACK_COMPONENT> --help or visit our CLI docs.
Registering a Stack
After registering each tool as the respective stack component, you can combine all of them into one stack using the zenml stack register command:
You can use zenml stack register --help to see a list of all possible arguments to the zenml stack register command, including a list of which option to use for which stack component.
When you register a stack, the corresponding artifact- and metadata store gets coupled, which means that these instances can not be use in different pairings. This is an intended behavior to avoid mismatches between the two stores. In order get more information about this interaction, please refer to our CLI docs..
Activating a Stack
Finally, to start using the stack you just registered, set it as active:
Now all your code is automatically executed using this stack.
Some advanced stack component flavors might require connecting to remote infrastructure components prior to running code on the stack. This can be done using zenml stack up. See the Managing Stack States section for more details.
Changing Stacks
If you have multiple stacks configured, you can switch between them using the zenml stack set command, similar to how you activate a stack.
Unregistering Stacks
To unregister (delete) a stack and all of its components, run
to delete the stack itself, followed by
to delete each of the individual stack components.
If you provisioned infrastructure related to the stack, make sure to deprovision it using zenml stack down --force before unregistering the stack. See the Managing Stack States section for more details.
Last updated
