Core Concepts

Core Concepts
ZenML consists of the following key components:
ZenML Architectural Overview
​
Repository
A repository is at the core of all ZenML activity. Every action that can be executed within ZenML must take place within a ZenML repository. ZenML repositories are inextricably tied to git. ZenML creates a .zen folder at the root of your repository to manage your assets and metadata.
Every ZenML project starts inside a ZenML repository. Think of it just like a normal Git repository, except that there are some added bonuses on top! A repository is at the core of all ZenML activity. Every action that can be executed within ZenML must take place within a ZenML repository. ZenML repositories are inextricably tied to git. ZenML creates a .zen folder at the root of your repository to manage your assets and metadata.
To create a ZenML repository, do the following after having installed ZenML:
1
zenml init
Copied!
Please make sure to be inside a valid git repository before executing the above!
The initialization creates a local .zen folder where various information about your local configuration lives, e.g., the active Stack that you are using to run pipelines.
Pipeline
Within your repository, you will have one or more pipelines as part of your experimentation workflow. A ZenML pipeline is a sequence of tasks that execute in a specific order and yield artifacts. The artifacts are stored within the artifact store and indexed via the metadata store. Each individual task within a pipeline is known as a step. The standard pipelines within ZenML are designed to have easy interfaces to add pre-decided steps, with the order also pre-decided. Other sorts of pipelines can be created as well from scratch.
Pipelines can be written as simple functions. They are created by using decorators appropriate to the specific use case you have. The moment it is run, a pipeline is compiled and passed directly to the orchestrator.
Pipelines are designed as simple functions. They are created by using decorators appropriate to the specific use case you have. The moment it is run, a pipeline is compiled and passed directly to the orchestrator, to be run in the orchestrator environment.
Within your repository, you will have one or more pipelines as part of your experimentation workflow. A ZenML pipeline is a sequence of tasks that execute in a specific order and yield artifacts. The artifacts are stored within the artifact store and indexed via the metadata store. Each individual task within a pipeline is known as a step. The standard pipelines (like TrainingPipeline) within ZenML are designed to have easy interfaces to add pre-decided steps, with the order also pre-decided. Other sorts of pipelines can be created as well from scratch.
1
@pipeline
2
def mnist_pipeline(
3
    importer,
4
    normalizer: normalizer,
5
    trainer,
6
    evaluator,
7
):
8
    # Link all the steps artifacts together
9
    X_train, y_train, X_test, y_test = importer()
10
    X_trained_normed, X_test_normed = normalizer(X_train=X_train, X_test=X_test)
11
    model = trainer(X_train=X_trained_normed, y_train=y_train)
12
    evaluator(X_test=X_test_normed, y_test=y_test, model=model)
13
​
14
​
15
# Initialise the pipeline
16
p = mnist_pipeline(
17
    importer=importer_mnist(),
18
    normalizer=normalizer(),
19
    trainer=trainer(config=TrainerConfig(epochs=1)),
20
    evaluator=evaluator(),
21
)
22
​
23
# Run the pipeline
24
p.run()
Copied!
Pipelines consist of many steps that define what actually happens to the data flowing through the pipelines.
Step
A step is a single piece or stage of a ZenML pipeline. Think of each step as being one of the nodes of the DAG. Steps are responsible for one aspect of processing or interacting with the data / artifacts in the pipeline. ZenML currently implements a basic step interface, but there will be other more customized interfaces (layered in a hierarchy) for specialized implementations. For example, broad steps like @trainer, @split and and so on.Conceptually, a Step is a discrete and independent part of a pipeline that is responsible for one particular aspect of data manipulation inside a ZenML pipeline.
A ZenML installation already comes with many standard steps found in zenml.core.steps.* for users to get started. For example, a SplitStep is responsible for splitting the data into various split's like train and eval for downstream steps to then use. However, in essence, virtually any Python function can be a ZenML step as well.
1
from zenml.steps import step
2
​
3
@step  # this is where the magic happens
4
def simplest_step_ever(basic_param_1: int, basic_param_2: str) -> int:
5
    return basic_param_1 + int(basic_param_2)
Copied!
There are only a few considerations for the parameters and return types.
All parameters passed into the signature must be typed. Similarly, if you're returning something, it must be also be typed with the return operator (->)
ZenML uses Pydantic for type checking and serialization under-the-hood, so all Pydantic types are supported [full list available soon].
While this is just a function with a decorator, it is not super useful. ZenML steps really get powerful when you put them together with data artifacts. Read about more of that here!
Artifact Store
An artifact store is a place where artifacts are stored. These artifacts may have been produced by the pipeline steps, or they may be the data first ingested into a pipeline via an ingestion step.
Artifact
Artifacts are the data that power your experimentation and model training. It is actually steps that produce artifacts, which are then stored in the artifact store. Artifacts are written in the signature of a step like so:
1
// Some code
2
def my_step(first_artifact: int, second_artifact: torch.nn.Module -> int:
3
    # first_artifact is an integer
4
    # second_artifact is a torch.nn.Module
5
    return 1
Copied!
Artifacts can be serialized and deserialized (i.e. written and read from the Artifact Store) in many different ways like TFRecords or saved model pickles, depending on what the step produces.The serialization and deserialization logic of artifacts is defined by  Materializers.
Materializers
A materializer defines how and where Artifacts live in between steps.
Parameter
When we think about steps as functions, we know they receive input in the form of artifacts. We also know that they produce output (also in the form of artifacts, stored in the artifact store). But steps also take parameters. The parameters that you pass into the steps are also (helpfully!) stored in the metadata store. This helps freeze the iterations of your experimentation workflow in time so you can return to them exactly as you ran them. Parameters can be passed in as a subclass of BaseStepConfig like so:
1
from zenml.steps.base_step_config import BaseStepConfig
2
​
3
class MyStepConfig(BaseStepConfig):
4
    basic_param_1: int = 1
5
    basic_param_2: str = 2
6
    
7
@step
8
def my_step(params: MyStepConfig):
9
    # user params here
10
    pass
Copied!
Metadata Store
The configuration of each pipeline, step, backend, and produced artifacts are all tracked within the metadata store. The metadata store is an SQL database, and can be sqlite or mysql.
Metadata
Metadata are the pieces of information tracked about the pipelines, experiments and configurations that you are are running with ZenML. Metadata are stored inside the metadata store.
Orchestrator
An orchestrator is a special kind of backend that manages the running of each step of the pipeline. Orchestrators administer the actual pipeline runs. You can think of it as the 'root' of any pipeline job that you run during your experimentation.
Stack
A stack is made up of the following three core components:
An Artifact Store
A Metadata Store
An Orchestrator (backend)
A ZenML stack also happens to be a Pydantic BaseSettings class, which means that there are multiple ways to use it.
1
zenml stack register MY_NEW_STACK
2
    --metadata_store my-new-metadata-store \
3
    --artifact_store my-new-artifact-store \ 
4
    --orchestrator my-new-orchestrator
Copied!
Backend (Executors)
Backends are the infrastructure and environments on which your steps run. There are different kinds of backends depending on the particular use case. COMING SOON
Tying Things All Together
ZenML's core abstractions are either close to or replicate completely the commonly-found abstractions found in the industry for pipeline-style workflows. As a data scientist, it perhaps isn't natural to think of your work from within this 'pipeline' abstraction, but we think you'll see the benefits if you try it out with some examples. Check out our Get Started guide to see an example of what ZenML will add to your current workflow!
Important considerations
Artifact stores and metadata stores can be configured per repository as well as per pipeline. However, only pipelines with the same artifact store and metadata store are comparable, and therefore should not change to maintain the benefits of caching and consistency across pipeline runs.
On a high level, when data is read from an artifact the results are persisted in your artifact store. An orchestrator reads the data from the artifact store and begins preprocessing - either itself or alternatively on a dedicated processing backend like Google Dataflow. Every pipeline step reads its predecessor's result artifacts from the artifact store and writes its own result artifacts to the artifact store. Once preprocessing is done, the orchestration begins the training of your model - again either itself or on a separate / dedicated training backend. The trained model will be persisted in the artifact store, and optionally passed on to a serving backend.
A few rules apply:
Every orchestrator (local, Google Cloud VMs, etc) can run all pipeline steps, including training.
Orchestrators have a selection of compatible processing backends.
Pipelines can be configured to utilize more powerful processing (e.g. distributed) and training (e.g. Google AI Platform) executors.
A quick example for large datasets makes this clearer. By default, your experiments will run locally. Pipelines that load large datasets would be severely bottlenecked, so you can configure Google Dataflow as a processing executor for distributed computation, and Google AI Platform as a training executor.
System design
The design choices in ZenML follow the understanding that production-ready model training pipelines need to be immutable, repeatable, discoverable, descriptive, and efficient. ZenML takes care of the orchestration of your pipelines, from sourcing data all the way to continuous training - no matter if it's running somewhere locally, in an on-premise data center, or in the cloud.
In different words, ZenML runs your ML code while taking care of the "Operations" for you. It takes care of:
Interfacing between the individual processing steps (splitting, transform, training).
Tracking of intermediate results and metadata
Caching your processing artifacts.
Parallelization of computing tasks.
Ensuring the immutability of your pipelines from data sourcing to model artifacts.
No matter where - cloud, on-prem, or locally.
Since production scenarios often look complex, ZenML is built with integrations in mind. ZenML will support a range of integrations for processing, training, and serving, and you can always add custom integrations via our extensible interfaces.
Quickstart
Why ZenML?
Last modified 49m ago
Export as PDF
Copy link
Contents
Core Concepts
Important considerations
System design