Compose pipelines

Composing your ZenML pipelines.

Sometimes it can be useful to extract some common functionality into separate functions in order to avoid code duplication. To facilitate this, ZenML allows you to compose your pipelines:

from zenml import pipeline

def data_loading_pipeline(mode: str):
    if mode == "train":
        data = training_data_loader_step()
        data = test_data_loader_step()
    processed_data = preprocessing_step(data)
    return processed_data

def training_pipeline():
    training_data = data_loading_pipeline(mode="train")
    model = training_step(data=training_data)
    test_data = data_loading_pipeline(mode="test")
    evaluation_step(model=model, data=test_data)

Calling a pipeline inside another pipeline does not actually trigger a separate run of the child pipeline but instead invokes the steps of the child pipeline to the parent pipeline.

Last updated