Trigger a pipeline from another pipeline

Trigger a pipeline from another pipeline.

This is a ZenML Pro only feature. Please sign up here get access. OSS users can only trigger a pipeline by calling the pipeline function inside their runner script.

Triggering a pipeline from another only works if you've created at least one run template for that pipeline.

import pandas as pd
from zenml import pipeline, step
from zenml.client import Client
from zenml.config.pipeline_run_configuration import PipelineRunConfiguration

@step  
def trainer(data_artifact_id: str):
    df = load_artifact(data_artifact_id)

@pipeline
def training_pipeline():
    trainer()

@step  
def load_data() -> pd.Dataframe:
    ...

@step  
def trigger_pipeline(df: UnmaterializedArtifact):
    # By using UnmaterializedArtifact we can get the ID of the artifact
    run_config = PipelineRunConfiguration(steps={"trainer": {"parameters": {"data_artifact_id": df.id}}})
    Client().trigger_pipeline("training_pipeline", run_configuration=run_config)

@pipeline  
def loads_data_and_triggers_training():
    df = load_data()
    trigger_pipeline(df)  # Will trigger the other pipeline

Read more about the PipelineRunConfiguration and trigger_pipeline function object in the SDK Docs.

Read more about Unmaterialized Artifacts here.

Learn how to run a pipeline directly from the Python Client

Learn how to run a pipeline from the REST API