Trigger a pipeline from another

Trigger a pipeline from another pipeline.

This is a ZenML Pro only feature. Please sign up here get access. OSS users can only trigger a pipeline by calling the pipeline function inside their runner script.

Triggering a pipeline from another only works with pipelines that are configured with a remote stack (i.e. at least a remote orchestrator, artifact store, and container registry). In order to trigger a pipeline from another, you can simply use the same syntax as you would if you wanted to trigger a pipeline from the Client directly.

import pandas as pd
from zenml import pipeline, step
from zenml.client import Client
from zenml.config.pipeline_run_configuration import PipelineRunConfiguration

@step  
def trainer(data_artifact_id: str):
    df = load_artifact(data_artifact_id)

@pipeline
def training_pipeline():
    trainer()

@step  
def load_data() -> pd.Dataframe:
    ...

@step  
def trigger_pipeline(df: UnmaterializedArtifact):
    # By using UnmaterializedArtifact we can get the ID of the artifact
    run_config = PipelineRunConfiguration(steps={"trainer": {"parameters": {"data_artifact_id": df.id}}})
    Client().trigger_pipeline("training_pipeline", run_configuration=run_config)

@pipeline  
def loads_data_and_triggers_training():
    df = load_data()
    trigger_pipeline(df)  # Will trigger the other pipeline

Read more about the PipelineRunConfiguration object in the SDK Docs.

Read more about Unmaterialized Artifacts here.

Learn how to run a pipeline directly from the Python Client

Learn how to run a pipeline from the REST API