Trigger a pipeline from another pipeline

Trigger a pipeline from another pipeline.

This is a ZenML Pro only feature. Please sign up here get access. OSS users can only trigger a pipeline by calling the pipeline function inside their runner script.

Triggering a pipeline from another only works with pipelines that are configured with a remote stack (i.e. at least a remote orchestrator, artifact store, and container registry). In order to trigger a pipeline from another, you can simply use the same syntax as you would if you wanted to trigger a pipeline from the Client directly.

import pandas as pd
from zenml import pipeline, step
from zenml.client import Client
from zenml.config.pipeline_run_configuration import PipelineRunConfiguration

@step  
def trainer(data_artifact_id: str):
    df = load_artifact(data_artifact_id)

@pipeline
def training_pipeline():
    trainer()

@step  
def load_data() -> pd.Dataframe:
    ...

@step  
def trigger_pipeline(df: UnmaterializedArtifact):
    # By using UnmaterializedArtifact we can get the ID of the artifact
    run_config = PipelineRunConfiguration(steps={"trainer": {"parameters": {"data_artifact_id": df.id}}})
    Client().trigger_pipeline("training_pipeline", run_configuration=run_config)

@pipeline  
def loads_data_and_triggers_training():
    df = load_data()
    trigger_pipeline(df)  # Will trigger the other pipeline

The pipeline that you're triggering (i.e. training_pipeline in the above example) has to have been run previously on a remote stack. In other words, the functionality to trigger a pipeline from another only works when a Docker image has previously been built for that pipeline. In most cases this will be because you ran the pipeline already, but in some cases you might have built the image separately.

Read more about the PipelineRunConfiguration and trigger_pipeline function object in the SDK Docs.

Read more about Unmaterialized Artifacts here.

Last updated