Scheduling

Learn how to set, pause and stop a schedule for pipelines.

Schedules don't work for all orchestrators. Here is a list of all supported orchestrators.

Orchestrator	Scheduling Support	Supported Schedule Types
AirflowOrchestrator	✅	Cron, Interval
AzureMLOrchestrator	✅	Cron, Interval
DatabricksOrchestrator	✅	Cron only
HyperAIOrchestrator	✅	Cron, One-time
KubeflowOrchestrator	✅	Cron, Interval
KubernetesOrchestrator	✅	Cron only
LocalOrchestrator	⛔️	N/A
LocalDockerOrchestrator	⛔️	N/A
SagemakerOrchestrator	✅	Cron, Interval, One-time
SkypilotAWSOrchestrator	⛔️	N/A
SkypilotAzureOrchestrator	⛔️	N/A
SkypilotGCPOrchestrator	⛔️	N/A
SkypilotLambdaOrchestrator	⛔️	N/A
TektonOrchestrator	⛔️	N/A
VertexOrchestrator	✅	Cron only

Check out our tutorial on scheduling for a practical guide on how to schedule a pipeline.

Set a schedule

from zenml.config.schedule import Schedule
from zenml import pipeline
from datetime import datetime

@pipeline()
def my_pipeline(...):
    ...

# Use cron expressions
schedule = Schedule(cron_expression="5 14 * * 3")
# or alternatively use human-readable notations
schedule = Schedule(start_time=datetime.now(), interval_second=1800)

my_pipeline = my_pipeline.with_options(schedule=schedule)
my_pipeline()

Check out our SDK docs to learn more about the different scheduling options.

Update/Pause/Stop a schedule

The way pipelines are scheduled depends on the orchestrator you are using. For example, if you are using Kubeflow, you can use the Kubeflow UI to stop or pause a scheduled run. However, the exact steps for stopping or pausing a scheduled run may vary depending on the orchestrator you are using. We recommend consulting the documentation for your orchestrator to learn the current method for stopping or pausing a scheduled run.

The normal pattern for updating a schedule is:

1. Find schedule on ZenML

2. Match schedule on orchestrator side and delete

3. Delete schedule on ZenML

4. Re-run pipeline with new schedule

A concrete example can be found on the GCP Vertex orchestrator docs, and this pattern can be adapted for other orchestrators as well.

Note that ZenML only gets involved to schedule a run, but maintaining the lifecycle of the schedule (as explained above) is the responsibility of the user. If you run a pipeline containing a schedule two times, two scheduled pipelines (with different/unique names) will be created.