Kubernetes Orchestrator

How to orchestrate pipelines with Kubernetes

This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.

The Kubernetes orchestrator is an orchestrator flavor provided with the ZenML kubernetes integration that runs your pipelines on a Kubernetes cluster.

When to use it

You should use the Kubernetes orchestrator if:

  • you're looking lightweight way of running your pipelines on Kubernetes.

  • you don't need a UI to list all your pipelines runs.

  • you're not willing to maintain Kubeflow Pipelines on your Kubernetes cluster.

  • you're not interested in paying for managed solutions like Vertex.

How to deploy it

The Kubernetes orchestrator requires a Kubernetes cluster in order to run. There are many ways to deploy a Kubernetes cluster using different cloud providers or on your custom infrastructure and we can't possibly cover all of them, but you can check out our cloud guide ZenML Cloud Guide for some complete stack deployments which use the Kubernetes orchestrator.

How to use it

To use the Kubernetes orchestrator, we need:

We can then register the orchestrator and use it in our active stack:

zenml orchestrator register <NAME> \
    --flavor=kubernetes \

# Add the orchestrator to the active stack
zenml stack update -o <NAME>

ZenML will build a Docker image called zenml-kubernetes which includes your code and use it to run your pipeline steps in Kubernetes. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.

If you decide you need the full flexibility of having a custom base image, you can update your existing orchestrator

zenml orchestrator update <NAME> \

or set it when registering a new Kubernetes orchestrator:

zenml orchestrator register <NAME> \
--flavor=kubernetes \

You can now run any ZenML pipeline using the Kubernetes orchestrator:

python file_that_runs_a_zenml_pipeline.py

A concrete example of using the Kubernetes orchestrator can be found here.

For more information and a full list of configurable attributes of the Kubernetes orchestrator, check out the API Docs.

Last updated