Kubeflow Orchestrator
How to orchestrate pipelines with Kubeflow
When to use it
You should use the Kubeflow orchestrator if:
you're looking for a proven production-grade orchestrator.
you're looking for a UI in which you can track your pipeline runs.
you're already using Kubernetes or are not afraid of setting up and maintaining a Kubernetes cluster.
you're willing to deploy and maintain Kubeflow Pipelines on your cluster.
How to deploy it
The Kubeflow orchestrator supports two different modes: Local
and remote
. In case you want to run the orchestrator on a local Kubernetes cluster running on your machine, there is no additional infrastructure setup necessary.
If you want to run your pipelines on a remote cluster instead, you'll need to set up a Kubernetes cluster and deploy Kubeflow Pipelines:
If one or more of the deployments are not in the Running
state, try increasing the number of nodes in your cluster.
If you're installing Kubeflow Pipelines manually, make sure the Kubernetes service is called exactly ml-pipeline
. This is a requirement for ZenML to connect to your Kubeflow Pipelines deployment.
How to use it
To use the Kubeflow orchestrator, we need:
The ZenML
kubeflow
integration installed. If you haven't done so, run
When using the Kubeflow orchestrator locally, you'll additionally need
The local Kubeflow Pipelines deployment requires more than 2 GB of RAM, so if you're using Docker Desktop make sure to update the resource limits in the preferences.
We can then register the orchestrator and use it in our active stack:
or set it when registering a new Kubeflow orchestrator:
Once the orchestrator is part of the active stack, we need to run zenml stack up
before running any pipelines. This command
forwards a port so you can view the Kubeflow UI in your browser.
(in the local case) uses K3D to provision a Kubernetes cluster on your machine and deploys Kubeflow Pipelines on it.
You can now run any ZenML pipeline using the Kubeflow orchestrator:
Last updated