Kubeflow Orchestrator
How to orchestrate pipelines with Kubeflow
This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.
The Kubeflow orchestrator is an orchestrator flavor provided with the ZenML kubeflow
integration that uses Kubeflow Pipelines to run your pipelines.
This component is only meant to be used within the context of remote ZenML deployment scenario. Usage with a local ZenML deployment may lead to unexpected behavior!
When to use it
You should use the Kubeflow orchestrator if:
you're looking for a proven production-grade orchestrator.
you're looking for a UI in which you can track your pipeline runs.
you're already using Kubernetes or are not afraid of setting up and maintaining a Kubernetes cluster.
you're willing to deploy and maintain Kubeflow Pipelines on your cluster.
How to deploy it
The Kubeflow orchestrator supports two different modes: Local
and remote
. In case you want to run the orchestrator on a local Kubernetes cluster running on your machine, there is no additional infrastructure setup necessary.
If you want to run your pipelines on a remote cluster instead, you'll need to set up a Kubernetes cluster and deploy Kubeflow Pipelines:
Have an existing AWS EKS cluster set up.
Make sure you have the AWS CLI set up.
Install Kubeflow Pipelines onto your cluster.
If one or more of the deployments are not in the Running
state, try increasing the number of nodes in your cluster.
If you're installing Kubeflow Pipelines manually, make sure the Kubernetes service is called exactly ml-pipeline
. This is a requirement for ZenML to connect to your Kubeflow Pipelines deployment.
How to use it
To use the Kubeflow orchestrator, we need:
The ZenML
kubeflow
integration installed. If you haven't done so, runDocker installed and running.
kubectl installed.
When using the Kubeflow orchestrator locally, you'll additionally need:
K3D installed to spin up a local Kubernetes cluster.
Terraform installed to set up the Kubernetes cluster with various deployments.
To run the pipeline on a local Kubeflow Pipelines deployment, you can use the ZenML Stack recipes to spin up a local Kubernetes cluster and install Kubeflow Pipelines on it. The stack recipe is called k3d-modular
and is available in the ZenML stack recipe repository. The recipe is modular, meaning that you can configured it to use different orchestrators, Model Deployers, and other tools.
To deploy the stack, run the following commands:
You can read more about the recipes in Stack Recipes. or check the recipe code in the ZenML Stack Recipe Repository.
The local Kubeflow Pipelines deployment requires more than 4 GB of RAM, and 30 GB of disk space, so if you are using Docker Desktop make sure to update the resource limits in the preferences.
ZenML will build a Docker image called <CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME>
which includes your code and use it to run your pipeline steps in Kubeflow. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.
You can now run any ZenML pipeline using the Kubeflow orchestrator:
Kubeflow UI
Kubeflow comes with its own UI that you can use to find further details about your pipeline runs, such as the logs of your steps. For any runs executed on Kubeflow, you can get the URL to the Kubeflow UI in Python using the following code snippet:
Additional configuration
For additional configuration of the Kubeflow orchestrator, you can pass KubeflowOrchestratorSettings
which allows you to configure (among others) the following attributes:
client_args
: Arguments to pass when initializing the KFP client.user_namespace
: The user namespace to use when creating experiments and runs.pod_settings
: Node selectors, affinity and tolerations to apply to the Kubernetes Pods running your pipline. These can be either specified using the Kubernetes model objects or as dictionaries.
Check out the API docs for a full list of available attributes and this docs page for more information on how to specify settings.
Enabling CUDA for GPU-backed hardware
Note that if you wish to use this orchestrator to run steps on a GPU, you will need to follow the instructions on this page to ensure that it works. It requires adding some extra settings customization and is essential to enable CUDA for the GPU to give its full acceleration.
Important Note for Multi-Tenancy Deployments
Kubeflow has a notion of multi-tenancy built into its deployment. Kubeflow's multi-user isolation simplifies user operations because each user only views and edited\s the Kubeflow components and model artifacts defined in their configuration.
Using the ZenML Kubeflow orchestrator on a multi-tenant deployment without any settings will result in the following error:
In order to get it to work, we need to leverage the KubeflowOrchestratorSettings
referenced above. By setting the namespace option, and by passing in the right authentication credentials to the Kubeflow Pipelines Client, we can make it work.
First, when registering your kubeflow orchestrator, please make sure to include the kubeflow_hostname
parameter. The kubeflow_hostname
must end with the /pipeline
post-fix.
Then, ensure that you use the pass the right settings before triggering a pipeline run. The following snippet will prove useful:
Note that the above is also currently not tested on all Kubeflow versions, so there might be further bugs with older Kubeflow versions. In this case, please reach out to us on Slack.
Using secrets in settings
The above example encoded the username and password in plain-text as settings. You can also set them as secrets.
And then you can use them in code:
See full documentation of using secrets within ZenML here.
A concrete example of using the Kubeflow orchestrator can be found here.
For more information and a full list of configurable attributes of the Kubeflow orchestrator, check out the API Docs.
Last updated