Kubeflow Orchestrator

Orchestrating your pipelines to run on Kubeflow.

PreviousLocal Docker Orchestrator NextKubernetes Orchestrator

Last updated 21 days ago

Was this helpful?

Kubeflow Orchestrator

Orchestrating your pipelines to run on Kubeflow.

The Kubeflow orchestrator is an flavor provided by the ZenML kubeflow integration that uses to run your pipelines.

This component is only meant to be used within the context of a . Usage with a local ZenML deployment may lead to unexpected behavior!

When to use it

You should use the Kubeflow orchestrator if:

you're looking for a proven production-grade orchestrator.
you're looking for a UI in which you can track your pipeline runs.
you're already using Kubernetes or are not afraid of setting up and maintaining a Kubernetes cluster.
you're willing to deploy and maintain Kubeflow Pipelines on your cluster.

How to deploy it

To run ZenML pipelines on Kubeflow, you'll need to set up a Kubernetes cluster and deploy Kubeflow Pipelines on it. This can be done in a variety of ways, depending on whether you want to use a cloud provider or your own infrastructure:

Have an existing AWS set up.
Make sure you have the set up.
Download and kubectl and configure it to talk to your EKS cluster using the following command:
```
aws eks --region REGION update-kubeconfig --name CLUSTER_NAME
```
Kubeflow Pipelines onto your cluster.
( optional) to grant ZenML Stack Components easy and secure access to the remote EKS cluster.

Have an existing GCP set up.
Make sure you have the set up first.
Download and kubectl and it to talk to your GKE cluster using the following command:
```
gcloud container clusters get-credentials CLUSTER_NAME
```
Kubeflow Pipelines onto your cluster.
( optional) to grant ZenML Stack Components easy and secure access to the remote GKE cluster.

Have an existing set up.
Make sure you have the set up first.
Download and kubectl and ensure that it talks to your AKS cluster using the following command:
```
az aks get-credentials --resource-group RESOURCE_GROUP --name CLUSTER_NAME
```
Kubeflow Pipelines onto your cluster.

Since Kubernetes v1.19, AKS has shifted to . However, the workflow controller installed with the Kubeflow installation has Docker set as the default runtime. In order to make your pipelines work, you have to change the value to one of the options listed , preferably k8sapi.

This change has to be made by editing the containerRuntimeExecutor property of the ConfigMap corresponding to the workflow controller. Run the following commands to first know what config map to change and then to edit it to reflect your new value:

kubectl get configmap -n kubeflow
kubectl edit configmap CONFIGMAP_NAME -n kubeflow
# This opens up an editor that can be used to make the change.

If one or more of the deployments are not in the Running state, try increasing the number of nodes in your cluster.

If you're installing Kubeflow Pipelines manually, make sure the Kubernetes service is called exactly ml-pipeline. This is a requirement for ZenML to connect to your Kubeflow Pipelines deployment.

How to use it

To use the Kubeflow orchestrator, we need:

The ZenML kubeflow integration installed. If you haven't done so, run
```
zenml integration install kubeflow
```

We can then register the orchestrator and use it in our active stack. This can be done in two ways:

# List all available Kubernetes clusters that can be accessed by service connectors
zenml service-connector list-resources --resource-type kubernetes-cluster -e
# Register the Kubeflow orchestrator and connect it to the remote Kubernetes cluster 
zenml orchestrator register <ORCHESTRATOR_NAME> --flavor kubeflow --connector <SERVICE_CONNECTOR_NAME> --resource-id <KUBERNETES_CLUSTER_NAME>
# Register a new stack with the orchestrator
zenml stack register <STACK_NAME> -o <ORCHESTRATOR_NAME> -a <ARTIFACT_STORE_NAME> -c <CONTAINER_REGISTRY_NAME> ... # Add other stack components as needed

The following example demonstrates how to register the orchestrator and connect it to a remote Kubernetes cluster using a Service Connector:

$ zenml service-connector list-resources --resource-type kubernetes-cluster -e
The following 'kubernetes-cluster' resources can be accessed by service connectors that you have configured:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━┓
┃             CONNECTOR ID             │ CONNECTOR NAME        │ CONNECTOR TYPE │ RESOURCE TYPE         │ RESOURCE NAMES      ┃
┠──────────────────────────────────────┼───────────────────────┼────────────────┼───────────────────────┼─────────────────────┨
┃ e33c9fac-5daa-48b2-87bb-0187d3782cde │ aws-iam-multi-eu      │ 🔶 aws         │ 🌀 kubernetes-cluster │ kubeflowmultitenant ┃
┃                                      │                       │                │                       │ zenbox              ┃
┠──────────────────────────────────────┼───────────────────────┼────────────────┼───────────────────────┼─────────────────────┨
┃ ed528d5a-d6cb-4fc4-bc52-c3d2d01643e5 │ aws-iam-multi-us      │ 🔶 aws         │ 🌀 kubernetes-cluster │ zenhacks-cluster    ┃
┠──────────────────────────────────────┼───────────────────────┼────────────────┼───────────────────────┼─────────────────────┨
┃ 1c54b32a-4889-4417-abbd-42d3ace3d03a │ gcp-sa-multi          │ 🔵 gcp         │ 🌀 kubernetes-cluster │ zenml-test-cluster  ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━┛

$ zenml orchestrator register aws-kubeflow --flavor kubeflow --connector aws-iam-multi-eu --resource-id zenhacks-cluster
Successfully registered orchestrator `aws-kubeflow`.
Successfully connected orchestrator `aws-kubeflow` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┓
┃             CONNECTOR ID             │ CONNECTOR NAME   │ CONNECTOR TYPE │ RESOURCE TYPE         │ RESOURCE NAMES   ┃
┠──────────────────────────────────────┼──────────────────┼────────────────┼───────────────────────┼──────────────────┨
┃ ed528d5a-d6cb-4fc4-bc52-c3d2d01643e5 │ aws-iam-multi-us │ 🔶 aws         │ 🌀 kubernetes-cluster │ zenhacks-cluster ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┛

# Create a new stack with the orchestrator
$ zenml stack register --set aws-kubeflow -o aws-kubeflow -a aws-s3 -c aws-ecr
Stack 'aws-kubeflow' successfully registered!
        Stack Configuration           
┏━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━┓
┃ COMPONENT_TYPE     │ COMPONENT_NAME  ┃
┠────────────────────┼─────────────────┨
┃ ARTIFACT_STORE     │ aws-s3          ┃
┠────────────────────┼─────────────────┨
┃ ORCHESTRATOR       │ aws-kubeflow    ┃
┠────────────────────┼─────────────────┨
┃ CONTAINER_REGISTRY │ aws-ecr         ┃
┗━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━┛
        'aws-kubeflow' stack         
No labels are set for this stack.
Stack 'aws-kubeflow' with id 'dab28f94-36ab-467a-863e-8718bbc1f060' is owned by user user.
Active global stack set to:'aws-kubeflow'

zenml orchestrator register <ORCHESTRATOR_NAME> --flavor=kubeflow --kubernetes_context=<KUBERNETES_CONTEXT>

# Register a new stack with the orchestrator
zenml stack register <STACK_NAME> -o <ORCHESTRATOR_NAME> -a <ARTIFACT_STORE_NAME> -c <CONTAINER_REGISTRY_NAME> ... # Add other stack components as needed

You can now run any ZenML pipeline using the Kubeflow orchestrator:

python file_that_runs_a_zenml_pipeline.py

Kubeflow UI

Kubeflow comes with its own UI that you can use to find further details about your pipeline runs, such as the logs of your steps. For any runs executed on Kubeflow, you can get the URL to the Kubeflow UI in Python using the following code snippet:

from zenml.client import Client

pipeline_run = Client().get_pipeline_run("<PIPELINE_RUN_NAME>")
orchestrator_url = pipeline_run.run_metadata["orchestrator_url"]

Additional configuration

For additional configuration of the Kubeflow orchestrator, you can pass KubeflowOrchestratorSettings which allows you to configure (among others) the following attributes:

client_args: Arguments to pass when initializing the KFP client.
user_namespace: The user namespace to use when creating experiments and runs.
pod_settings: Node selectors, affinity, and tolerations to apply to the Kubernetes Pods running your pipeline. These can be either specified using the Kubernetes model objects or as dictionaries.

from zenml.integrations.kubeflow.flavors.kubeflow_orchestrator_flavor import KubeflowOrchestratorSettings
from kubernetes.client.models import V1Toleration

kubeflow_settings = KubeflowOrchestratorSettings(
    client_args={},
    user_namespace="my_namespace",
    pod_settings={
        "affinity": {
            "nodeAffinity": {
                "requiredDuringSchedulingIgnoredDuringExecution": {
                    "nodeSelectorTerms": [
                        {
                            "matchExpressions": [
                                {
                                    "key": "node.kubernetes.io/name",
                                    "operator": "In",
                                    "values": ["my_powerful_node_group"],
                                }
                            ]
                        }
                    ]
                }
            }
        },
        "tolerations": [
            V1Toleration(
                key="node.kubernetes.io/name",
                operator="Equal",
                value="",
                effect="NoSchedule"
            )
        ]
    }
)


@pipeline(
    settings={
        "orchestrator": kubeflow_settings
    }
)


...

Enabling CUDA for GPU-backed hardware

Important Note for Multi-Tenancy Deployments

Using the ZenML Kubeflow orchestrator on a multi-tenant deployment without any settings will result in the following error:

HTTP response body: {"error":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.","code":3,"message":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by 
namespace.","details":[{"@type":"type.googleapis.com/api.Error","error_message":"Invalid resource references for experiment. ListExperiment requires filtering by namespace.","error_details":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace."}]}

In order to get it to work, we need to leverage the KubeflowOrchestratorSettings referenced above. By setting the namespace option, and by passing in the right authentication credentials to the Kubeflow Pipelines Client, we can make it work.

First, when registering your Kubeflow orchestrator, please make sure to include the kubeflow_hostname parameter. The kubeflow_hostname must end with the /pipeline post-fix.

zenml orchestrator register <NAME> \
    --flavor=kubeflow \
    --kubeflow_hostname=<KUBEFLOW_HOSTNAME> # e.g. https://mykubeflow.example.com/pipeline

Then, ensure that you use the pass the right settings before triggering a pipeline run. The following snippet will prove useful:

import requests

from zenml.client import Client
from zenml.integrations.kubeflow.flavors.kubeflow_orchestrator_flavor import (
    KubeflowOrchestratorSettings,
)

NAMESPACE = "namespace_name"  # This is the user namespace for the profile you want to use
USERNAME = "admin"  # This is the username for the profile you want to use
PASSWORD = "abc123"  # This is the password for the profile you want to use

# Use client_username and client_password and ZenML will automatically fetch a session cookie
kubeflow_settings = KubeflowOrchestratorSettings(
    client_username=USERNAME,
    client_password=PASSWORD,
    user_namespace=NAMESPACE
)


# You can also pass the cookie in `client_args` directly
# kubeflow_settings = KubeflowOrchestratorSettings(
#     client_args={"cookies": session_cookie}, user_namespace=NAMESPACE
# )

@pipeline(
    settings={
        "orchestrator": kubeflow_settings
    }
)

:
...

if "__name__" == "__main__":
# Run the pipeline

Using secrets in settings

The above example encoded the username and password in plain text as settings. You can also set them as secrets.

zenml secret create kubeflow_secret \
    --username=admin \
    --password=abc123

And then you can use them in code:

# Use client_username and client_password and ZenML will automatically fetch a session cookie
kubeflow_settings = KubeflowOrchestratorSettings(
    client_username="{{kubeflow_secret.username}}",  # secret reference
    client_password="{{kubeflow_secret.password}}",  # secret reference
    user_namespace="namespace_name"
)

PreviousLocal Docker Orchestrator NextKubernetes Orchestrator

Last updated 21 days ago

Was this helpful?