Last updated
Was this helpful?
Last updated
Was this helpful?
is a powerful and flexible open-source framework for creating CI/CD systems, allowing developers to build, test, and deploy across cloud providers and on-premise systems.
This component is only meant to be used within the context of a . Usage with a local ZenML deployment may lead to unexpected behavior!
You should use the Tekton orchestrator if:
you're looking for a proven production-grade orchestrator.
you're looking for a UI in which you can track your pipeline runs.
you're already using Kubernetes or are not afraid of setting up and maintaining a Kubernetes cluster.
you're willing to deploy and maintain Tekton Pipelines on your cluster.
You'll first need to set up a Kubernetes cluster and deploy Tekton Pipelines:
A remote ZenML server. See the for more information.
Have an existing AWS set up.
Make sure you have the set up.
Download and kubectl
and it to talk to your EKS cluster using the following command:
Tekton Pipelines onto your cluster.
If one or more of the deployments are not in the Running
state, try increasing the number of nodes in your cluster.
ZenML has only been tested with Tekton Pipelines >=0.38.3 and may not work with previous versions.
To use the Tekton orchestrator, we need:
The ZenML tekton
integration installed. If you haven't done so, run
The name of your Kubernetes context which points to your remote cluster. Run kubectl config get-contexts
to see a list of available contexts.
We can then register the orchestrator and use it in our active stack. This can be done in two ways:
You can now run any ZenML pipeline using the Tekton orchestrator:
Tekton comes with its own UI that you can use to find further details about your pipeline runs, such as the logs of your steps.
To find the Tekton UI endpoint, we can use the following command:
For additional configuration of the Tekton orchestrator, you can pass TektonOrchestratorSettings
which allows you to configure node selectors, affinity, and tolerations to apply to the Kubernetes Pods running your pipeline. These can be either specified using the Kubernetes model objects or as dictionaries.
If your pipelines steps have certain hardware requirements, you can specify them as ResourceSettings
:
These settings can then be specified on either pipeline-level or step-level:
installed and running.
Tekton pipelines deployed on a remote cluster. See the for more information.
A as part of your stack.
A as part of your stack.
installed and the name of the Kubernetes configuration context which points to the target cluster (i.e. runkubectl config get-contexts
to see a list of available contexts). This is optional (see below).
It is recommended that you set up and use it to connect ZenML Stack Components to the remote Kubernetes cluster, especially If you are using a Kubernetes cluster managed by a cloud provider like AWS, GCP or Azure, This guarantees that your Stack is fully portable on other environments and your pipelines are fully reproducible.
If you have configured to access the remote Kubernetes cluster, you no longer need to set the kubernetes_context
attribute to a local kubectl
context. In fact, you don't need the local Kubernetes CLI at all. You can instead:
if you don't have a Service Connector on hand and you don't want to , the local Kubernetes kubectl
client needs to be configured with a configuration context pointing to the remote cluster. The kubernetes_context
stack component must also be configured with the value of that context:
ZenML will build a Docker image called <CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME>
which includes your code and use it to run your pipeline steps in Tekton. Check out if you want to learn more about how ZenML builds these images and how you can customize them.
Check out the for a full list of available attributes and for more information on how to specify settings.
For more information and a full list of configurable attributes of the Tekton orchestrator, check out the .
Note that if you wish to use this orchestrator to run steps on a GPU, you will need to follow to ensure that it works. It requires adding some extra settings customization and is essential to enable CUDA for the GPU to give its full acceleration.
Orchestrating your pipelines to run on Tekton.