Tekton Orchestrator
How to orchestrate pipelines with Tekton
This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.
The Tekton orchestrator is an orchestrator flavor provided with the ZenML tekton
integration that uses Tekton Pipelines to run your pipelines.
When to use it
You should use the Tekton orchestrator if:
you're looking for a proven production-grade orchestrator.
you're looking for a UI in which you can track your pipeline runs.
you're already using Kubernetes or are not afraid of setting up and maintaining a Kubernetes cluster.
you're willing to deploy and maintain Tekton Pipelines on your cluster.
How to deploy it
You'll first need to set up a Kubernetes cluster and deploy Tekton Pipelines:
Have an existing AWS EKS cluster set up.
Make sure you have the AWS CLI set up.
Install Tekton Pipelines onto your cluster.
If one or more of the deployments are not in the Running
state, try increasing the number of nodes in your cluster.
ZenML has only been tested with Tekton Pipelines >=0.38.3 and may not work with previous versions.
How to use it
To use the Tekton orchestrator, we need:
The ZenML
tekton
integration installed. If you haven't done so, runDocker installed and running.
kubectl installed.
Tekton pipelines deployed on a remote cluster. See the deployment section for more information.
The name of your Kubernetes context which points to your remote cluster. Run
kubectl config get-contexts
to see a list of available contexts.A remote artifact store as part of your stack.
A remote metadata store as part of your stack. We suggest using a MySQL metatadata store.
A remote container registry as part of your stack.
We can then register the orchestrator and use it in our active stack:
ZenML will build a Docker image called <CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME>
which includes your code and use it to run your pipeline steps in Tekton. Check out this page if you want to learn more about how ZenML builds these images and how you can customize them.
Once the orchestrator is part of the active stack, we need to run zenml stack up
before running any pipelines. This command forwards a port so you can view the Tekton UI in your browser.
You can now run any ZenML pipeline using the Tekton orchestrator:
A concrete example of using the Tekton orchestrator can be found here.
For more information and a full list of configurable attributes of the Tekton orchestrator, check out the API Docs.
Last updated