Productionalize with CI/CD/CT

How to include ZenML in the CI/CD/CT workflows

This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.

Productionalize with CI/CD/CT

In many production settings, it is undesirable to have all developers accessing and running on the production environments. Therefore, it is best to "gate" certain pipelines running on production stacks. This can be achieved through the concept of Continuous Integration and Continuous Deployment (CI/CD).

CI/CD represents an approach to continuously integrate new code, data, and features into existing pipelines, validate the quality through automated testing, and then automatically deploy improved and tested ML pipelines into production. Practically, this means that the running of the pipeline on a certain stack should be done not on a local developer machine, but rather than in CI job, using services like Jenkins , GitHub Actions, Circle CI and more.

The Git Workflow

Here is how such a CI/CD system could look like:

How to use ZenML with CI/CD frameworks

You can find a practical example that uses the visualized flow here. In particular, take a look at the workflow file here.

In essence this is what happens:

1) User writes and pushes code to their code repository

2) The CI/CD Framework of the code repository starts a worker

3) Within this worker zenml and other requirements will be installed

4 - 6) The worker connects to the ZenML server and triggers a pipeline run

7 - 11) The pipeline is now executed on the stack of choice

Best Practices

• Create a dedicated user for the CI/CD workflow.

• Always make sure to call zenml connect in your worker during setup with the above user. Make sure the credentials are secrets in your workflow.

• Create a staging stack when merging to a develop branch, and trigger the production workflow only when merging to the main branch.

• Understand how containerization works in ZenML, and leverage advantages such as build caching and code repository.