Productionalize with CI/CD/CT

How to include ZenML in the CI/CD/CT workflows

In many production settings, it is undesirable to have all developers accessing and running on the production environments. Therefore, it is best to "gate" certain pipelines running on production stacks. This can be achieved through the concept of Continuous Integration and Continuous Deployment (CI/CD).
CI/CD represents an approach to continuously integrate new code, data, and features into existing pipelines, validate the quality through automated testing, and then automatically deploy improved and tested ML pipelines into production. Practically, this means that the running of the pipeline on a certain stack should be done not on a local developer machine, but rather than in a CI job, using services like Jenkins, GitHub Actions, Circle CI, and more.
The Git Workflow
Here is what such a CI/CD system could look like:
How to use ZenML with CI/CD frameworks
You can find a practical example that uses the visualized flow here. In particular, take a look at the workflow file here.
In essence, this is what happens:
1) User writes and pushes code to their code repository
2) The CI/CD Framework of the code repository starts a worker
3) Within this worker zenml and other requirements will be installed
4 - 6) The worker connects to the ZenML server and triggers a pipeline run
7 - 11) The pipeline is now executed on the stack of choice
Best Practices
Create a dedicated user for the CI/CD workflow.
Always make sure to call zenml connect in your worker during setup with the above user. Make sure the credentials are secrets in your workflow.
Create a staging stack when merging to a develop branch, and trigger the production workflow only when merging to the main branch.
Understand how containerization works in ZenML, and leverage advantages such as build-caching and code repository.

Contribute flavors or components

Implement a custom stack component

Last modified 3d ago