Implement a custom integration
Creating an external integration and contributing to ZenML
One of the main goals of ZenML is to find some semblance of order in the ever-growing MLOps landscape. ZenML already provides numerous integrations into many popular tools, and allows you to come up with ways to implement your own stack component flavors in order to fill in any gaps that are remaining.
However, what if you want to make your extension of ZenML part of the main codebase, to share it with others? If you are such a person, e.g., a tooling provider in the ML/MLOps space, or just want to contribute a tooling integration to ZenML, this guide is intended for you.
Step 1: Plan out your integration
In the previous page, we looked at the categories and abstractions that core ZenML defines. In order to create a new integration into ZenML, you would need to first find the categories that your integration belongs to. The list of categories can be found here as well.
Note that one integration may belong to different categories: For example, the cloud integrations (AWS/GCP/Azure) contain container registries, artifact stores etc.
Step 2: Create individual stack component flavors
Each category selected above would correspond to a stack component type. You can now start developing individual stack component flavors for this type by following the detailed instructions on the respective pages.
Before you package your new components into an integration, you may want to use/test them as a regular custom flavor. For instance, if you are developing a custom orchestrator and your flavor class MyOrchestratorFlavor
is defined in flavors/my_flavor.py
, you can register it by using:
ZenML resolves the flavor class by taking the path where you initialized zenml (via zenml init
) as the starting point of resolution. Therefore, please ensure you follow the best practice of initializing zenml at the root of your repository.
If ZenML does not find an initialized ZenML repository in any parent directory, it will default to the current working directory, but usually it's better to not have to rely on this mechanism, and initialize zenml at the root.
Afterward, you should see the new flavor in the list of available flavors:
See the docs on extensibility of the different components here or get inspired by the many integrations that are already implemented such as the MLflow experiment tracker.
Step 3: Create an integration class
Once you are finished with your flavor implementations, you can start the process of packaging them into your integration and ultimately the base ZenML package. Follow this checklist to prepare everything:
1. Clone Repo
Once your stack components work as a custom flavor, you can now clone the main zenml repository and follow the contributing guide to set up your local environment for develop.
2. Create the integration directory
All integrations live within src/zenml/integrations/
in their own sub-folder. You should create a new folder in this directory with the name of your integration.
An example integration directory would be structured as follows:
3. Define the name of your integration in constants
In zenml/integrations/constants.py
, add:
This will be the name of the integration when you run:
4. Create the integration class __init__.py
In src/zenml/integrations/<YOUR_INTEGRATION>/init__.py
you must now create a new class, which is a subclass of the Integration
class, set some important attributes (NAME
and REQUIREMENTS
), and overwrite the flavors
class method.
Have a look at the MLflow Integration as an example for how it is done.
5. Import in all the right places
The Integration itself must be imported within src/zenml/integrations/__init__.py
.
You can now create a PR to ZenML and wait for the core maintainers to take a look. Thank you so much for your contribution to the codebase, rock on! 💜
Last updated