Develop a Custom Experiment Tracker
How to develop a custom experiment tracker
Base abstraction in progress!
We are actively working on the base abstraction for the Experiment Tracker, which will be available soon. As a result, their extension is not recommended at the moment. When you are selecting an Experiment Tracker for your stack, you can use one of the existing flavors.
If you need to implement your own Experiment Tracker flavor, you can still do so, but keep in mind that you may have to refactor it when the base abstraction is released.
If you want to create your own custom flavor for an experiment tracker, you can follow the following steps:
- 1.Create a class which inherits from the
BaseExperimentTrackerclass and implement the abstract methods.
- 2.If you need any configuration, create a class which inherits from the
BaseExperimentTrackerConfigclass add your configuration parameters.
- 3.Bring both of the implementation and the configuration together by inheriting from the
Once you are done with the implementation, you can register it through the CLI. Please ensure you point to the flavor class via dot notation:
zenml experiment-tracker flavor register <path.to.MyExperimentTrackerFlavor>
For example, if your flavor class
MyExperimentTrackerFlavoris defined in
flavors/my_flavor.py, you'd register it by doing:
zenml experiment-tracker flavor register flavors.my_flavor.MyExperimentTrackerFlavor
ZenML resolves the flavor class by taking the path where you initialized zenml (via
zenml init) as the starting point of resolution. Therefore, please ensure you follow the best practice of initializing zenml at the root of your repository.
If ZenML does not find an initialized ZenML repository in any parent directory, it will default to the current working directory, but usually its better to not have to rely on this mechanism, and initialize zenml at the root.
Afterwards, you should see the new flavor in the list of available flavors:
zenml experiment-tracker flavor list
It is important to draw attention to when and how these base abstractions are coming into play in a ZenML workflow.
- The CustomExperimentTrackerFlavor class is imported and utilized upon the creation of the custom flavor through the CLI.
- The CustomExperimentTrackerConfig class is imported when someone tries to register/update a stack component with this custom flavor. Especially, during the registration process of the stack component, the config will be used to validate the values given by the user. As
Configobject are inherently
pydanticobjects, you can also add your own custom validators here.
- The CustomExperimentTracker only comes into play when the component is ultimately in use.
The design behind this interaction lets us separate the configuration of the flavor from its implementation. This way we can register flavors and components even when the major dependencies behind their implementation are not installed in our local setting (assuming the
CustomExperimentTrackerConfigare implemented in a different module/path than the actual