Develop a custom data validator
How to develop a custom data validator
Develop a custom data validator
Base abstraction in progress!
If you need to implement your own Data Validator flavor, you can still do so, but keep in mind that you may have to refactor it when the base abstraction is updated.
Build your own custom data validator
If you want to implement your own custom Data Validator, you can follow the following steps:
If you need any configuration, you can create a class which inherits from the
BaseDataValidatorConfig
class.Bring both of these classes together by inheriting from the
BaseDataValidatorFlavor
.(Optional) You should also provide some standard steps that others can easily insert into their pipelines for instant access to data validation features.
Once you are done with the implementation, you can register it through the CLI. Please ensure you point to the flavor class via dot notation:
For example, if your flavor class MyDataValidatorFlavor
is defined in flavors/my_flavor.py
, you'd register it by doing:
If ZenML does not find an initialized ZenML repository in any parent directory, it will default to the current working directory, but usually it's better to not have to rely on this mechanism, and initialize zenml at the root.
Afterwards, you should see the new flavor in the list of available flavors:
It is important to draw attention to when and how these base abstractions are coming into play in a ZenML workflow.
The CustomDataValidatorFlavor class is imported and utilized upon the creation of the custom flavor through the CLI.
The CustomDataValidatorConfig class is imported when someone tries to register/update a stack component with this custom flavor. Especially, during the registration process of the stack component, the config will be used to validate the values given by the user. As
Config
object are inherentlypydantic
objects, you can also add your own custom validators here.The CustomDataValidator only comes into play when the component is ultimately in use.
The design behind this interaction lets us separate the configuration of the flavor from its implementation. This way we can register flavors and components even when the major dependencies behind their implementation are not installed in our local setting (assuming the CustomDataValidatorFlavor
and the CustomDataValidatorConfig
are implemented in a different module/path than the actual CustomDataValidator
).
Last updated