Configure Automated Caching
How automated caching works in ZenML
Caching in ZenML
ZenML comes with caching enabled by default. Since ZenML automatically tracks and versions all inputs, outputs, and parameters of steps and pipelines, ZenML will not re-execute steps within the same pipeline on subsequent pipeline runs as long as there is no change in these three.
Currently, the caching does not automatically detect changes within the file system or on external APIs. Make sure to set caching to False
on steps that depend on external inputs or if the step should run regardless of caching.
Configuring caching behavior of your pipelines
Although caching is desirable in many circumstances, one might want to disable it in certain instances. For example, if you are quickly prototyping with changing step definitions or you have an external API state change in your function that ZenML does not detect.
There are multiple ways to take control of when and where caching is used:
Disabling caching for the entire pipeline
On a pipeline level the caching policy can be set as a parameter within the decorator.
If caching is explicitly turned off on a pipeline level, all steps are run without caching, even if caching is set to True
for single steps.
Disabling caching for individual steps
Caching can also be explicitly turned off at a step level. You might want to turn off caching for steps that take external input (like fetching data from an API or File IO).
Dynamically disabling caching for a pipeline run
Sometimes you want to have control over caching at runtime instead of defaulting to the backed in configurations of your pipeline and its steps. ZenML offers a way to override all caching settings of the pipeline at runtime.
Code Example
Last updated