Artifact saving and loading
Saving and loading artifacts manually.
Whenever you run a ZenML pipeline, all outputs of your steps are automatically saved as artifacts into your artifact store and subsequent steps automatically load them in again. For most use cases, this is the recommended way of saving or loading artifacts. However, there are a few exceptions that are enabled by the following advanced concepts:
Using External Artifacts, you can pass existing artifacts from other pipeline runs into your steps. You can also use them to pass complex Python objects that were not loaded by any other steps of your pipeline.
You can also save and load artifacts manually without defining them as inputs or outputs of your pipeline steps. This is primarily useful for creating and inspecting ZenML artifacts outside of your pipeline runs, e.g., if parts of your ML workflow are still done manually in Jupyter notebooks or the like.
External Artifacts
External artifacts can be used to pass values to steps that are neither JSON serializable nor produced by an upstream step:
You can also use an ExternalArtifact
to pass an artifact stored in the ZenML database. Search can be performed in one of the following two ways:
By providing an artifact ID.
By providing an artifact name and version. If no version is provided, the latest version of that artifact will be used.
Optionally, you can configure the ExternalArtifact
to use a custom materializer for your data or disable artifact metadata and visualizations. Check out the SDK docs for all available options.
Using an ExternalArtifact
with input data for your step automatically disables caching for the step.
Saving and Loading Artifacts Manually
You can save and load ZenML artifacts manually by using the zenml.save_artifact()
and zenml.load_artifact()
util functions:
It is also possible to use these functions inside your ZenML steps. However, it is usually cleaner to return the artifacts as outputs of your step to save them, or to use External Artifacts to load them instead.
Last updated