Metadata Stores
Tracking the execution of your pipelines/steps.
This is an older version of the ZenML documentation. To check the latest version please visit https://docs.zenml.io
The configuration of each pipeline, step and produced artifacts are all tracked within the metadata store. ZenML puts a lot of emphasis on guaranteed tracking of inputs across pipeline steps. The strict, fully automated, and deeply built-in tracking enables some powerful features - e.g. reproducibility.
Base abstraction in progress!
We are actively working on the base abstraction for the metadata stores, which will be available soon. As a result, their extension is not possible at the moment. When you are selecting a metadata store for your stack, you can use one of the flavors listed down below.
List of available metadata stores
In its current version, ZenML features three metadata store flavors, namely the sqlite
, kubeflow
, and mysql
flavors. These flavors are to be used in different contexts, but in general, we suggest to use the mysql
flavor for most use cases.
sqlite
: This is the default store. Only use this for running ZenML locally.kubeflow
: Kubeflow boasts an internal metadata store, which ZenML can leverage. This flavor of metadata store is only to be used in combination with theKubeflowOrchestrator
.mysql
: To be used for most other production settings, including with theKubeflowOrchestrator
or other orchestrators that do not support an internal metadata store.
Flavor | Integration | |
---|---|---|
sqlite |
| |
mysql |
| |
kubeflow | kubeflow | |
kubernetes | kubernetes |
If you would like to see the available flavors for metadata stores, you can use the command:
Last updated