0.9.0
Developer Guide
Advanced Guide
Extending ZenML
Links & Resources
Reference
Artifact Stores
Setting up the storage for your artifacts
In ZenML, all the inputs and outputs which go through a step are treated as artifacts and as its name suggests, an
ArtifactStore is a place where these artifacts get stored.Before reading this chapter, make sure that you are familiar with the concept of stacks, stack components and their flavors.
Base Abstraction
The artifact store establishes one of the main components in every ZenML stack. Now, let us take a deeper dive into the fundamentals behind its abstraction, namely the
BaseArtifactStore:- 1.As it is the base class for a specific type of StackComponent, it inherits from the StackComponent class. This sets the
TYPEvariable to a StackComponentType. TheFLAVORclass variable needs to be set in the specific subclass. - 2.As ZenML only supports filesystem-based artifact stores, it features an instance configuration parameter called
path, which will indicate the root path of the artifact store. When creating an instance of any flavor of anArtifactStore, users will have to define this parameter. - 3.Moreover, there is an empty class variable called
SUPPORTED_SCHEMESthat needs to be defined in every flavor implementation. It indicates the supported filepath schemes for the corresponding implementation. For instance, for the Azure artifact store, this set will be defined as{"abfs://", "az://"}. - 4.Lastly, the base class features a set of
abstractmethods:open,copyfile,exists,glob,isdir,listdir,makedirs,mkdir,remove,rename,rmtree,stat,walk. In the implementation of everyArtifactStoreflavor, it is required to define these methods with respect to the flavor at hand.
Putting all these considerations together, we end up with the following implementation:
1
from zenml.enums import StackComponentType
2
from zenml.stack import StackComponent
3
​
4
PathType = Union[bytes, str]
5
​
6
class BaseArtifactStore(StackComponent):
7
"""Base class for all ZenML artifact stores."""
8
​
9
# --- Instance configuration ---
10
path: str # The root path of the artifact store.
11
​
12
# --- Class variables ---
13
TYPE: ClassVar[StackComponentType] = StackComponentType.ARTIFACT_STORE
14
SUPPORTED_SCHEMES: ClassVar[Set[str]]
15
​
16
# --- User interface ---
17
@abstractmethod
18
def open(self, name: PathType, mode: str = "r") -> Any:
19
"""Open a file at the given path."""
20
​
21
@abstractmethod
22
def copyfile(
23
self, src: PathType, dst: PathType, overwrite: bool = False
24
) -> None:
25
"""Copy a file from the source to the destination."""
26
​
27
@abstractmethod
28
def exists(self, path: PathType) -> bool:
29
"""Returns `True` if the given path exists."""
30
​
31
@abstractmethod
32
def glob(self, pattern: PathType) -> List[PathType]:
33
"""Return the paths that match a glob pattern."""
34
​
35
@abstractmethod
36
def isdir(self, path: PathType) -> bool:
37
"""Returns whether the given path points to a directory."""
38
​
39
@abstractmethod
40
def listdir(self, path: PathType) -> List[PathType]:
41
"""Returns a list of files under a given directory in the filesystem."""
42
​
43
@abstractmethod
44
def makedirs(self, path: PathType) -> None:
45
"""Make a directory at the given path, recursively creating parents."""
46
​
47
@abstractmethod
48
def mkdir(self, path: PathType) -> None:
49
"""Make a directory at the given path; parent directory must exist."""
50
​
51
@abstractmethod
52
def remove(self, path: PathType) -> None:
53
"""Remove the file at the given path. Dangerous operation."""
54
​
55
@abstractmethod
56
def rename(
57
self, src: PathType, dst: PathType, overwrite: bool = False
58
) -> None:
59
"""Rename source file to destination file."""
60
​
61
@abstractmethod
62
def rmtree(self, path: PathType) -> None:
63
"""Deletes dir recursively. Dangerous operation."""
64
​
65
@abstractmethod
66
def stat(self, path: PathType) -> Any:
67
"""Return the stat descriptor for a given file path."""
68
​
69
@abstractmethod
70
def walk(
71
self,
72
top: PathType,
73
topdown: bool = True,
74
onerror: Optional[Callable[..., None]] = None,
75
) -> Iterable[Tuple[PathType, List[PathType], List[PathType]]]:
76
"""Return an iterator that walks the contents of the given directory."""
Copied!
This is a slimmed-down version of the base implementation which aims to highlight the abstraction layer. In order to see the full implementation and get the complete docstrings, please check the API docs.
The effect on the
zenml.io.fileioIf you created an instance of an artifact store, added it to your stack and activated the stack, it will create a filesystem each time you run a ZenML pipeline and make it available to the
zenml.io.fileio module.This means that when you utilize a method such as
fileio.open(...) with a filepath which starts with one of the SUPPORTED_SCHEMES within your steps or materializers, it will be able to use the open(...) method that you defined within your artifact store.List of available artifact stores
Out of the box, ZenML comes with a
LocalArtifactStore implementation, which is a simple implementation for a local setup.Moreover, additional artifact stores can be found in specific
integration modules, such as the GCPArtifactStore in the gcp integration and the AzureArtifactStore in the azure integration.Text
Flavor
Integration
If you would like to see the available flavors for artifact stores, you can use the command:
1
zenml artifact-store flavor list
Copied!
Build your own custom artifact store
If you want to create your own custom flavor for an artifact store, you can follow the following steps:
- 1.Create a class which inherits from the
BaseArtifactStore. - 2.Define the
FLAVORclass variable. - 3.Implement the
abstactmethods based on your desired filesystem.
Once you are done with the implementation, you can register it through the CLI as:
1
zenml artifact-store flavor register <THE-SOURCE-PATH-OF-YOUR-ARTIFACT-STORE>
Copied!
Last modified 10d ago
Export as PDF
Copy link
Edit on GitHub