Repo & Config

ZenML has two main locations where it stores information on the machine where it is used. These are the Global Config and the Repository. The latter is also referred to as the .zen folder.
The Global Config
Most of the information stored by ZenML on a machine, such as the global settings, the configured ZenML Profiles and even the configured Stacks and Stack Components, is kept in a folder commonly referred to as the ZenML Global Config Directory or the ZenML Config Path. The location of this folder depends on the operating system type and the current system user, but is usually located in the following locations:
Linux: ~/.config/zenml
Mac: ~/Library/Application Support/ZenML
Windows: C:\Users\%USERNAME%\AppData\Local\ZenML
The default location may be overridden by setting the ZENML_CONFIG_PATH environment variable to a custom value. The current location of the Global Config Directory used on a system can be retrieved by running the following command:
1
python -c 'from zenml.utils.io_utils import get_global_config_directory; print(get_global_config_directory())'
Copied!
Manually altering or deleting the files and folders stored under the ZenML Global Config Directory is not recommended, as this can break the internal consistency of the ZenML configuration. As an alternative, ZenML provides CLI commands that can be used to manage the information stored there:
zenml analytics - manage the analytics settings
zenml profile - manage configuration Profiles
zenml stack - manage Stacks
zenml <stack-component> - manage Stack Components
zenml clean - to be used only in case of emergency, to bring the ZenML configuration back to its default factory state
The first time that ZenML is run on a machine, it creates the Global Config Directory and initializes the default configuration in it, along with a default Profile and Stack:
1
$ zenml stack list
2
Unable to find ZenML repository in your current working directory (/home/stefan)
3
or any parent directories. If you want to use an existing repository which is in
4
a different location, set the environment variable 'ZENML_REPOSITORY_PATH'. If
5
you want to create a new repository, run zenml init.
6
Initializing the ZenML global configuration version to 0.7.3
7
Creating default profile...
8
Initializing profile default...
9
Registering default stack...
10
Registered stack component with type 'orchestrator' and name 'default'.
11
Registered stack component with type 'metadata_store' and name 'default'.
12
Registered stack component with type 'artifact_store' and name 'default'.
13
Registered stack with name 'default'.
14
Created and activated default profile.
15
Running without an active repository root.
16
Running with active profile: 'default' (global)
17
┏━━━━━━━━┯━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
18
┃ ACTIVE │ STACK NAME │ ARTIFACT_STORE │ METADATA_STORE │ ORCHESTRATOR ┃
19
┠────────┼────────────┼────────────────┼────────────────┼──────────────┨
20
┃   👉   │ default    │ default        │ default        │ default      ┃
21
┗━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
Copied!
The following is an example of the layout of the Global Config Directory immediately after initialization:
1
/home/stefan/.config/zenml   <- Global Config Directory
2
├── config.yaml              <- Global Configuration Settings
3
├── local_stores             <- Every Stack component that stores information
4
|   |                        locally will have its own subdirectory here.
5
|   |                        
6
│   └── 09fcdb1c-4079-4d20-afdb-957965405863   <- Local Store path for the `default`
7
|                                              local Artifact Store
8
|
9
└── profiles                 <- root path where Profiles data (stacks, components,
10
    |                        etc) are stored by default. Every Profile will have
11
    |                        its own subdirectory here, unless the Profile is
12
    |                        configured with a custom configuration path.
13
    |
14
    └── default              <- configuration folder for the `default` Profile.
15
        ├── artifact_stores
16
        │   └── default.yaml
17
        ├── metadata_stores
18
        │   └── default.yaml
19
        ├── orchestrators
20
        │   └── default.yaml
21
        └── stacks.yaml
Copied!
As shown above, the Global Config Directory stores the following information:
1.The global.yaml file stores the global configuration settings: the unique ZenML user ID, the active Profile, the analytics related options and a list of all configured Profiles, along with their configuration attributes, such as the active Stack set for each Profile. This is an example of the global.yaml file contents immediately after initialization:
1
activated_profile: default
2
analytics_opt_in: true
3
profiles:
4
default:
5
    active_stack: default
6
    active_user: default
7
    name: default
8
    store_type: local
9
    store_url: file:///home/stefan/.config/zenml/profiles/default
10
user_id: 4b773740-fddc-46ee-938e-1c78a075cfc7
11
user_metadata: null
12
version: 0.7.3
Copied!
2.The local_stores directory is where some "local" flavors of Stack Components, such as the local Artifact Store, the sqlite Metadata Store or the local Secrets Manager persist data locally. Every local Stack Component will have its own subdirectory here named after the Stack Component's unique UUID. One notable example is the local Artifact Store flavor that, when part of the active Stack, stores all the artifacts generated by Pipeline runs in the designated local directory.
3.The profiles directory is used as a default root path location where ZenML stores information about the Stacks, Stack Components, custom Stack Component flavors etc. that are configured under each Profile. Every Profile will have its own subdirectory here, unless the Profile is explicitly created with a custom configuration path. (See the zenml profile command and the section on ZenML Profiles for more information about Profiles.)
In addition to the above, you may also find the following files and folders under the Global Config Directory, depending on what you do with ZenML:
zenml_examples - used as a local cache by the zenml example command, where the pulled ZenML examples are stored.
kubeflow - this is where the Kubeflow orchestrators that are part of a Stack store some of their configuration and logs.
The ZenML Repository
A ZenML Repository designates a folder that contains a complete set of Python scripts and modules as well as any associated files that are needed to run ZenML pipelines. Any such folder can be registered as a ZenML Repository root by running the zenml init command from that folder or passing its location as an argument to that same command, e.g.:
1
[email protected]:/tmp/zenml$ zenml init
2
ZenML repository initialized at /tmp/zenml.
3
The local active profile was initialized to 'default' and the local active stack to 'default'. This local configuration will only take effect when you're running ZenML from the initialized repository root, or from a subdirectory. For more information on profile and stack configuration, please visit https://docs.zenml.io.
Copied!
The Repository designation plays a double role in ZenML:
it is used by ZenML to identify which files it must copy into the environments it builds that are used to execute pipeline steps remotely, such as container images.
it stores local configuration parameters that can be set independently of the global configuration: the Repository active Profile and active Stack. These settings are in effect when ZenML code is executed while the current working directory is the Repository root or one of its sub-folders. For more information on setting the active Profile and Stack local to a Repository, please visit the ZenML Profiles section.
A ZenML Repository is easily identifiable by the presence of a (hidden) .zen directory located in the root folder. The .zen directory contains a single config.yaml file that stores the local settings:
1
active_profile_name: default
2
active_stack_name: default
Copied!
It is recommended to use the zenml init command to initialize a ZenML Repository in the same location of your custom Python source tree where you would normally point PYTHONPATH, especially if your Python code relies on a hierarchy of modules spread out across multiple sub-folders.
ZenML CLI commands and ZenML code will display a warning if they are not running in the context of a ZenML Repository, e.g.:
1
[email protected]:/tmp$ zenml stack list
2
Unable to find ZenML repository in your current working directory (/tmp) or any parent directories. If you want to use an existing repository which is in a different location, set the environment variable 'ZENML_REPOSITORY_PATH'. If you want to create a new repository, run zenml init.
3
Running without an active repository root.
4
Running with active profile: 'default' (global)
5
┏━━━━━━━━┯━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
6
┃ ACTIVE │ STACK NAME │ ARTIFACT_STORE │ METADATA_STORE │ ORCHESTRATOR ┃
7
┠────────┼────────────┼────────────────┼────────────────┼──────────────┨
8
┃   👉   │ default    │ default        │ default        │ default      ┃
9
┗━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
Copied!
To remove the Repository designation from a folder, simply delete the .zen subdirectory in that folder.
The GlobalConfiguration and Repository Singletons
ZenML provides two singleton objects that can be used to access and manage the information stored in the Global Config Directory and the current Repository.
To access the global configuration settings and manage Profiles, you can use the zenml.config.global_config.GlobalConfiguration singleton, while the zenml.repository.Repository singleton acts as the central point of management for Stacks, Stack Components, Stack Component flavors and other associated ZenML concepts. The following are examples of how to use these singletons to perform various operations:
to retrieve the global configuration, active Profile and active Stack:
1
from zenml.config.global_config import GlobalConfiguration
2
from zenml.repository import Repository
3
​
4
repo = Repository()
5
config = GlobalConfiguration()
6
​
7
# the unique user UUID set
8
user_id = config.user_id
9
​
10
# the active profile name
11
profile = repo.active_profile_name
12
​
13
# the active profile object
14
profile = repo.active_profile
15
​
16
# the name of the active stack
17
active_stack_name = config.active_stack_name
18
​
19
# the active stack object
20
active_stack = repo.active_stack
Copied!
to create a new Profile and set it as the active one:
1
from zenml.repository import Repository
2
from zenml.config.global_config import GlobalConfiguration
3
from zenml.config.profile_config import ProfileConfiguration
4
​
5
repo = Repository()
6
config = GlobalConfiguration()
7
​
8
profile = ProfileConfiguration(name="local")
9
config.add_or_update_profile(profile)
10
​
11
repo.activate_profile("local")
Copied!
access the active Stack
1
from zenml.repository import Repository
2
​
3
repo = Repository()
4
stack = repo.active_stack
5
​
6
print(stack.name)
7
print(stack.orchestrator.name)
8
print(stack.artifact_store.name)
9
print(stack.artifact_store.path)
10
print(stack.metadata_store.name)
11
print(stack.metadata_store.uri)
Copied!
register and set a new Stack
1
from zenml.repository import Repository
2
from zenml.artifact_stores import LocalArtifactStore
3
from zenml.metadata_stores import SQLiteMetadataStore
4
from zenml.orchestrators import LocalOrchestrator
5
from zenml.stack import Stack
6
​
7
repo = Repository()
8
​
9
orchestrator = LocalOrchestrator(name="local")
10
metadata_store = SQLiteMetadataStore(
11
    name="local",
12
    uri="/tmp/zenml/zenml.db",
13
)
14
artifact_store = LocalArtifactStore(
15
    name="local",
16
    path="/tmp/zenml/artifacts",
17
)
18
stack = Stack(
19
    name="local",
20
    orchestrator=orchestrator,
21
    metadata_store=metadata_store,
22
    artifact_store=artifact_store,
23
)
24
​
25
repo.register_stack(stack)
26
repo.activate_stack(stack.name)
Copied!
To explore all possible operations that can be performed via the GlobalConfiguration and Repository singletons, please consult the API docs sections on GlobalConfiguration and Repository.
Developer Guide - Previous
Class Based API
Next - Advanced Guide
Stacks, Components, Flavors
Last modified 10d ago
Export as PDF
Copy link
Edit on GitHub
Contents
The Global Config
The ZenML Repository
The GlobalConfiguration and Repository Singletons