Secret Managers
Using secrets across your ZenML pipeline.
Most projects involving either cloud infrastructure or of a certain complexity will involve secrets of some kind. ZenML provides you a Secrets Manager as stack component to help manage and use these secrets for your pipeline runs.
Before reading this chapter, make sure that you are familiar with the concept of stacks, stack components and their flavors.

Base Abstraction

The secret manager acts as the one-stop shop for all the secrets to which your pipeline or stack components might need access.
  1. 1.
    As it is the base class for a specific type of StackComponent, it inherits from the StackComponent class. This sets the TYPE variable to the secrets manager . The FLAVOR class variable needs to be set in the specific subclass.
  2. 2.
    The BaseSecretsManager implements the interface for a set of CRUD operations as abstractmethods: register_secret, get_secret, get_all_secret_keys, update_secret, delete_secret, delete_all_secrets. In the implementation of every SecretsManager flavor, it is required to define these methods with respect to the flavor at hand.
1
from abc import ABC, abstractmethod
2
from typing import ClassVar, List
3
4
from zenml.enums import StackComponentType
5
from zenml.secret.base_secret import BaseSecretSchema
6
from zenml.stack import StackComponent
7
8
class BaseSecretsManager(StackComponent, ABC):
9
"""Base class for all ZenML secrets managers."""
10
11
# Class configuration
12
TYPE: ClassVar[StackComponentType] = StackComponentType.SECRETS_MANAGER
13
FLAVOR: ClassVar[str]
14
15
@abstractmethod
16
def register_secret(self, secret: BaseSecretSchema) -> None:
17
"""Registers a new secret."""
18
19
@abstractmethod
20
def get_secret(self, secret_name: str) -> BaseSecretSchema:
21
"""Gets the value of a secret."""
22
23
@abstractmethod
24
def get_all_secret_keys(self) -> List[str]:
25
"""Get all secret keys."""
26
27
@abstractmethod
28
def update_secret(self, secret: BaseSecretSchema) -> None:
29
"""Update an existing secret."""
30
31
@abstractmethod
32
def delete_secret(self, secret_name: str) -> None:
33
"""Delete an existing secret."""
34
35
@abstractmethod
36
def delete_all_secrets(self, force: bool = False) -> None:
37
"""Delete all existing secrets."""
Copied!
This is a slimmed-down version of the base implementation which aims to highlight the abstraction layer. In order to see the full implementation and get the complete docstrings, please check the API docs.

List of available secrets managers

Out-of-the-box, ZenML comes with a LocalSecretsManager implementation, which is a simple implementation for a local setup. This secrets manager simply saves secrets into a local yaml file with base64 encoding. This implementation is not intended for production use.
For production use cases some more flavors can be found in specific integrations modules, such as the GCPSecretsManager in the gcp integration, the AWSSecretsManager in the aws integration and the AzureSecretsManager in the azure integration.
Text
Flavor
Integration
local
built-in
aws
aws
gcp
gcp
azure
azure
github
github
If you would like to see the available flavors for secret managers, you can use the command:
1
zenml secrets-manager flavor list
Copied!

Build your own custom secrets manager

If you want to create your own custom flavor for a secrets manager, you can follow the following steps:
  1. 1.
    Create a class which inherits from the BaseSecretsManager.
  2. 2.
    Define the FLAVOR class variable.
  3. 3.
    Implement the abstactmethods based on the API of your desired secrets manager
Once you are done with the implementation, you can register it through the CLI as:
1
zenml secrets-manager flavor register <THE-SOURCE-PATH-OF-YOUR-SECRETS-MANAGER>
Copied!

Some additional implementation details

Different providers in the space of secrets manager have different definitions of what constitutes a secret. While some providers consider a single key-value pair a secret: ('secret_name': 'secret_value'), other providers have a slightly different definition. For them, a secret is a collection of key-value pairs: {'some_username': 'user_name_1', 'some_pwd': '1234'}.
ZenML falls into the second category. The implementation of the different methods should reflect this convention. In case the specific implementation interfaces with a secrets manager that uses the other definition of a secret, working with tags can be helpful. See the GCPSecretsManager for inspiration.

SecretSchemas

One way the ZenML expands on the notion of secrets as dictionaries is the secret schema. A secret schema allows the user to create and use a specific template. A schema could, for example, require the combination of a username, password and token. All schemas must sub-class from the BaseSecretSchema.
  1. 1.
    All Secret Schemas will need to have a defined TYPE.
  2. 2.
    The required and optional keys of the secret need to be defined as class variables.
1
class BaseSecretSchema(BaseModel, ABC):
2
name: str
3
TYPE: ClassVar[str]
4
5
@property
6
def content(self) -> Dict[str, Any]:
7
...
Copied!
One such schema could look like this.
1
from typing import ClassVar, Optional
2
3
from zenml.secret import register_secret_schema_class
4
from zenml.secret.base_secret import BaseSecretSchema
5
6
EXAMPLE_SECRET_SCHEMA_TYPE = "example"
7
8
@register_secret_schema_class
9
class ExampleSecretSchema(BaseSecretSchema):
10
11
TYPE: ClassVar[str] = EXAMPLE_SECRET_SCHEMA_TYPE
12
13
username: str
14
password: str
15
token: Optional[str]
Copied!