Develop a Custom Secrets Manager
How to develop a custom secrets manager
This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.
Base Abstraction
The secret manager acts as the one-stop shop for all the secrets to which your pipeline or stack components might need access.
As it is the base class for a specific type of
StackComponent
, it inherits from the StackComponent class. This sets theTYPE
variable to the secrets manager . TheFLAVOR
class variable needs to be set in the specific subclass.The
BaseSecretsManager
implements the interface for a set of CRUD operations asabstractmethod
s:register_secret
,get_secret
,get_all_secret_keys
,update_secret
,delete_secret
,delete_all_secrets
. In the implementation of everySecretsManager
flavor, it is required to define these methods with respect to the flavor at hand.
This is a slimmed-down version of the base implementation which aims to highlight the abstraction layer. In order to see the full implementation and get the complete docstrings, please check the API docs.
Build your own custom secrets manager
If you want to create your own custom flavor for a secrets manager, you can follow the following steps:
Create a class which inherits from the
BaseSecretsManager
.Define the
FLAVOR
class variable.Implement the
abstactmethod
s based on the API of your desired secrets manager
Once you are done with the implementation, you can register it through the CLI as:
Some additional implementation details
Different providers in the space of secrets manager have different definitions of what constitutes a secret. While some providers consider a single key-value pair a secret: ('secret_name': 'secret_value'
), other providers have a slightly different definition. For them, a secret is a collection of key-value pairs: {'some_username': 'user_name_1', 'some_pwd': '1234'}
.
ZenML falls into the second category. The implementation of the different methods should reflect this convention. In case the specific implementation interfaces with a secrets manager that uses the other definition of a secret, working with tags can be helpful. See the GCPSecretsManager
for inspiration.
SecretSchemas
One way that ZenML expands on the notion of secrets as dictionaries is the secret schema. A secret schema allows the user to create and use a specific template. A schema could, for example, require the combination of a username, password and token. All schemas must sub-class from the BaseSecretSchema
.
All Secret Schemas will need to have a defined
TYPE
.The required and optional keys of the secret need to be defined as class variables.
One such schema could look like this.
Last updated