Secrets Management
How to register and use secrets
What is a ZenML secret
Centralized secrets store
ZenML provides a centralized secrets management system that allows you to register and manage secrets in a secure way. When you are using a local ZenML deployment, the secrets are stored in the local SQLite database. If you are connected to a local or remote ZenML server, the secrets are stored in one of the centralized secrets management back-ends that the server is connected to, but all access to the secrets is done through the ZenML server API.
AWS Secrets Manager
GCP Secret Manager
Azure Key Vault
HashiCorp Vault
How to create a secret with the CLI
To create a secret with name <SECRET_NAME>
and a key-value pair, you can run the following CLI command:
Alternatively, you can start an interactive creation (in which ZenML will query you for the secret keys and values) by passing the --interactive/-i
parameter:
For secret values that are too big to pass as a command line argument, or have special characters, you can also use the special @
syntax to indicate to ZenML that the value needs to be read from a file:
How to create a secret with the ZenML Client API
The ZenML client API offers a programmatic interface to create, e.g.:
Secrets scoping
ZenML secrets can be scoped to a workspace or a user. This allows you to create secrets that are only accessible to a specific workspace or user.
By default, all created secrets are scoped to the active workspace. To create a secret and scope it to your active user instead, you can pass the --scope
argument to the CLI command:
Scopes also act as individual namespaces. When you are referencing a secret by name in your pipelines and stacks, ZenML will first look for a secret with that name scoped to the active user, and if it doesn't find one, it will look for one in the active workspace.
Secrets management with Secrets Managers
Managing secrets through a secrets manager stack component suffers from a number of limitations, some of which are:
even with a secrets manager configured in your active stack, if you are using a secrets manager flavor with a cloud back-end (e.g. AWS, GCP or Azure), you still need to configure all your ZenML clients with the authentication credentials required to access the back-end directly. This is not only an inconvenience, it is also a security risk, because it basically represents a large attack surface. With centralized secrets management, you only need to configure the ZenML server to access the cloud back-end.
How to register a secret
To register a secret with name <SECRET_NAME>
and a key-value pair, you can then run the following CLI command:
Alternatively, you can start an interactive registration (in which ZenML will query you for the secret keys and values) by passing the --interactive/-i
parameter:
For secret values that are too big to pass as a command line argument, or have special characters, you can also use the special @
syntax to indicate to ZenML that the value needs to be read from a file:
Interactively register missing secrets for your stack
How to use registered secrets
Reference secrets in stack component attributes and settings
Some of the components in your stack require you to configure them with sensitive information like passwords or tokens so they can connect to the underlying infrastructure. Secret references allow you to configure these components in a secure way by not specifying the value directly but instead referencing a secret by providing the secret name and key. Referencing a secret for the value of any string attribute of your stack components, simply specify the attribute using the following syntax: {{<SECRET_NAME>.<SECRET_KEY>}}
For example:
When using secret references in your stack, ZenML will validate that all secrets and keys referenced in your stack components exist before running a pipeline. This helps us fail early so your pipeline doesn't fail after running for some time due to some missing secret.
This validation by default needs to fetch and read every secret to make sure that both the secret and the specified key-value pair exist. This can take quite some time and might fail if you don't have the permissions to read secrets.
You can use the environment variable ZENML_SECRET_VALIDATION_LEVEL
to disable or control the degree to which ZenML validates your secrets:
Setting it to
NONE
disables any validation.Setting it to
SECRET_EXISTS
only validates the existence of secrets. This might be useful if the machine you're running on only has permissions to list secrets but not actually read their values.Setting it to
SECRET_AND_KEY_EXISTS
(the default) validates both the secret existence as well as the existence of the exact key-value pair.
Fetch secret values in a step
If you are using a Secrets Manager to manage secrets, you can access the secrets manager directly from within your steps through the StepContext
. This allows you to use your secrets for querying APIs from within your step without hard-coding your access keys. Don't forget to make the appropriate decision regarding caching as it will be disabled by default when the StepContext
is passed into the step.
This will only work if the environment that your orchestrator uses to execute steps has access to the secrets manager. For example a local secrets manager will not work in combination with a remote orchestrator.
Last updated