Secrets Management
How to reference secrets when configuring stack components
This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.
Some of the components in your stack require you to configure them with sensitive information like passwords or tokens so they can connect to the underlying infrastructure. Secret references allow you to configure these components in a secure way by not specifying the value directly but instead referencing a secret. To reference a secret in any string attribute of your stack components, simply specify the attribute value using the following syntax:
For example:
Register missing secrets for your stack
When using components with secret references in your stack, you need to make sure that the stack contains a secrets manager and all the referenced secrets exist in this secrets manager. To make this process easier, you can use the following CLI command to interactively register all secrets for a stack:
How to use it
In the CLI
A full guide on using the CLI interface to register, access, update and delete secrets is available here.
A ZenML secret is a grouping of key-value pairs which are defined by a schema. An AWS SecretSchema, for example, has key-value pairs for AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
as well as an optional AWS_SESSION_TOKEN
. If you don't specify a schema when registering a secret, ZenML will use the ArbitrarySecretSchema
, a schema where arbitrary keys are allowed.
Note that there are two ways you can register or update your secrets. If you wish to do so interactively, passing the secret name in as an argument (as in the following example) will initiate an interactive process:
If you wish to specify key-value pairs using command line arguments, you can do so instead:
For secret values that are too big to pass as a command line argument, or have special characters, you can also use the special @
syntax to indicate to ZenML that the value needs to be read from a file:
In a ZenML Step
You can access the secrets manager directly from within your steps through the StepContext
. This allows you to use your secrets for querying APIs from within your step without hard-coding your access keys. Don't forget to make the appropriate decision regarding caching as it will be disabled by default when the StepContext
is passed into the step.
This will only work if the environment that your orchestrator uses to execute steps has access to the secrets manager. For example a local secrets manager will not work in combination with a remote orchestrator.
To read a more detailed guide about how Secret Managers function in ZenML, click here.
Secret validation
Before running a pipeline, ZenML will validate your stack and make sure that all secrets and keys referenced in your stack components exist. This helps us fail early so your pipeline doesn't fail after running for some time due to some missing secret.
This validation by default needs to fetch and read every secret to make sure that both the secret and the specified key-value pair exist. This can take quite some time and might fail if you don't have the permissions to read secrets.
You can use the environment variable ZENML_SECRET_VALIDATION_LEVEL
to disable or control the degree to which ZenML validates your secrets:
Setting it to
NONE
disables any validation.Setting it to
SECRET_EXISTS
only validates the existence of secrets. This might be useful if the machine you're running on only has permissions to list secrets but not actually read their values.Setting it to
SECRET_AND_KEY_EXISTS
(the default) validates both the secret existence as well as the existence of the exact key-value pair.
Last updated