0.9.0
Developer Guide
Advanced Guide
Extending ZenML
Links & Resources
Reference
Manage your Secrets
ZenML provides functionality to store secrets locally and with AWS.
Most projects involving either cloud infrastructure or of a certain complexity will involve secrets of some kind. You use secrets, for example, when connecting to AWS, which requires an
access_key_id and a secret_access_key which is usually stored in your ~/.aws/credentials file.You might find you need to access those secrets from within your Kubernetes cluster as it runs individual steps, or you might just want a centralized location for the storage of secrets across your project. ZenML offers a basic local secrets manager and an integration with the managed AWS Secrets Manager.
A ZenML Secret is a grouping of key-value pairs. These are accessed and administered via the ZenML Secret Manager (a stack component).
Secrets are distinguished by having different schemas. An AWS SecretSchema, for example, has key-value pairs for
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as well as an optional AWS_SESSION_TOKEN. If you don't specify a schema at the point of registration, ZenML will set the schema as ArbitrarySecretSchema, a kind of default schema where things that aren't attached to a grouping can be stored.Registering a secrets manager
For early development purposes ZenML provides a local secrets manager which uses a YAML file to store base64 encoded secret. If you want to instead use the AWS or GCP Secrets Manager as a non-local flavor that is also possible with ZenML.
To register a local secrets manager, use the CLI interface:
1
zenml secrets-manager register SECRETS_MANAGER_NAME --flavor=local
Copied!
You will then need to add the secrets manager to a new stack that you register, for example:
1
zenml stack register STACK_NAME \
2
-m METADATA_STORE_NAME \
3
-a ARTIFACT_STORE_NAME \
4
-o ORCHESTRATOR_NAME \
5
-x SECRETS_MANAGER_NAME \
6
--set
Copied!
Interacting with the Secrets Manager
In the CLI
A full guide on using the CLI interface to register, access, update and delete secrets is available here.
Note that there are two ways you can register or update your secrets. If you wish to do so interactively, simply passing the secret name in as an argument (as in the following example) will initiate an interactive process:
1
zenml secret register SECRET_NAME -i
Copied!
If you wish to specify key-value pairs using command line arguments, you can do so instead:
1
zenml secret register SECRET_NAME --key1=value1 --key2=value2
Copied!
For secret values that are too big to pass as a command line argument, or have special characters, you can also use the special
@ syntax to indicate to ZenML that the value needs to be read from a file:1
zenml secret register SECRET_NAME --attr_from_literal=value \
2
--attr_from_file=@path/to/file.txt ...
Copied!
In a ZenML Step
You can access the secrets manager directly from within your steps through the
StepContext. This allows you to use your secrets for querying APIs from within your step without hard-coding your access keys. Don't forget to make the appropriate decision regarding caching as it will be disabled by default when the StepContext is passed into the step.1
from zenml.steps import step, StepContext
2
​
3
​
4
@step(enable_cache=True)
5
def secret_loader(
6
context: StepContext,
7
) -> None:
8
"""Load the example secret from the secret manager."""
9
# Load Secret from active secret manager. This will fail if no secret
10
# manager is active or if that secret does not exist
11
retrieved_secret = context.stack.secrets_manager.get_secret(<SECRET_NAME>)
12
​
13
# retrieved_secret.content will contain a dictionary with all Key-Value
14
# pairs within your secret.
15
return
Copied!
This will only work if your orchestrator has access to the secret manager. For example a
local_secrets_manager will not work in combination with a remote orchestrator like kubeflow pipelines.Secret Schemas
The concept of secret schemas exists to support strongly typed secrets that validate which keys can be configured for a given secret and which values are allowed for those keys.
Secret schemas are available as builtin schemas, or loaded when an integration is installed. Custom schemas can also be defined by sub-classing the
zenml.secret.BaseSecretSchema class. For example, the following is the builtin schema defined for the MySQL Metadata Store secrets:1
from typing import ClassVar, Optional
2
​
3
from zenml.secret.base_secret import BaseSecretSchema
4
​
5
MYSQL_METADATA_STORE_SCHEMA_TYPE = "mysql"
6
​
7
class MYSQLSecretSchema(BaseSecretSchema):
8
TYPE: ClassVar[str] = MYSQL_METADATA_STORE_SCHEMA_TYPE
9
​
10
user: Optional[str]
11
password: Optional[str]
12
ssl_ca: Optional[str]
13
ssl_cert: Optional[str]
14
ssl_key: Optional[str]
15
ssl_verify_server_cert: Optional[bool] = False
Copied!
To register a secret regulated by a schema, the
--schema argument must be passed to the zenml secret register command:1
zenml secret register mysql_secret --schema=mysql --user=user --password=password
2
--ssl_ca=@./ca.pem --ssl_verify_server_cert=true
Copied!
The keys and values passed to the CLI are validated using regular Pydantic rules:
- optional attributes don't need to be passed to the CLI and will be set to their default value if omitted
- required attributes must be passed to the CLI or an error will be raised
- all values must be a valid string representation of the data type indicated in the schema (i.e. that can be converted to the type indicated) or an error will be raised
Using Secrets in a Kubeflow environment
ZenML will handle passing secrets down through the various stages of a Kubeflow pipeline, so your secrets will be accessible wherever your code is running.
Note: The Secrets Manager as currently implemented does not work with our Airflow orchestrator integration. Let us know if you would like us to prioritize adding this in!
To pass a particular secret as part of the environment available to a pipeline, include a list of your secret names as an extra argument when you are defining your pipeline, as in the following example (taken from the corresponding Kubeflow example):
1
from zenml.pipelines import pipeline
2
from zenml.integrations.constants import TENSORFLOW
3
​
4
@pipeline(required_integrations=[TENSORFLOW], secrets=["aws"], enable_cache=True)
5
def mnist_pipeline(
6
importer,
7
normalizer,
8
trainer,
9
evaluator,
10
):
11
# Link all the steps together
12
X_train, X_test, y_train, y_test = importer()
13
X_trained_normed, X_test_normed = normalizer(X_train=X_train, X_test=X_test)
14
model = trainer(X_train=X_trained_normed, y_train=y_train)
15
evaluator(X_test=X_test_normed, y_test=y_test, model=model)
Copied!
Secrets are made available to steps regardless of whether you're using a local secret store or non-local AWS/GCP Secrets Manager.
Using the AWS Secrets Manager
Amazon offers a managed secrets manager to store and use secrets for AWS services. If your stack is primarily running on AWS, you can use our integration to interact with it. Before getting started with the AWS secret manager you'll need to make sure to have your AWS credentials set up locally and you have access to a service account with read/write permissions to the secrets manager. This guide can help you get started.
With this set up, using the AWS Secrets Manager is just as easy as using the local version. Make sure that the integration is installed first, and then register your secrets manager in the following way:
1
zenml integration install aws
2
zenml secrets-manager register AWS_SECRETS_MANAGER_NAME --flavor=aws
Copied!
If you are using the ZenML Kubeflow integration for your orchestrator, you can then access the keys and their corresponding values for all the secrets you imported in the pipeline definition (as mentioned above). The keys that you used when creating the secret will be capitalized when they are passed down into the images used by Kubeflow. For example, in the case of accessing the
aws secret referenced above, you would get the value for the aws_secret_access_key key with the following code (within a step):1
import os
2
​
3
os.environ.get('AWS_SECRET_ACCESS_KEY')
Copied!
Note that some secrets will get used by your stack implicitly. For example, in the case of when you are using an AWS S3 artifact store, the environment variables passed down will be used to confirm access.
Using the GCP Secrets Manager
Google offers a managed secret manager to store and use secrets for GCP services. If your stack is primarily running on GCP, you can use our integration to interact with it.
Before getting started with the GCP secret manager you'll need to make sure to have your GCP credentials set up locally, and you have access to a service account with read/write permissions to the secrets manager. This guide can help you get started.
With this set up, using the GCP Secret Manager is just as easy as using the local version. Make sure that the integration is installed first, and then register your secrets manager in the following way:
1
zenml integration install gcp_secrets_manager
2
zenml secrets-manager register GCP_SECRETS_MANAGER_NAME -t gcp \
3
--project_id=<ID_OF_YOUR_PROJECT>
Copied!
The Project ID refers to the GCP project of your secrets manager. This is how you can find the project ID of your project.
If you are using the ZenML Kubeflow integration for your orchestrator, you can then access the keys and their corresponding values for all the secrets you imported in the pipeline definition (as mentioned above). The keys that you used when creating the secret will be capitalized when they are passed down into the images used by Kubeflow. For example, in the case of accessing the
aws secret referenced above, you would get the value for the aws_secret_access_key key with the following code (within a step):1
import os
2
​
3
os.environ.get('AWS_SECRET_ACCESS_KEY')
Copied!
Note that some secrets will get used by your stack implicitly. For example, in the case of when you are using an GCP artifact store, the environment variables passed down will be used to confirm access.
Using the Azure Secrets Manager
Azure offers a managed secrets manager (known as 'Azure Key Vault') to store and use secrets for Azure services. If your stack is primarily running on Microsoft's Azure cloud platform, you can use our integration to interact with it.
Before getting started with the Azure secret manager you'll need to make sure to have your Azure credentials set up locally, and you have access to a service account with read/write permissions to the secrets manager. This guide can help you get started. You'll want to create a new key vault as described in the link, or use whatever vault you already have created.
With this set up, using the Azure Key Vault is just as easy as using the local version. Make sure that the integration is installed first, and then register your secrets manager in the following way:
1
zenml integration install azure
2
zenml secrets-manager register AZURE_SECRETS_MANAGER_NAME -f azure \
3
--key_vault_name=<YOUR_KEY_VAULT_NAME>
4
​
5
# update your default stack, for example
6
zenml stack update default -x AZURE_SECRETS_MANAGER_NAME
Copied!
The Key Vault name refers to the name of your Key Vault as created within Azure.
If you are using the ZenML Kubeflow integration for your orchestrator, you can then access the keys and their corresponding values for all the secrets you imported in the pipeline definition (as mentioned above). The keys that you used when creating the secret will be capitalized when they are passed down into the images used by Kubeflow. For example, in the case of accessing the
aws secret referenced above, you would get the value for the aws_secret_access_key key with the following code (within a step):1
import os
2
​
3
os.environ.get('AWS_SECRET_ACCESS_KEY')
Copied!
Note that some secrets will get used by your stack implicitly. For example, in the case of when you are using an AWS S3 artifact store, the environment variables passed down will be used to confirm access.