Environment Variables
Configuring environment variables.
Environment variables can be configured to be available at runtime during step execution. ZenML provides two ways to set environment variables:
Plain text environment variables: Configure key-value pairs directly
Secrets as environment variables: Use ZenML secrets where the secret values become environment variables. Check out this page for more information on secret management in ZenML.
Configuration levels
Environment variables and secrets can be configured at different levels with increasing precedence:
Stack components - Available for all pipelines executed on stacks containing this component
Stack - Available for all pipelines executed on this stack
Pipeline - Available for all steps in this pipeline
Step - Available only for this specific step
Automatic environment variable injection
When executing a pipeline, ZenML automatically scans your local environment for any variables that start with the __ZENML__
prefix and adds them to the pipeline environment. The prefix is removed during this process.
For example, if you set:
export __ZENML__MY_VAR=my_value
It will be available in your steps as follows:
import os
from zenml import step
@step
def my_step():
my_var = os.environ["MY_VAR"] # "my_value"
Configuring environment variables on stack components
Configure environment variables and secrets that will be available for all pipelines executed on stacks containing this component.
# Configure environment variables
zenml orchestrator update <ORCHESTRATOR_NAME> --env <KEY>=<VALUE>
# Remove environment variables (set empty value)
zenml orchestrator update <ORCHESTRATOR_NAME> --env <KEY>=
# Attach secrets (secret values become environment variables)
zenml orchestrator update <ORCHESTRATOR_NAME> --secret <SECRET_NAME_OR_ID>
# Remove secrets
zenml orchestrator update <ORCHESTRATOR_NAME> --remove-secret <SECRET_NAME_OR_ID>
Setting environment variables on stacks
Configure environment variables and secrets for all pipelines executed on this stack.
# Configure environment variables
zenml stack update <STACK_NAME> --env <KEY>=<VALUE>
# Remove environment variables
zenml stack update <STACK_NAME> --env <KEY>=
# Attach secrets
zenml stack update <STACK_NAME> --secret <SECRET_NAME_OR_ID>
# Remove secrets
zenml stack update <STACK_NAME> --remove-secret <SECRET_NAME_OR_ID>
Configuring environment variables on pipelines
Configure environment variables and secrets for all steps of a pipeline. See this page for more details on how to configure pipelines.
from zenml import pipeline
# On the decorator
@pipeline(
environment={
"<KEY>": "<VALUE>",
"<KEY>": "<VALUE>"
},
secrets=["<SECRET_NAME_OR_ID>", "<SECRET_NAME_OR_ID>"]
)
def my_pipeline():
...
# Using the `with_options(...)` method
my_pipeline = my_pipeline.with_options(
environment={
"<KEY>": "<VALUE>",
"<KEY>": "<VALUE>"
},
secrets=["<SECRET_NAME_OR_ID>", "<SECRET_NAME_OR_ID>"]
)
Setting environment variables on steps
Configure environment variables and secrets for individual steps. See this page for more details on how to configure steps.
from zenml import step
# On the decorator
@step(
environment={
"<KEY>": "<VALUE>",
"<KEY>": "<VALUE>"
},
secrets=["<SECRET_NAME_OR_ID>"]
)
def my_step() -> str:
...
# Using the `with_options(...)` method
my_step = my_step.with_options(
environment={
"<KEY>": "<VALUE>",
"<KEY>": "<VALUE>"
},
secrets=["<SECRET_NAME_OR_ID>", "<SECRET_NAME_OR_ID>"]
)
When environment variables are set
The timing of when environment variables are set depends on the orchestrator being used:
The Databricks and Lightning orchestrators will set the environment variables right before your step code is being executed
All other orchestrators set environment variables already at container startup time
Last updated
Was this helpful?