Fetch metadata within steps

Accessing meta information in real-time within your pipeline.

Using the StepContext

To find information about the pipeline or step that is currently running, you can use the zenml.get_step_context() function to access the StepContext of your step:

from zenml import step, pipeline, get_step_context

@step
def my_step():
    step_context = get_step_context()
    pipeline_name = step_context.pipeline.name
    run_name = step_context.pipeline_run.name
    step_name = step_context.step_run.name

Furthermore, you can also use the StepContext to find out where the outputs of your current step will be stored and which Materializer class will be used to save them:

@step
def my_step():
    step_context = get_step_context()
    # Get the URI where the output will be saved.
    uri = step_context.get_output_artifact_uri()

    # Get the materializer that will be used to save the output.
    materializer = step_context.get_output_materializer() 

See the API Docs for more information on which attributes and methods the StepContext provides.

Getting artifact and model metadata

As described in the metadata guide, the metadata can be fetched with the client, and this is how you would use it to fetch it within a step.

from zenml.client import Client
from zenml import step, pipeline

@step
def my_step():
    client = Client()
    # Directly fetch an artifact
    output = client.get_artifact_version("my_dataset", "my_version")
    output.run_metadata["accuracy"].value

You can also use the active model to get the model metadata, or the associated artifacts directly as described in the starter guide:

from zenml import step, pipeline, get_step_context

@step
def my_step():
    # Get active step context
    mv = get_step_context().model

    # Get metadata
    print(mv.run_metadata["metadata_key"].value)

    # Directly fetch an artifact
    output = mv.get_artifact("my_dataset", "my_version")
    output.run_metadata["accuracy"].value

Fetching secret values in a step

ZenML secrets are groupings of key-value pairs which are securely stored in the ZenML secrets store. Additionally, a secret always has a name that allows you to fetch or reference them in your pipelines and stacks. In order to learn more about how to configure and create secrets, please refer to the platform guide on secrets.

You can access secrets directly from within your steps through the ZenML Client API. This allows you to use your secrets for querying APIs from within your step without hard-coding your access keys:

from zenml import step
from zenml.client import Client


@step
def secret_loader() -> None:
    """Load the example secret from the server."""
    # Fetch the secret from ZenML.
    secret = Client().get_secret(<SECRET_NAME>)

    # `secret.secret_values` will contain a dictionary with all key-value
    # pairs within your secret.
    authenticate_to_some_api(
        username=secret.secret_values["username"],
        password=secret.secret_values["password"],
    )
    ...