Pipeline Configuration

Setting step parameters using a config file

In addition to setting parameters for your pipeline steps in code, ZenML also allows you to use a configuration yaml file. This configuration file must follow the following structure:
1
steps:
2
step_name:
3
parameters:
4
parameter_name: parameter_value
5
some_other_parameter_name: 2
6
some_other_step_name:
7
...
Copied!
Use the configuration file by calling the pipeline method with_config(...):
1
@pipeline
2
def my_pipeline(...):
3
...
4
5
pipeline_instance = my_pipeline(...).with_config("path_to_config.yaml")
6
pipeline_instance.run()
Copied!

Naming a pipeline run

When running a pipeline by calling my_pipeline.run(), ZenML uses the current date and time as the name for the pipeline run. In order to change the name for a run, simply pass it as a parameter to the run() function:
1
my_pipeline.run(run_name="custom_pipeline_run_name")
Copied!
Pipeline run names must be unique, so make sure to compute it dynamically if you plan to run your pipeline multiple times.
Once the pipeline run is finished we can easily access this specific run during our post-execution workflow:
1
from zenml.core.repo import Repository
2
3
repo = Repository()
4
pipeline = repo.get_pipeline(pipeline_name="my_pipeline")
5
run = pipeline.get_run(run_name="custom_pipeline_run_name")
Copied!
Last modified 30d ago