Fetching pipelines
Inspecting a finished pipeline run and its outputs.
Last updated
Inspecting a finished pipeline run and its outputs.
Last updated
Once a pipeline run has been completed, we can access the corresponding information in code, which enables the following:
Loading artifacts like models or datasets saved by previous runs
Accessing metadata or configurations of previous runs
Programmatically inspecting the lineage of pipeline runs and their artifacts
The hierarchy of pipelines, runs, steps, and artifacts is as follows:
As you can see from the diagram, there are many layers of 1-to-N relationships.
Let us investigate how to traverse this hierarchy level by level:
After you have run a pipeline at least once, you can also fetch the pipeline via the Client.get_pipeline()
method.
Check out the ZenML Client Documentation for more information on the Client
class and its purpose.
If you're not sure which pipeline you need to fetch, you can find a list of all registered pipelines in the ZenML dashboard, or list them programmatically either via the Client or the CLI.
You can use the Client.list_pipelines()
method to get a list of all pipelines registered in ZenML:
Each pipeline can be executed many times, resulting in several Runs.
You can get a list of all runs of a pipeline using the runs
property of the pipeline:
The result will be a list of the most recent runs of this pipeline, ordered from newest to oldest.
Alternatively, you can also use the pipeline_model.get_runs()
method which allows you to specify detailed parameters for filtering or pagination. See the ZenML SDK Docs for more information.
To access the most recent run of a pipeline, you can either use the last_run
property or access it through the runs
list:
If your most recent runs have failed, and you want to find the last run that has succeeded, you can use the last_successful_run
property instead.
Calling a pipeline executes it and then returns the response of the freshly executed run.
The run that you get back is the model stored in the ZenML database at the point of the method call. This means the pipeline run is still initializing and no steps have been run. To get the latest state can get a refreshed version from the client:
If you already know the exact run that you want to fetch (e.g., from looking at the dashboard), you can use the Client.get_pipeline_run()
method to fetch the run directly without having to query the pipeline first:
Similar to pipelines, you can query runs by either ID, name, or name prefix, and you can also discover runs through the Client or CLI via the Client.list_pipeline_runs()
or zenml pipeline runs list
commands.
Each run has a collection of useful information which can help you reproduce your runs. In the following, you can find a list of some of the most useful pipeline run information, but there is much more available. See the PipelineRunResponse
definition for a comprehensive list.
The status of a pipeline run. There are five possible states: initialized, failed, completed, running, and cached.
The pipeline_configuration
is an object that contains all configurations of the pipeline and pipeline run, including the pipeline-level settings, which we will learn more about later:
Depending on the stack components you use, you might have additional component-specific metadata associated with your run, such as the URL to the UI of a remote orchestrator. You can access this component-specific metadata via the run_metadata
attribute:
If you're only calling each step once inside your pipeline, the invocation ID will be the same as the name of your step. For more complex pipelines, check out this page to learn more about the invocation ID.
If you are using our VS Code extension, you can easily view your pipeline runs by opening the sidebar (click on the ZenML icon). You can then click on any particular pipeline run to see its status and some other metadata. If you want to delete a run, you can also do so from the same sidebar view.
Similar to the run, you can use the step
object to access a variety of useful information:
The parameters used to run the step via step.config.parameters
,
The step-level settings via step.config.settings
,
Component-specific step metadata, such as the URL of an experiment tracker or model deployer, via step.run_metadata
See the StepRunResponse
definition for a comprehensive list of available information.
Each step of a pipeline run can have multiple output and input artifacts that we can inspect via the outputs
and inputs
properties.
To inspect the output artifacts of a step, you can use the outputs
attribute, which is a dictionary that can be indexed using the name of an output. Alternatively, if your step only has a single output, you can use the output
property as a shortcut directly:
Similarly, you can use the inputs
and input
properties to get the input artifacts of a step instead.
Check out this page to see what the output names of your steps are and how to customize them.
Note that the output of a step corresponds to a specific artifact version.
If you'd like to fetch an artifact or an artifact version directly, it is easy to do so with the Client
:
Regardless of how one fetches it, each artifact contains a lot of general information about the artifact as well as datatype-specific metadata and visualizations.
All output artifacts saved through ZenML will automatically have certain datatype-specific metadata saved with them. NumPy Arrays, for instance, always have their storage size, shape
, dtype
, and some statistical properties saved with them. You can access such metadata via the run_metadata
attribute of an output, e.g.:
We will talk more about metadata in the next section.
ZenML automatically saves visualizations for many common data types. Using the visualize()
method you can programmatically show these visualizations in Jupyter notebooks:
If you're not in a Jupyter notebook, you can simply view the visualizations in the ZenML dashboard by running zenml login --local
and clicking on the respective artifact in the pipeline run DAG instead. Check out the artifact visualization page to learn more about how to build and view artifact visualizations in ZenML!
While most of this document has focused on fetching objects after a pipeline run has been completed, the same logic can also be used within the context of a running pipeline.
This is often desirable in cases where a pipeline is running continuously over time and decisions have to be made according to older runs.
For example, this is how we can fetch the last pipeline run of the same pipeline from within a ZenML step:
As shown in the example, we can get additional information about the current run using the StepContext
, which is explained in more detail in the advanced docs.
This section combines all the code from this section into one simple script that you can use to see the concepts discussed above: