Post Execution Workflow

Post-execution workflow

After executing a pipeline, the user needs to be able to fetch it from history and perform certain tasks. This page captures these workflows at an orbital level.

Component Hierarchy

In the context of a post-execution workflow, there is an implied hierarchy of some basic ZenML components:
1
pipelines -> runs -> steps -> outputs
2
3
# where -> implies a 1-many relationship.
Copied!
The highest level pipeline object is also related to the stack component implicitly.

Hierarchy in Code

Pipelines

1
pipelines = repo.get_pipelines() # get all pipeline, can be filtered by name etc via params
2
pipeline = pipeline[0]
Copied!

Runs

1
runs = pipeline.get_runs() # all runs of a pipeline, can be filtered by name etc.
2
run = runs[0]
Copied!

Steps

1
# at this point we switch from the `get_` paradigm to properties
2
steps = run.steps # all steps of a pipeline
3
step = steps[0] || steps.step_name
Copied!

Outputs

1
outputs = step.outputs # all outputs of a step: these are all Artifacts
2
output = outputs.text_artifact_name # artifact output
3
4
>>> output.type_
5
TextArtifact
6
>> output.materializers
7
8
# OR
9
output = outputs[0] # basic output
10
>> output.type_
11
int
12
>> output.value
13
4
Copied!

Visuals

Materializing outputs (or inputs)

Once an output_artifact is acquired from history, one can visualize it with any chosen Materializer.
1
df = output.materializers.pandas.read() # can Read TextArtifact into Pandas DF
2
df.head()
Copied!

Seeing statistics and schema

1
stats = output.materializers.statistics.read()
2
stats # visualize facet
3
4
schema = output.materializers.schema.read()
5
schema # visualize schema
Copied!

Retrieving Model

1
model = output.materializers.keras.read()
2
model # visualize facet
Copied!
Last modified 2h ago