Best Practices

Best practices, recommendations, and tips from the ZenML team

Recommended Repository Structure
├── notebooks                   <- All notebooks in one place
│   ├── *.ipynb         
├── pipelines                   <- All pipelines in one place
│   ├── training_pipeline
│   │   ├── .dockerignore
│   │   ├── config.yaml
│   │   ├── Dockerfile
│   │   ├── training_pipeline.py
│   │   ├── requirements.txt
│   ├── deployment_pipeline
│   │   ├── ...
├── steps                       <- All steps in one place
│   ├── loader_step
│   │   ├── loader_step.py
│   ├── training_step
│   │   ├── ...
├── .dockerignore 
├── .gitignore
├── config.yaml
├── Dockerfile
├── README.md
├── requirements.txt
├── run.py
└── setup.sh
Best Practices and Tips
Nest pipeline_instance.run() in if __name__ == "__main__"
pipeline_instance = training_pipeline(...)
​
if __name__ == "__main__":
    pipeline_instance.run()
This ensures that loading the pipeline from elsewhere does not also run it.
Never call the pipeline instance pipeline or a step instance step
Doing this will overwrite the imported pipeline and step decorators and lead to failures at later stages if more steps and pipelines are decorated there.
from zenml.pipelines import pipeline
​
​
@pipeline
def first_pipeline():
    ...
​
​
pipeline = first_pipeline(...)
​
​
@pipeline  # The code will fail here
def second_pipeline():
    ...
Explicitly set enable_cache at the @pipeline level
Caching is enabled by default for ZenML Pipelines. It is good to be explicit, though, so that it is clear when looking at the code if caching is enabled or disabled for any given pipeline.
Explicitly disable caching when loading data from filesystem or external APIs
ZenML inherently uses caching. However, this caching relies on changes of input artifacts to invalidate the cache. In case a step has external data sources like external APIs or file systems, caching should be disabled explicitly for the step.
Enable cache explicitly for steps that have a context argument, if they don't invalidate the caching behavior
Cache is implicitly disabled for steps that have a context argument, because it is assumed that you might use the step context to retrieve artifacts from the artifact store that are unrelated to the current step. However, if that is not the case, and your step logic doesn't invalidate the caching behavior, it would be better to explicitly enable the cache for your step.
Use unique pipeline names across projects/workspaces
Pipeline names are their unique identifiers, so using the same name for different pipelines will create a mixed history of runs between the two pipelines.
Check which integrations are required for registering a stack component
You can do so by running zenml flavor list and installing the missing integration(s) with zenml integration install.
Initialize the ZenML repository in the root of the source code tree of a project, even if it's optional
This will set the ZenML project root for the project and create a local configuration. The advantage is that you create and maintain your active stack on a project level.
Include a .dockerignore in the ZenML repository to exclude files and folders from the container images built by ZenML for containerized environments
Containerized Orchestrators and Step Operators load your complete project files into a Docker image for execution. To speed up the process and reduce Docker image sizes, exclude all unnecessary files (like data, virtual environments, git repos, etc.) within the .dockerignore.
Use get_pipeline(pipeline=...) instead of indexing ([0]) to retrieve previous pipelines
When inspecting pipeline runs it is tempting to access the pipeline views directly by their index, but the pipelines are sorted in descending order of their creation time, so the pipeline at [0] might not be the one you are expecting.
from zenml.post_execution import get_pipeline
​
first_pipeline.run()
second_pipeline.run()
first_pipeline.run()
​
get_pipelines()
>>> [PipelineView('first_pipeline'), PipelineView('second_pipeline')]
​
# This is the recommended explicit way to retrieve your specific pipeline 
# using the pipeline class if you have it at hand
get_pipeline(pipeline=first_pipeline)
​
# Alternatively you can also use the name of the pipeline
get_pipeline(pipeline="first_pipeline")
Have your imports relative to your .zen directory OR have your imports relative to the root of your repository in cases when you don't have a .zen directory (=> which means to have the runner at the root of your repository)
ZenML uses the .zen repository root to resolve the class path of your functions and classes in a way that is portable across different types of environments such as containers. If a repository is not present, the location of the main Python module is used as an implicit repository root.
Put your runners in the root of the repository
Putting your pipeline runners in the root of the repository ensures that all imports that are defined relative to the project root resolve for the pipeline runner.
Tips
Use zenml GROUP explain to explain what everything is
For a practical example on all of the above, please check out ZenML Projects which are practical end-to-end projects built using ZenML.

Develop a Custom Model Registry

Next - Guidelines

Global Configuration

Last modified 2d ago

Best Practices

Recommended Repository Structure

Best Practices and Tips

Nest `pipeline_instance.run()` in `if name == "main"`

Never call the pipeline instance `pipeline` or a step instance `step`

Explicitly set `enable_cache` at the `@pipeline` level

Explicitly disable caching when loading data from filesystem or external APIs

Enable cache explicitly for steps that have a `context` argument, if they don't invalidate the caching behavior

Use unique pipeline names across projects/workspaces

Check which integrations are required for registering a stack component

Initialize the ZenML repository in the root of the source code tree of a project, even if it's optional

Include a `.dockerignore` in the ZenML repository to exclude files and folders from the container images built by ZenML for containerized environments

Use `get_pipeline(pipeline=...)` instead of indexing (`[0]`) to retrieve previous pipelines

Have your imports relative to your `.zen` directory OR have your imports relative to the root of your repository in cases when you don't have a `.zen` directory (=> which means to have the runner at the root of your repository)

Put your runners in the root of the repository

Tips