Use config files to develop locally
Create different variants of your pipeline for local development and production.
When developing ZenML pipelines, it's often beneficial to have different variants of your pipeline for local development and production environments. This approach allows you to iterate quickly during development while maintaining a full-scale setup for production. While configuration files are one way to achieve this, you can also implement this directly in your code.
There are several ways to create different variants of your pipeline:
Using configuration files
Implementing variants in code
Using environment variables
Let's explore each of these methods:
Using configuration files
ZenML allows you to specify pipeline and step configurations using YAML files. Here's an example:
This config file sets up a development variant by using a smaller dataset and disabling caching.
To apply this configuration to your pipeline, use the with_options(config_path=<PATH_TO_CONFIG>)
pattern:
You can create separate configuration files for development and production:
config_dev.yaml
: Configuration for local developmentconfig_prod.yaml
: Configuration for production
Implementing variants in code
You can also create pipeline variants directly in your code:
This approach allows you to switch between development and production variants using a simple boolean flag.
Using environment variables
You can use environment variables to determine which variant to run:
Run your pipeline with: ZENML_ENVIRONMENT=dev python run.py
or ZENML_ENVIRONMENT=prod python run.py
.
Development variant considerations
When creating a development variant of your pipeline, consider optimizing these aspects for faster iteration and debugging:
Use smaller datasets for quicker runs
Specify a local stack for execution
Reduce number of training epochs
Decrease batch size
Use a smaller base model
For example, in a configuration file:
Or in code:
By creating different variants of your pipeline, you can quickly test and debug your code locally with a lightweight setup, while maintaining a full-scale configuration for production execution. This approach streamlines your development workflow and allows for efficient iteration without compromising your production pipeline.
Last updated