Low Level API
Build production ML pipelines from the simple step interface.
The Low Level ZenML API is defined by the primitive @step and @pipeline decorators. These should be used when the High Level API is too inflexible for the use-case at hand, and one requires more control over individual steps and connecting them in pipelines.
A user may also mix-and-match the Low Level API with the High Level API: All standard data types and steps that are applicable in the High Level API can be used with the Low Level API as well!
In order to illustrate how the Low Level API functions, we'll do a simple exercise to create a training pipeline from scratch (without any High Level components) and take it all the way to deployment.
Here is what we'll do in this guide:
  • Create a MNIST pipeline that trains using TensorFlow (Keras) (similar to the Quickstart).
  • Swap out implementations of the trainer and evaluator steps with scikit-learn.
  • Persist our interim data artifacts in SQL tables rather than on files.
  • Read from a dynamically changing datasource rather than a static one.
  • Deploy the pipeline on Airflow.
If you just want to see all of the code for each chapter of the guide, head over to the GitHub version
If not, then get your environment ready and follow along!

Set up locally

In order to run the chapters of the guide, you need to install and initialize ZenML:
1
# install CLI (order matters, please install in this order.)
2
pip install zenml
3
pip install tensorflow
4
pip install apache_airflow==2.2.0
5
pip install sqlalchemy
6
pip install sklearn
7
8
# pull example
9
zenml example pull low_level_guide
10
11
# temporary for version <0.5.2
12
cd zenml_examples
13
git checkout main
14
15
# back to normal
16
cd low_level_guide
17
18
# initialize
19
git init
20
zenml init
Copied!
In general, to run each chapter you can do:
1
python chapter_*.py # for the chapter of your choice
Copied!
Note before executing each chapter, make sure to clean the old chapter artifact and metadata store:
1
rm -rf .zen
2
zenml init # start again
Copied!

Clean up

In order to clean up, delete the remaining zenml references.
1
rm -rf zenml_examples
Copied!
Press next to start the first chapter!
Last modified 20d ago
Export as PDF
Copy link