Migration guide 0.58.2 → 0.60.0
How to migrate from ZenML 0.58.2 to 0.60.0 (Pydantic 2 edition).
ZenML now uses Pydantic v2. 🥳
This upgrade comes with a set of critical updates. While your user experience mostly remains unaffected, you might see unexpected behavior due to the changes in our dependencies. Moreover, since Pydantic v2 provides a slightly stricter validation process, you might end up bumping into some validation errors which was not caught before, but it is all for the better 🙂 If you run into any other errors, please let us know either on GitHub or on our Slack.
Changes in some of the critical dependencies
SQLModel is one of the core dependencies of ZenML and prior to this upgrade, we were utilizing version
0.0.8
. However, this version is relatively outdated and incompatible with Pydantic v2. Within the scope of this upgrade, we upgraded it to0.0.18
.Due to the change in the SQLModel version, we also had to upgrade our SQLAlchemy dependency from V1 to v2. While this does not affect the way that you are using ZenML, if you are using SQLAlchemy in your environment, you might have to migrate your code as well. For a detailed list of changes, feel free to check their migration guide.
Changes in pydantic
pydantic
Pydantic v2 brings a lot of new and exciting changes to the table. The core logic now uses Rust and it is much faster and more efficient in terms of performance. On top of it, the main concepts like model design, configuration, validation, or serialization now include a lot of new cool features. If you are using pydantic
in your workflow and are interested in the new changes, you can check the brilliant migration guide provided by the pydantic
team to see the full list of changes.
Changes in our integrations changes
Much like ZenML, pydantic
is an important dependency in many other Python packages. That’s why conducting this upgrade helped us unlock a new version for several ZenML integration dependencies. Additionally, in some instances, we had to adapt the functionality of the integration to keep it compatible with pydantic
. So, if you are using any of these integrations, please go through the changes.
Airflow
As mentioned above upgrading our pydantic
dependency meant we had to upgrade our sqlmodel
dependency. Upgrading our sqlmodel
dependency meant we had to upgrade our sqlalchemy
dependency as well. Unfortunately, apache-airflow
is still using sqlalchemy
v1 and is incompatible with pydantic v2. As a solution, we have removed the dependencies of the airflow
integration. Now, you can use ZenML to create your Airflow pipelines and use a separate environment to run them with Airflow. You can check the updated docs right here.
AWS
Some of our integrations now require protobuf
4. Since our previous sagemaker
version (2.117.0
) did not support protobof
4, we could not pair it with these new integrations. Thankfully sagemaker
started supporting protobuf
4 with version 2.172.0
and relaxing its dependency solved the compatibility issue.
Evidently
The old version of our evidently
integration was not compatible with Pydantic v2. They started supporting it starting from version 0.4.16
. As their latest version is 0.4.22
, the new dependency of the integration is limited between these two versions.
Feast
Our previous implementation of the feast
integration was not compatible with Pydantic v2 due to the extra redis
dependency we were using. This extra dependency is now removed and the feast
integration is working as intended.
GCP
The previous version of the Kubeflow dependency (kfp==1.8.22
) in our GCP integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the kfp
dependency to v2 (which has no dependencies on the Pydantic library). This is why you may see some functional changes in the vertex step operator and orchestrator. If you would like to go through the changes in the kfp
library, you can find the migration guide here.
Great Expectations
Great Expectations started supporting Pydantic v2 starting from version 0.17.15
and they are closing in on their 1.0
release. Since this release might include a lot of big changes, we adjusted the dependency in our integration to great-expectations>=0.17.15,<1.0
. We will try to keep it updated in the future once they release the 1.0
version
Kubeflow
Similar to the GCP integration, the previous version of the kubeflow dependency (kfp==1.8.22
) in our kubeflow
integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the kfp
dependency to v2 (which has no dependencies on the Pydantic library). If you would like to go through the changes in the kfp
library, you can find the migration guide here. ( We also are considering adding an alternative version of this integration so our users can keep using kfp
V1 in their environment. Stay tuned for any updates.)
MLflow
mlflow
is compatible with both Pydantic V1 and v2. However, due to a known issue, if you install zenml
first and then do zenml integration install mlflow -y
, it downgrades pydantic
to V1. This is why we manually added the same duplicated pydantic
requirement in the integration definition as well. Keep in mind that the mlflow
library is still using some features of pydantic
V1 which are deprecated. So, if the integration is installed in your environment, you might run into some deprecation warnings.
Label Studio
While we were working on updating our pydantic
dependency, the label-studio-sdk
has released its 1.0 version. In this new version, pydantic
v2 is also supported. The implementation and documentation of our Label Studio integration have been updated accordingly.
Skypilot
With the switch to pydantic
v2, the implementation of our skypilot
integration mostly remained untouched. However, due to an incompatibility between the new version pydantic
and the azurecli
, the skypilot[azure]
flavor can not be installed at the same time, thus our skypilot_azure
integration is currently deactivated. We are working on fixing this issue and if you are using this integration in your workflows, we recommend staying on the previous version of ZenML until we can solve this issue.
Tensorflow
The new version of pydantic
creates a drift between tensorflow
and typing_extensions
packages and relaxing the dependencies here resolves the issue. At the same time, the upgrade to kfp
v2 (in integrations like kubeflow
, tekton
, or gcp
) bumps our protobuf
dependency from 3.X
to 4.X
. To stay compatible with this requirement, the installed version of tensorflow
needs to be >=2.12.0
. While this change solves the dependency issues in most settings, we have bumped into some errors while using tensorflow
2.12.0 on Python 3.8 on Ubuntu. If you would like to use this integration, please consider using a higher Python version.
Tekton
Similar to the gcp
and kubeflow
integrations, the old version of our tekton
integration was not compatible with pydantic
V1 due to its kfp
dependency. With the switch from kfp
V1 to v2, we have adapted our implementation to use the new version of kfp
library and updated our documentation accordingly.
Due to all aforementioned changes, when you upgrade ZenML to 0.60.0, you might run into some dependency issues, especially if you were previously using an integration which was not supporting Pydantic v2 before. In such cases, we highly recommend setting up a fresh Python environment.
Last updated