Handling dependencies
How to handle issues with conflicting dependencies
This page documents a some of the common issues that arise when using ZenML with other libraries.
When using ZenML with other libraries, you may encounter issues with conflicting dependencies. ZenML aims to be stack- and integration-agnostic, allowing you to run your pipelines using the tools that make sense for your problems. With this flexibility comes the possibility of dependency conflicts.
ZenML allows you to install dependencies required by integrations through the zenml integration install ...
command. This is a convenient way to install dependencies for a specific integration, but it can also lead to dependency conflicts if you are using other libraries in your environment. An easy way to see if the ZenML requirements are still met (after installing any extra dependencies required by your work) by running zenml integration list
and checking that your desired integrations still bear the green tick symbol denoting that all requirements are met.
Suggestions for Resolving Dependency Conflicts
Use a tool like pip-compile
for reproducibility
pip-compile
for reproducibilityConsider using a tool like pip-compile
(available through the pip-tools
package) to compile your dependencies into a static requirements.txt
file that can be used across environments. (If you are using uv
, you might want to use uv pip compile
as an alternative.)
For a practical example and explanation of using pip-compile
to address exactly this need, see our 'gitflow' repository and workflow to learn more.
Use pip check
to discover dependency conflicts
pip check
to discover dependency conflictsRunning pip check
will verify that your environment's dependencies are compatible with one another. If not, you will see a list of the conflicts. This may or may not be a problem or something that will prevent you from moving forward with your specific use case, but it is certainly worth being aware of whether this is the case.
Well-known dependency resolution issues
Some of ZenML's integrations come with strict dependency and package version requirements. We try to keep these dependency requirements ranges as wide as possible for the integrations developed by ZenML, but it is not always possible to make this work completely smoothly. Here is one of the known issues:
click
: ZenML currently requiresclick~=8.0.3
for its CLI. This is on account of another dependency of ZenML. Using versions ofclick
in your own project that are greater than 8.0.3 may cause unanticipated behaviors.
Manually bypassing ZenML's integration installation
It is possible to skip ZenML's integration installation process and install dependencies manually. This is not recommended, but it is possible and can be run at your own risk.
Note that the zenml integration install ...
command runs a pip install ...
under the hood as part of its implementation, taking the dependencies listed in the integration object and installing them. For example, zenml integration install gcp
will run pip install "kfp==1.8.16" "gcsfs" "google-cloud-secret-manager" ...
and so on, since they are specified in the integration definition.
To do this, you will need to install the dependencies for the integration you want to use manually. You can find the dependencies for the integrations by running the following:
You can then amend and tweak those requirements as you see fit. Note that if you are using a remote orchestrator, you would then have to place the updated versions for the dependencies in a DockerSettings
object (described in detail here) which will then make sure everything is working as you need.
Last updated