🐞Debug and solve issues
A guide to debug common issues and get help.
Last updated
A guide to debug common issues and get help.
Last updated
If you stumbled upon this page, chances are you're facing issues with using ZenML. This page documents suggestions and best practices to let you debug, get help, and solve issues quickly.
We suggest going through the following checklist before asking for help:
Search on Slack using the built-in Slack search function at the top of the page.
Search on GitHub issues.
Search the docs using the search bar in the top right corner of the page.
Check out the common errors section below.
Understand the problem by studying the additional logs and client/server logs.
Chances are you'd find your answers there. If you can't find any clue, then it's time to post your question on Slack.
When posting on Slack it's useful to provide the following information (when applicable) so that we get a complete picture before jumping into solutions.
Let us know relevant information about your system. We recommend running the following in your terminal and attaching the output to your question.
You can optionally include information about specific packages where you're having problems by using the -p option. For example, if you're having problems with the tensorflow package, you can run:
The output should look something like this:
System information provides more context to your issue and also eliminates the need for anyone to ask when they're trying to help. This increases the chances of your question getting answered and saves everyone's time.
Tell us briefly:
What were you trying to achieve?
What did you expect to happen?
What actually happened?
Walk us through how to reproduce the same error you had step-by-step, whenever possible. Use the format you prefer. Write it in text or record a video, whichever lets you get the issue at hand across to us!
As a general rule of thumb, always attach relevant log outputs and the full error traceback to help us understand what happened under the hood. If the full error traceback does not fit into a text message, attach a file or use a service like Pastebin or Github's Gist.
Along with the error traceback, we recommend to always share the output of the following commands:
zenml status
zenml stack describe
When applicable, also attach logs of the orchestrator. For example, if you're using the Kubeflow orchestrator, include the logs of the pod that was running the step that failed.
Usually, the default log you see in your terminal is sufficient, in the event it's not, then it's useful to provide additional logs. Additional logs are not shown by default, you'll have to toggle an environment variable for it. Read the next section to find out how.
4.1 Additional logs
When the default logs are not helpful, ambiguous, or do not point you to the root of the issue, you can toggle the value of the ZENML_LOGGING_VERBOSITY environment variable to change the type of logs shown. The default value of ZENML_LOGGING_VERBOSITY environment variable is:
You can pick other values such as WARN, ERROR, CRITICAL, DEBUG to change what's shown in the logs. And export the environment variable in your terminal. For example in Linux:
Read more about how to set environment variables for:
When facing a ZenML Server-related issue, you can view the logs of the server to introspect deeper. To achieve this, run:
The logs from a healthy server should look something like this:
This section documents frequently encountered errors among users and solutions to each.
Typically, the error presents itself as:
If you restarted your machine after deploying ZenML then you have to run zenml up again after each restart. Local ZenML deployments don't survive machine restarts.
This happens when a step configuration is too long. We changed the limit from 4K to 65K chars, but it could still happen if you have excessively long strings in your config.
This is also a common error you might encounter when you do not have the necessary stack components registered on the stack. For example:
In the above error snippet, the step on line 24 expects an experiment tracker but could not find it on the stack. To solve it, register an experiment tracker of your choice on the stack. For instance:
and update your stack with the experiment tracker:
This also applies to all other stack components.
