Troubleshoot your ZenML server
Troubleshooting tips for your ZenML deployment
In this document, we will go over some common issues that you might face when deploying ZenML and how to solve them.
Viewing logs
Analyzing logs is a great way to debug issues. Depending on whether you have a Kubernetes (using Helm or zenml deploy
) or a Docker deployment, you can view the logs in different ways.
If you are using Kubernetes, you can view the logs of the ZenML server using the following method:
Check all pods that are running your ZenML deployment.
If you see that the pods aren't running, you can use the command below to get the logs for all pods at once.
Note that the error can either be from the zenml-db-init
container that connects to the MySQL database or from the zenml
container that runs the server code. If the get pods command shows that the pod is failing in the Init
state then use zenml-db-init
as the container name, otherwise use zenml
.
You can also use the --tail
flag to limit the number of lines to show or the --follow
flag to follow the logs in real-time.
Fixing database connection problems
If you are using a MySQL database, you might face issues connecting to it. The logs from the zenml-db-init
container should give you a good idea of what the problem is. Here are some common issues and how to fix them:
If you see an error like
ERROR 1045 (28000): Access denied for user <USER> using password YES
, it means that the username or password is incorrect. Make sure that the username and password are correctly set for whatever deployment method you are using.If you see an error like
ERROR 2003 (HY000): Can't connect to MySQL server on <HOST> (<IP>)
, it means that the host is incorrect. Make sure that the host is correctly set for whatever deployment method you are using.
You can test the connection and the credentials by running the following command from your machine:
If you are using a Kubernetes deployment, you can use the kubectl port-forward
command to forward the MySQL port to your local machine. This will allow you to connect to the database from your machine.
Fixing database initialization problems
If you’ve migrated from a newer ZenML version to an older version and see errors like Revision not found
in your zenml-db-init
logs, one way out is to drop the database and create a new one with the same name.
Log in to your MySQL instance.
Drop the database for the server.
Create the database with the same name.
Restart the Kubernetes pods or the docker container running your server to trigger the database initialization again.
Last updated