ZenServer
Use ZenML in a Collaborative Setting with the ZenServer.
This is an older version of the ZenML documentation. To check the latest version please visit https://docs.zenml.io
The ZenML Server
Sometimes, you may need to exchange or collaborate on Stack configurations with other developers or even just have your Stacks available on multiple machines. While you can always zip up your Profile files or check the local ZenStore files into version control, a more elegant solution is to have some kind of service accessible from anywhere in your network that provides a REST API that can be used to store and access your Stacks over the network.
The ZenServer, short for ZenML Server, is a distributed client-server ZenML deployment scenario in which multiple ZenML clients can connect to a remote service that provides persistent storage and acts as a central management hub for all ZenML operations involving Stack configurations, Stack Components and other ZenML objects.
Working with a ZenServer involves two main aspects: deploying the ZenServer somewhere and connecting to it from your ZenML client. Keep reading to learn more about how to configure your ZenML client to connect to a remote ZenServer instance, or jump straight to the available ZenServer deployment options by visiting the relevant sections:
(more to come ...)
The ZenServer is still undergoing heavy development. Some features are in Alpha state and may change in future releases. We are also working on providing more deployment and lifecycle management options for the ZenServer that will expand the currently supported deployment use-cases with improved scalability, security and robustness.
Interacting with a Remote ZenServer
The ZenML Profile and ZenStore are the main configuration concepts behind the ZenServer architecture. To connect your ZenML client to a remote ZenServer instance, you need the URL of the ZenServer REST API and a user that has been configured on the server.
Connecting to the ZenServer is just a matter of creating a ZenML Profile backed by a ZenStore of type rest
and pointing it to the ZenServer REST API URL, e.g.:
All ZenML pipelines executed while the ZenServer backed Profile is active will use the Stack and Stack Component definitions stored on the server.
Running the ZenServer Locally
The ZenServer can be deployed locally on any machine, as a means of assessing its functionality without going through the pains of provisioning a complicated infrastructure setup. Before we get started, we need to make sure that all the necessary dependencies for the ZenServer are installed. To do that, we install ZenML with the server extras like this:
Starting a local ZenServer instance is as simple as typing zenml server up
. This will start the server as a background daemon process accessible on your local machine, by default on port 8000:
The local ZenServer exposes a local REST API through which clients can access the same data available in your active Profile. You can also specify a different profile by passing the --profile=$PROFILE_NAME
command-line argument, or change the HTTP port and address on which the ZenServer is accepting requests using --port=$PORT
and --ip-address=$IP_ADDRESS
.The following example creates a new SQL profile named zen-server
and starts a ZenServer with that profile that listens to port 8080 on all interfaces. This is a typical use case where you might want to expose the ZenServer to other machines in your network:
The status of the local ZenServer instance can be checked at any time using:
and shut down using:
To launch the ZenServer manually, without using the ZenML CLI, you can use the uvicorn
command line launch utility. For example, the equivalent of the previous zenml server up
example would be:
Deploy the ZenServer with Docker
The ZenServer can be deployed as a Docker container. To persist the ZenStore information between container restarts, the Docker container should be configured to mount a host volume where its global configuration is kept. The following is an example of starting a Docker container with a ZenServer instance exposed on the local port 8080 and using a local zenserver
folder to persist the ZenML data between container restarts:
ZenServer User Management
All clients connecting to a ZenServer instance must use a user account. This is currently available only as a rudimentary user management system until the ZenServer is fully developed.
A default
user is automatically created when the ZenServer is started. The ZenML CLI can be used to register additional users, either directly into the Profile that the ZenServer is running on, or remotely, from any ZenML client:
ZenServer API Reference
For more details on the ZenServer REST API, spin up the service and visit http://$SERVICE_URL/docs
to see the OpenAPI specification.
Last updated