ZenServer

Sometimes, you may need to exchange or collaborate on Stack configurations with other developers or even just have your Stacks available on multiple machines. While you can always zip up your Profile files or check the local ZenStore files into version control, a more elegant solution is to have some kind of service accessible from anywhere in your network that provides a REST API that can be used to store and access your Stacks over the network.
The ZenServer, short for ZenML Server, is a distributed client-server ZenML deployment scenario in which multiple ZenML clients can connect to a remote service that provides persistent storage and acts as a central management hub for all ZenML operations involving Stack configurations, Stack Components and other ZenML objects.
Working with a ZenServer involves two main aspects: deploying the ZenServer somewhere and connecting to it from your ZenML client. Keep reading to learn more about how to configure your ZenML client to connect to a remote ZenServer instance, or jump straight to the available ZenServer deployment options by visiting the relevant sections:
​run a ZenServer locally​
​deploy ZenServer with Docker​
(more to come ...)
The ZenServer is still undergoing heavy development. Some features are in Alpha state and may change in future releases. We are also working on providing more deployment and lifecycle management options for the ZenServer that will expand the currently supported deployment use-cases with improved scalability, security and robustness.
Interacting with a Remote ZenServer
The ZenML Profile and ZenStore are the main configuration concepts behind the ZenServer architecture. To connect your ZenML client to a remote ZenServer instance, you need the URL of the ZenServer REST API and a user that has been configured on the server.
Connecting to the ZenServer is just a matter of creating a ZenML Profile backed by a ZenStore of type rest and pointing it to the ZenServer REST API URL, e.g.:
1
$ zenml profile create zenml-remote -t rest --url http://192.168.178.40:8080 --user default
2
Running without an active repository root.
3
Running with active profile: 'default' (global)
4
Initializing profile zenml-remote...
5
Profile 'zenml-remote' successfully created.
6
​
7
$ zenml profile set zenml-remote 
8
Running without an active repository root.
9
Running with active profile: 'devel' (global)
10
Active profile changed to: 'zenml-remote'
Copied!
All ZenML pipelines executed while the ZenServer backed Profile is active will use the Stack and Stack Component definitions stored on the server.
Running the ZenServer Locally
The ZenServer can be deployed locally on any machine, as a means of assessing its functionality without going through the pains of provisioning a complicated infrastructure setup. Before we get started, we need to make sure that all the necessary dependencies for the ZenServer are installed. To do that, we install ZenML with the server extras like this:
1
pip install zenml[server]
Copied!
Starting a local ZenServer instance is as simple as typing zenml server up. This will start the server as a background daemon process accessible on your local machine, by default on port 8000:
1
$ zenml server up
2
Running without an active repository root.
3
Starting a new ZenServer local instance.
4
ZenServer running at 'http://127.0.0.1:8000/'.
Copied!
The local ZenServer exposes a local REST API through which clients can access the same data available in your active Profile. You can also specify a different profile by passing the --profile=$PROFILE_NAME command-line argument, or change the HTTP port and address on which the ZenServer is accepting requests using --port=$PORT and --ip-address=$IP_ADDRESS.The following example creates a new SQL profile named zen-server and starts a ZenServer with that profile that listens to port 8080 on all interfaces. This is a typical use case where you might want to expose the ZenServer to other machines in your network:
1
$ zenml server down
2
Shutting down the local ZenService instance.
3
​
4
$ zenml profile create -t sql zen-server
5
Running without an active repository root.
6
Running with active profile: 'devel' (global)
7
Initializing profile zen-server...
8
Registering default stack...
9
Registered stack with name 'default'.
10
Profile 'zen-server' successfully created.
11
​
12
$ zenml server up --port 8080 --ip-address 0.0.0.0 --profile zen-server
13
Running without an active repository root.
14
Starting a new ZenServer local instance.
15
ZenServer running at 'http://0.0.0.0:8080/'.
Copied!
The status of the local ZenServer instance can be checked at any time using:
1
$ zenml server status
2
The ZenServer status is active and running at http://0.0.0.0:8080/..
Copied!
and shut down using:
1
$ zenml server down
2
Shutting down the local ZenService instance.
Copied!
To launch the ZenServer manually, without using the ZenML CLI, you can use the uvicorn command line launch utility. For example, the equivalent of the previous zenml server up example would be:
1
ZENML_PROFILE_NAME=zen-server uvicorn zenml.zen_server.zen_server_api:app \
2
    --port 8080 --host 0.0.0.0
Copied!
Deploy the ZenServer with Docker
The ZenServer can be deployed as a Docker container. To persist the ZenStore information between container restarts, the Docker container should be configured to mount a host volume where its global configuration is kept. The following is an example of starting a Docker container with a ZenServer instance exposed on the local port 8080 and using a local zenserver folder to persist the ZenML data between container restarts:
1
mkdir zenserver
2
docker run -it -d -p 8080:8000 \
3
    -v $PWD/zenserver:/zenserver \
4
    -e ZENML_CONFIG_PATH=/zenserver \
5
    zenmldocker/zenml \
6
    uvicorn zenml.zen_server.zen_server_api:app --host 0.0.0.0
Copied!
ZenServer User Management
All clients connecting to a ZenServer instance must use a user account. This is currently available only as a rudimentary user management system until the ZenServer is fully developed.
A default user is automatically created when the ZenServer is started. The ZenML CLI can be used to register additional users, either directly into the Profile that the ZenServer is running on, or remotely, from any ZenML client:
1
$ zenml profile describe
2
Running without an active repository root.
3
Running with active profile: 'zenml-remote' (global)
4
'zenml-remote' Profile Configuration (ACTIVE)
5
┏━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
6
┃ PROPERTY     │ VALUE                      ┃
7
┠──────────────┼────────────────────────────┨
8
┃ NAME         │ zenml-remote               ┃
9
┠──────────────┼────────────────────────────┨
10
┃ STORE_URL    │ http://192.168.178.40:8080 ┃
11
┠──────────────┼────────────────────────────┨
12
┃ STORE_TYPE   │ rest                       ┃
13
┠──────────────┼────────────────────────────┨
14
┃ ACTIVE_STACK │ default                    ┃
15
┠──────────────┼────────────────────────────┨
16
┃ ACTIVE_USER  │ default                    ┃
17
┗━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
18
​
19
$ zenml user list
20
Running without an active repository root.
21
Running with active profile: 'zenml-remote' (global)
22
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┓
23
┃                  ID                  │ CREATION_DATE              │ NAME    ┃
24
┠──────────────────────────────────────┼────────────────────────────┼─────────┨
25
┃ 085cb51e-37cf-4661-a8cb-ec885b218387 │ 2022-05-16 18:12:16.355773 │ default ┃
26
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┛
27
​
28
$ zenml user create aria
29
Running without an active repository root.
30
Running with active profile: 'zenml-remote' (global)
31
​
32
$ zenml user list
33
Running without an active repository root.
34
Running with active profile: 'zenml-remote' (global)
35
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┓
36
┃                  ID                  │ CREATION_DATE              │ NAME    ┃
37
┠──────────────────────────────────────┼────────────────────────────┼─────────┨
38
┃ 085cb51e-37cf-4661-a8cb-ec885b218387 │ 2022-05-16 18:12:16.355773 │ default ┃
39
┠──────────────────────────────────────┼────────────────────────────┼─────────┨
40
┃ 16dfc5da-9462-4a36-8502-15f0204501cf │ 2022-05-16 18:39:09.461282 │ aria    ┃
41
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┛
Copied!
ZenServer API Reference
For more details on the ZenServer REST API, spin up the service and visit http://$SERVICE_URL/docs to see the OpenAPI specification.

Collaborate - Previous

Share with Profiles

Next - Links & Resources

ZenML Links

Last modified 10d ago

Export as PDF

Copy link

Edit on GitHub

Contents

Interacting with a Remote ZenServer

Running the ZenServer Locally

Deploy the ZenServer with Docker

ZenServer User Management

ZenServer API Reference