Model Deployers
Managing the models deployed by your pipelines
Model deployers are stack components responsible for online model serving. Online serving is the process of hosting and loading machine-learning models as part of a managed web service and providing access to the models through an API endpoint like HTTP or GRPC. Once deployed, you can send inference requests to the model through the web service's API and receive fast, low-latency responses.
When present in a stack, the model deployer also acts as a registry for models that are served with ZenML. You can use the model deployer to list all models that are currently deployed for online inference or filtered according to a particular pipeline run or step, or to suspend, resume or delete an external model server managed through ZenML.
Before reading this chapter, make sure that you are familiar with the concept of stacks, stack components and their flavors.

Base Abstraction

In ZenML, the base abstraction of the model deployer is built on top of three major criteria:
  1. 1.
    It needs to contain all the stack-related configuration attributes required to interact with the remote model serving tool, service or platform (e.g. hostnames, URLs, references to credentials, other client-related configuration parameters).
  2. 2.
    It needs to implement the continuous deployment logic necessary to deploy models in a way that updates an existing model server that is already serving a previous version of the same model instead of creating a new model server for every new model version (see the deploy_model abstract method). This functionality can be consumed directly from ZenML pipeline steps, but it can also be used outside the pipeline to deploy ad-hoc models. It is also usually coupled with a standard model deployer step, implemented by each integration, that hides the details of the deployment process from the user.
  3. 3.
    It needs to act as a ZenML BaseService registry, where every BaseService instance is used as an internal representation of a remote model server (see the find_model_server abstract method). To achieve this, it must be able to re-create the configuration of a BaseService from information that is persisted externally, alongside or even as part of the remote model server configuration itself. For example, for model servers that are implemented as Kubernetes resources, the BaseService instances can be serialized and saved as Kubernetes resource annotations. This allows the model deployer to keep track of all externally running model servers and to re-create their corresponding BaseService instance representations at any given time. The model deployer also defines methods that implement basic life-cycle management on remote model servers outside the coverage of a pipeline (see stop_model_server, start_model_server and delete_model_server).
Putting all these considerations together, we end up with the following interface:
1
from abc import ABC, abstractmethod
2
from typing import ClassVar, Dict, Generator, List, Optional
3
from uuid import UUID
4
5
from zenml.enums import StackComponentType
6
from zenml.services import BaseService, ServiceConfig
7
from zenml.stack import StackComponent
8
9
DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT = 300
10
11
12
class BaseModelDeployer(StackComponent, ABC):
13
"""Base class for all ZenML model deployers."""
14
15
# Class variables
16
TYPE: ClassVar[StackComponentType] = StackComponentType.MODEL_DEPLOYER
17
18
@abstractmethod
19
def deploy_model(
20
self,
21
config: ServiceConfig,
22
replace: bool = False,
23
timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,
24
) -> BaseService:
25
"""Abstract method to deploy a model."""
26
27
@staticmethod
28
@abstractmethod
29
def get_model_server_info(
30
service: BaseService,
31
) -> Dict[str, Optional[str]]:
32
"""Give implementation-specific way to extract relevant model server
33
properties for the user."""
34
35
@abstractmethod
36
def find_model_server(
37
self,
38
running: bool = False,
39
service_uuid: Optional[UUID] = None,
40
pipeline_name: Optional[str] = None,
41
pipeline_run_id: Optional[str] = None,
42
pipeline_step_name: Optional[str] = None,
43
model_name: Optional[str] = None,
44
model_uri: Optional[str] = None,
45
model_type: Optional[str] = None,
46
) -> List[BaseService]:
47
"""Abstract method to find one or more model servers that match the
48
given criteria."""
49
50
@abstractmethod
51
def stop_model_server(
52
self,
53
uuid: UUID,
54
timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,
55
force: bool = False,
56
) -> None:
57
"""Abstract method to stop a model server."""
58
59
@abstractmethod
60
def start_model_server(
61
self,
62
uuid: UUID,
63
timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,
64
) -> None:
65
"""Abstract method to start a model server."""
66
67
@abstractmethod
68
def delete_model_server(
69
self,
70
uuid: UUID,
71
timeout: int = DEFAULT_DEPLOYMENT_START_STOP_TIMEOUT,
72
force: bool = False,
73
) -> None:
74
"""Abstract method to delete a model server."""
Copied!
This is a slimmed-down version of the base implementation which aims to highlight the abstraction layer. In order to see the full implementation and get the complete docstrings, please check the API docs.

List of available model deployers

In its current version, ZenML features two integrations, namely the mlflow and the seldon integrations, with model deployers as stack components. In order to get more information on how you can use these flavors in your stack, please check the corresponding pages in the API docs linked below.
Text
Flavor
Integration
mlflow
mlflow
seldon
seldon
If you would like to see the available flavors for model deployers, you can use the command:
1
zenml model-deployer flavor list
Copied!

Building your own model deployers

If you want to create your own custom flavor for a model deployer, you can follow the following steps:
  1. 1.
    Create a class which inherits from the BaseModelDeployer.
  2. 2.
    Define the FLAVOR class variable.
  3. 3.
    Implement the abstactmethods based on the API of your desired model deployer.
Once you are done with the implementation, you can register it through the CLI as:
1
zenml model-deployer flavor register <THE-SOURCE-PATH-OF-YOUR-MODEL_DEPLOYER>
Copied!