Introduction
Connect your ZenML deployment to a cloud provider and other infrastructure services and resources.
A production-grade MLOps platform involves interactions between a diverse combination of third-party libraries and external services sourced from various different vendors. One of the most daunting hurdles in building and operating an MLOps platform composed of multiple components is configuring and maintaining uninterrupted and secured access to the infrastructure resources and services that it consumes.
In layman's terms, your pipeline code needs to "connect" to a handful of different services to run successfully and do what it's designed to do. For example, it might need to connect to a private AWS S3 bucket to read and store artifacts, a Kubernetes cluster to execute steps with Kubeflow or Tekton, and a private GCR container registry to build and store container images. ZenML makes this possible by allowing you to configure authentication information and credentials embedded directly into your Stack Components, but this doesn't scale well when you have more than a few Stack Components and has many other disadvantages related to usability and security.
Gaining access to infrastructure resources and services requires knowledge about the different authentication and authorization mechanisms and involves configuring and maintaining valid credentials. It gets even more complicated when these different services need to access each other. For instance, the Kubernetes container running your pipeline step needs access to the S3 bucket to store artifacts or needs to access a cloud service like AWS SageMaker, VertexAI, or AzureML to run a CPU/GPU intensive task like training a model.
The challenge comes from setting up and implementing proper authentication and authorization with the best security practices in mind, while at the same time keeping this complexity away from the day-to-day routines of coding and running pipelines.
The hard-to-swallow truth is there is no single standard that unifies all authentication and authorization-related matters or a single, well-defined set of security best practices that you can follow. However, with ZenML you get the next best thing, an abstraction that keeps the complexity of authentication and authorization away from your code and makes it easier to tackle them: the ZenML Service Connectors.

A representative use-case
The range of features covered by Service Connectors is extensive and going through the entire Service Connector Guide can be overwhelming. If all you want is to get a quick overview of how Service Connectors work and what they can do for you, this section is for you.
This is a representative example of how you would use a Service Connector to connect ZenML to a cloud service. This example uses the AWS Service Connector to connect ZenML to an AWS S3 bucket and then link an S3 Artifact Store Stack Component to it.
Some details about the current alternatives to using Service Connectors and their drawbacks are provided below. Feel free to skip them if you are already familiar with them or just want to get to the good part.
Without Service Connectors, credentials are stored directly in the Stack Component configuration or ZenML Secret and are directly used in the runtime environment. The Stack Component implementation is directly responsible for validating credentials, authenticating and connecting to the infrastructure service. This is illustrated in the following diagram:

When Service Connectors are involved in the authentication and authorization process, they can act as brokers. The credentials validation and authentication process takes place on the ZenML server. In most cases, the main credentials never have to leave the ZenML server as the Service Connector automatically converts them into short-lived credentials with a reduced set of privileges and issues these credentials to clients. Furthermore, multiple Stack Components of different flavors can use the same Service Connector to access different types or resources with the same credentials:

In working with Service Connectors, the first step is usually finding out what types of resources you can connect ZenML to. Maybe you have already planned out the infrastructure options for your MLOps platform and are looking to find out whether ZenML can accommodate them. Or perhaps you want to use a particular Stack Component flavor in your Stack and are wondering whether you can use a Service Connector to connect it to external resources.
Listing the available Service Connector Types will give you a good idea of what you can do with Service Connectors:
zenml service-connector list-types
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━┯━━━━━━━━┓
┃ NAME │ TYPE │ RESOURCE TYPES │ AUTH METHODS │ LOCAL │ REMOTE ┃
┠──────────────────────────────┼───────────────┼───────────────────────┼──────────────────┼───────┼────────┨
┃ Kubernetes Service Connector │ 🌀 kubernetes │ 🌀 kubernetes-cluster │ password │ ✅ │ ✅ ┃
┃ │ │ │ token │ │ ┃
┠──────────────────────────────┼───────────────┼───────────────────────┼──────────────────┼───────┼────────┨
┃ Docker Service Connector │ 🐳 docker │ 🐳 docker-registry │ password │ ✅ │ ✅ ┃
┠──────────────────────────────┼───────────────┼───────────────────────┼──────────────────┼───────┼────────┨
┃ AWS Service Connector │ 🔶 aws │ 🔶 aws-generic │ implicit │ ✅ │ ✅ ┃
┃ │ │ 📦 s3-bucket │ secret-key │ │ ┃
┃ │ │ 🌀 kubernetes-cluster │ sts-token │ │ ┃
┃ │ │ 🐳 docker-registry │ iam-role │ │ ┃
┃ │ │ │ session-token │ │ ┃
┃ │ │ │ federation-token │ │ ┃
┠──────────────────────────────┼───────────────┼───────────────────────┼──────────────────┼───────┼────────┨
┃ GCP Service Connector │ 🔵 gcp │ 🔵 gcp-generic │ implicit │ ✅ │ ✅ ┃
┃ │ │ 📦 gcs-bucket │ user-account │ │ ┃
┃ │ │ 🌀 kubernetes-cluster │ service-account │ │ ┃
┃ │ │ 🐳 docker-registry │ oauth2-token │ │ ┃
┃ │ │ │ impersonation │ │ ┃
┠──────────────────────────────┼───────────────┼───────────────────────┼──────────────────┼───────┼────────┨
┃ HyperAI Service Connector │ 🤖 hyperai │ 🤖 hyperai-instance │ rsa-key │ ✅ │ ✅ ┃
┃ │ │ │ dsa-key │ │ ┃
┃ │ │ │ ecdsa-key │ │ ┃
┃ │ │ │ ed25519-key │ │ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┷━━━━━━━┷━━━━━━━━┛
Service Connector Types are also displayed in the dashboard during the configuration of a new Service Connector:
The cloud provider of choice for our example is AWS and we're looking to hook up an S3 bucket to an S3 Artifact Store Stack Component. We'll use the AWS Service Connector Type.
Not all Stack Components support being linked to a Service Connector. This is indicated in the flavor description of each Stack Component. Our example uses the S3 Artifact Store, which does support it:
$ zenml artifact-store flavor describe s3
Configuration class: S3ArtifactStoreConfig
[...]
This flavor supports connecting to external resources with a Service Connector. It requires a 's3-bucket' resource. You can get a list of all available connectors and the compatible resources that they can
access by running:
'zenml service-connector list-resources --resource-type s3-bucket'
If no compatible Service Connectors are yet registered, you can register a new one by running:
'zenml service-connector register -i'
The second step is registering a Service Connector that effectively enables ZenML to authenticate to and access one or more remote resources. This step is best handled by someone with some infrastructure knowledge, but there are sane defaults and auto-detection mechanisms built into most Service Connectors that can make this a walk in the park even for the uninitiated. For our simple example, we're registering an AWS Service Connector with AWS credentials automatically lifted up from your local host, giving ZenML access to the same resources that you can access from your local machine through the AWS CLI.
This step assumes the AWS CLI is already installed and set up with credentials on your machine (e.g. by running aws configure
).
zenml service-connector register aws-s3 --type aws --auto-configure --resource-type s3-bucket
⠼ Registering service connector 'aws-s3'...
Successfully registered service connector `aws-s3` with access to the following resources:
┏━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠───────────────┼───────────────────────────────────────┨
┃ 📦 s3-bucket │ s3://aws-ia-mwaa-715803424590 ┃
┃ │ s3://zenbytes-bucket ┃
┃ │ s3://zenfiles ┃
┃ │ s3://zenml-demos ┃
┃ │ s3://zenml-generative-chat ┃
┃ │ s3://zenml-public-datasets ┃
┃ │ s3://zenml-public-swagger-spec ┃
┗━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
The CLI validates and shows all S3 buckets that can be accessed with the auto-discovered credentials.
The next step in this journey is configuring and connecting one (or more) Stack Components to a remote resource via the Service Connector registered in the previous step. This is as easy as saying "I want this S3 Artifact Store to use the s3://my-bucket
S3 bucket" and doesn't require any knowledge whatsoever about the authentication mechanisms or even the provenance of those resources. The following example creates an S3 Artifact store and connects it to an S3 bucket with the earlier connector:
zenml artifact-store register s3-zenfiles --flavor s3 --path=s3://zenfiles
zenml artifact-store connect s3-zenfiles --connector aws-s3
$ zenml artifact-store register s3-zenfiles --flavor s3 --path=s3://zenfiles
Successfully registered artifact_store `s3-zenfiles`.
$ zenml artifact-store connect s3-zenfiles --connector aws-s3
Successfully connected artifact store `s3-zenfiles` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼───────────────┼────────────────┨
┃ 96a92154-4ec7-4722-bc18-21eeeadb8a4f │ aws-s3 │ 🔶 aws │ 📦 s3-bucket │ s3://zenfiles ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
The S3 Artifact Store Stack Component we just connected to the infrastructure is now ready to be used in a stack to run a pipeline:
zenml stack register s3-zenfiles -o default -a s3-zenfiles --set
A simple pipeline could look like this:
from zenml import step, pipeline
@step
def simple_step_one() -> str:
"""Simple step one."""
return "Hello World!"
@step
def simple_step_two(msg: str) -> None:
"""Simple step two."""
print(msg)
@pipeline
def simple_pipeline() -> None:
"""Define single step pipeline."""
message = simple_step_one()
simple_step_two(msg=message)
if __name__ == "__main__":
simple_pipeline()
Save this as run.py
and run it with the following command:
python run.py
Running pipeline simple_pipeline on stack s3-zenfiles (caching enabled)
Step simple_step_one has started.
Step simple_step_one has finished in 1.065s.
Step simple_step_two has started.
Hello World!
Step simple_step_two has finished in 5.681s.
Pipeline run simple_pipeline-2023_06_15-19_29_42_159831 has finished in 12.522s.
Dashboard URL: http://127.0.0.1:8237/default/pipelines/8267b0bc-9cbd-42ac-9b56-4d18275bdbb4/runs
This example is just a simple demonstration of how to use Service Connectors to connect ZenML Stack Components to your infrastructure. The range of features and possibilities is much larger. ZenML ships with built-in Service Connectors able to connect and authenticate to AWS, GCP, and Azure and offers many different authentication methods and security best practices. Follow the resources below for more information.
🪄 The complete guide to Service Connectors
Everything you need to know to unlock the power of Service Connectors in your project.
✅ Security Best Practices
Best practices concerning the various authentication methods implemented by Service Connectors.
🐋 Docker Service Connector
Use the Docker Service Connector to connect ZenML to a generic Docker container registry.
🌀 Kubernetes Service Connector
Use the Kubernetes Service Connector to connect ZenML to a generic Kubernetes cluster.
Last updated
Was this helpful?