Connect ZenML to infrastructure
Connect your ZenML deployment to a cloud provider and other infrastructure services and resources.
This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.
Connect ZenML to infrastructure
A production-grade MLOps platform involves interactions between a diverse combination of third-party libraries and external services sourced from various different vendors. One of the most daunting hurdles in building and operating an MLOps platform composed of multiple components is configuring and maintaining uninterrupted and secured access to the infrastructure resources and services that it consumes.
In layman's terms, your pipeline code needs to "connect" to a handful of different services to run successfully and do what it's designed to do. For example, it might need to connect to a private AWS S3 bucket to read and store artifacts, a Kubernetes cluster to execute steps with Kubeflow or Tekton, and a private GCR container registry to build and store container images. ZenML makes this possible by allowing you to configure authentication information and credentials embedded directly into your Stack Components, but this doesn't scale well when you have more than a few Stack Components and has many other disadvantages related to usability and security.
Gaining access to infrastructure resources and services requires knowledge about the different authentication and authorization mechanisms and involves configuring and maintaining valid credentials. It gets even more complicated when these different services need to access each other. For instance, the Kubernetes container running your pipeline step needs access to the S3 bucket to store artifacts or needs to access a cloud service like AWS SageMaker, VertexAI, or AzureML to run a CPU/GPU intensive task like training a model.
The challenge comes from setting up and implementing proper authentication and authorization with the best security practices in mind, while at the same time keeping this complexity away from the day-to-day routines of coding and running pipelines.
The hard-to-swallow truth is there is no single standard that unifies all authentication and authorization-related matters or a single, well-defined set of security best practices that you can follow. However, with ZenML you get the next best thing, an abstraction that keeps the complexity of authentication and authorization away from your code and makes it easier to tackle them: the ZenML Service Connectors.
Discover Service Connectors with a glance into their features and use-cases.
Understand the paradigm and workflow of connecting ZenML to external resources with Service Connectors.
Everything you need to know to unlock the power of Service Connectors in your project.
Best practices concerning the various authentication methods implemented by Service Connectors.
Use the Docker Service Connector to connect ZenML to a generic Docker container registry.
Use the Kubernetes Service Connector to connect ZenML to a generic Kubernetes cluster.
Use the AWS Service Connector to connect ZenML to AWS cloud resources.
Use the GCP Service Connector to connect ZenML to a GCP cloud resources.
Last updated
