GCP Service Connector
Configuring GCP Service Connectors to connect ZenML to GCP resources such as GCS buckets, GKE Kubernetes clusters, and GCR container registries.
The ZenML GCP Service Connector facilitates the authentication and access to managed GCP services and resources. These encompass a range of resources, including GCS buckets, GAR and GCR container repositories, and GKE clusters. The connector provides support for various authentication methods, including GCP user accounts, service accounts, short-lived OAuth 2.0 tokens, and implicit authentication.
To ensure heightened security measures, this connector always issues short-lived OAuth 2.0 tokens to clients instead of long-lived credentials unless explicitly configured to do otherwise. Furthermore, it includes automatic configuration and detection of credentials locally configured through the GCP CLI.
This connector serves as a general means of accessing any GCP service by issuing OAuth 2.0 credential objects to clients. Additionally, the connector can handle specialized authentication for GCS, Docker, and Kubernetes Python clients. It also allows for the configuration of local Docker and Kubernetes CLIs.
$ zenml service-connector list-types --type gcp┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━┯━━━━━━━━┓
┃ NAME │ TYPE │ RESOURCE TYPES │ AUTH METHODS │ LOCAL │ REMOTE ┃
┠───────────────────────┼────────┼───────────────────────┼──────────────────┼───────┼────────┨
┃ GCP Service Connector │ 🔵 gcp │ 🔵 gcp-generic │ implicit │ ✅ │ ✅ ┃
┃ │ │ 📦 gcs-bucket │ user-account │ │ ┃
┃ │ │ 🌀 kubernetes-cluster │ service-account │ │ ┃
┃ │ │ 🐳 docker-registry │ external-account │ │ ┃
┃ │ │ │ oauth2-token │ │ ┃
┃ │ │ │ impersonation │ │ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┷━━━━━━━┷━━━━━━━━┛Prerequisites
The GCP Service Connector is part of the GCP ZenML integration. You can either install the entire integration or use a PyPI extra to install it independently of the integration:
pip install "zenml[connectors-gcp]"installs only prerequisites for the GCP Service Connector Typezenml integration install gcpinstalls the entire GCP ZenML integration
It is not required to install and set up the GCP CLI on your local machine to use the GCP Service Connector to link Stack Components to GCP resources and services. However, it is recommended to do so if you are looking for a quick setup that includes using the auto-configuration Service Connector features.
Resource Types
Generic GCP resource
This resource type allows Stack Components to use the GCP Service Connector to connect to any GCP service or resource. When used by Stack Components, they are provided a Python google-auth credentials object populated with a GCP OAuth 2.0 token. This credentials object can then be used to create GCP Python clients for any particular GCP service.
This generic GCP resource type is meant to be used with Stack Components that are not represented by one of the other, more specific resource types like GCS buckets, Kubernetes clusters, or Docker registries. For example, it can be used with the Google Cloud Image Builder stack component, or the Vertex AI Orchestrator and Step Operator. It should be accompanied by a matching set of GCP permissions that allow access to the set of remote resources required by the client and Stack Component (see the documentation of each Stack Component for more details).
The resource name represents the GCP project that the connector is authorized to access.
GCS bucket
Allows Stack Components to connect to GCS buckets. When used by Stack Components, they are provided a pre-configured GCS Python client instance.
The configured credentials must have at least the following GCP permissions associated with the GCS buckets that it can access:
storage.buckets.liststorage.buckets.getstorage.objects.createstorage.objects.deletestorage.objects.getstorage.objects.liststorage.objects.update
For example, the GCP Storage Object Admin role includes all of the required permissions, but it also includes additional permissions that are not required by the connector. Follow the principle of least privilege by creating a custom role with only the specific permissions listed above, or scope the Storage Object Admin role to specific buckets rather than using it project-wide.
If set, the resource name must identify a GCS bucket using one of the following formats:
GCS bucket URI (canonical resource name): gs://{bucket-name}
GCS bucket name: {bucket-name}
GKE Kubernetes cluster
Allows Stack Components to access a GKE cluster as a standard Kubernetes cluster resource. When used by Stack Components, they are provided a pre-authenticated Python Kubernetes client instance.
The configured credentials must have at least the following GCP permissions associated with the GKE clusters that it can access:
container.clusters.listcontainer.clusters.get
In addition to the above permissions, the credentials should include permissions to connect to and use the GKE cluster (i.e. some or all permissions in the Kubernetes Engine Developer role).
If set, the resource name must identify a GKE cluster using one of the following formats:
GKE cluster name:
{cluster-name}
GKE cluster names are project scoped. The connector can only be used to access GKE clusters in the GCP project that it is configured to use.
GAR container registry (including legacy GCR support)
Important Notice: Google Container Registry is being replaced by Artifact Registry**. Please start using Artifact Registry for your containers. As per Google's documentation, "after May 15, 2024, Artifact Registry will host images for the gcr.io domain in Google Cloud projects without previous Container Registry usage. After March 18, 2025, Container Registry will be shut down.".
Support for legacy GCR registries is still included in the GCP service connector. Users that already have GCP service connectors configured to access GCR registries may continue to use them without taking any action. However, it is recommended to transition to Google Artifact Registries as soon as possible by following the GCP guide on this subject and making the following updates to ZenML GCP Service Connectors that are used to access GCR resources:
add the IAM permissions documented here to the GCP Service Connector credentials to enable them to access the Artifact Registries.
users may keep the gcr.io GCR URLs already configured in the GCP Service Connectors as well as those used in linked Container Registry stack components given that these domains are redirected by Google to GAR as covered in the GCR transition guide. Alternatively, users may update the GCP Service Connector configuration and/or the Container Registry stack components to use the replacement Artifact Registry URLs.
The GCP Service Connector will list the legacy GCR registries as accessible for a GCP project even if the GCP Service Connector credentials do not grant access to GCR registries. This is required for backwards-compatibility and will be removed in a future release.
Allows Stack Components to access a Google Artifact Registry as a standard Docker registry resource. When used by Stack Components, they are provided a pre-authenticated Python Docker client instance.
The configured credentials must have at least the following GCP permissions:
artifactregistry.repositories.createOnPushartifactregistry.repositories.downloadArtifactsartifactregistry.repositories.getartifactregistry.repositories.listartifactregistry.repositories.readViaVirtualRepositoryartifactregistry.repositories.uploadArtifactsartifactregistry.locations.list
The Artifact Registry Create-on-Push Writer role includes all of the above permissions.
This resource type also includes legacy GCR container registry support. When used with GCR registries, the configured credentials must have at least the following GCP permissions:
storage.buckets.getstorage.multipartUploads.abortstorage.multipartUploads.createstorage.multipartUploads.liststorage.multipartUploads.listPartsstorage.objects.createstorage.objects.deletestorage.objects.list
The Storage Legacy Bucket Writer role includes all of the above permissions while at the same time restricting access to only the GCR buckets.
If set, the resource name must identify a GAR or GCR registry using one of the following formats:
Google Artifact Registry repository URI:
[https://]<region>-docker.pkg.dev/<project-id>/<registry-id>[/<repository-name>]Google Artifact Registry name:
projects/<project-id>/locations/<location>/repositories/<repository-id>(legacy) GCR repository URI:
[https://][us.|eu.|asia.]gcr.io/<project-id>[/<repository-name>]
The connector can only be used to access GAR and GCR registries in the GCP project that it is configured to use.
Authentication Methods
Implicit authentication
Implicit authentication to GCP services using Application Default Credentials.
This method may constitute a security risk, because it can give users access to the same cloud resources and services that the ZenML Server itself is configured to access. For this reason, all implicit authentication methods are disabled by default and need to be explicitly enabled by setting the ZENML_ENABLE_IMPLICIT_AUTH_METHODS environment variable or the helm chart enableImplicitAuthMethods configuration option to true in the ZenML deployment.
This authentication method doesn't require any credentials to be explicitly configured. It automatically discovers and uses credentials from one of the following sources:
environment variables (GOOGLE_APPLICATION_CREDENTIALS)
local ADC credential files set up by running
gcloud auth application-default login(e.g.~/.config/gcloud/application_default_credentials.json).a GCP service account attached to the resource where the ZenML server is running. Only works when running the ZenML server on a GCP resource with a service account attached to it or when using Workload Identity (e.g. GKE cluster).
This is the quickest and easiest way to authenticate to GCP services. However, the results depend on how ZenML is deployed and the environment where it is used and is thus not fully reproducible:
when used with the default local ZenML deployment or a local ZenML server, the credentials are those set up on your machine (i.e. by running
gcloud auth application-default loginor setting theGOOGLE_APPLICATION_CREDENTIALSenvironment variable to point to a service account key JSON file).when connected to a ZenML server, this method only works if the ZenML server is deployed in GCP and will use the service account attached to the GCP resource where the ZenML server is running (e.g. a GKE cluster). The service account permissions may need to be adjusted to allow listing and accessing/describing the GCP resources that the connector is configured to access.
Note that the discovered credentials inherit the full set of permissions of the local GCP CLI credentials or service account attached to the ZenML server GCP workload. Depending on the extent of those permissions, this authentication method might not be suitable for production use, as it can lead to accidental privilege escalation. Instead, it is recommended to use the Service Account Key or Service Account Impersonation authentication methods to restrict the permissions that are granted to the connector clients.
To find out more about Application Default Credentials, see the GCP ADC documentation.
A GCP project is required and the connector may only be used to access GCP resources in the specified project. When used remotely in a GCP workload, the configured project has to be the same as the project of the attached service account.
GCP User Account
Long-lived GCP credentials consist of a GCP user account and its credentials.
This method requires GCP user account credentials like those generated by the gcloud auth application-default login command.
By default, the GCP connector generates temporary OAuth 2.0 tokens from the user account credentials and distributes them to clients. The tokens have a limited lifetime of 1 hour. This behavior can be disabled by setting the generate_temporary_tokens configuration option to False, in which case, the connector will distribute the user account credentials JSON to clients instead (not recommended).
This method is preferred during development and testing due to its simplicity and ease of use. It is not recommended as a direct authentication method for production use cases because the clients are granted the full set of permissions of the GCP user account. For production, it is recommended to use the GCP Service Account or GCP Service Account Impersonation authentication methods.
A GCP project is required and the connector may only be used to access GCP resources in the specified project.
If you already have the local GCP CLI set up with these credentials, they will be automatically picked up when auto-configuration is used (see the example below).
GCP Service Account
Long-lived GCP credentials consisting of a GCP service account and its credentials.
This method requires a GCP service account and a service account key JSON created for it.
By default, the GCP connector generates temporary OAuth 2.0 tokens from the service account credentials and distributes them to clients. The tokens have a limited lifetime of 1 hour. This behavior can be disabled by setting the generate_temporary_tokens configuration option to False, in which case, the connector will distribute the service account credentials JSON to clients instead (not recommended).
A GCP project is required and the connector may only be used to access GCP resources in the specified project. If the project_id is not provided, the connector will use the one extracted from the service account key JSON.
If you already have the GOOGLE_APPLICATION_CREDENTIALS environment variable configured to point to a service account key JSON file, it will be automatically picked up when auto-configuration is used.
GCP Service Account impersonation
Generates temporary STS credentials by impersonating another GCP service account.
The connector needs to be configured with the email address of the target GCP service account to be impersonated, accompanied by a GCP service account key JSON for the primary service account. The primary service account must have permission to generate tokens for the target service account (i.e. the Service Account Token Creator role). The connector will generate temporary OAuth 2.0 tokens upon request by using GCP direct service account impersonation. The tokens have a configurable limited lifetime of up to 1 hour.
The best practice implemented with this authentication scheme is to keep the set of permissions associated with the primary service account down to the bare minimum and grant permissions to the privilege-bearing service account instead.
A GCP project is required and the connector may only be used to access GCP resources in the specified project.
If you already have the GOOGLE_APPLICATION_CREDENTIALS environment variable configured to point to the primary service account key JSON file, it will be automatically picked up when auto-configuration is used.
External Account (GCP Workload Identity)
Use GCP workload identity federation to authenticate to GCP services using AWS IAM credentials, Azure Active Directory credentials or generic OIDC tokens.
This authentication method only requires a GCP workload identity external account JSON file that only contains the configuration for the external account without any sensitive credentials. It allows implementing a two layer authentication scheme that keeps the set of permissions associated with implicit credentials down to the bare minimum and grants permissions to the privilege-bearing GCP service account instead.
This authentication method can be used to authenticate to GCP services using credentials from other cloud providers or identity providers. When used with workloads running on AWS or Azure, it involves automatically picking up credentials from the AWS IAM or Azure AD identity associated with the workload and using them to authenticate to GCP services. This means that the result depends on the environment where the ZenML server is deployed and is thus not fully reproducible.
When used with AWS or Azure implicit in-cloud authentication, this method may constitute a security risk, because it can give users access to the identity (e.g. AWS IAM role or Azure AD principal) implicitly associated with the environment where the ZenML server is running. For this reason, all implicit authentication methods are disabled by default and need to be explicitly enabled by setting the ZENML_ENABLE_IMPLICIT_AUTH_METHODS environment variable or the helm chart enableImplicitAuthMethods configuration option to true in the ZenML deployment.
By default, the GCP connector generates temporary OAuth 2.0 tokens from the external account credentials and distributes them to clients. The tokens have a limited lifetime of 1 hour. This behavior can be disabled by setting the generate_temporary_tokens configuration option to False, in which case, the connector will distribute the external account credentials JSON to clients instead (not recommended).
A GCP project is required and the connector may only be used to access GCP resources in the specified roject. This project must be the same as the one for which the external account was configured.
If you already have the GOOGLE_APPLICATION_CREDENTIALS environment variable configured to point to an external account key JSON file, it will be automatically picked up when auto-configuration is used.
GCP OAuth 2.0 token
Uses temporary OAuth 2.0 tokens explicitly configured by the user.
This method has the major limitation that the user must regularly generate new tokens and update the connector configuration as OAuth 2.0 tokens expire. On the other hand, this method is ideal in cases where the connector only needs to be used for a short period of time, such as sharing access temporarily with someone else in your team.
Using any of the other authentication methods will automatically generate and refresh OAuth 2.0 tokens for clients upon request.
A GCP project is required and the connector may only be used to access GCP resources in the specified project.
Auto-configuration
The GCP Service Connector allows auto-discovering and fetching credentials and configuration set up by the GCP CLI on your local host.
Local client provisioning
The local gcloud CLI, the Kubernetes kubectl CLI and the Docker CLI can be configured with credentials extracted from or generated by a compatible GCP Service Connector. Please note that unlike the configuration made possible through the GCP CLI, the Kubernetes and Docker credentials issued by the GCP Service Connector have a short lifetime and will need to be regularly refreshed. This is a byproduct of implementing a high-security profile.
Stack Components use
The GCS Artifact Store Stack Component can be connected to a remote GCS bucket through a GCP Service Connector.
The Google Cloud Image Builder Stack Component, VertexAI Orchestrator, and VertexAI Step Operator can be connected and use the resources of a target GCP project through a GCP Service Connector.
The GCP Service Connector can also be used with any Orchestrator or Model Deployer stack component flavor that relies on Kubernetes clusters to manage workloads. This allows GKE Kubernetes container workloads to be managed without the need to configure and maintain explicit GCP or Kubernetes kubectl configuration contexts and credentials in the target environment or in the Stack Component itself.
Similarly, Container Registry Stack Components can be connected to a Google Artifact Registry or GCR Container Registry through a GCP Service Connector. This allows container images to be built and published to GAR or GCR container registries without the need to configure explicit GCP credentials in the target environment or the Stack Component.
End-to-end examples
Last updated
Was this helpful?