Registering Existing Infrastructure with ZenML - A Guide for Terraform Users
Terraform is a powerful tool for managing infrastructure as code, and is by far the most popular IaC tool. Many companies already have existing Terraform setups, and it is often desirable to integrate ZenML with this setup.
We already got a glimpse on how to deploy a cloud stack with Terraform using existing Terraform modules that are maintained by the ZenML team. While this is a great solution for quickly getting started, it might not always be suitable for your use case.
This guide is for advanced users who want to manage their own custom Terraform code but want to use ZenML to manage their stacks. For this, the ZenML provider is a better choice.
Understanding the Two-Phase Approach
When working with ZenML stacks, there are two distinct phases:
Infrastructure Deployment: Creating cloud resources (typically handled by platform teams)
ZenML Registration: Registering these resources as ZenML stack components
While our official modules (zenml-stack/aws, zenml-stack/gcp, zenml-stack/azure) handle both phases, you might already have infrastructure deployed. Let's explore how to register existing infrastructure with ZenML.
Phase 1: Infrastructure Deployment
You likely already have this handled in your existing Terraform configurations:
# Example of existing GCP infrastructureresource "google_storage_bucket" "ml_artifacts" { name ="company-ml-artifacts" location ="US"}resource "google_artifact_registry_repository" "ml_containers" { repository_id ="ml-containers" format ="DOCKER"}
Phase 2: ZenML Registration
Setup the ZenML Provider
First, configure the ZenML provider to communicate with your ZenML server:
terraform {required_providers { zenml = { source ="zenml-io/zenml" } }}provider "zenml" {# Configuration options will be loaded from environment variables:# ZENML_SERVER_URL# ZENML_API_KEY}
To generate a API key, use the command:
zenmlservice-accountcreate<SERVICE_ACCOUNT_NAME>
You can learn more about how to generate a ZENML_API_KEY via service accounts here.
Create the service connectors
The key to successful registration is proper authentication between the components. Service connectors are ZenML's way of managing this:
# First, create a service connectorresource "zenml_service_connector" "gcp_connector" { name ="gcp-${var.environment}-connector" type ="gcp" auth_method ="service-account" configuration = { project_id = var.project_id service_account_json =file("service-account.json") }}# Create a stack component referencing the connectorresource "zenml_stack_component" "artifact_store" { name ="existing-artifact-store" type ="artifact_store" flavor ="gcp" configuration = { path ="gs://${google_storage_bucket.ml_artifacts.name}" } connector_id = zenml_service_connector.gcp_connector.id}
# outputs.tfoutput "stack_id" { description ="ID of the created ZenML stack" value = zenml_stack.gcp_stack.id}output "stack_name" { description ="Name of the created ZenML stack" value = zenml_stack.gcp_stack.name}output "artifact_store_path" { description ="GCS path for artifacts" value ="${google_storage_bucket.artifacts.name}/artifacts"}output "container_registry_uri" { description ="URI of the container registry" value = "${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.containers.repository_id}"
}
Step 4: terraform.tfvars Configuration
Create a terraform.tfvars file (remember to never commit this to version control):