Registering Existing Infrastructure with ZenML - A Guide for Terraform Users
Terraform is a powerful tool for managing infrastructure as code, and is by far the most popular IaC tool. Many companies already have existing Terraform setups, and it is often desirable to integrate ZenML with this setup.
We already got a glimpse on how to deploy a cloud stack with Terraform using existing Terraform modules that are maintained by the ZenML team. While this is a great solution for quickly getting started, it might not always be suitable for your use case.
This guide is for advanced users who want to manage their own custom Terraform code but want to use ZenML to manage their stacks. For this, the ZenML provider is a better choice.
Understanding the Two-Phase Approach
When working with ZenML stacks, there are two distinct phases:
Infrastructure Deployment: Creating cloud resources (typically handled by platform teams)
ZenML Registration: Registering these resources as ZenML stack components
While our official modules (zenml-stack/aws, zenml-stack/gcp, zenml-stack/azure) handle both phases, you might already have infrastructure deployed. Let's explore how to register existing infrastructure with ZenML.
Phase 1: Infrastructure Deployment
You likely already have this handled in your existing Terraform configurations:
# Example of existing GCP infrastructure
resource "google_storage_bucket" "ml_artifacts" {
name = "company-ml-artifacts"
location = "US"
}
resource "google_artifact_registry_repository" "ml_containers" {
repository_id = "ml-containers"
format = "DOCKER"
}
Phase 2: ZenML Registration
Setup the ZenML Provider
First, configure the ZenML provider to communicate with your ZenML server:
terraform {
required_providers {
zenml = {
source = "zenml-io/zenml"
}
}
}
provider "zenml" {
# Configuration options will be loaded from environment variables:
# ZENML_SERVER_URL
# ZENML_API_KEY
}
# outputs.tf
output "stack_id" {
description = "ID of the created ZenML stack"
value = zenml_stack.gcp_stack.id
}
output "stack_name" {
description = "Name of the created ZenML stack"
value = zenml_stack.gcp_stack.name
}
output "artifact_store_path" {
description = "GCS path for artifacts"
value = "${google_storage_bucket.artifacts.name}/artifacts"
}
output "container_registry_uri" {
description = "URI of the container registry"
value = "${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.containers.repository_id}"
}
Step 4: terraform.tfvars Configuration
Create a terraform.tfvars file (remember to never commit this to version control):