Register a cloud stack

Seamlessly register a cloud stack by using existing infrastructure

In ZenML, the stack is a fundamental concept that represents the configuration of your infrastructure. In a normal workflow, creating a stack requires you to first deploy the necessary pieces of infrastructure and then define them as stack components in ZenML with proper authentication.

Especially in a remote setting, this process can be challenging and time-consuming, and it may create multi-faceted problems. This is why we implemented a feature called the stack wizard, that allows you to browse through your existing infrastructure and use it to register a ZenML cloud stack.

If you do not have the required infrastructure pieces already deployed on your cloud, you can also use the 1-click deployment tool to build your cloud stack.

Alternatively, if you prefer to have more control over where and how resources are provisioned in your cloud, you can use one of our Terraform modules to manage your infrastructure as code yourself.

How to use the Stack Wizard?

The stack wizard is available to you by both our CLI and our dashboard.

If you are using the dashboard, the stack wizard is available through the stacks page.

The new stacks page

Here you can click on "+ New Stack" and choose the option "Use existing Cloud".

New stack options

Next, you have to select the cloud provider that you want to work with.

Stack Wizard Cloud Selection

Choose one of the possible authentication methods based on your provider and fill in the required fields.

Wizard Example
AWS: Authentication methods

If you select aws as your cloud provider, and you haven't selected a connector or declined auto-configuration, you will be prompted to select an authentication method for your cloud connector.

Available authentication methods for AWS
                  Available authentication methods for AWS                   
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Choice  ┃ Name                           ┃ Required                       ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [0]     │ AWS Secret Key                 │ aws_access_key_id  (AWS Access │
│         │                                │ Key ID)                        │
│         │                                │ aws_secret_access_key  (AWS    │
│         │                                │ Secret Access Key)             │
│         │                                │ region  (AWS Region)           │
│         │                                │                                │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [1]     │ AWS STS Token                  │ aws_access_key_id  (AWS Access │
│         │                                │ Key ID)                        │
│         │                                │ aws_secret_access_key  (AWS    │
│         │                                │ Secret Access Key)             │
│         │                                │ aws_session_token  (AWS        │
│         │                                │ Session Token)                 │
│         │                                │ region  (AWS Region)           │
│         │                                │                                │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [2]     │ AWS IAM Role                   │ aws_access_key_id  (AWS Access │
│         │                                │ Key ID)                        │
│         │                                │ aws_secret_access_key  (AWS    │
│         │                                │ Secret Access Key)             │
│         │                                │ region  (AWS Region)           │
│         │                                │ role_arn  (AWS IAM Role ARN)   │
│         │                                │                                │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [3]     │ AWS Session Token              │ aws_access_key_id  (AWS Access │
│         │                                │ Key ID)                        │
│         │                                │ aws_secret_access_key  (AWS    │
│         │                                │ Secret Access Key)             │
│         │                                │ region  (AWS Region)           │
│         │                                │                                │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [4]     │ AWS Federation Token           │ aws_access_key_id  (AWS Access │
│         │                                │ Key ID)                        │
│         │                                │ aws_secret_access_key  (AWS    │
│         │                                │ Secret Access Key)             │
│         │                                │ region  (AWS Region)           │
│         │                                │                                │
└─────────┴────────────────────────────────┴────────────────────────────────┘
GCP: Authentication methods

If you select gcp as your cloud provider, and you haven't selected a connector or declined auto-configuration, you will be prompted to select an authentication method for your cloud connector.

Available authentication methods for GCP
                  Available authentication methods for GCP                   
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Choice  ┃ Name                           ┃ Required                       ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [0]     │ GCP User Account               │ user_account_json  (GCP User   │
│         │                                │ Account Credentials JSON       │
│         │                                │ optionally base64 encoded.)    │
│         │                                │ project_id  (GCP Project ID    │
│         │                                │ where the target resource is   │
│         │                                │ located.)                      │
│         │                                │                                │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [1]     │ GCP Service Account            │ service_account_json  (GCP     │
│         │                                │ Service Account Key JSON       │
│         │                                │ optionally base64 encoded.)    │
│         │                                │                                │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [2]     │ GCP External Account           │ external_account_json  (GCP    │
│         │                                │ External Account JSON          │
│         │                                │ optionally base64 encoded.)    │
│         │                                │ project_id  (GCP Project ID    │
│         │                                │ where the target resource is   │
│         │                                │ located.)                      │
│         │                                │                                │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [3]     │ GCP Oauth 2.0 Token            │ token  (GCP OAuth 2.0 Token)   │
│         │                                │ project_id  (GCP Project ID    │
│         │                                │ where the target resource is   │
│         │                                │ located.)                      │
│         │                                │                                │
├─────────┼────────────────────────────────┼────────────────────────────────┤
│ [4]     │ GCP Service Account            │ service_account_json  (GCP     │
│         │ Impersonation                  │ Service Account Key JSON       │
│         │                                │ optionally base64 encoded.)    │
│         │                                │ target_principal  (GCP Service │
│         │                                │ Account Email to impersonate)  │
│         │                                │                                │
└─────────┴────────────────────────────────┴────────────────────────────────┘
Azure: Authentication methods

If you select azure as your cloud provider, and you haven't selected a connector or declined auto-configuration, you will be prompted to select an authentication method for your cloud connector.

Available authentication methods for Azure
    Available authentication methods for AZURE                         
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Choice ┃ Name                    ┃ Required                           ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ [0]    │ Azure Service Principal │ client_secret  (Service principal  │
│        │                         │ client secret)                     │
│        │                         │ tenant_id  (Azure Tenant ID)       │
│        │                         │ client_id  (Azure Client ID)       │
│        │                         │                                    │
├────────┼─────────────────────────┼────────────────────────────────────┤
│ [1]    │ Azure Access Token      │ token  (Azure Access Token)        │
│        │                         │                                    │
└────────┴─────────────────────────┴────────────────────────────────────┘

From this step forward, ZenML will show you different selections of resources that you can use from your existing infrastructure so that you can create the required stack components such as an artifact store, an orchestrator, and a container registry.

There you have it! Through the wizard, you just registered a cloud stack and, you can start running your pipelines on a remote setting.

ZenML Scarf

Last updated

Was this helpful?