In ZenML, the stack is a fundamental concept that represents the configuration of your infrastructure. In a normal workflow, creating a stack requires you to first deploy the necessary pieces of infrastructure and then define them as stack components in ZenML with proper authentication.
Especially in a remote setting, this process can be challenging and time-consuming, and it may create multi-faceted problems. This is why we implemented a feature called the stack wizard, that allows you to browse through your existing infrastructure and use it to register a ZenML cloud stack.
Alternatively, if you prefer to have more control over where and how resources are provisioned in your cloud, you can use one of our Terraform modules to manage your infrastructure as code yourself.
How to use the Stack Wizard?
The stack wizard is available to you by both our CLI and our dashboard.
If you are using the dashboard, the stack wizard is available through the stacks page.
Here you can click on "+ New Stack" and choose the option "Use existing Cloud".
Next, you have to select the cloud provider that you want to work with.
Choose one of the possible authentication methods based on your provider and fill in the required fields.
AWS: Authentication methods
There are several different methods to authenticate with AWS:
On the dashboard, the stack wizard only works with AWS. We are working on bringing support to GCP and Azure as well. If you would like to use these providers, you can still use the CLI or stay tuned for further updates.
From this step forward, ZenML will show you different selections of resources that you can use from your existing infrastructure so that you can create the required stack components such as an artifact store, an orchestrator, and a container registry.
In order to register a remote stack over the CLI with the stack wizard, you can use the following command:
zenmlstackregister<STACK_NAME>-p{aws|gcp|azure}
To register the cloud stack, the first thing that the wizard needs is a service connector. You can either use an existing connector by providing its ID or name -sc <SERVICE_CONNECTOR_ID_OR_NAME> (CLI-Only) or the wizard will create one for you.
Similar to the service connector, if you use the CLI, you can also use existing stack components. However, this is only possible if these components are already configured with the same service connector that you provided through the parameter described above.
Define Service Connector
As the very first step the configuration wizard will check if the selected cloud provider credentials can be acquired automatically from the local environment. If the credentials are found, you will be offered to use them or proceed to manual configuration.
Example prompt for AWS auto-configuration
AWS cloud service connector has detected connection
credentials in your environment.
Would you like to use these credentials or create a new
configuration by providing connection details? [y/n] (y):
If you decline auto-configuration next you might be offered the list of already created service connectors available on the server: pick one of them and proceed or pick 0 to create a new one.
AWS: Authentication methods
If you select aws as your cloud provider, and you haven't selected a connector or declined auto-configuration, you will be prompted to select an authentication method for your cloud connector.
If you select gcp as your cloud provider, and you haven't selected a connector or declined auto-configuration, you will be prompted to select an authentication method for your cloud connector.
If you select azure as your cloud provider, and you haven't selected a connector or declined auto-configuration, you will be prompted to select an authentication method for your cloud connector.
