Deploy a cloud stack

Deploy a cloud stack from scratch with a single click

In ZenML, the stack is a fundamental concept that represents the configuration of your infrastructure. In a normal workflow, creating a stack requires you to first deploy the necessary pieces of infrastructure and then define them as stack components in ZenML with proper authentication.

Especially in a remote setting, this process can be challenging and time-consuming, and it may create multi-faceted problems. This is why we implemented a feature, that allows you to deploy the necessary pieces of infrastructure on your selected cloud provider and get you started on remote stack with a single click.

If you have the required infrastructure pieces already deployed on your cloud, you can also use the stack wizard to seamlessly register your stack.

How to use the 1-click deployment tool?

The first thing that you need in order to use this feature is a deployed instance of ZenML (not a local server via zenml up). If you do not already have it set up for you, feel free to learn how to do so here.

Once you are connected to your deployed ZenML instance, you can use the 1-click deployment tool either through the dashboard or the CLI:

In order to create a remote stack over the dashboard go to the stacks page on the dashboard and click "+ New Stack".

Since we will be deploying it from scratch, select "New Infrastructure" on the next page:

Currently, the 1-click deployment only works on AWS and GCP. We are working on bringing support to Azure as well. Stay in touch for further updates.

AWS

If you choose aws as your provider, you will see a page where you will have to select a region and a name for your new stack:

Once the configuration is finished, you will see a deployment page:

Clicking on the "Deploy in AWS" button will redirect you to a Cloud Formation page on AWS Console.

You will have to log in to your AWS account, review and confirm the pre-filled configuration and create the stack.

GCP

If you choose gcp as your provider, you will see a page where you will have to select a region and a name for your new stack:

Once the configuration is finished, you will see a deployment page:

Make note of the configuration values provided to you in the ZenML dashboard. You will need these in the next step.

Clicking on the "Deploy in GCP" button will redirect you to a Cloud Shell session on GCP.

The Cloud Shell session will warn you that the ZenML GitHub repository is untrusted. We recommend that you review the contents of the repository and then check the Trust repo checkbox to proceed with the deployment, otherwise the Cloud Shell session will not be authenticated to access your GCP projects. You will also get a chance to review the scripts that will be executed in the Cloud Shell session before proceeding.

After the Cloud Shell session starts, you will be guided through the process of authenticating with GCP, configuring your deployment, and finally provisioning the resources for your new GCP stack using Deployment Manager.

First, you will be asked to create or choose an existing GCP project with billing enabled and to configure your terminal with the selected project:

Next, you will be asked to configure your deployment by pasting the configuration values that were provided to you earlier in the ZenML dashboard. You may need to switch back to the ZenML dashboard to copy these values if you did not do so earlier:

You can take this opportunity to review the script that will be executed at the next step. You will notice that this script starts by enabling some necessary GCP service APIs and configuring some basic permissions for the service accounts involved in the stack deployment, and then deploys the stack using a GCP Deployment Manager template. You can proceed with the deployment by running the script in your terminal:

The script will deploy a GCP Deployment Manager template that provisions the necessary resources for your new GCP stack and automatically registers the stack with your ZenML server. You can monitor the progress of the deployment in your GCP console:

Once the deployment is complete, you may close the Cloud Shell session and return to the ZenML dashboard to view the newly created stack:

What will be deployed?

Here is an overview of the infrastructure that the 1-click deployment will prepare for you based on your cloud provider:

Resources

  • An S3 bucket that will be used as a ZenML Artifact Store.

  • An ECR container registry that will be used as a ZenML Container Registry.

  • Permissions to use SageMaker as a ZenML Orchestrator.

  • An IAM user and IAM role with the minimum necessary permissions to access the resources listed above.

  • An AWS access key used to give access to ZenML to connect to the above resources through a ZenML service connector.

Permissions

The configured IAM service account and AWS access key will grant ZenML the following AWS permissions in your AWS account:

  • S3 Bucket:

    • s3:ListBucket

    • s3:GetObject

    • s3:PutObject

    • s3:DeleteObject

  • ECR Repository:

    • ecr:DescribeRepositories

    • ecr:ListRepositories

    • ecr:DescribeRegistry

    • ecr:BatchGetImage

    • ecr:DescribeImages

    • ecr:BatchCheckLayerAvailability

    • ecr:GetDownloadUrlForLayer

    • ecr:InitiateLayerUpload

    • ecr:UploadLayerPart

    • ecr:CompleteLayerUpload

    • ecr:PutImage

    • ecr:GetAuthorizationToken

  • SageMaker (Client):

    • sagemaker:CreatePipeline

    • sagemaker:StartPipelineExecution

    • sagemaker:DescribePipeline

    • sagemaker:DescribePipelineExecution

  • SageMaker (Jobs):

    • AmazonSageMakerFullAccess

There you have it! With a single click, you just deployed a cloud stack and, you can start running your pipelines on a remote setting.

Last updated