How to set up a minimal stack on Amazon Web Services (AWS)
This is an older version of the ZenML documentation. To read and view the latest version please visit this up-to-date URL.
Getting started with AWS
To get started using ZenML on the cloud, you need some basic infrastructure up and running which ZenML can use to run your pipelines. This step-by-step guide explains how to set up a basic cloud stack on AWS.
This guide represents one of many ways to create a cloud stack on AWS. You can customize this by adding additional components of replacing one of the components described in this guide.
All the AWS setup steps can either be done using the AWS UI or CLI. Simply select the tab for your preferred option and let's get started. First open up a terminal which we'll use to store some values along the way which we'll need to configure our ZenML stack later.
Select a descriptive name and a region. Let's also store these values in our terminal:
REGION=<REGION># for example us-west-1S3_BUCKET_NAME=<S3_BUCKET_NAME>
# Set a name for your bucket and the AWS region for your resources# Select one of the region codes for <REGION>: https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints
REGION=<REGION>S3_BUCKET_NAME=zenml-artifact-storeawss3apicreate-bucket--bucket=$S3_BUCKET_NAME \--region=$REGION \--create-bucket-configuration=LocationConstraint=$REGION
Select your new database and note down its endpoint:
RDS_MYSQL_ENDPOINT=<RDS_MYSQL_ENDPOINT>
Click on the active VPC security group, select Inbound rules and click on Edit inbound rules
Add a new rule with type MYSQL/Aurora and source Anywhere-IPv4. (Note: You can also restrict this to more limited IP address ranges or security groups if you want to limit access to your database.)
Go back to your database page and click on Modify in the top right.
In the Connectivity section, open the Advanced configuration and enable public access.
# Set values for the database identifier and username/password to access itMYSQL_DATABASE_ID=zenml-metadata-storeRDS_MYSQL_USERNAME=adminRDS_MYSQL_PASSWORD=<RDS_MYSQL_PASSWORD>awsrdscreate-db-instance--engine=mysql \--db-instance-class=db.t3.micro \--allocated-storage=20 \--publicly-accessible \--db-instance-identifier=$MYSQL_DATABASE_ID \--region=$REGION \--master-username=$RDS_MYSQL_USERNAME \--master-user-password=$RDS_MYSQL_PASSWORD# Wait until the database is createdawsrdswaitdb-instance-available--db-instance-identifier=$MYSQL_DATABASE_ID \--region=$REGION# Fetch the endpointRDS_MYSQL_ENDPOINT=$(awsrdsdescribe-db-instances--query='DBInstances[0].Endpoint.Address' \--output=text \--db-instance-identifier=$MYSQL_DATABASE_ID \--region=$REGION)# Fetch the security group idSECURITY_GROUP_ID=$(awsrdsdescribe-db-instances--query='DBInstances[0].VpcSecurityGroups[0].VpcSecurityGroupId' \--output=text --db-instance-identifier=$MYSQL_DATABASE_ID \ --region=$REGION)awsec2authorize-security-group-ingress \--protocol=tcp \--port=3306 \--cidr=0.0.0.0/0 \--group-id=$SECURITY_GROUP_ID \--region=$REGION
Follow this guide to create an Amazon EKS cluster role. We'll refer to this role as the cluster role in following steps.
Follow this guide to create an Amazon EC2 node role. We'll refer to this role as the node role in following steps.
Go to the IAM website, select Roles and edit the node role role.
Click on Add permissions and select Attach policies.
Attach the SecretsManagerReadWrite, and AmazonS3FullAccess policies to the role. The node role should now have the following attached policies: AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_Policy, SecretsManagerReadWrite and AmazonS3FullAccess.
After all of this setup, you're now ready to run any ZenML pipeline on AWS!
Quick setup
If you're looking for a way to get started quickly, we've combined all the commands so you can copy-paste them and execute them in a single go. You'll only need to set values for the <REGION> and <RDS_MYSQL_PASSWORD> right at the beginning before executing the rest.
Quick setup commands
# Select one of the region codes for <REGION>: https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints
REGION=<REGION># Choose a secure password for your database admin account. Make sure it includes:# - at least 8 printable ASCII characters# - no slash, single or double quotes or @ signsRDS_MYSQL_PASSWORD=<RDS_MYSQL_PASSWORD># Other parameters (we've set some defaults for these but feel free to change them):S3_BUCKET_NAME=zenml-artifact-storeMYSQL_DATABASE_ID=zenml-metadata-storeRDS_MYSQL_USERNAME=adminEKS_CLUSTER_NAME=zenml-eks-clusterNODEGROUP_NAME=zenml-eks-cluster-nodesEKS_ROLE_NAME=ZenMLEKSRoleEC2_ROLE_NAME=ZenMLEKSNodeRoleawss3apicreate-bucket--bucket=$S3_BUCKET_NAME \--region=$REGION \--create-bucket-configuration=LocationConstraint=$REGIONawsrdscreate-db-instance--engine=mysql \--db-instance-class=db.t3.micro \--allocated-storage=20 \--publicly-accessible \--db-instance-identifier=$MYSQL_DATABASE_ID \--region=$REGION \--master-username=$RDS_MYSQL_USERNAME \--master-user-password=$RDS_MYSQL_PASSWORD# Wait until the database is createdawsrdswaitdb-instance-available--db-instance-identifier=$MYSQL_DATABASE_ID \--region=$REGION# Fetch the endpointRDS_MYSQL_ENDPOINT=$(awsrdsdescribe-db-instances--query='DBInstances[0].Endpoint.Address' \--output=text \--db-instance-identifier=$MYSQL_DATABASE_ID \--region=$REGION)# Fetch the security group idSECURITY_GROUP_ID=$(awsrdsdescribe-db-instances--query='DBInstances[0].VpcSecurityGroups[0].VpcSecurityGroupId' \--output=text --db-instance-identifier=$MYSQL_DATABASE_ID \ --region=$REGION)awsec2authorize-security-group-ingress \--protocol=tcp \--port=3306 \--cidr=0.0.0.0/0 \--group-id=$SECURITY_GROUP_ID \--region=$REGIONawsecrcreate-repository--repository-name=zenml-kubernetes--region=$REGIONREGISTRY_ID=$(awsecrdescribe-registry--region=$REGION--query=registryId--output=text)ECR_URI="$REGISTRY_ID.dkr.ecr.$REGION.amazonaws.com"EKS_POLICY_JSON='{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "eks.amazonaws.com" }, "Action": "sts:AssumeRole" } ]}'awsiamcreate-role \--role-name=$EKS_ROLE_NAME \--assume-role-policy-document="$EKS_POLICY_JSON"awsiamattach-role-policy \--policy-arn='arn:aws:iam::aws:policy/AmazonEKSClusterPolicy' \--role-name=$EKS_ROLE_NAMEEC2_POLICY_JSON='{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ]}'awsiamcreate-role \--role-name=$EC2_ROLE_NAME \--assume-role-policy-document="$EC2_POLICY_JSON"awsiamattach-role-policy \--policy-arn='arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy' \--role-name=$EC2_ROLE_NAMEawsiamattach-role-policy \--policy-arn='arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly' \--role-name=$EC2_ROLE_NAMEawsiamattach-role-policy \--policy-arn='arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy' \--role-name=$EC2_ROLE_NAMEawsiamattach-role-policy \--policy-arn='arn:aws:iam::aws:policy/SecretsManagerReadWrite' \--role-name=$EC2_ROLE_NAMEawsiamattach-role-policy \--policy-arn='arn:aws:iam::aws:policy/AmazonS3FullAccess' \--role-name=$EC2_ROLE_NAME# Get the role ARN'sEKS_ROLE_ARN=$(awsiamget-role--role-name=$EKS_ROLE_NAME--query='Role.Arn' --output=text)EC2_ROLE_ARN=$(awsiamget-role--role-name=$EC2_ROLE_NAME--query='Role.Arn' --output=text)# Get default VPC IDVPC_ID=$(awsec2describe-vpcs--filters='Name=is-default,Values=true' \--query='Vpcs[0].VpcId' \--output=text \--region=$REGION)# Get subnet IDsSUBNET_IDS=$(awsec2describe-subnets--region=$REGION \--filters="Name=vpc-id,Values=$VPC_ID" \--query='Subnets[*].SubnetId' \--output=json)awsekscreate-cluster--region=$REGION \--name=$EKS_CLUSTER_NAME \--role-arn=$EKS_ROLE_ARN \--resources-vpc-config="{\"subnetIds\": $SUBNET_IDS}"# Wait until the cluster is activeawsekswaitcluster-active--name=$EKS_CLUSTER_NAME \--region=$REGIONawsekscreate-nodegroup--region=$REGION \--cluster-name=$EKS_CLUSTER_NAME \--nodegroup-name=$NODEGROUP_NAME \--node-role=$EC2_ROLE_ARN \--subnets="$SUBNET_IDS"# Wait until the node group is activeawsekswaitnodegroup-active--cluster-name=$EKS_CLUSTER_NAME \--nodegroup-name=$NODEGROUP_NAME \--region=$REGION# ZenML stack setupzenmlartifact-storeregisters3_store \--flavor=s3 \--path=s3://$S3_BUCKET_NAMEzenmlcontainer-registryregisterecr_registry \--flavor=aws \--uri=$ECR_URIawsecrget-login-password--region $REGION |dockerlogin--usernameAWS--password-stdin $ECR_URIzenmlmetadata-storeregisterrds_mysql \--flavor=mysql \--database=zenml \--secret=rds_authentication \--host=$RDS_MYSQL_ENDPOINTzenmlsecrets-managerregisteraws_secrets_manager \--flavor=aws \--region_name=$REGIONawseks--region=$REGIONupdate-kubeconfig--name=$EKS_CLUSTER_NAMEkubectlcreatenamespacezenmlzenmlorchestratorregistereks_kubernetes_orchestrator \--flavor=kubernetes \--kubernetes_context=$(kubectlconfigcurrent-context)zenmlstackregisterkubernetes_stack \-oeks_kubernetes_orchestrator \-as3_store \-mrds_mysql \-cecr_registry \-xaws_secrets_manager \--setzenmlsecrets-managersecretregisterrds_authentication \--schema=mysql \--user=$RDS_MYSQL_USERNAME \--password=$RDS_MYSQL_PASSWORD