Set Up a Minimal MLOps Stack on AWS
How to set up a minimal stack on Amazon Web Services (AWS)
To get started using ZenML on the cloud, you need some basic infrastructure up and running which ZenML can use to run your pipelines. This step-by-step guide explains how to set up a basic cloud stack on AWS.
This guide represents one of many ways to create a cloud stack on AWS. You can customize this by adding additional components of replacing one of the components described in this guide.

Prerequisites

  • Docker installed and running.
  • kubectl installed.
  • The AWS CLI installed and authenticated.
  • ZenML and the integrations for this tutorial stack installed:
    pip install zenml
    zenml integration install aws s3 kubernetes

Setting up the AWS resources

All the AWS setup steps can either be done using the AWS UI or CLI. Simply select the tab for your preferred option and let's get started. First open up a terminal which we'll use to store some values along the way which we'll need to configure our ZenML stack later.

Artifact Store (S3 bucket)

AWS UI
AWS CLI
  • Go to the S3 website.
  • Click on Create bucket.
  • Select a descriptive name and a region. Let's also store these values in our terminal:
    REGION=<REGION> # for example us-west-1
    S3_BUCKET_NAME=<S3_BUCKET_NAME>
# Set a name for your bucket and the AWS region for your resources
# Select one of the region codes for <REGION>: https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints
REGION=<REGION>
S3_BUCKET_NAME=zenml-artifact-store
aws s3api create-bucket --bucket=$S3_BUCKET_NAME \
--region=$REGION \
--create-bucket-configuration=LocationConstraint=$REGION

Metadata Store (RDS MySQL database)

AWS UI
AWS CLI
  • Go to the RDS website.
  • Make sure the correct region is selected on the top right (this region must be the same for all following steps).
  • Click on Create database.
  • Select Easy Create, MySQL, Free tier and enter values for your database name, username and password.
  • Note down the username and password:
    RDS_MYSQL_USERNAME=<RDS_MYSQL_USERNAME>
    RDS_MYSQL_PASSWORD=<RDS_MYSQL_PASSWORD>
  • Wait until the deployment is finished.
  • Select your new database and note down its endpoint:
    RDS_MYSQL_ENDPOINT=<RDS_MYSQL_ENDPOINT>
  • Click on the active VPC security group, select Inbound rules and click on Edit inbound rules
  • Add a new rule with type MYSQL/Aurora and source Anywhere-IPv4. (Note: You can also restrict this to more limited IP address ranges or security groups if you want to limit access to your database.)
  • Go back to your database page and click on Modify in the top right.
  • In the Connectivity section, open the Advanced configuration and enable public access.
# Set values for the database identifier and username/password to access it
MYSQL_DATABASE_ID=zenml-metadata-store
RDS_MYSQL_USERNAME=admin
RDS_MYSQL_PASSWORD=<RDS_MYSQL_PASSWORD>
aws rds create-db-instance --engine=mysql \
--db-instance-class=db.t3.micro \
--allocated-storage=20 \
--publicly-accessible \
--db-instance-identifier=$MYSQL_DATABASE_ID \
--region=$REGION \
--master-username=$RDS_MYSQL_USERNAME \
--master-user-password=$RDS_MYSQL_PASSWORD
# Wait until the database is created
aws rds wait db-instance-available --db-instance-identifier=$MYSQL_DATABASE_ID \
--region=$REGION
# Fetch the endpoint
RDS_MYSQL_ENDPOINT=$(aws rds describe-db-instances --query='DBInstances[0].Endpoint.Address' \
--output=text \
--db-instance-identifier=$MYSQL_DATABASE_ID \
--region=$REGION)
# Fetch the security group id
SECURITY_GROUP_ID=$(aws rds describe-db-instances --query='DBInstances[0].VpcSecurityGroups[0].VpcSecurityGroupId' \
--output=text
--db-instance-identifier=$MYSQL_DATABASE_ID \
--region=$REGION)
aws ec2 authorize-security-group-ingress \
--protocol=tcp \
--port=3306 \
--cidr=0.0.0.0/0 \
--group-id=$SECURITY_GROUP_ID \
--region=$REGION

Container Registry (ECR)

AWS UI
AWS CLI
  • Go to the ECR website.
  • Make sure the correct region is selected on the top right.
  • Click on Create repository.
  • Create a private repository called zenml-kubernetes with default settings.
  • Note down the URI of your registry:
    # This should be the prefix of your just created repository URI,
    # e.g. 714803424590.dkr.ecr.eu-west-1.amazonaws.com
    ECR_URI=<ECR_URI>
aws ecr create-repository --repository-name=zenml-kubernetes --region=$REGION
REGISTRY_ID=$(aws ecr describe-registry --region=$REGION --query=registryId --output=text)
ECR_URI="$REGISTRY_ID.dkr.ecr.$REGION.amazonaws.com"

Orchestrator (EKS)

AWS UI
AWS CLI
  • Follow this guide to create an Amazon EKS cluster role. We'll refer to this role as the cluster role in following steps.
  • Follow this guide to create an Amazon EC2 node role. We'll refer to this role as the node role in following steps.
  • Go to the IAM website, select Roles and edit the node role role.
  • Click on Add permissions and select Attach policies.
  • Attach the SecretsManagerReadWrite, and AmazonS3FullAccess policies to the role. The node role should now have the following attached policies: AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_Policy, SecretsManagerReadWrite and AmazonS3FullAccess.
  • Go to the EKS website.
  • Make sure the correct region is selected on the top right.
  • Click on Add cluster and select Create.
  • Enter a name and select the cluster role for Cluster service role.
  • Keep the default values for the networking and logging steps and create the cluster.
  • Note down the cluster name:
    EKS_CLUSTER_NAME=<EKS_CLUSTER_NAME>
  • After the cluster is created, select it and click on Add node group in the Compute tab.
  • Enter a name and select the node role.
  • Keep all other default values and create the node group.
# Choose names for your EKS cluster, node group and their respective roles
EKS_CLUSTER_NAME=zenml-eks-cluster
NODEGROUP_NAME=zenml-eks-cluster-nodes
EKS_ROLE_NAME=ZenMLEKSRole
EC2_ROLE_NAME=ZenMLEKSNodeRole
EKS_POLICY_JSON='{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "eks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'
aws iam create-role \
--role-name=$EKS_ROLE_NAME \
--assume-role-policy-document="$EKS_POLICY_JSON"
aws iam attach-role-policy \
--policy-arn='arn:aws:iam::aws:policy/AmazonEKSClusterPolicy' \
--role-name=$EKS_ROLE_NAME
EC2_POLICY_JSON='{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'
aws iam create-role \
--role-name=$EC2_ROLE_NAME \
--assume-role-policy-document="$EC2_POLICY_JSON"
aws iam attach-role-policy \
--policy-arn='arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy' \
--role-name=$EC2_ROLE_NAME
aws iam attach-role-policy \
--policy-arn='arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly' \
--role-name=$EC2_ROLE_NAME
aws iam attach-role-policy \
--policy-arn='arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy' \
--role-name=$EC2_ROLE_NAME
aws iam attach-role-policy \
--policy-arn='arn:aws:iam::aws:policy/SecretsManagerReadWrite' \
--role-name=$EC2_ROLE_NAME
aws iam attach-role-policy \
--policy-arn='arn:aws:iam::aws:policy/AmazonS3FullAccess' \
--role-name=$EC2_ROLE_NAME
# Get the role ARN's
EKS_ROLE_ARN=$(aws iam get-role --role-name=$EKS_ROLE_NAME --query='Role.Arn' --output=text)
EC2_ROLE_ARN=$(aws iam get-role --role-name=$EC2_ROLE_NAME --query='Role.Arn' --output=text)
# Get default VPC ID
VPC_ID=$(aws ec2 describe-vpcs --filters='Name=is-default,Values=true' \
--query='Vpcs[0].VpcId' \
--output=text \
--region=$REGION)
# Get subnet IDs
SUBNET_IDS=$(aws ec2 describe-subnets --region=$REGION \
--filters="Name=vpc-id,Values=$VPC_ID" \
--query='Subnets[*].SubnetId' \
--output=json)
aws eks create-cluster --region=$REGION \
--name=$EKS_CLUSTER_NAME \
--role-arn=$EKS_ROLE_ARN \
--resources-vpc-config="{\"subnetIds\": $SUBNET_IDS}"
# Wait until the cluster is active
aws eks wait cluster-active --name=$EKS_CLUSTER_NAME \
--region=$REGION
aws eks create-nodegroup --region=$REGION \
--cluster-name=$EKS_CLUSTER_NAME \
--nodegroup-name=$NODEGROUP_NAME \
--node-role=$EC2_ROLE_ARN \
--subnets="$SUBNET_IDS"
# Wait until the node group is active
aws eks wait nodegroup-active --cluster-name=$EKS_CLUSTER_NAME \
--nodegroup-name=$NODEGROUP_NAME \
--region=$REGION

Register the ZenML stack

  • Register the artifact store:
    zenml artifact-store register s3_store \
    --flavor=s3 \
    --path=s3://$S3_BUCKET_NAME
  • Register the container registry and authenticate your local docker client
    zenml container-registry register ecr_registry \
    --flavor=aws \
    --uri=$ECR_URI
    aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ECR_URI
  • Register the metadata store:
    zenml metadata-store register rds_mysql \
    --flavor=mysql \
    --database=zenml \
    --secret=rds_authentication \
    --host=$RDS_MYSQL_ENDPOINT
  • Register the secrets manager:
    zenml secrets-manager register aws_secrets_manager \
    --flavor=aws \
    --region_name=$REGION
  • Configure your kubectl client and register the orchestrator:
    aws eks --region=$REGION update-kubeconfig --name=$EKS_CLUSTER_NAME
    kubectl create namespace zenml
    zenml orchestrator register eks_kubernetes_orchestrator \
    --flavor=kubernetes \
    --kubernetes_context=$(kubectl config current-context)
  • Register the ZenML stack and activate it:
    zenml stack register kubernetes_stack \
    -o eks_kubernetes_orchestrator \
    -a s3_store \
    -m rds_mysql \
    -c ecr_registry \
    -x aws_secrets_manager \
    --set
  • Register the secret for authenticating with your MySQL database:
    zenml secret register rds_authentication \
    --schema=mysql \
    --user=$RDS_MYSQL_USERNAME \
    --password=$RDS_MYSQL_PASSWORD
After all of this setup, you're now ready to run any ZenML pipeline on AWS!

Quick setup

If you're looking for a way to get started quickly, we've combined all the commands so you can copy-paste them and execute them in a single go. You'll only need to set values for the <REGION> and <RDS_MYSQL_PASSWORD> right at the beginning before executing the rest.
Quick setup commands