Amazon Simple Cloud Storage (S3)

Storing artifacts in an AWS S3 bucket.

The S3 Artifact Store is an Artifact Store flavor provided with the S3 ZenML integration that uses the AWS S3 managed object storage service or one of the self-hosted S3 alternatives, such as MinIO or Ceph RGW, to store artifacts in an S3 compatible object storage backend.

When would you want to use it?

Running ZenML pipelines with the local Artifact Store is usually sufficient if you just want to evaluate ZenML or get started quickly without incurring the trouble and the cost of employing cloud storage services in your stack. However, the local Artifact Store becomes insufficient or unsuitable if you have more elaborate needs for your project:

if you want to share your pipeline run results with other team members or stakeholders inside or outside your organization
if you have other components in your stack that are running remotely (e.g. a Kubeflow or Kubernetes Orchestrator running in a public cloud).
if you outgrow what your local machine can offer in terms of storage space and need to use some form of private or public storage service that is shared with others
if you are running pipelines at scale and need an Artifact Store that can handle the demands of production-grade MLOps

In all these cases, you need an Artifact Store that is backed by a form of public cloud or self-hosted shared object storage service.

You should use the S3 Artifact Store when you decide to keep your ZenML artifacts in a shared object storage and if you have access to the AWS S3 managed service or one of the S3 compatible alternatives (e.g. Minio, Ceph RGW). You should consider one of the other Artifact Store flavors if you don't have access to an S3-compatible service.

How do you deploy it?

Would you like to skip ahead and deploy a full ZenML cloud stack already, including an S3 Artifact Store? Check out the in-browser stack deployment wizard, the stack registration wizard, or the ZenML AWS Terraform module for a shortcut on how to deploy & register this stack component.

The S3 Artifact Store flavor is provided by the S3 ZenML integration, you need to install it on your local machine to be able to register an S3 Artifact Store and add it to your stack:

zenml integration install s3 -y

The only configuration parameter mandatory for registering an S3 Artifact Store is the root path URI, which needs to point to an S3 bucket and take the form s3://bucket-name. Please read the documentation relevant to the S3 service that you are using on how to create an S3 bucket. For example, the AWS S3 documentation is available here.

With the URI to your S3 bucket known, registering an S3 Artifact Store and using it in a stack can be done as follows:

# Register the S3 artifact-store
zenml artifact-store register s3_store -f s3 --path=s3://bucket-name

# Register and set a stack with the new artifact store
zenml stack register custom_stack -a s3_store ... --set

Depending on your use case, however, you may also need to provide additional configuration parameters pertaining to authentication or pass advanced configuration parameters to match your S3-compatible service or deployment scenario.

Authentication Methods

Integrating and using an S3-compatible Artifact Store in your pipelines is not possible without employing some form of authentication. If you're looking for a quick way to get started locally, you can use the Implicit Authentication method. However, the recommended way to authenticate to the AWS cloud platform is through an AWS Service Connector. This is particularly useful if you are configuring ZenML stacks that combine the S3 Artifact Store with other remote stack components also running in AWS.

This method uses the implicit AWS authentication available in the environment where the ZenML code is running. On your local machine, this is the quickest way to configure an S3 Artifact Store. You don't need to supply credentials explicitly when you register the S3 Artifact Store, as it leverages the local credentials and configuration that the AWS CLI stores on your local machine. However, you will need to install and set up the AWS CLI on your machine as a prerequisite, as covered in the AWS CLI documentation, before you register the S3 Artifact Store.

Certain dashboard functionality, such as visualizing or deleting artifacts, is not available when using an implicitly authenticated artifact store together with a deployed ZenML server because the ZenML server will not have permission to access the filesystem.

The implicit authentication method also needs to be coordinated with other stack components that are highly dependent on the Artifact Store and need to interact with it directly to work. If these components are not running on your machine, they do not have access to the local AWS CLI configuration and will encounter authentication failures while trying to access the S3 Artifact Store:

Orchestrators need to access the Artifact Store to manage pipeline artifacts
Step Operators need to access the Artifact Store to manage step-level artifacts
Model Deployers need to access the Artifact Store to load served models

To enable these use-cases, it is recommended to use an AWS Service Connector to link your S3 Artifact Store to the remote S3 bucket.

To set up the S3 Artifact Store to authenticate to AWS and access an S3 bucket, it is recommended to leverage the many features provided by the AWS Service Connector such as auto-configuration, best security practices regarding long-lived credentials and fine-grained access control and reusing the same credentials across multiple stack components.

If you don't already have an AWS Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command. You have the option to configure an AWS Service Connector that can be used to access more than one S3 bucket or even more than one type of AWS resource:

zenml service-connector register --type aws -i

A non-interactive CLI example that leverages the AWS CLI configuration on your local machine to auto-configure an AWS Service Connector targeting a single S3 bucket is:

zenml service-connector register <CONNECTOR_NAME> --type aws --resource-type s3-bucket --resource-name <S3_BUCKET_NAME> --auto-configure

Example Command Output

$ zenml service-connector register s3-zenfiles --type aws --resource-type s3-bucket --resource-id s3://zenfiles --auto-configure
⠸ Registering service connector 's3-zenfiles'...
Successfully registered service connector `s3-zenfiles` with access to the following resources:
┏━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠───────────────┼────────────────┨
┃ 📦 s3-bucket  │ s3://zenfiles  ┃
┗━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛

Note: Please remember to grant the entity associated with your AWS credentials permissions to read and write to your S3 bucket as well as to list accessible S3 buckets. For a full list of permissions required to use an AWS Service Connector to access one or more S3 buckets, please refer to the AWS Service Connector S3 bucket resource type documentation or read the documentation available in the interactive CLI commands and dashboard. The AWS Service Connector supports many different authentication methods with different levels of security and convenience. You should pick the one that best fits your use case.

If you already have one or more AWS Service Connectors configured in your ZenML deployment, you can check which of them can be used to access the S3 bucket you want to use for your S3 Artifact Store by running e.g.:

zenml service-connector list-resources --resource-type s3-bucket

Example Command Output

The following 's3-bucket' resources can be accessed by service connectors that you have configured:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃             CONNECTOR ID             │ CONNECTOR NAME       │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES                                 ┃
┠──────────────────────────────────────┼──────────────────────┼────────────────┼───────────────┼────────────────────────────────────────────────┨
┃ aeed6507-f94c-4329-8bc2-52b85cd8d94d │ aws-s3               │ 🔶 aws         │ 📦 s3-bucket  │ s3://zenfiles                                  ┃
┠──────────────────────────────────────┼──────────────────────┼────────────────┼───────────────┼────────────────────────────────────────────────┨
┃ 9a810521-ef41-4e45-bb48-8569c5943dc6 │ aws-implicit         │ 🔶 aws         │ 📦 s3-bucket  │ s3://sagemaker-studio-907999144431-m11qlsdyqr8 ┃
┃                                      │                      │                │               │ s3://sagemaker-studio-d8a14tvjsmb              ┃
┠──────────────────────────────────────┼──────────────────────┼────────────────┼───────────────┼────────────────────────────────────────────────┨
┃ 37c97fa0-fa47-4d55-9970-e2aa6e1b50cf │ aws-secret-key       │ 🔶 aws         │ 📦 s3-bucket  │ s3://zenfiles                                  ┃
┃                                      │                      │                │               │ s3://zenml-demos                               ┃
┃                                      │                      │                │               │ s3://zenml-generative-chat                     ┃
┃                                      │                      │                │               │ s3://zenml-public-datasets                     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

After having set up or decided on an AWS Service Connector to use to connect to the target S3 bucket, you can register the S3 Artifact Store as follows:

# Register the S3 artifact-store and reference the target S3 bucket
zenml artifact-store register <S3_STORE_NAME> -f s3 \
    --path='s3://your-bucket'

# Connect the S3 artifact-store to the target bucket via an AWS Service Connector
zenml artifact-store connect <S3_STORE_NAME> -i

A non-interactive version that connects the S3 Artifact Store to a target S3 bucket through an AWS Service Connector:

zenml artifact-store connect <S3_STORE_NAME> --connector <CONNECTOR_ID>

Example Command Output

$ zenml artifact-store connect s3-zenfiles --connector s3-zenfiles
Successfully connected artifact store `s3-zenfiles` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃             CONNECTOR ID             │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼───────────────┼────────────────┨
┃ c4ee3f0a-bc69-4c79-9a74-297b2dd47d50 │ s3-zenfiles    │ 🔶 aws         │ 📦 s3-bucket  │ s3://zenfiles  ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛

As a final step, you can use the S3 Artifact Store in a ZenML Stack:

# Register and set a stack with the new artifact store
zenml stack register <STACK_NAME> -a <S3_STORE_NAME> ... --set

When you register the S3 Artifact Store, you can generate an AWS access key, store it in a ZenML Secret and then reference it in the Artifact Store configuration.

This method has some advantages over the implicit authentication method:

you don't need to install and configure the AWS CLI on your host
you don't need to care about enabling your other stack components (orchestrators, step operators, and model deployers) to have access to the artifact store through IAM roles and policies
you can combine the S3 artifact store with other stack components that are not running in AWS

Note: When you create the IAM user for your AWS access key, please remember to grant the created IAM user permissions to read and write to your S3 bucket (i.e. at a minimum: s3:PutObject, s3:GetObject, s3:ListBucket, s3:DeleteObject, s3:GetBucketVersioning, s3:ListBucketVersions, s3:DeleteObjectVersion)

After having set up the IAM user and generated the access key, as described in the AWS documentation, you can register the S3 Artifact Store as follows:

# Store the AWS access key in a ZenML secret
zenml secret create s3_secret \
    --aws_access_key_id='<YOUR_S3_ACCESS_KEY_ID>' \
    --aws_secret_access_key='<YOUR_S3_SECRET_KEY>'

# Register the S3 artifact-store and reference the ZenML secret
zenml artifact-store register s3_store -f s3 \
    --path='s3://your-bucket' \
    --authentication_secret=s3_secret

# Register and set a stack with the new artifact store
zenml stack register custom_stack -a s3_store ... --set

Advanced Configuration

The S3 Artifact Store accepts a range of advanced configuration options that can be used to further customize how ZenML connects to the S3 storage service that you are using. These are accessible via the client_kwargs, config_kwargs and s3_additional_kwargs configuration attributes and are passed transparently to the underlying S3Fs library:

client_kwargs: arguments that will be transparently passed to the botocore client . You can use it to configure parameters like endpoint_url and region_name when connecting to an S3-compatible endpoint (e.g. Minio).
config_kwargs: advanced parameters passed to botocore.client.Config.
s3_additional_kwargs: advanced parameters that are used when calling S3 API, typically used for things like ServerSideEncryption and ACL.

To include these advanced parameters in your Artifact Store configuration, pass them using JSON format during registration, e.g.:

zenml artifact-store register minio_store -f s3 \
    --path='s3://minio_bucket' \
    --authentication_secret=s3_secret \
    --client_kwargs='{"endpoint_url": "http://minio.cluster.local:9000", "region_name": "us-east-1"}'

For more, up-to-date information on the S3 Artifact Store implementation and its configuration, you can have a look at the SDK docs .

How do you use it?

Aside from the fact that the artifacts are stored in an S3 compatible backend, using the S3 Artifact Store is no different than using any other flavor of Artifact Store.

PreviousLocal Artifact Store NextGoogle Cloud Storage (GCS)

Last updated 23 days ago