Track Metadata in the Cloud
Use a MySQL database to track your metadata non-locally
While the local SQLite-based Metadata Store is a great default to get you started quickly, you will eventually need to move towards a deployed, shared and scalable database, for example once you switch to remote orchestration. This database will ideally be accessible both from your local machine, your orchestrator and the individual worker nodes of your orchestrator. A deployed MySQL Database can tick all these boxes.

Registering the MySQL Metadata Store

For the purpose of these docs it is assumed that the MySQL database is set up already. In order to connect to your MySQL instance there are a few parameters that you will need to register a Metadata Store stack component.
In all cases you will need to set these three fields:
  • host - The host IP address is the public IP address of your MySQL instance
  • port - The port at which to reach the MySQL instance (default is 3306)
  • database - The name of the database that will be used
For authentication, you'll have the choice of setting username and password directly to a stack component (not recommended for production setting), or you'll be able to register a secret in your secrets manager that allows you to also supply SSL certificates for more secure connections.
Basic Authentication
Advanced Authentication
This is not recommended for production settings as the password is easily accessible on your machine in clear text and communication with the database is unencrypted.
1
# Register the mysql-metadata-store
2
zenml metadata-store register mysql_metadata_store --flavor=mysql \
3
--host=`<DB_HOST_IP>` --port=`<DB_PORT>` --database=`<DB_NAME>` \
4
--username=`<DB_USERNAME>` --password`<DB_PASSWORD>`
5
6
# Register and set a stack with the new metadata store
7
zenml stack register cloud_stack -m mysql_metadata_store ... --set
Copied!
For this option you will need to make sure to create and download all three SSL certificates so that you can add them to a secret within the secrets manager of the stack that the MySQL Metadata Store is a part of.
1
# Register the mysql-metadata-store
2
zenml metadata-store register mysql_metadata_store --flavor=mysql \
3
--host=`<DB_HOST_IP>` --port=`<DB_PORT>` --database=`<DB_NAME>` \
4
--secret=mysql_secret
5
6
# Register a Secrets Manager
7
zenml secrets-manager register cloud_secrets_manager \
8
--flavor=`<FLAVOR_OF_YOUR_CHOIC>` ...
9
10
# Register and set a stack with the new metadata store and secret manager
11
zenml stack register cloud_stack -m mysql_metadata_store ... \
12
-x cloud_secrets_manager --set
13
14
zenml secret register mysql_secret --schema=mysql \
15
--user=`<DB_USERNAME>` --password=`<DB_PASSWORD>` \
16
--ssl_ca=@`</PATH/TO/DOWNLOADED/SERVER-CA>` \
17
--ssl_cert=@`</PATH/TO/DOWNLOADED/CLIENT-CERT>` \
18
--ssl_key=@`</PATH/TO/DOWNLOADED/CLIENT-KEY>`
Copied!
Depending on the cloud provider that you are running on, you will need to make sure the different stack components have the correct permissions and roles to access one another. The orchestrator in particular will need to have access to the secrets manager.
Export as PDF
Copy link
Edit on GitHub