Keep your pipelines and dashboard clean
Learn how to keep your pipeline runs clean during development.
When developing pipelines, it's common to run and debug them multiple times. To avoid cluttering the server with these development runs, ZenML provides several options:
Run locally
One of the easiest ways to avoid cluttering a shared server / dashboard is to disconnect your client from the remote server and simply spin up a local server:
Note that there are some limitations to this approach, particularly if you want to use remote infrastructure, but if there are local runs that you can do without the need for remote infrastructure, this can be a quick and easy way to keep things clean. When you're ready to reconnect to the server to continue with your shared runs, you can simply run zenml login <remote-url>
again.
Pipeline Runs
Unlisted Runs
Pipeline runs can be created without being explicitly associated with a pipeline by passing the unlisted
parameter when running a pipeline:
Unlisted runs are not displayed on the pipeline's page in the dashboard (though they are displayed in the pipeline run section), keeping the pipeline's history clean and focused on the pipelines that matter most.
Deleting Pipeline Runs
If you want to delete a specific pipeline run, you can use a script like this:
If you want to delete all pipeline runs in the last 24 hours, for example, you could run a script like this:
For different time ranges you can update this as appropriate.
Pipelines
Deleting Pipelines
Pipelines that are no longer needed can be deleted using the command:
This allows you to start fresh with a new pipeline, removing all previous runs associated with the deleted pipeline. This is a slightly more drastic approach, but it can sometimes be useful to keep the development environment clean.
Unique Pipeline Names
Pipelines can be given unique names each time they are run to uniquely identify them. This helps differentiate between multiple iterations of the same pipeline during development.
By default ZenML generates names automatically based on the current date and time, but you can pass in a run_name
when defining the pipeline:
Note that pipeline names must be unique. For more information on this feature, see the documentation on naming pipeline runs.
Models
Models are something that you have to explicitly register or pass in as you define your pipeline, so to run a pipeline without it being attached to a model is fairly straightforward: simply don't do the things specified in our documentation on registering models.
In order to delete a model or a specific model version, you can use the CLI or Python SDK to accomplish this. As an example, to delete all versions of a model, you can use:
See the full documentation on how to delete models.
Artifacts
Pruning artifacts
If you want to delete artifacts that are no longer referenced by any pipeline runs, you can use the following CLI command:
By default, this method deletes artifacts physically from the underlying artifact store AND also the entry in the database. You can control this behavior by using the --only-artifact
and --only-metadata
flags.
For more information, see the documentation for this artifact pruning feature.
Cleaning your environment
As a more drastic measure, the zenml clean
command can be used to start from scratch on your local machine. This will:
delete all pipelines, pipeline runs and associated metadata
delete all artifacts
There is also a --local
flag that you can set if you want to delete local files relating to the active stack. Note that zenml clean
does not delete artifacts and pipelines on the server; it only deletes the local data and metadata.
By utilizing these options, you can maintain a clean and organized pipeline dashboard, focusing on the runs that matter most for your project.
Last updated