Trigger a pipeline from REST API

Trigger a pipeline from the rest API.

This is a ZenML Pro only feature. Please sign up here get access. OSS users can only trigger a pipeline by calling the pipeline function inside their runner script.

Triggering a pipeline from the REST API only works with pipelines that are already built with a remote stack (i.e. at least a remote orchestrator, artifact store, and container registry). Most of the times, this means you can only re-run a pipeline with a remote stack using a different configuration.

As a pre-requisite, you need a pipeline name and version. After you have it, there are three calls that need to be made in order to trigger a pipeline from the REST API:

  1. GET /pipelines?name=<PIPELINE_NAME>&version=<PIPELINE_VERSION> -> This returns a response, where a <PIPELINE_ID> can be copied

  2. GET /pipeline_builds?pipeline_id=<PIPELINE_ID> -> This returns a list of responses where a <BUILD_ID> can be chosen

  3. POST /pipeline_builds/<BUILD_ID>/runs -> This runs the pipeline. You can pass the PipelineRunConfiguration in the body

The above set of REST calls means that you can only trigger a pipeline that has been built before on a remote stack.

A worked example

Learn how to get a bearer token for the curl commands here.

Here is an example. Let's say would we like to re-run a pipeline called training. We first query the /pipelines endpoint:

curl -X 'GET' \
  '<YOUR_ZENML_SERVER_URL>/api/v1/pipelines?hydrate=false&name=training&version=45' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer <YOUR_TOKEN>'

We can take the ID from any object in the list of responses. In this case, the <PIPELINE_ID> is c953985e-650a-4cbf-a03a-e49463f58473 in the response.

After this, we take the pipeline ID and call the /pipeline_builds?pipeline_id=<PIPELINE_ID> API:

curl -X 'GET' \
  '<YOUR_ZENML_SERVER_URL>/api/v1/pipeline_builds?hydrate=false&logical_operator=and&page=1&size=20&pipeline_id=b826b714-a9b3-461c-9a6e-1bde3df3241d' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer <YOUR_TOKEN>'

We can now take the <BUILD_ID> from this response. Here it is b826b714-a9b3-461c-9a6e-1bde3df3241d.

Finally, we can use the build ID to trigger the pipeline with a different configuration:

curl -X 'POST' \
  '<YOUR_ZENML_SERVER_URL>/api/v1/pipeline_builds/b826b714-a9b3-461c-9a6e-1bde3df3241d/runs' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <YOUR_TOKEN>' \
  -d '{
  "steps": {"model_trainer": {"parameters": {"model_type": "rf"}}}
}'

The pipeline that you're triggering has to have been run previously on a remote stack. In other words, the functionality to trigger a pipeline from another only works when a Docker image has previously been built for that pipeline. In most cases this will be because you ran the pipeline already, but in some cases you might have built the image separately.

A positive response means your pipeline has been re-triggered with a different config!

Last updated