# Welcome to ZenML

Build production ML pipelines and production-grade AI agents with ZenML and Kitaru.

ZenML is an open-source framework for developing, evaluating, and deploying your entire AI portfolio: classical ML, LLM pipelines, and AI agents alike. It brings the battle-tested engineering principles of production ML to everything you ship, so you don't maintain one toolchain for models and another for agents.

That story has two open-source projects behind it:

* **ZenML** is the MLOps framework: portable, production-ready **pipelines** for ML and LLM workloads, with versioned artifacts, caching, and infrastructure abstracted behind [stacks](https://docs.zenml.io/stacks).
* [**Kitaru**](https://docs.zenml.io/kitaru) is for **AI agents**: traces you can run, not just read. Replay a real run against your real code with one thing changed — a cheaper model, a different prompt — diff the two, and roll the winner across recent runs. Recorded tool calls are answered from the recording, which is what makes the replay faithful.

Each works on its own. You can run ZenML and never touch Kitaru, or pick up Kitaru purely to put one agent's runs under replay. The split is clean: ZenML is for ML pipelines; Kitaru is for agents, with its own lightweight server (one `docker compose up`) and workers that execute replays in your environment.

### What are you building?

Pick the path that matches your work. Neither path requires the other, and adopting the second one later doesn't mean starting over.

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>ML Pipelines → ZenML</strong></td><td>Build, version, and deploy classical ML and LLM pipelines. Start with Hello World, then the Starter guide.</td><td><a href="/files/qdeGD5qx5b8tXE5MV5SE">/files/qdeGD5qx5b8tXE5MV5SE</a></td><td><a href="/pages/hMvaw3pJYHjAQ2mQuRGb">/pages/hMvaw3pJYHjAQ2mQuRGb</a></td></tr><tr><td><strong>AI Agents → Kitaru</strong></td><td>Record every run as a session, replay a real run against your real code with one change, and keep the version that wins on cost, latency, and quality. Start with the Kitaru quickstart.</td><td><a href="/files/YfFG4p4r8KCz7DiIUP1s">/files/YfFG4p4r8KCz7DiIUP1s</a></td><td><a href="https://docs.zenml.io/kitaru/getting-started/quickstart">https://docs.zenml.io/kitaru/getting-started/quickstart</a></td></tr></tbody></table>

### How these docs are organized

The documentation is split into spaces — the tabs at the top of this page. Knowing what lives where saves you a lot of searching:

| Space                                                                                                               | What you'll find there                                                                                                |
| ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| **ZenML** (you are here)                                                                                            | The pipelines framework: installation, core concepts, deployment, and how-to guides                                   |
| [**Kitaru**](https://docs.zenml.io/kitaru)                                                                          | The agents project: quickstart, recording and replaying sessions, evaluators and experiments, framework adapters      |
| [**Learn**](https://docs.zenml.io/user-guides)                                                                      | Narrative guides for both projects: Starter, Production, LLMOps, and Agents tracks, plus tutorials and best practices |
| [**Stacks**](https://docs.zenml.io/stacks)                                                                          | The infrastructure components — orchestrators, artifact stores, and more — that both pipelines and agents run on      |
| [**SDK reference**](https://docs.zenml.io/sdk-reference) / [**API reference**](https://docs.zenml.io/api-reference) | Client and REST API references, organized per project                                                                 |
| [**Changelog**](https://docs.zenml.io/changelog)                                                                    | Release notes, version by version                                                                                     |

### First steps

Whichever path you picked, the first steps are the same shape: install, run something real, then learn the concepts.

|               | ML Pipelines (ZenML)                            | AI Agents (Kitaru)                                                        |
| ------------- | ----------------------------------------------- | ------------------------------------------------------------------------- |
| **Install**   | [Installation](/getting-started/installation)   | [Installation](https://docs.zenml.io/kitaru/getting-started/installation) |
| **First run** | [Hello World](/getting-started/hello-world)     | [Quickstart](https://docs.zenml.io/kitaru/getting-started/quickstart)     |
| **Concepts**  | [Core Concepts](/getting-started/core-concepts) | [Core Concepts](https://docs.zenml.io/kitaru/concepts)                    |

If you use AI coding tools, see [LLM tooling](/reference/llms-txt) for ZenML's MCP server and Agent Skills (including `zenml-scoping` and `zenml-pipeline-authoring`).

### Guides

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Starter Guide</strong></td><td>Get started with ZenML fundamentals and set up your first pipeline</td><td><a href="/files/qyUjfZIIZ6gu6KN2d1i0">/files/qyUjfZIIZ6gu6KN2d1i0</a></td><td><a href="https://docs.zenml.io/user-guides/starter-guide">https://docs.zenml.io/user-guides/starter-guide</a></td></tr><tr><td><strong>Production Guide</strong></td><td>Move your ML pipelines from development to production</td><td><a href="/files/MdXKmxkxJ3qzq1imbM3J">/files/MdXKmxkxJ3qzq1imbM3J</a></td><td><a href="https://docs.zenml.io/user-guides/production-guide">https://docs.zenml.io/user-guides/production-guide</a></td></tr><tr><td><strong>Kitaru</strong></td><td>Replay a real run with one change, and improve agents across a cohort of real runs</td><td><a href="/files/YfFG4p4r8KCz7DiIUP1s">/files/YfFG4p4r8KCz7DiIUP1s</a></td><td><a href="https://docs.zenml.io/kitaru">https://docs.zenml.io/kitaru</a></td></tr></tbody></table>

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Installation

Installing ZenML and getting started.

{% stepper %}
{% step %}

#### Install ZenML

ZenML currently supports **Python 3.10, 3.11, 3.12, 3.13, and 3.14**. Please make sure that you are using a supported Python version.

{% tabs %}
{% tab title="Base package" %}
**ZenML** is a Python package that can be installed using `pip` or other Python package managers:

```shell
pip install zenml
```

{% hint style="warning" %}
Installing the base package only allows you to connect to a [deployed ZenML server](/deploying-zenml/deploying-zenml). If you want to use ZenML purely locally, install it with the `local` extra:

```shell
pip install 'zenml[local]'
```

{% endhint %}
{% endtab %}

{% tab title="Local Dashboard" %}
If you want to use the [ZenML dashboard](https://github.com/zenml-io/zenml-dashboard) locally, you need to install ZenML with the `server` extra:

```shell
pip install 'zenml[server]'
```

{% hint style="warning" %}
If you want to run a local server while running on a Mac with Apple Silicon (M1, M2, M3, M4), you should set the following environment variable:

```bash
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
```

You can read more about this [here](http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html).
{% endhint %}
{% endtab %}

{% tab title="Jupyter Notebooks" %}
If you write your ZenML pipelines ins Jupyter notebooks, we recommend installing ZenML with the `jupyter` extra which includes improved CLI output and logs:

```shell
pip install 'zenml[jupyter]'
```

{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}

#### Verifying Installations

Once the installation is completed, you can check whether the installation was successful either through Bash or Python:

{% tabs %}
{% tab title="Bash" %}

```bash
zenml version
```

{% endtab %}

{% tab title="Python" %}

```python
import zenml

print(zenml.__version__)
```

{% endtab %}
{% endtabs %}

If you would like to learn more about the current release, please visit our [PyPi package page.](https://pypi.org/project/zenml)
{% endstep %}
{% endstepper %}

## Running with Docker

`zenml` is also available as a Docker image hosted publicly on [DockerHub](https://hub.docker.com/r/zenmldocker/zenml). Use the following command to get started in a bash environment with `zenml` available:

```shell
docker run -it zenmldocker/zenml /bin/bash
```

If you would like to run the ZenML server with Docker:

```shell
docker run -it -d -p 8080:8080 zenmldocker/zenml-server
```

## Starting the local server

By default, ZenML runs without a server connected to a local database on your machine. If you want to access the dashboard locally, you need to start a local server:

```shell
# Make sure to have the `server` extra installed
pip install "zenml[server]"
zenml login --local  # opens the dashboard locally 
```

However, advanced ZenML features are dependent on a centrally deployed ZenML server accessible to other MLOps stack components. You can read more about it [here](/deploying-zenml/deploying-zenml). For the deployment of ZenML, you have the option to either [self-host](/deploying-zenml/deploying-zenml) it or register for a free [ZenML Pro](https://zenml.io/pro?utm_source=docs\&utm_medium=referral_link\&utm_campaign=cloud_promotion\&utm_content=signup_link) account.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Hello World

Your first ML pipeline with ZenML - from local development to cloud deployment in minutes.

This guide will help you build and deploy your first ZenML pipeline, starting locally and then transitioning to the cloud without changing your code. The same principles you'll learn here apply whether you're building classical ML models or AI agents.

{% stepper %}
{% step %}

#### Install ZenML

Start by installing ZenML in a fresh Python environment:

```bash
pip install 'zenml[server]'
zenml login
```

This gives you access to both the ZenML Python SDK and CLI tools. It also surfaces the ZenML dashboard + connects it to your local client.
{% endstep %}

{% step %}

#### Write your first pipeline

Create a simple `run.py` file with a basic workflow:

<pre class="language-python"><code class="lang-python">from zenml import step, pipeline


@step
def basic_step() -> str:
    """A simple step that returns a greeting message."""
    return "Hello World!"


@pipeline
def basic_pipeline() -> str:
    """A simple pipeline with just one step."""
    greeting = basic_step()
    return greeting


if __name__ == "__main__":
<strong>    basic_pipeline()
</strong></code></pre>

Run this pipeline in batch mode locally:

```bash
python run.py
```

You will see ZenML automatically tracks the execution and stores artifacts. View these on the CLI or on the dashboard.
{% endstep %}

{% step %}

#### Create a Pipeline Snapshot (Optional but Recommended)

Before deploying, you can create a **snapshot** - an immutable, reproducible version of your pipeline including code, configuration, and container images:

```bash
# Create a snapshot of your pipeline
zenml pipeline snapshot create run.basic_pipeline --name my_snapshot
```

Snapshots are powerful because they:

* **Freeze your pipeline state** - Ensure the exact same pipeline always runs
* **Enable parameterization** - Run the same snapshot with different inputs
* **Support team collaboration** - Share ready-to-use pipeline configurations
* **Integrate with automation** - Trigger from dashboards, APIs, or CI/CD systems

[Learn more about Snapshots](/concepts/snapshots)
{% endstep %}

{% step %}

#### Deploy your pipeline as a real-time service

ZenML can deploy your pipeline (or snapshot) as a persistent HTTP service for real-time inference:

```bash
# Deploy your pipeline directly
zenml pipeline deploy run.basic_pipeline --name my_deployment

# OR deploy a snapshot (if you created one above)
zenml pipeline snapshot deploy my_snapshot --deployment my_deployment
```

Your pipeline now runs as a production-ready service! This is perfect for serving predictions to web apps, powering AI agents, or handling real-time requests.

**Key insight**: When you deploy a pipeline directly with `zenml pipeline deploy`, ZenML automatically creates an implicit snapshot behind the scenes, ensuring reproducibility.

[Learn more about Pipeline Deployments](/concepts/deployment)
{% endstep %}

{% step %}

#### Set up a ZenML Server (For Remote Infrastructure)

To use remote infrastructure (cloud deployers, orchestrators, artifact stores), you need to deploy a ZenML server to manage your pipelines centrally. You can use [ZenML Pro](https://zenml.io/pro) (managed, 14-day free trial) or [deploy it yourself](/deploying-zenml/deploying-zenml) (self-hosted, open-source).

Connect your local environment:

```bash
zenml login
zenml project set <PROJECT_NAME>
```

Once connected, you'll have a centralized dashboard to manage infrastructure, collaborate with team members, and schedule pipeline runs.
{% endstep %}

{% step %}

#### Create your first remote stack (Optional)

A "stack" in ZenML represents the infrastructure where your pipelines run. You can now scale from local development to cloud infrastructure without changing any code.

<figure><img src="/files/0yqTLz3KIyBMxAgQgClo" alt="ZenML Stack Deployment Options"><figcaption><p>Stack deployment options</p></figcaption></figure>

Remote stacks can include:

* [**Remote Deployers**](https://docs.zenml.io/stacks/stack-components/deployers) ([AWS App Runner](https://docs.zenml.io/stacks/stack-components/deployers/aws-app-runner), [GCP Cloud Run](https://docs.zenml.io/stacks/stack-components/deployers/gcp-cloud-run), [Azure Container Instances](https://docs.zenml.io/stacks/stack-components/container-registries/azure)) - for deploying your pipelines as scalable HTTP services on the cloud
* [**Remote Orchestrators**](https://docs.zenml.io/stacks/stack-components/orchestrators) ([Kubernetes](https://docs.zenml.io/stacks/stack-components/orchestrators/kubernetes), [GCP Vertex AI](https://docs.zenml.io/stacks/stack-components/orchestrators/vertex), [AWS SageMaker](https://docs.zenml.io/stacks/stack-components/orchestrators/sagemaker)) - for running batch pipelines at scale
* [**Remote Artifact Stores**](https://docs.zenml.io/stacks/stack-components/artifact-stores) ([S3](https://docs.zenml.io/stacks/stack-components/artifact-stores/s3), [GCS](https://docs.zenml.io/stacks/stack-components/artifact-stores/gcp), [Azure Blob](https://docs.zenml.io/stacks/stack-components/artifact-stores/azure)) - for storing and versioning pipeline artifacts

The fastest way to create a cloud stack is through the **Infrastructure-as-Code** option, which uses Terraform to deploy cloud resources and register them as a ZenML stack.

You'll need:

* [Terraform](https://developer.hashicorp.com/terraform/install) version 1.9+ installed locally
* Authentication configured for your preferred cloud provider (AWS, GCP, or Azure)
* Appropriate permissions to create resources in your cloud account

```bash
# Create a remote stack using the deployment wizard
zenml stack register <STACK_NAME> \
  --deployer <DEPLOYER_NAME> \
  --orchestrator <ORCHESTRATOR_NAME> \
  --artifact-store <ARTIFACT_STORE_NAME>
```

The wizard will guide you through each step.
{% endstep %}

{% step %}

#### Deploy and run on remote infrastructure

Once you have a remote stack, you can:

1. **Deploy your service to the cloud** - Your deployment runs on managed cloud infrastructure:

```bash
zenml stack set <REMOTE_STACK_NAME>
zenml pipeline deploy run.basic_pipeline --name my_production_deployment
```

2. **Run batch pipelines at scale** - Use the same code with a cloud orchestrator:

```bash
zenml stack set <REMOTE_STACK_NAME>
python run.py  # Automatically runs on cloud infrastructure
```

ZenML handles packaging code, building containers, orchestrating execution, and tracking artifacts automatically across all cloud providers.

<figure><img src="/files/kXHKVEqKR9MkN1QmDwK3" alt="Pipeline Run in ZenML Dashboard"><figcaption><p>Your pipeline in the ZenML Pro Dashboard</p></figcaption></figure>
{% endstep %}

{% step %}

#### What's next?

Congratulations! You've just experienced the core value proposition of ZenML:

* **Write Once, Run Anywhere**: The same code runs locally during development and in the cloud for production
* **Unified Framework**: Use the same MLOps principles for both classical ML models and AI agents
* **Separation of Concerns**: Infrastructure configuration and ML code are completely decoupled, enabling independent evolution of each
* **Full Tracking**: Every run, artifact, and model is automatically versioned and tracked - whether it's a scikit-learn model or a multi-agent system

To continue your ZenML journey, explore these key topics:

**For All AI Workloads:**

* **AI-assisted pipeline building**: Use ZenML Agent Skills like `zenml-scoping` (architecture planning) and `zenml-pipeline-authoring` (implementation guidance) — see [LLM tooling](/reference/llms-txt)
* **Pipeline Development**: Discover advanced features like [scheduling](/concepts/steps_and_pipelines/advanced_features#scheduling) and [caching](/concepts/steps_and_pipelines/advanced_features#caching)
* **Artifact Management**: Learn how ZenML [stores, versions, and tracks your data](/concepts/artifacts) automatically
* **Organization**: Use [tags](/concepts/tags) and [metadata](/concepts/metadata) to keep your AI projects structured

**For LLMs and AI Agents:**

* **LLMOps Guide**: Write your [first AI pipeline](/getting-started/your-first-ai-pipeline) for agent development patterns
* **Deploying Agents**: To see an example of a deployed document extraction agent, see the [deploying agents](https://github.com/zenml-io/zenml/tree/main/examples/deploying_agent/README.md) example
* **Agent Outer Loop**: See the [Agent Outer Loop](https://github.com/zenml-io/zenml/tree/main/examples/agent_outer_loop/README.md) example to learn about training classifiers and improving agents through feedback loops
* **Agent Evaluation**: Learn to [systematically evaluate](https://github.com/zenml-io/zenml/tree/main/examples/agent_comparison/README.md) and compare different agent architectures
* **Prompt Management**: Version and track prompts, tools, and agent configurations as [artifacts](/concepts/artifacts)

**Infrastructure & Deployment:**

* **Containerization**: Understand how ZenML [handles containerization](/concepts/containerization) for reproducible execution
* **Stacks & Infrastructure**: Explore the concepts behind [stacks](/concepts/stack_components) and [service connectors](/concepts/service_connectors) for authentication
* **Secrets Management**: Learn how to [handle sensitive information](/concepts/secrets) securely
* **Snapshots**: Create [reusable snapshots](/concepts/snapshots) for standardized workflows
  {% endstep %}
  {% endstepper %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Your First AI Pipeline

Choose your path and build your first pipeline with ZenML in minutes.

### Your First AI Pipeline

ZenML pipelines work the same for **classical ML**, **AI agents**, and **hybrid approaches**. Choose your path below to get started:

{% hint style="info" %}
Why ZenML pipelines?

* **Reproducible & portable**: Run the same code locally or on the cloud by switching stacks.
* **One approach for models and agents**: Steps, pipelines, and artifacts work for sklearn, classical ML, and LLMs alike.
* **Observe by default**: Lineage and step metadata (e.g., latency, tokens, metrics) are tracked and visible in the dashboard.
  {% endhint %}

{% hint style="info" %}
**Plan and build pipelines with AI coding agents:** If you use Claude Code, Codex, Copilot, Cursor, or similar tools, ZenML provides Agent Skills that can scope your idea into a realistic multi-pipeline plan (`zenml-scoping`) and help you implement steps and pipelines (`zenml-pipeline-authoring`). See [LLM tooling](/reference/llms-txt) for installation instructions.
{% endhint %}

***

### What do you want to build?

Choose one of the paths below. The same ZenML pipeline pattern works for all of them—the difference is in your steps and how you orchestrate them.

* [**Build AI Agents**](#path-1-build-ai-agents) - Use LLMs and tools to create autonomous agents
* [**Build Classical ML Pipelines**](#path-2-build-classical-ml-pipelines) - Train and serve ML models with scikit-learn, TensorFlow, or PyTorch
* [**Build Hybrid Systems**](#path-3-build-hybrid-systems) - Combine ML classifiers with agents

***

### Path 1: Build AI Agents

Use large language models, prompts, and tools to build intelligent autonomous agents that can reason, take action, and interact with your systems.

{% hint style="info" %}
**Two ways to run agents.** This path runs your agent *inside a pipeline*, which is ideal for batch workloads and evaluation. If the agent itself runs in production, use [Kitaru](https://docs.zenml.io/kitaru): it records a run that actually happened as a replayable session, so you can replay it faithfully against your real code with one thing changed (a cheaper model, a different prompt), diff the two, and keep the version that wins. Recorded tool calls are answered from the session on replay, which is what makes it faithful — and safe. The split is clean: ZenML is for ML pipelines, Kitaru is for agents.
{% endhint %}

#### Architecture example

```mermaid
---
config:
  layout: elk
  theme: mc
---
flowchart TB
  U["CLI / curl / web UI"] --> D["ZenML Deployment<br/>(doc_analyzer)"]

  subgraph PIPE["Pipeline: doc_analyzer"]
    I["ingest_document_step"]
    A["analyze_document_step"]
    R["render_analysis_report_step"]
    I --> A --> R
  end

  D --> PIPE

  subgraph STACK["Stack"]
    OR[("Deployer")]
    AR[("Artifact Store")]
  end

  PIPE --> AR
  D --> OR
```

<details>

<summary><strong>View Quick Start &#x26; Examples</strong></summary>

#### Quick start

```bash
git clone --depth 1 https://github.com/zenml-io/zenml.git
cd zenml/examples/deploying_agent
uv pip install -r requirements.txt
```

Then follow the guide in [`examples/deploying_agent`](https://github.com/zenml-io/zenml/tree/main/examples/deploying_agent/README.md):

1. **Define your steps**: Use LLM APIs (OpenAI, Claude, etc.) to build reasoning steps
2. **Deploy as HTTP service**: Turn your agent into a managed endpoint
3. **Invoke and monitor**: Use the CLI, curl, or the embedded web UI to interact with your agent
4. **Inspect traces**: View agent reasoning, tool calls, and metadata in the ZenML dashboard

#### Example output

* Automated document analysis (see `deploying_agent`)
* Multi-turn chatbots with context
* Autonomous workflows with tool integrations
* Agentic RAG systems with retrieval steps

#### Related examples

* [**agent\_outer\_loop**](https://github.com/zenml-io/zenml/tree/main/examples/agent_outer_loop/README.md): Combine ML classifiers with agents for hybrid intelligent systems
* [**agentic\_hitl\_pipeline**](https://github.com/zenml-io/zenml/tree/main/examples/agentic_hitl_pipeline/README.md): Add dynamic fan-out and human approval to an agent workflow
* [**agent\_comparison**](https://github.com/zenml-io/zenml/tree/main/examples/agent_comparison/README.md): Compare different agent architectures and LLM providers
* [**agent\_framework\_integrations**](https://github.com/zenml-io/zenml/tree/main/examples/agent_framework_integrations/README.md): Integrate with popular agent frameworks
* [**llm\_finetuning**](https://github.com/zenml-io/zenml/tree/main/examples/llm_finetuning/README.md): Fine-tune LLMs for specialized tasks

</details>

***

### Path 2: Build Classical ML Pipelines

Use scikit-learn, TensorFlow, PyTorch, or other ML frameworks to build data processing, feature engineering, training, and inference pipelines.

#### Architecture example

```mermaid
---
config:
  layout: elk
  theme: mc
---
flowchart TB
  subgraph TRAIN["Training"]
    D["generate_churn_data"]
    T["train_churn_model"]
    D --> T
  end

  subgraph INFER["Inference"]
    P["predict_churn"]
  end

  U["Customer Features<br/>(curl / SDK)"] --> INFER

  subgraph STACK["Stack"]
    OR[("Orchestrator")]
    AR[("Artifact Store")]
    DE[("Deployer")]
  end

  TRAIN --> AR
  TRAIN --> OR
  INFER --> DE
  INFER --> AR
```

<details>

<summary><strong>View Quick Start &#x26; Examples</strong></summary>

#### Quick start

```bash
git clone --depth 1 https://github.com/zenml-io/zenml.git
cd zenml/examples/deploying_ml_model
uv pip install -r requirements.txt
```

Then follow the guide in [`examples/deploying_ml_model`](https://github.com/zenml-io/zenml/tree/main/examples/deploying_ml_model/README.md):

1. **Build your pipeline**: Data loading → preprocessing → training → evaluation
2. **Deploy the model**: Serve your trained model as a real-time HTTP endpoint
3. **Monitor performance**: Track predictions, latency, and data drift in the dashboard
4. **Iterate**: Retrain and redeploy without code changes—just switch your orchestrator

#### Example output

* Predictive models (regression, classification)
* Time series forecasting
* NLP pipelines (sentiment analysis, text classification)
* Computer vision workflows
* Model scoring and ranking systems

#### Related examples

* [**e2e**](https://github.com/zenml-io/zenml/tree/main/examples/e2e/README.md): End-to-end ML pipeline with data validation and model deployment
* [**e2e\_nlp**](https://github.com/zenml-io/zenml/tree/main/examples/e2e_nlp/README.md): Domain-specific NLP pipeline example
* [**mlops\_starter**](https://github.com/zenml-io/zenml/tree/main/examples/mlops_starter/README.md): Production-ready MLOps setup with monitoring and governance

</details>

***

### Path 3: Build Hybrid Systems

Combine classical ML models and AI agents in a single pipeline. For example, use a classifier to route requests to specialized agents, or use agents to augment ML predictions.

#### Architecture example

```mermaid
---
config:
  layout: elk
  theme: mc
---
flowchart TB
  U["Customer Input<br/>(curl / SDK)"] --> SA["Agent Service"]

  subgraph TRAIN["Training"]
    D["load_data"]
    T["train_classifier"]
    D --> T
  end

  subgraph SERVE["Serving"]
    C["classify_intent"]
    R["generate_response"]
    C --> R
  end

  SA --> SERVE

  subgraph STACK["Stack"]
    OR[("Orchestrator")]
    AR[("Artifact Store")]
    DE[("Deployer")]
  end

  TRAIN --> AR
  TRAIN --> OR
  SERVE --> AR
  SERVE --> DE
```

<details>

<summary><strong>View Quick Start &#x26; Examples</strong></summary>

#### Quick start

```bash
git clone --depth 1 https://github.com/zenml-io/zenml.git
cd zenml/examples/agent_outer_loop
uv pip install -r requirements.txt
```

Then follow the guide in [`examples/agent_outer_loop`](https://github.com/zenml-io/zenml/tree/main/examples/agent_outer_loop/README.md):

1. **Define both components**: Classical ML classifier + AI agent steps
2. **Wire them together**: Use the classifier output to influence agent behavior
3. **Deploy as one service**: The entire hybrid system becomes a single endpoint
4. **Monitor both**: Track ML metrics and agent traces in the same dashboard

#### Example output

* Intent classification with specialized agent handling
* Upgrade paths: generic agent → train classifier → automatic routing
* Ensemble systems combining multiple models and agents
* Fact-checking pipelines with verification steps

#### Related examples

* [**agent\_outer\_loop**](https://github.com/zenml-io/zenml/tree/main/examples/agent_outer_loop/README.md): Full hybrid example with automatic intent detection
* [**deploying\_agent**](https://github.com/zenml-io/zenml/tree/main/examples/deploying_agent/README.md): Start here for the agent piece
* [**deploying\_ml\_model**](https://github.com/zenml-io/zenml/tree/main/examples/deploying_ml_model/README.md): Start here for the ML piece

</details>

***

### Common Next Steps

Once you've chosen your path and gotten your first pipeline running:

#### Deploy remotely

All three paths use the same deployment pattern. Configure a remote stack and deploy:

```bash
# Create a remote stack (e.g., AWS)
zenml stack register my-remote-stack \
  --orchestrator aws-sagemaker \
  --artifact-store s3-bucket \
  --deployer aws

# Set it and deploy—your code doesn't change
zenml stack set my-remote-stack
```

Run in batch mode with:

```bash
python run.py
```

Deploy as a real-time endpoint with:

```bash
zenml pipeline deploy pipelines.my_pipeline.my_pipeline --config deploy_config.yaml
```

See [Deploying ZenML](/deploying-zenml/deploying-zenml) for cloud setup details.

#### View the dashboard

Start the dashboard to explore your pipeline runs:

```bash
zenml login
```

In the dashboard, you'll see:

* **Pipeline DAGs**: Visual representation of your steps and data flow
* **Artifacts**: Versioned outputs from each step (models, reports, traces)
* **Metadata**: Latency, tokens, metrics, or custom metadata you track
* **Timeline view**: Compare step durations and identify bottlenecks

### Core Concepts Recap

Regardless of which path you choose:

* [**Pipelines**](/concepts/steps_and_pipelines) - Orchestrate your workflow steps with automatic tracking
* [**Steps**](/concepts/steps_and_pipelines) - Modular, reusable units (data loading, model training, LLM inference, etc.)
* [**Artifacts**](/concepts/artifacts) - Versioned outputs (models, predictions, traces, reports) with automatic logging
* [**Stacks**](/concepts/stack_components) - Switch execution environments (local, remote, cloud) without code changes
* [**Deployments**](/concepts/deployment) - Turn pipelines into HTTP services with built-in UIs and monitoring

For deeper dives, explore the [Concepts](/concepts/steps_and_pipelines) section in the docs.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Core Concepts

Discovering the core concepts behind ZenML.

![A diagram of core concepts of ZenML OSS](/files/GYRS4UEmdV932nlC52as)

**ZenML** is a unified, extensible, open-source MLOps framework for creating portable, production-ready **MLOps pipelines**. It's built for data scientists, ML Engineers, and MLOps Developers to collaborate as they develop to production. By extending the battle-tested principles you rely on for classical ML to the new world of AI agents, ZenML serves as one platform to develop, evaluate, and deploy your entire AI portfolio - from decision trees to complex multi-agent systems. In order to achieve this goal, ZenML introduces various concepts for different aspects of ML workflows and AI agent development, and we can categorize these concepts under three different threads:

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td><mark style="color:purple;"><strong>1. Development</strong></mark></td><td>As a developer, how do I design my machine learning workflows?</td><td></td><td><a href="#1-development">#1-development</a></td><td><a href="/files/qfMWieEtYiiE5rToygPk">/files/qfMWieEtYiiE5rToygPk</a></td></tr><tr><td><mark style="color:purple;"><strong>2. Execution</strong></mark></td><td>While executing, how do my workflows utilize the large landscape of MLOps tooling/infrastructure?</td><td></td><td><a href="#2-execution">#2-execution</a></td><td><a href="/files/hAlxc8uYi7OafZaWhVtf">/files/hAlxc8uYi7OafZaWhVtf</a></td></tr><tr><td><mark style="color:purple;"><strong>3. Management</strong></mark></td><td>How do I establish and maintain a production-grade and efficient solution?</td><td></td><td><a href="#3-management">#3-management</a></td><td><a href="/files/JHhPPRaCv8jTmVjqXXEG">/files/JHhPPRaCv8jTmVjqXXEG</a></td></tr></tbody></table>

{% embed url="<https://www.youtube.com/embed/iCB4KNjl5vs>" %}
If you prefer visual learning, this short video demonstrates the key concepts covered below.
{% endembed %}

## 1. Development

First, let's look at the main concepts that play a role during the development stage of ML workflows and AI agent pipelines with ZenML.

#### Step

Steps are functions annotated with the `@step` decorator. The easiest one could look like this.

```python
from zenml import step

@step
def step_1() -> str:
    """Returns a string."""
    return "world"
```

These functions can also have inputs and outputs. For ZenML to work properly, these should preferably be typed.

```python
from zenml import step

@step(enable_cache=False)
def step_2(input_one: str, input_two: str) -> str:
    """Combines the two strings passed in."""
    combined_str = f"{input_one} {input_two}"
    return combined_str

@step
def evaluate_agent_response(prompt: str, test_query: str) -> dict:
    """Evaluates an AI agent's response to a test query."""
    response = call_llm_agent(prompt, test_query)
    return {"query": test_query, "response": response, "quality_score": 0.95}
```

#### Pipelines

At its core, ZenML follows a pipeline-based workflow for your projects. A **pipeline** consists of a series of **steps**, organized in any order that makes sense for your use case.

![Representation of a pipeline dag.](/files/cy6APDn1m7FGwWcxiow3)

As seen in the image, a step might use the outputs from a previous step and thus must wait until the previous step is completed before starting. This is something you can keep in mind when organizing your steps.

Pipelines and steps are defined in code using Python *decorators* or *classes*. This is where the core business logic and value of your work live, and you will spend most of your time defining these two things.

Even though pipelines are simple Python functions, you are only allowed to call steps within this function. The inputs for steps called within a pipeline can either be the outputs of previous steps or alternatively, you can pass in values directly or map them onto pipeline parameters (as long as they're JSON-serializable). Similarly, you can return values from a pipeline that are step outputs as long as they are JSON-serializable.

```python
from zenml import pipeline

@pipeline
def my_pipeline():
    output_step_one = step_1()
    step_2(input_one="hello", input_two=output_step_one)

@pipeline
def agent_evaluation_pipeline(query: str = "What is machine learning?") -> str:
    """An AI agent evaluation pipeline."""
    prompt = "You are a helpful assistant. Please answer: {query}"
    evaluation_result = evaluate_agent_response(prompt, query)
    return evaluation_result
```

{% hint style="info" %}
If you use an AI coding agent, ZenML Agent Skills can help you scope a pipeline architecture (`zenml-scoping`) and then implement steps and pipelines correctly (`zenml-pipeline-authoring`). See [LLM tooling](/reference/llms-txt).
{% endhint %}

Executing the Pipeline is as easy as calling the function that you decorated with the `@pipeline` decorator.

```python
if __name__ == "__main__":
    my_pipeline()
    agent_evaluation_pipeline(query="What is an LLM?")
```

#### Artifacts

Artifacts represent the data that goes through your steps as inputs and outputs, and they are automatically tracked and stored by ZenML in the artifact store. They are produced by and circulated among steps whenever your step returns an object or a value. This means the data is not passed between steps in memory. Rather, when the execution of a step is completed, they are written to storage, and when a new step gets executed, they are loaded from storage.

Artifacts can be traditional ML data (datasets, models, metrics) or AI agent components (prompt templates, agent configurations, evaluation results). The same artifact system seamlessly handles both use cases.

The serialization and deserialization logic of artifacts is defined by [Materializers](/concepts/artifacts/materializers).

#### Models

Models are used to represent the outputs of a training process along with all metadata associated with that output. In other words: models in ZenML are more broadly defined as the weights as well as any associated information. This includes traditional ML models (scikit-learn, PyTorch, etc.) and AI agent configurations (prompt templates, tool definitions, multi-agent system architectures). Models are first-class citizens in ZenML and as such viewing and using them is unified and centralized in the ZenML API, client, as well as on the [ZenML Pro](https://zenml.io/pro) dashboard.

#### Materializers

Materializers define how artifacts live in between steps. More precisely, they define how data of a particular type can be serialized/deserialized, so that the steps are able to load the input data and store the output data.

All materializers use the base abstraction called the `BaseMaterializer` class. While ZenML comes built-in with various implementations of materializers for different datatypes, if you are using a library or a tool that doesn't work with our built-in options, you can write [your own custom materializer](/concepts/artifacts/materializers) to ensure that your data can be passed from step to step.

#### Parameters & Settings

When we think about steps as functions, we know they receive input in the form of artifacts. We also know that they produce output (in the form of artifacts, stored in the artifact store). But steps also take parameters. The parameters that you pass into the steps are also (helpfully!) stored by ZenML. This helps freeze the iterations of your experimentation workflow in time, so you can return to them exactly as you run them. On top of the parameters that you provide for your steps, you can also use different `Setting`s to configure runtime configurations for your infrastructure and pipelines.

#### Model and model versions

ZenML exposes the concept of a `Model`, which consists of multiple different model versions. A model version represents a unified view of the ML models that are created, tracked, and managed as part of a ZenML project. Model versions link all other entities to a centralized view.

## 2. Execution

Once you have implemented your workflow by using the concepts described above, you can focus your attention on the execution of the pipeline run.

#### Stacks & Components

When you want to execute a pipeline run with ZenML, **Stacks** come into play. A **Stack** is a collection of **stack components**, where each component represents the respective configuration regarding a particular function in your MLOps pipeline, such as pipeline orchestration or deployment systems, artifact repositories and container registries.

Pipelines can be executed in two ways: in **batch mode** (traditional execution through an orchestrator) or in **online mode** (long-running HTTP servers that can be invoked via REST API calls). Deploying pipelines for online mode execution allows you to serve your ML workflows as real-time endpoints, making them accessible for live inference and interactive use cases.

For instance, if you take a close look at the default local stack of ZenML, you will see two components that are **required** in every stack in ZenML, namely an *orchestrator* and an *artifact store*. Additional components like *deployers* can be added to enable specific functionality such as deploying pipelines as HTTP endpoints.

![ZenML running code on the Local Stack.](/files/iE8LmDLsiIorG7aq8E6S)

{% hint style="info" %}
Keep in mind that each one of these components is built on top of base abstractions and is completely extensible.
{% endhint %}

#### Orchestrator

An **Orchestrator** is a workhorse that coordinates all the steps to run in a pipeline in batch mode. Since pipelines can be set up with complex combinations of steps with various asynchronous dependencies between them, the orchestrator acts as the component that decides what steps to run and when to run them.

ZenML comes with a default *local orchestrator* designed to run on your local machine. This is useful, especially during the exploration phase of your project. You don't have to rent a cloud instance just to try out basic things.

#### Artifact Store

An **Artifact Store** is a component that houses all data that passes through the pipeline as inputs and outputs. Each artifact that gets stored in the artifact store is tracked and versioned and this allows for extremely useful features like data caching, which speeds up your workflows.

Similar to the orchestrator, ZenML comes with a default *local artifact store* designed to run on your local machine. This is useful, especially during the exploration phase of your project. You don't have to set up a cloud storage system to try out basic things.

#### Deployer

A **Deployer** is a stack component that manages the deployment of pipelines as long-running HTTP servers useful for online mode execution. Unlike orchestrators that execute pipelines in batch mode, deployers can create and manage persistent services that wrap your pipeline in a web application, usually containerized, allowing it to be invoked through HTTP requests.

ZenML comes with a *Docker deployer* that can run deployments on your local machine as Docker containers, making it easy to test and develop real-time pipeline endpoints before moving to production infrastructure.

#### Flavor

ZenML provides a dedicated base abstraction for each stack component type. These abstractions are used to develop solutions, called **Flavors**, tailored to specific use cases/tools. With ZenML installed, you get access to a variety of built-in and integrated Flavors for each component type, but users can also leverage the base abstractions to create their own custom flavors.

#### Stack Switching

When it comes to production-grade solutions, it is rarely enough to just run your workflow locally without including any cloud infrastructure.

Thanks to the separation between the pipeline code and the stack in ZenML, you can easily switch your stack independently from your code. For instance, all it would take you to switch from an experimental local stack running on your machine to a remote stack that employs a full-fledged cloud infrastructure is a single CLI command.

#### Pipeline Snapshot

A **Pipeline Snapshot** is an immutable snapshot of your pipeline that includes the pipeline DAG, code, configuration, and container images. Snapshots can be run from the server or dashboard, and can also be [deployed](#deployment).

#### Pipeline Run

A **Pipeline Run** is a record of a pipeline execution. When you run a pipeline using an orchestrator, a pipeline run is created tracking information about the execution such as the status, the artifacts and metadata produced by the pipeline and all its steps. When a pipeline is deployed for online mode execution, a pipeline run is similarly created for every HTTP request made to it.

#### Deployment

A **Deployment** is a running instance of a pipeline deployed as an HTTP endpoint. When you deploy a pipeline using a deployer, it becomes a long-running service that can be invoked through REST API calls. Each HTTP request to a deployment triggers a new pipeline run, creating the same artifacts and metadata tracking as traditional batch pipeline executions. This enables real-time inference, interactive ML workflows, and seamless integration with web applications and external services.

## 3. Management

In order to benefit from the aforementioned core concepts to their fullest extent, it is essential to deploy and manage a production-grade environment that interacts with your ZenML installation.

#### ZenML Server

To use *stack components* that are running remotely on a cloud infrastructure, you need to deploy a [**ZenML Server**](https://docs.zenml.io/user-guides/production-guide/deploying-zenml) so it can communicate with these stack components and run your pipelines. The server is also responsible for managing ZenML business entities like pipelines, steps, models, etc.

![Visualization of the relationship between code and infrastructure.](/files/pjOpwRaO5QM3SvLt3f6u)

#### Server Deployment

In order to benefit from the advantages of using a deployed ZenML server, you can either choose to use the [**ZenML Pro SaaS offering**](https://docs.zenml.io/pro)**,** which provides a control plane for you to create managed instances of ZenML servers, or [deploy it in your self-hosted environment](/deploying-zenml/deploying-zenml).

#### Metadata Tracking

On top of the communication with the stack components, the **ZenML Server** also keeps track of all the bits of metadata around a pipeline run. With a ZenML server, you are able to access all of your previous experiments with the associated details. This is extremely helpful in troubleshooting.

#### Secrets

The **ZenML Server** also acts as a [centralized secrets store](/deploying-zenml/deploying-zenml/secret-management) that safely and securely stores sensitive data, such as credentials used to access the services that are part of your stack. It can be configured to use a variety of different backends for this purpose, such as the AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and Hashicorp Vault.

Secrets are sensitive data that you don't want to store in your code or configure alongside your stacks and pipelines. ZenML includes a [centralized secrets store](/deploying-zenml/deploying-zenml/secret-management) that you can use to store and access your secrets securely.

#### Collaboration

Collaboration is a crucial aspect of any MLOps team as they often need to bring together individuals with diverse skills and expertise to create a cohesive and effective workflow for machine learning projects and AI agent development. A successful MLOps team requires seamless collaboration between data scientists, engineers, and DevOps professionals to develop, train, deploy, and maintain both traditional ML models and AI agent systems.

With a deployed **ZenML Server**, users have the ability to create their own teams and project structures. They can easily share pipelines, runs, stacks, and other resources, streamlining the workflow and promoting teamwork across the entire AI development lifecycle.

#### Dashboard

The **ZenML Dashboard** also communicates with **the ZenML Server** to visualize your *pipelines*, *stacks*, and *stack components*. The dashboard serves as a visual interface to showcase collaboration with ZenML. You can invite *users* and share your stacks with them.

When you start working with ZenML, you'll start with a local ZenML setup, and when you want to transition, you will need to [deploy ZenML](/deploying-zenml/deploying-zenml). Don't worry though, there is a one-click way to do it, which we'll learn about later.

#### VS Code Extension

ZenML also provides a [VS Code extension](https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode) that allows you to interact with your ZenML stacks, runs, and server directly from your VS Code editor. If you're working on code in your editor, you can easily switch and inspect the stacks you're using, delete and inspect pipelines as well as even switch stacks.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# System Architecture

Different variations of the ZenML architecture depending on your needs.

This guide walks through the various ways that ZenML can be deployed, from self-hosted OSS to\
SaaS to self-hosted ZenML Pro!

## ZenML OSS (Self-hosted)

{% hint style="info" %}
This page is intended as a high-level overview. To learn more about how to deploy ZenML OSS, read [this guide](/deploying-zenml/deploying-zenml).
{% endhint %}

A ZenML OSS deployment consists of the following moving pieces:

* **ZenML OSS Server**: This is a FastAPI app that manages metadata of pipelines, artifacts, stacks, etc. Note: In ZenML Pro, the notion of a ZenML server is replaced with what is known as a "Workspace". For all intents and purposes, consider a ZenML Workspace to be a ZenML OSS server that comes with more functionality.
* **OSS Metadata Store**: This is where all ZenML workspace metadata is stored, including ML metadata such as tracking and versioning information about pipelines and models.
* **OSS Dashboard**: This is a ReactJS app that shows pipelines, runs, etc.
* **Secrets Store**: All secrets and credentials required to access customer infrastructure services are stored in a secure secrets store. The ZenML Pro API has access to these secrets and uses them to access customer infrastructure services on behalf of the ZenML Pro. The secrets store can be hosted either by the ZenML Pro or by the customer.

![ZenML OSS server deployment architecture](/files/vLV9sq58On9Kf3xoZj0S)

ZenML OSS is free with Apache 2.0 license. Learn how to deploy it [here](/deploying-zenml/deploying-zenml).

{% hint style="info" %}
To learn more about the core concepts for ZenML OSS, go [here](/getting-started/core-concepts).
{% endhint %}

## ZenML Pro (SaaS or Self-hosted)

{% hint style="info" %}
If you're interested in assessing ZenML Pro SaaS, you can create a [free account](https://zenml.io/pro?utm_source=docs\&utm_medium=referral_link\&utm_campaign=cloud_promotion\&utm_content=signup_link).

If you would like to self-host ZenML Pro, please [book a demo](https://zenml.io/book-a-demo).
{% endhint %}

The above deployment can be augmented with the ZenML Pro components:

* **ZenML Pro Control Plane**: This is the central controlling entity of all workspaces.
* **Pro Dashboard**: This is a dashboard that builds on top of the OSS dashboard and adds further functionality.
* **Pro Metadata Store**: This is a PostgreSQL database where all ZenML Pro-related metadata is stored, such as roles, permissions, teams, and workspace management-related data.
* **Pro Add-ons**: These are Python modules injected into the OSS Server for enhanced functionality.
* **Identity Provider**: ZenML Pro offers flexible authentication options. In cloud-hosted deployments, it integrates with [Auth0](https://auth0.com/), allowing users to log in via social media or corporate credentials. For self-hosted deployments, customers can configure their own identity management solution, with ZenML Pro supporting custom OIDC provider integration. This allows organizations to leverage their existing identity infrastructure for authentication and authorization, whether using the cloud service or deploying on-premises.

![ZenML Pro deployment architecture](/files/l8texmPSw5X8R7q0d3wN)

ZenML Pro offers many additional features to increase your team's productivity. No matter your specific needs, the hosting options for ZenML Pro range from easy SaaS integration to completely air-gapped deployments on your own infrastructure.

You might have noticed that this architecture builds on top of the ZenML OSS system architecture. Therefore, if you already have ZenML OSS deployed, it is easy to enroll it as part of a ZenML Pro deployment!

The above components interact with other MLOps stack components, secrets, and data in the following scenarios described below.

{% hint style="info" %}
To learn more about the core concepts for ZenML Pro, go [here](https://docs.zenml.io/pro/core-concepts)
{% endhint %}

### ZenML Pro SaaS Architecture

![ZenML Pro SaaS deployment with ZenML secret store](/files/z69w3B1O0qRAHpXkbqm9)

For the ZenML Pro SaaS deployment case, all ZenML services are hosted on infrastructure hosted by the ZenML Team. Customer secrets and credentials required to access customer infrastructure are stored and managed by the ZenML Pro Control Plane.

On the ZenML Pro infrastructure, only ML *metadata* (e.g. pipeline and model tracking and versioning information) is stored. All the actual ML data artifacts (e.g. data produced or consumed by pipeline steps, logs and visualizations, models) are stored on the customer cloud. This can be set up quite easily by configuring an [artifact store](https://docs.zenml.io/stacks/artifact-stores) with your MLOps stack.

Your workspace only needs permissions to read from this data to display artifacts on the ZenML dashboard. The workspace also needs direct access to parts of the customer infrastructure services to support dashboard control plane features such as CI/CD, triggering and running pipelines, triggering model deployments and so on.

The advantage of this setup is that it is a fully-managed service, and is very easy to get started with. However, for some clients, even some metadata can be sensitive; these clients should refer to the other architecture diagram.

### ZenML Pro Hybrid SaaS

![ZenML Pro self-hosted deployment](/files/K3bRKxtFztLReNE9mxhX)

The partially self-hosted architecture offers a balanced approach that combines the benefits of cloud-hosted control with on-premises data sovereignty. In this configuration, while the ZenML Pro control plane remains hosted by ZenML (handling user management, authentication, RBAC and global workspace coordination), all other components - including services, data, and secrets - are deployed within your own cloud infrastructure.

This hybrid model is particularly well-suited for organizations with:

* A centralized MLOps or Platform team responsible for standardizing ML practices
* Multiple business units or teams that require autonomy over their data and infrastructure
* Strict security requirements where workspaces must operate behind VPN/corporate firewalls
* Compliance requirements that mandate keeping sensitive data and ML artifact metadata within company infrastructure
* Need for customization of workspace configurations while maintaining centralized governance

The key advantages of this setup include:

* Simplified user management through the ZenML-hosted control plane
* Complete data sovereignty - sensitive data and ML artifacts remain within your infrastructure
* Secure networking - workspaces communicate through outbound-only connections via VPN/private networks
* Ability to customize and configure workspaces according to specific team needs
* Reduced operational overhead compared to fully self-hosted deployments
* Reduced maintenance burden - all control plane updates and maintenance are handled by ZenML This architecture strikes a balance between convenience and control, making it a popular choice for enterprises looking to standardize their MLOps practices while maintaining sovereignty.

### ZenML Pro Self-Hosted Architecture

![ZenML Pro self-hosted deployment](/files/EUIlNsvVKYeiC4hLlKz6)

In the case of self-hosting ZenML Pro, all services, data, and secrets are deployed on the customer\
cloud. This is meant for customers who require completely air-gapped deployments, for the tightest security standards. [Reach out to us](mailto:cloud@zenml.io) if you want to set this up.

<details>

<summary>Detailed Architecture Diagram for self-hosted ZenML Pro deployment</summary>

<img src="/files/v32jmgM25olBNURgnTuJ" alt="ZenML Pro self-hosted deployment details" data-size="original">

</details>

Are you interested in ZenML Pro? [Sign up](https://zenml.io/pro/?utm_source=docs\&utm_medium=referral_link\&utm_campaign=cloud_promotion\&utm_content=signup_link) and get access with a free trial now!

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>

## Data Implications Across Deployment Scenarios

| Deployment Scenario         | Data Location                                                                                                                               | Data Movement                                                                                                                                                        | Data Access                                                                                                                                                                                                                       | Data Isolation                                                                                                               |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **ZenML OSS (Self-hosted)** | All data remains on customer infrastructure: both ML metadata in OSS Metadata Store and actual ML data artifacts in customer Artifact Store | Data stays within customer boundary; moves between pipeline steps via the Orchestrator                                                                               | Accessible only through customer infrastructure; no ZenML-managed components have access                                                                                                                                          | Complete data isolation from ZenML-managed services                                                                          |
| **ZenML Pro SaaS**          | ML metadata in ZenML-hosted DB; Actual ML data artifacts in customer Artifact Store; Secrets in ZenML-managed Secret Store                  | Metadata flows to ZenML Pro Control Plane; ML data artifacts stay on customer infrastructure; ZenML services access customer infrastructure using stored credentials | ZenML Pro has access to the customer secrets that are explicitly stored; Workspace optionally needs read access to artifact store for dashboard display; No actual ML data moves to ZenML infrastructure unless explicitly shared | Only metadata and credentials are stored on ZenML infrastructure; actual ML data remains isolated on customer infrastructure |
| **ZenML Pro Hybrid SaaS**   | Control Plane on ZenML infrastructure; Workspace, DB, Secret Store, Orchestrator, and Artifact Store on customer infrastructure             | Only authentication/authorization data flows to ZenML; All ML data and metadata stays on customer infrastructure                                                     | ZenML Control Plane has limited access to user management data; No access to actual ML data or metadata; Customer maintains all data access controls                                                                              | Strong data isolation with only authentication events crossing boundary. Allows securing access via VPN/private networks.    |
| **ZenML Pro Self-Hosted**   | All components run on customer infrastructure                                                                                               | All data movement contained within customer infrastructure boundary                                                                                                  | No external access to any data; completely air-gapped operation possible                                                                                                                                                          | Complete data isolation; ZenML has no access to any customer data                                                            |


# Deploy

Why do we need to deploy ZenML?

![ZenML OSS server deployment architecture](/files/vLV9sq58On9Kf3xoZj0S)

Moving your ZenML Server to a production environment offers several benefits over staying local:

1. **Scalability**: Production environments are designed to handle large-scale workloads, allowing your models to process more data and deliver faster results.
2. **Reliability**: Production-grade infrastructure ensures high availability and fault tolerance, minimizing downtime and ensuring consistent performance.
3. **Collaboration**: A shared production environment enables seamless collaboration between team members, making it easier to iterate on models and share insights.

Despite these advantages, transitioning to production can be challenging due to the complexities involved in setting up the needed infrastructure.

## Components

A ZenML deployment consists of multiple infrastructure components:

* [FastAPI server](https://github.com/zenml-io/zenml/tree/main/src/zenml/zen_server/README.md) backed with a SQLite or MySQL database
* [Python Client](https://github.com/zenml-io/zenml/tree/main/src/zenml/README.md)
* An [open-source companion ReactJS](https://github.com/zenml-io/zenml-dashboard) dashboard
* (Optional) [ZenML Pro API + Database + ZenML Pro dashboard](/getting-started/system-architectures)

You can read more in-depth about the system architecture of ZenML [here](/getting-started/system-architectures).\
This documentation page will focus on the components required to deploy ZenML OSS.

<details>

<summary>Details on the ZenML Python Client</summary>

The ZenML client is a Python package that you can install on your machine. It is used to interact with the ZenML server. You can install it using the `pip` command as outlined [here](/getting-started/installation).

This Python package gives you [the `zenml` command-line interface](https://sdkdocs.zenml.io/latest/cli.html) which you can use to interact with the ZenML server for common tasks like managing stacks, setting up secrets, and so on. It also gives you the general framework that lets you [author and deploy pipelines](https://docs.zenml.io/user-guides/starter-guide) and so forth.

If you want to have more fine-grained control and access to the metadata that ZenML manages, you can use the Python SDK to access the API. This allows you to create your own custom automations and scripts and is the most common way teams access the metadata stored in the ZenML server. The full documentation for the Python SDK can be found [here](https://sdkdocs.zenml.io/latest/). The full HTTP [API documentation](https://docs.zenml.io/api-reference) can also be found by adding the`/doc` suffix to the URL when accessing your deployed ZenML server.

</details>

### Deployment scenarios

When you first get started with ZenML, you have the following architecture on your machine.

![ZenML default local configuration](/files/KI3wRplpwhZAyLliO8vl)

The SQLite database that you can see in this diagram is used to store information about pipelines, pipeline runs, stacks, and other configurations. This default setup allows you to get started and try out the core features, but you won't be able to use cloud-based components like serverless orchestrators and so on.

Users can run the `zenml login --local` command to spin up a local ZenML OSS server to serve the dashboard. For the local OSS server option, the `zenml login --local` command implicitly connects the client to the server. The diagram for this looks as follows:

![ZenML with a local ZenML OSS Server](/files/7syQVJyGgq5ECaNW4Xp5)

In order to move into production, the ZenML server needs to be deployed somewhere centrally so that the different cloud stack components can read from and write to the server. Additionally, this also allows all your team members to connect to it and share stacks and pipelines.

![ZenML centrally deployed for multiple users](/files/jBIwZNBsYtmWQF0MZ97D)

You connect to your deployed ZenML server using the `zenml login` command, and then you have the full benefits and power of ZenML. You can use all the cloud-based components, your metadata will be stored and synchronized across all the users of the server, and you can leverage features like centralized logs storage and pipeline artifact visualization.

## How to deploy ZenML

Deploying the ZenML Server is a crucial step towards transitioning to a production-grade environment for your machine learning projects. By setting up a deployed ZenML Server instance, you gain access to powerful features, allowing you to use stacks with remote components, centrally track progress, collaborate effectively, and achieve reproducible results.

Currently, there are two main options to access a deployed ZenML server:

1. **Managed deployment:** With [ZenML Pro](https://docs.zenml.io/pro) offering you can utilize a control plane to create ZenML servers, also known as [workspaces](https://docs.zenml.io/pro/core-concepts/workspaces). These workspaces are managed and maintained by ZenML's dedicated team, alleviating the burden of server management from your end. Importantly, your data remains securely within your stack, and ZenML's role is primarily to handle tracking of metadata and server maintenance.
2. **Self-hosted Deployment:** Alternatively, you have the ability to deploy ZenML on your own self-hosted environment. This can be achieved through various methods, including using [Docker](/deploying-zenml/deploying-zenml/deploy-with-docker), [Helm](/deploying-zenml/deploying-zenml/deploy-with-helm), or [HuggingFace Spaces](/deploying-zenml/deploying-zenml/deploy-using-huggingface-spaces). We also offer our Pro version for self-hosted deployments, so you can use our full paid feature set while staying fully in control with an air-gapped solution on your infrastructure.

Both options offer distinct advantages, allowing you to choose the deployment approach that best aligns with your organization's needs and infrastructure preferences. Whichever path you select, ZenML facilitates a seamless and efficient way to take advantage of the ZenML Server and enhance your machine learning workflows for production-level success.

### Options for deploying ZenML

Documentation for the various deployment strategies can be found in the following pages below (in our 'how-to' guides):

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden></th><th data-hidden data-type="content-ref"></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:purple;"><strong>Deploying ZenML using ZenML Pro</strong></mark></td><td>Deploying ZenML using ZenML Pro.</td><td><a href="/files/EK2aEzs80xfZmzvsgrbo">/files/EK2aEzs80xfZmzvsgrbo</a></td><td></td><td></td><td><a href="https://docs.zenml.io/pro/deployments/scenarios">https://docs.zenml.io/pro/deployments/scenarios</a></td></tr><tr><td><mark style="color:purple;"><strong>Deploy with Docker</strong></mark></td><td>Deploying ZenML in a Docker container.</td><td><a href="/files/q2V1f2kUvozN5biA6DSu">/files/q2V1f2kUvozN5biA6DSu</a></td><td></td><td></td><td><a href="/pages/UjlFvcqv6kAeMZQKQLVF">/pages/UjlFvcqv6kAeMZQKQLVF</a></td></tr><tr><td><mark style="color:purple;"><strong>Deploy with Helm</strong></mark></td><td>Deploying ZenML in a Kubernetes cluster with Helm.</td><td><a href="/files/8Yf9ED21HnwXtgI0Rk4l">/files/8Yf9ED21HnwXtgI0Rk4l</a></td><td></td><td></td><td><a href="/pages/bxQvlzr5MAUROsUnWdUR">/pages/bxQvlzr5MAUROsUnWdUR</a></td></tr><tr><td><mark style="color:purple;"><strong>Deploy with HuggingFace Spaces</strong></mark></td><td>Deploying ZenML to Hugging Face Spaces.</td><td><a href="/files/cTRq3SCjBJ9PKmtl16ZE">/files/cTRq3SCjBJ9PKmtl16ZE</a></td><td></td><td></td><td><a href="/pages/ETA8TToFaUeSGVISttsv">/pages/ETA8TToFaUeSGVISttsv</a></td></tr></tbody></table>

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Deploy with Docker

Deploying ZenML in a Docker container.

The ZenML server container image is available at [`zenmldocker/zenml-server`](https://hub.docker.com/r/zenmldocker/zenml/) and can be used to deploy ZenML with a container management or orchestration tool like Docker and docker-compose, or a serverless platform like [Cloud Run](https://cloud.google.com/run), [Container Apps](https://learn.microsoft.com/en-us/azure/container-apps/overview), and more! This guide walks you through the various configuration options that the ZenML server container expects as well as a few deployment use cases.

## Try it out locally first

If you're just looking for a quick way to deploy the ZenML server using a container, without going through the hassle of interacting with a container management tool like Docker and manually configuring your container, you can use the ZenML CLI to do so. You only need to have Docker installed and running on your machine:

```bash
zenml login --local --docker
```

This command deploys a ZenML server locally in a Docker container, then connects your client to it. Similar to running plain `zenml login --local`, the server and the local ZenML client share the same SQLite database.

The rest of this guide is addressed to advanced users who are looking to manually deploy and manage a containerized ZenML server.

## ZenML server configuration options

If you're planning on deploying a custom containerized ZenML server yourself, you probably need to configure some settings for it like the **database** it should use, the **default user details,** and more. The ZenML server container image uses sensible defaults, so you can simply start a container without worrying too much about the configuration. However, if you're looking to connect the ZenML server to an external MySQL database or secrets management service, to persist the internal SQLite database, or simply want to control other settings like the default account, you can do so by customizing the container's environment variables.

The following environment variables can be passed to the container:

* **ZENML\_STORE\_URL**: This URL should point to an SQLite database file *mounted in the container*, or to a MySQL-compatible database service *reachable from the container*. It takes one of these forms:

  ```
  sqlite:////path/to/zenml.db
  ```

  or:

  ```
  mysql://username:password@host:port/database
  ```
* **ZENML\_STORE\_SSL\_CA**: This can be set to a custom server CA certificate in use by the MySQL database service. Only valid when `ZENML_STORE_URL` points to a MySQL database that uses SSL-secured connections. The variable can be set either to the path where the certificate file is mounted inside the container or to the certificate contents themselves.
* **ZENML\_STORE\_SSL\_CERT**: This can be set to a client SSL certificate required to connect to the MySQL database service. Only valid when `ZENML_STORE_URL` points to a MySQL database that uses SSL-secured connections and requires client SSL certificates. The variable can be set either to the path where the certificate file is mounted inside the container or to the certificate contents themselves. This variable also requires `ZENML_STORE_SSL_KEY` to be set.
* **ZENML\_STORE\_SSL\_KEY**: This can be set to a client SSL private key required to connect to the MySQL database service. Only valid when `ZENML_STORE_URL` points to a MySQL database that uses SSL-secured connections and requires client SSL certificates. The variable can be set either to the path where the certificate file is mounted inside the container or to the certificate contents themselves. This variable also requires `ZENML_STORE_SSL_CERT` to be set.
* **ZENML\_STORE\_SSL\_VERIFY\_SERVER\_CERT**: This boolean variable controls whether the SSL certificate in use by the MySQL server is verified. Only valid when `ZENML_STORE_URL` points to a MySQL database that uses SSL-secured connections. Defaults to `False`.
* **ZENML\_LOGGING\_VERBOSITY**: Controls the verbosity of server logs inside the container. It can be set to `NOTSET`, `ERROR`, `WARN`, `INFO` (default), `DEBUG`, or `CRITICAL`.
* **ZENML\_CONSOLE\_LOGGING\_FORMAT**: Controls the server container stdout/stderr log format. Set it to `console` (default) for structured text output, `json` for JSON formatted logs, or a valid Python `%`-style logging format string for custom console output. `ZENML_LOGGING_FORMAT` is still accepted as a deprecated alias.
* **ZENML\_LOGGING\_COLORS\_DISABLED**: Set this to `true` to disable ANSI color output in console logs. Colors are enabled by default.
* **ZENML\_SERVER\_OTEL\_EXPORTER\_OTLP\_ENDPOINT**: Set this to the base OTLP/HTTP endpoint of an OpenTelemetry-compatible backend to export ZenML server traces, metrics, and logs. The standard `OTEL_EXPORTER_OTLP_ENDPOINT` variable is also supported as a fallback. ZenML appends `/v1/traces`, `/v1/metrics`, and `/v1/logs` for each signal unless you set per-signal endpoint variables such as `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, or their matching `ZENML_SERVER_OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT` names. If no base or per-signal endpoint is set, server OpenTelemetry instrumentation is disabled. The server logs export to the OTel backend is independent of `ZENML_CONSOLE_LOGGING_FORMAT`: logs are exported as OTLP records with structured attributes derived from the underlying log record, not as console-formatted text or JSON.
* **ZENML\_SERVER\_OTEL\_SERVICE\_NAME**: The service name reported in OpenTelemetry resource attributes for server traces, metrics, and logs. The standard `OTEL_SERVICE_NAME` variable is also supported as a fallback. Defaults to `zenml-server` for self-hosted deployments, or to the ZenML Pro workspace name for cloud deployments when not explicitly set.
* **ZENML\_STORE\_BACKUP\_STRATEGY**: This variable controls the database backup strategy used by the ZenML server. See the [Database backup and recovery](#database-backup-and-recovery) section for more details about this feature and other related environment variables. Defaults to `in-memory`.
* **ZENML\_SERVER\_RATE\_LIMIT\_ENABLED**: This variable controls the rate limiting for ZenML API (currently only for the `LOGIN` endpoint). It is disabled by default, so set it to `1` only if you need to enable rate limiting. To determine unique users a `X_FORWARDED_FOR` header or `request.client.host` is used, so before enabling this make sure that your network configuration is associating proper information with your clients in order to avoid disruptions for legitimate requests.
* **ZENML\_SERVER\_LOGIN\_RATE\_LIMIT\_MINUTE**: If rate limiting is enabled, this variable controls how many requests will be allowed to query the login endpoint in a one minute interval. Set it to a desired integer value; defaults to `5`.
* **ZENML\_SERVER\_LOGIN\_RATE\_LIMIT\_DAY**: If rate limiting is enabled, this variable controls how many requests will be allowed to query the login endpoint in an interval of day interval. Set it to a desired integer value; defaults to `1000`.

If none of the `ZENML_STORE_*` variables are set, the container will default to creating and using an SQLite database file stored at `/zenml/.zenconfig/local_stores/default_zen_store/zenml.db` inside the container. The `/zenml/.zenconfig/local_stores` base path where the default SQLite database is located can optionally be overridden by setting the `ZENML_LOCAL_STORES_PATH` environment variable to point to a different path (e.g. a persistent volume or directory that is mounted from the host).

### Secret store environment variables

Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as [a secrets store backend](/deploying-zenml/deploying-zenml/secret-management) where secret values are stored. If you want to use an external secrets management service like the AWS Secrets Manager, GCP Secrets Manager, Azure Key Vault, HashiCorp Vault or even your custom Secrets Store back-end implementation instead, you need to configure it explicitly using Docker environment variables. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you will also need to provide the credentials needed to access the secrets management service API.

> **Important:** If you are updating the configuration of your ZenML Server container to use a different secrets store back-end or location, you should follow [the documented secrets migration strategy](/deploying-zenml/deploying-zenml/secret-management#secrets-migration-strategy) to minimize downtime and to ensure that existing secrets are also properly migrated.

{% tabs %}
{% tab title="Default" %}
The SQL database is used as the default secret store location. You only need to configure these options if you want to change the default behavior.

It is particularly recommended to enable encryption at rest for the SQL database if you plan on using it as a secrets store backend. You'll have to configure the secret key used to encrypt the secret values. If not set, encryption will not be used and passwords will be stored unencrypted in the database.

* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `sql` in order to explicitly set this type of secret store.
* **ZENML\_SECRETS\_STORE\_ENCRYPTION\_KEY**: the secret key used to encrypt all secrets stored in the SQL secrets store. It is recommended to set this to a random string with a length of at least 32 characters, e.g.:

  ```python
  from secrets import token_hex
  token_hex(32)
  ```

  or:

  ```shell
  openssl rand -hex 32
  ```

> **Important:** If you configure encryption for your SQL database secrets store, you should keep the `ZENML_SECRETS_STORE_ENCRYPTION_KEY` value somewhere safe and secure, as it will always be required by the ZenML server to decrypt the secrets in the database. If you lose the encryption key, you will not be able to decrypt the secrets in the database and will have to reset them.
> {% endtab %}

{% tab title="AWS" %}
These configuration options are only relevant if you're using the AWS Secrets Manager as the secrets store backend.

* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `aws` in order to set this type of secret store.

The AWS Secrets Store uses the ZenML AWS Service Connector under the hood to authenticate with the AWS Secrets Manager API. This means that you can use any of the [authentication methods supported by the AWS Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector#authentication-methods) to authenticate with the AWS Secrets Manager API.

The minimum set of permissions that must be attached to the implicit or configured AWS credentials are: `secretsmanager:CreateSecret`, `secretsmanager:GetSecretValue`, `secretsmanager:DescribeSecret`, `secretsmanager:PutSecretValue`, `secretsmanager:TagResource` and `secretsmanager:DeleteSecret` and they must be associated with secrets that have a name starting with `zenml/` in the target region and account. The following IAM policy example can be used as a starting point:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ZenMLSecretsStore",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:CreateSecret",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:PutSecretValue",
                "secretsmanager:TagResource",
                "secretsmanager:DeleteSecret"
            ],
            "Resource": "arn:aws:secretsmanager:<AWS-region>:<AWS-account-id>:secret:zenml/*"
        }
    ]
}
```

The following configuration options are supported:

* **ZENML\_SECRETS\_STORE\_AUTH\_METHOD**: The AWS Service Connector authentication method to use (e.g. `secret-key` or `iam-role`).
* **ZENML\_SECRETS\_STORE\_AUTH\_CONFIG**: The AWS Service Connector configuration, in JSON format (e.g. `{"aws_access_key_id":"<aws-key-id>","aws_secret_access_key":"<aws-secret-key>","region":"<aws-region>"}`).

> **Note:** The remaining configuration options are deprecated and may be removed in a future release. Instead, you should set the `ZENML_SECRETS_STORE_AUTH_METHOD` and `ZENML_SECRETS_STORE_AUTH_CONFIG` variables to use the AWS Service Connector authentication method.

* **ZENML\_SECRETS\_STORE\_REGION\_NAME**: The AWS region to use. This must be set to the region where the AWS Secrets Manager service that you want to use is located.
* **ZENML\_SECRETS\_STORE\_AWS\_ACCESS\_KEY\_ID**: The AWS access key ID to use for authentication. This must be set to a valid AWS access key ID that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable.
* **ZENML\_SECRETS\_STORE\_AWS\_SECRET\_ACCESS\_KEY**: The AWS secret access key to use for authentication. This must be set to a valid AWS secret access key that has access to the AWS Secrets Manager service that you want to use. If you are using an IAM role attached to an EKS cluster to authenticate, you can omit this variable.
  {% endtab %}

{% tab title="GCP" %}
These configuration options are only relevant if you're using the GCP Secrets Manager as the secrets store backend.

* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `gcp` in order to set this type of secret store.

The GCP Secrets Store uses the ZenML GCP Service Connector under the hood to authenticate with the GCP Secrets Manager API. This means that you can use any of the [authentication methods supported by the GCP Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector#authentication-methods) to authenticate with the GCP Secrets Manager API.

The minimum set of permissions that must be attached to the implicit or configured GCP credentials are as follows:

* `secretmanager.secrets.create` for the target GCP project (i.e. no condition on the name prefix)
* `secretmanager.secrets.get`, `secretmanager.secrets.update`, `secretmanager.versions.access`, `secretmanager.versions.add` and `secretmanager.secrets.delete` for the target GCP project and for secrets that have a name starting with `zenml-`

This can be achieved by creating two custom IAM roles and attaching them to the principal (e.g. user or service account) that will be used to access the GCP Secrets Manager API with a condition configured when attaching the second role to limit access to secrets with a name prefix of `zenml-`. The following `gcloud` CLI command examples can be used as a starting point:

```bash
gcloud iam roles create ZenMLServerSecretsStoreCreator \
  --project <your GCP project ID> \
  --title "ZenML Server Secrets Store Creator" \
  --description "Allow the ZenML Server to create new secrets" \
  --stage GA \
  --permissions "secretmanager.secrets.create"

gcloud iam roles create ZenMLServerSecretsStoreEditor \
  --project <your GCP project ID> \
  --title "ZenML Server Secrets Store Editor" \
  --description "Allow the ZenML Server to manage its secrets" \
  --stage GA \
  --permissions "secretmanager.secrets.get,secretmanager.secrets.update,secretmanager.versions.access,secretmanager.versions.add,secretmanager.secrets.delete"

gcloud projects add-iam-policy-binding <your GCP project ID> \
  --member serviceAccount:<your GCP service account email> \
  --role projects/<your GCP project ID>/roles/ZenMLServerSecretsStoreCreator \
  --condition None

# NOTE: use the GCP project NUMBER, not the project ID in the condition
gcloud projects add-iam-policy-binding <your GCP project ID> \
  --member serviceAccount:<your GCP service account email> \
  --role projects/<your GCP project ID>/roles/ZenMLServerSecretsStoreEditor \
  --condition 'title=limit_access_zenml,description="Limit access to secrets with prefix zenml-",expression=resource.name.startsWith("projects/<your GCP project NUMBER>/secrets/zenml-")'
```

The following configuration options are supported:

* **ZENML\_SECRETS\_STORE\_AUTH\_METHOD**: The GCP Service Connector authentication method to use (e.g. `service-account`).
* **ZENML\_SECRETS\_STORE\_AUTH\_CONFIG**: The GCP Service Connector configuration, in JSON format (e.g. `{"project_id":"my-project","service_account_json":{ ... }}`).

> **Note:** The remaining configuration options are deprecated and may be removed in a future release. Instead, you should set the `ZENML_SECRETS_STORE_AUTH_METHOD` and `ZENML_SECRETS_STORE_AUTH_CONFIG` variables to use the GCP Service Connector authentication method.

* **ZENML\_SECRETS\_STORE\_PROJECT\_ID**: The GCP project ID to use. This must be set to the project ID where the GCP Secrets Manager service that you want to use is located.
* **GOOGLE\_APPLICATION\_CREDENTIALS**: The path to the GCP service account credentials file to use for authentication. This must be set to a valid GCP service account credentials file that has access to the GCP Secrets Manager service that you want to use. If you are using a GCP service account attached to a GKE cluster to authenticate, you can omit this variable. NOTE: the path to the credentials file must be mounted into the container.
  {% endtab %}

{% tab title="Azure" %}
These configuration options are only relevant if you're using Azure Key Vault as the secrets store backend.

* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `azure` in order to set this type of secret store.
* **ZENML\_SECRETS\_STORE\_KEY\_VAULT\_NAME**: The name of the Azure Key Vault. This must be set to point to the Azure Key Vault instance that you want to use.

The Azure Secrets Store uses the ZenML Azure Service Connector under the hood to authenticate with the Azure Key Vault API. This means that you can use any of the [authentication methods supported by the Azure Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector#authentication-methods) to authenticate with the Azure Key Vault API. The following configuration options are supported:

* **ZENML\_SECRETS\_STORE\_AUTH\_METHOD**: The Azure Service Connector authentication method to use (e.g. `service-account`).
* **ZENML\_SECRETS\_STORE\_AUTH\_CONFIG**: The Azure Service Connector configuration, in JSON format (e.g. `{"tenant_id":"my-tenant-id","client_id":"my-client-id","client_secret": "my-client-secret"}`).

> **Note:** The remaining configuration options are deprecated and may be removed in a future release. Instead, you should set the `ZENML_SECRETS_STORE_AUTH_METHOD` and `ZENML_SECRETS_STORE_AUTH_CONFIG` variables to use the Azure Service Connector authentication method.

* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_ID**: The Azure application service principal client ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_CLIENT\_SECRET**: The Azure application service principal client secret to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable.
* **ZENML\_SECRETS\_STORE\_AZURE\_TENANT\_ID**: The Azure application service principal tenant ID to use to authenticate with the Azure Key Vault API. If you are running the ZenML server hosted in Azure and are using a managed identity to access the Azure Key Vault service, you can omit this variable.
  {% endtab %}

{% tab title="Hashicorp" %}
These configuration options are only relevant if you're using Hashicorp Vault as the secrets store backend.

* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `hashicorp` in order to set this type of secret store.
* **ZENML\_SECRETS\_STORE\_VAULT\_ADDR**: The URL of the HashiCorp Vault server to connect to. NOTE: this is the same as setting the `VAULT_ADDR` environment variable.
* **ZENML\_SECRETS\_STORE\_VAULT\_NAMESPACE**: The Vault Enterprise namespace. Not required for Vault OSS. NOTE: this is the same as setting the `VAULT_NAMESPACE` environment variable.
* **ZENML\_SECRETS\_STORE\_MOUNT\_POINT**: The mount point to use for the HashiCorp Vault secrets store. If not set, the default value of `secret` will be used.
* **ZENML\_SECRETS\_STORE\_VAULT\_AUTH\_METHOD**: The authentication method to use to authenticate with the HashiCorp Vault server. One of: `token`, `app_role`, `aws`. Defaults to `token` if not set.
* **ZENML\_SECRETS\_STORE\_VAULT\_AUTH\_MOUNT\_POINT**: The mount point to use for the authentication method. If not set, the default value specific to the authentication method will be used.
* **ZENML\_SECRETS\_STORE\_VAULT\_TOKEN**: The token to use to authenticate with the HashiCorp Vault server. Mandatory if the authentication method is `token`. NOTE: this is the same as setting the `VAULT_TOKEN` environment variable.
* **ZENML\_SECRETS\_STORE\_VAULT\_APP\_ROLE\_ID**: The role ID to use for the app role authentication method. Mandatory if the authentication method is `app_role`.
* **ZENML\_SECRETS\_STORE\_VAULT\_APP\_SECRET\_ID**: The secret ID to use for the app role authentication method. Mandatory if the authentication method is `app_role`.
* **ZENML\_SECRETS\_STORE\_VAULT\_AWS\_ROLE**: The AWS role to use for the AWS authentication method. Only relevant if the authentication method is `aws`.
* **ZENML\_SECRETS\_STORE\_VAULT\_AWS\_HEADER\_VALUE**: The AWS header value to use for the AWS authentication method. Only relevant if the authentication method is `aws`.
* **ZENML\_SECRETS\_STORE\_MAX\_VERSIONS**: The maximum number of secret versions to keep for each Vault secret. If not set, the default value of 1 will be used (only the latest version will be kept).
  {% endtab %}

{% tab title="Custom" %}
These configuration options are only relevant if you're using a custom secrets store backend implementation. For this to work, you must have [a custom implementation of the secrets store API](/deploying-zenml/deploying-zenml/custom-secret-stores) in the form of a class derived from `zenml.zen_stores.secrets_stores.base_secrets_store.BaseSecretsStore`. This class must be importable from within the ZenML server container, which means you most likely need to mount the directory containing the class into the container or build a custom container image that contains the class.

The following configuration option is required:

* **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `custom` in order to set this type of secret store.
* **ZENML\_SECRETS\_STORE\_CLASS\_PATH**: The fully qualified path to the class that implements the custom secrets store API (e.g. `my_package.my_module.MySecretsStore`).

If your custom secrets store implementation requires additional configuration options, you can pass them as environment variables using the following naming convention:

* `ZENML_SECRETS_STORE_<OPTION_NAME>`: The name of the option to pass to the custom secrets store class. The option name must be in uppercase and any hyphens (`-`) must be replaced with underscores (`_`). ZenML will automatically convert the environment variable name to the corresponding option name by removing the prefix and converting the remaining characters to lowercase. For example, the environment variable `ZENML_SECRETS_STORE_MY_OPTION` will be converted to the option name `my_option` and passed to the custom secrets store class configuration.
  {% endtab %}
  {% endtabs %}

{% hint style="info" %}
**ZENML\_SECRETS\_STORE\_TYPE**: Set this variable to `none`to disable the secrets store functionality altogether.
{% endhint %}

#### Backup secrets store

[A backup secrets store](/deploying-zenml/deploying-zenml/secret-management#backup-secrets-store) back-end may be configured for high-availability and backup purposes. or as an intermediate step in the process of [migrating secrets to a different external location or secrets manager provider](/deploying-zenml/deploying-zenml/secret-management#secrets-migration-strategy).

To configure a backup secrets store in the Docker container, use the same approach and instructions documented for the primary secrets store, but set the `**ZENML\_BACKUP\_SECRETS\_STORE\***` environment variables instead of `**ZENML\_SECRETS\_STORE\***`, e.g.:

```yaml
ZENML_BACKUP_SECRETS_STORE_TYPE: aws
ZENML_BACKUP_SECRETS_STORE_AUTH_METHOD: secret-key
ZENML_BACKUP_SECRETS_STORE_AUTH_CONFIG: '{"aws_access_key_id":"<aws-key-id>", "aws_secret_access_key","<aws-secret-key>","role_arn": "<aws-role-arn>"}`'
```

### Advanced server configuration options

These configuration options are not required for most use cases, but can be useful in certain scenarios that require mirroring the same ZenML server configuration across multiple container instances (e.g. a Kubernetes deployment with multiple replicas):

* **ZENML\_SERVER\_JWT\_SECRET\_KEY**: This is a secret key used to sign JWT tokens used for authentication. If not explicitly set, a random key is generated automatically by the server on startup and stored in the server's global configuration. This should be set to a random string with a recommended length of at least 32 characters, e.g.:

  ```python
  from secrets import token_hex
  token_hex(32)
  ```

  or:

  ```shell
  openssl rand -hex 32
  ```

The environment variables starting with *ZENML\_SERVER\_SECURE\_HEADERS\_*\* can be used to enable, disable or set custom values for security headers in the ZenML server's HTTP responses. The following values can be set for any of the supported secure headers configuration options:

* `enabled`, `on`, `true` or `yes` - enables the secure header with the default value.
* `disabled`, `off`, `false`, `none` or `no` - disables the secure header entirely, so that it is not set in the ZenML server's HTTP responses.
* any other value - sets the secure header to the specified value.

The following secure headers environment variables are supported:

* **ZENML\_SERVER\_SECURE\_HEADERS\_SERVER**: The `Server` HTTP header value used to identify the server. The default value is the ZenML server ID.
* **ZENML\_SERVER\_SECURE\_HEADERS\_HSTS**: The `Strict-Transport-Security` HTTP header value. The default value is `max-age=63072000; includeSubDomains`.
* **ZENML\_SERVER\_SECURE\_HEADERS\_XFO**: The `X-Frame-Options` HTTP header value. The default value is `SAMEORIGIN`.
* **ZENML\_SERVER\_SECURE\_HEADERS\_CONTENT**: The `X-Content-Type-Options` HTTP header value. The default value is `nosniff`.
* **ZENML\_SERVER\_SECURE\_HEADERS\_CSP**: The `Content-Security-Policy` HTTP header value. This is by default set to a strict CSP policy that only allows content from the origins required by the ZenML dashboard. NOTE: customizing this header is discouraged, as it may cause the ZenML dashboard to malfunction.
* **ZENML\_SERVER\_SECURE\_HEADERS\_REFERRER**: The `Referrer-Policy` HTTP header value. The default value is `no-referrer-when-downgrade`.
* **ZENML\_SERVER\_SECURE\_HEADERS\_CACHE**: The `Cache-Control` HTTP header value. The default value is `no-store, no-cache, must-revalidate`.
* **ZENML\_SERVER\_SECURE\_HEADERS\_PERMISSIONS**: The `Permissions-Policy` HTTP header value. The default value is `accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()`.

If you prefer to activate the server automatically during the initial deployment and also automate the creation of the initial admin user account, this legacy behavior can be brought back by setting the following environment variables:

* **ZENML\_SERVER\_AUTO\_ACTIVATE**: Set this to `1` to automatically activate the server and create the initial admin user account when the server is first deployed. Defaults to `0`.
* **ZENML\_DEFAULT\_USER\_NAME**: The name of the initial admin user account created by the server on the first deployment, during database initialization. Defaults to `default`.
* **ZENML\_DEFAULT\_USER\_PASSWORD**: The password to use for the initial admin user account. Defaults to an empty password value, if not set.

## Run the ZenML server with Docker

As previously mentioned, the ZenML server container image uses sensible defaults for most configuration options. This means that you can simply run the container with Docker without any additional configuration and it will work out of the box for most use cases:

```bash
docker run -it -d -p 8080:8080 --name zenml zenmldocker/zenml-server
```

> **Note:** It is recommended to use a ZenML container image version that matches the version of your client, to avoid any potential API incompatibilities (e.g. `zenmldocker/zenml-server:0.21.1` instead of `zenmldocker/zenml-server`).

The above command will start a containerized ZenML server running on your machine that uses a temporary SQLite database file stored in the container. Temporary means that the database and all its contents (stacks, pipelines, pipeline runs, etc.) will be lost when the container is removed with `docker rm`.

You need to visit the ZenML dashboard at `http://localhost:8080` and activate the server by creating an initial admin user account. You can then connect your client to the server with the web login flow:

```shell
$ zenml login http://localhost:8080
Connecting to: 'http://localhost:8080'...
If your browser did not open automatically, please open the following URL into your browser to proceed with the authentication:

http://localhost:8080/devices/verify?device_id=f7a7333a-3ef0-4f39-85a9-f190279456d3&user_code=9375f5cdfdaf36772ce981fe3ee6172c

Successfully logged in.
Creating default stack for user 'default'...
Updated the global store configuration.
```

{% hint style="info" %}
The `localhost` URL **will** work, even if you are using Docker-backed ZenML orchestrators in your stack, like [the local Docker orchestrator](https://docs.zenml.io/stacks/orchestrators/local-docker) or [a locally deployed Kubeflow orchestrator](https://docs.zenml.io/stacks/orchestrators/kubeflow).

ZenML makes use of specialized DNS entries such as `host.docker.internal` and `host.k3d.internal` to make the ZenML server accessible from the pipeline steps running inside other Docker containers on the same machine.
{% endhint %}

You can manage the container with the usual Docker commands:

* `docker logs zenml` to view the server logs
* `docker stop zenml` to stop the server
* `docker start zenml` to start the server again
* `docker rm zenml` to remove the container

If you are looking for a customized ZenML server Docker deployment, you can configure one or more of [the supported environment variables](#zenml-server-configuration-options) and then pass them to the container using the `docker run` `--env` or `--env-file` arguments (see the [Docker documentation](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file) for more details). For example:

```shell
docker run -it -d -p 8080:8080 --name zenml \
    --env ZENML_STORE_URL=mysql://username:password@host:port/database \
    zenmldocker/zenml-server
```

If you're looking for a quick way to run both the ZenML server and a MySQL database with Docker, you can [deploy the ZenML server with Docker Compose](#zenml-server-with-docker-compose).

The rest of this guide covers various advanced use cases for running the ZenML server with Docker.

### Persisting the SQLite database

Depending on your use case, you may also want to mount a persistent volume or directory from the host into the container to store the ZenML SQLite database file. This can be done using the `--mount` flag (see the [Docker documentation](https://docs.docker.com/storage/volumes/) for more details). For example:

```shell
mkdir zenml-server
docker run -it -d -p 8080:8080 --name zenml \
    --mount type=bind,source=$PWD/zenml-server,target=/zenml/.zenconfig/local_stores/default_zen_store \
    zenmldocker/zenml-server
```

This deployment has the advantage that the SQLite database file is persisted even when the container is removed with `docker rm`.

### Docker MySQL database

As a recommended alternative to the SQLite database, you can run a MySQL database service as another Docker container and connect the ZenML server container to it.

A command like the following can be run to start the containerized MySQL database service:

```shell
docker run --name mysql -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password mysql:8.0
```

If you also wish to persist the MySQL database data, you can mount a persistent volume or directory from the host into the container using the `--mount` flag, e.g.:

```shell
mkdir mysql-data
docker run --name mysql -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password \
    --mount type=bind,source=$PWD/mysql-data,target=/var/lib/mysql \
    mysql:8.0
```

Configuring the ZenML server container to connect to the MySQL database is just a matter of setting the `ZENML_STORE_URL` environment variable. We use the special `host.docker.internal` DNS name that is resolved from within the Docker containers to the gateway IP address used by the Docker network (see the [Docker documentation](https://docs.docker.com/desktop/networking/#use-cases-and-workarounds-for-all-platforms) for more details). On Linux, this needs to be explicitly enabled in the `docker run` command with the `--add-host` argument:

```shell
docker run -it -d -p 8080:8080 --name zenml \
    --add-host host.docker.internal:host-gateway \
    --env ZENML_STORE_URL=mysql://root:password@host.docker.internal/zenml \
    zenmldocker/zenml-server
```

You need to visit the ZenML dashboard at `http://localhost:8080` and activate the server by creating an initial admin user account. You can then connect your client to the server with the web login flow:

```shell
zenml login http://localhost:8080
```

### Direct MySQL database connection

This scenario is similar to the previous one, but instead of running a ZenML server, the client is configured to connect directly to a MySQL database running in a Docker container.

As previously covered, the containerized MySQL database service can be started with a command like the following:

```shell
docker run --name mysql -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password mysql:8.0
```

The ZenML client on the host machine can then be configured to connect directly to the database with a slightly different `zenml login` command:

```shell
zenml login mysql://root:password@127.0.0.1/zenml
```

> **Note** The `localhost` hostname will not work with MySQL databases. You need to use the `127.0.0.1` IP address instead.

### ZenML server with `docker-compose`

Docker compose offers a simpler way of managing multi-container setups on your local machine, which is the case for instance if you are looking to deploy the ZenML server container and connect it to a MySQL database service also running in a Docker container.

To use Docker Compose, you need to [install the docker-compose plugin](https://docs.docker.com/compose/install/linux/) on your machine first.

A `docker-compose.yml` file like the one below can be used to start and manage the ZenML server container and the MySQL database service all at once:

```yaml
version: "3.9"

services:
  mysql:
    image: mysql:8.0
    ports:
      - 3306:3306
    environment:
      - MYSQL_ROOT_PASSWORD=password
  zenml:
    image: zenmldocker/zenml-server
    ports:
      - "8080:8080"
    environment:
      - ZENML_STORE_URL=mysql://root:password@host.docker.internal/zenml
    links:
      - mysql
    depends_on:
      - mysql
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: on-failure
```

Note the following:

* `ZENML_STORE_URL` is set to the special Docker `host.docker.internal` hostname to instruct the server to connect to the database over the Docker network.
* The `extra_hosts` section is needed on Linux to make the `host.docker.internal` hostname resolvable from the ZenML server container.

To start the containers, run the following command from the directory where the `docker-compose.yml` file is located:

```shell
docker compose -p zenml up  -d
```

or, if you need to use a different filename or path:

```shell
docker compose -f /path/to/docker-compose.yml -p zenml up -d
```

You need to visit the ZenML dashboard at `http://localhost:8080` to activate the server by creating an initial admin account. You can then connect your client to the server with the web login flow:

```shell
zenml login http://localhost:8080
```

Tearing down the installation is as simple as running:

```shell
docker compose -p zenml down
```

## Database backup and recovery

An automated database backup and recovery feature is enabled by default for all Docker deployments. The ZenML server will automatically back up the database in-memory before every database schema migration and restore it if the migration fails.

{% hint style="info" %}
The database backup automatically created by the ZenML server is only temporary and only used as an immediate recovery in case of database migration failures. It is not meant to be used as a long-term backup solution. If you need to back up your database for long-term storage, you should use a dedicated backup solution.
{% endhint %}

Several database backup strategies are supported, depending on where and how the backup is stored. The strategy can be configured by means of the `ZENML_STORE_BACKUP_STRATEGY` environment variable:

* `disabled` - no backup is performed
* `in-memory` - the database schema and data are stored in memory. This is the fastest backup strategy, but the backup is not persisted across container restarts, so no manual intervention is possible in case the automatic DB recovery fails after a failed DB migration. Adequate memory resources should be allocated to the ZenML server container when using this backup strategy with larger databases. This is the default backup strategy.
* `database` - the database is copied to a backup database in the same database server. This requires the `ZENML_STORE_BACKUP_DATABASE` environment variable to be set to the name of the backup database. This backup strategy is only supported for MySQL compatible databases and the user specified in the database URL must have permissions to manage (create, drop, and modify) the backup database in addition to the main database.
* `dump-file` - the database schema and data are dumped to a filesystem location inside the ZenML server container. This location can be customized by means of the `ZENML_STORE_BACKUP_DIRECTORY` environment variable. When this strategy is configured, users should mount a host directory in the container and point the `ZENML_STORE_BACKUP_DIRECTORY` variable to where it's mounted inside the container. If a host directory is not mounted, the dump file will be stored in the container's filesystem and will be lost when the container is removed.
* `mydumper` - the database is backed up using mydumper/myloader. This requires the `mydumper` and `myloader` utilities to be installed in the ZenML server container. The `ZENML_STORE_MYDUMPER_THREADS`, `ZENML_STORE_MYDUMPER_COMPRESS`, `ZENML_STORE_MYDUMPER_EXTRA_ARGS`, `ZENML_STORE_MYLOADER_THREADS`, and `ZENML_STORE_MYLOADER_EXTRA_ARGS` environment variables can be used to configure the backup and restore processes.
* `custom` - use a custom backup engine. This requires the `ZENML_STORE_CUSTOM_BACKUP_ENGINE` environment variable to be set to the class path of the custom backup engine. The class should extend from the `zenml.zen_stores.migrations.backup.base_backup_engine.BaseBackupEngine` base class and be importable from the container image that you are using for the ZenML server. Arguments for the custom backup engine can be passed using the `ZENML_STORE_CUSTOM_BACKUP_ENGINE_CONFIG` environment variable.

The following additional rules are applied concerning the creation and lifetime of the backup:

* a backup is not attempted if the database doesn't need to undergo a migration (e.g. when the ZenML server is upgraded to a new version that doesn't require a database schema change or if the ZenML version doesn't change at all).
* a backup file or database is created before every database migration attempt (i.e. when the container starts). If a backup already exists (i.e. persisted in a mounted host directory or backup database), it is NOT overwritten. Instead, the existing backup is used to rollback the database to the previous state in case the migration fails again.
* the persistent backup file or database is cleaned up after the migration is completed successfully or if the database doesn't need to undergo a migration. This includes backups created by previous failed migration attempts.
* the persistent backup file or database is NOT cleaned up after a failed migration. This allows the user to manually inspect and/or apply the backup if the automatic recovery fails.

{% hint style="warning" %}
When running in production where database sizes are large, you should use the `mydumper` backup strategy or write your own custom backup engine. The other backup strategies are not recommended because they are inefficient and will take a long time and consume a lot of resources to handle large databases.
{% endhint %}

The following example shows how to deploy the ZenML server to use a mounted host directory to persist the database backup file during a database migration:

```shell
mkdir mysql-data

docker run --name mysql -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password \
    --mount type=bind,source=$PWD/mysql-data,target=/var/lib/mysql \
    mysql:8.0

docker run -it -d -p 8080:8080 --name zenml \
    --add-host host.docker.internal:host-gateway \
    --mount type=bind,source=$PWD/mysql-data,target=/db-dump \
    --env ZENML_STORE_URL=mysql://root:password@host.docker.internal/zenml \
    --env ZENML_STORE_BACKUP_STRATEGY=dump-file \
    --env ZENML_STORE_BACKUP_DIRECTORY=/db-dump \
    zenmldocker/zenml-server
```

## Troubleshooting

You can check the logs of the container to verify if the server is up and, depending on where you have deployed it, you can also access the dashboard at a `localhost` port (if running locally) or through some other service that exposes your container to the internet.

### CLI Docker deployments

If you used the `zenml login --local --docker` CLI command to deploy the Docker ZenML server, you can check the logs with the command:

```shell
zenml logs -f
```

### Manual Docker deployments

If you used the `docker run` command to manually deploy the Docker ZenML server, you can check the logs with the command:

```shell
docker logs zenml -f
```

If you used the `docker compose` command to manually deploy the Docker ZenML server, you can check the logs with the command:

```shell
docker compose -p zenml logs -f
```

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Deploy with Helm

Deploying ZenML in a Kubernetes cluster with Helm.

If you wish to manually deploy and manage ZenML in a Kubernetes cluster of your choice, ZenML also includes a Helm chart among its available deployment options.

The chart uses standard Kubernetes APIs and works on managed cloud clusters (EKS, GKE, AKS), enterprise platforms like Red Hat OpenShift and SUSE Rancher, and self-managed clusters on-premises or in your own cloud.

You can find the chart on this [ArtifactHub repository](https://artifacthub.io/packages/helm/zenml/zenml), along with the templates, default values and instructions on how to install it. Read on to find detailed explanations on prerequisites, configuration, and deployment scenarios.

## Prerequisites

You'll need the following:

* A Kubernetes cluster (version 1.21 or higher recommended)
* Optional, but recommended: a MySQL-compatible database reachable from the Kubernetes cluster (e.g. one of the managed databases offered by Google Cloud, AWS, or Azure). A MySQL server version of 8.0 or higher is required
* the [Kubernetes client](https://kubernetes.io/docs/tasks/tools/#kubectl) already installed on your machine and configured to access your cluster
* [Helm](https://helm.sh/docs/intro/install/) installed on your machine
* Optional: an external Secrets Manager service (e.g. one of the managed secrets management services offered by Google Cloud, AWS, Azure, or HashiCorp Vault). By default, ZenML stores secrets inside the SQL database that it's connected to, but you also have the option of using an external cloud Secrets Manager service if you already happen to use one of those cloud or service providers

## ZenML Helm Configuration

You can start by taking a look at the [`values.yaml` file](https://artifacthub.io/packages/helm/zenml/zenml?modal=values) and familiarize yourself with some of the configuration settings that you can customize for your ZenML deployment.

In addition to tools and infrastructure, you will also need to collect and [prepare information related to your database](#collect-information-from-your-sql-database-service) and [information related to your external secrets management service](#collect-information-from-your-secrets-management-service) to be used for the Helm chart configuration and you may also want to install additional [optional services in your cluster](#optional-cluster-services).

When you are ready, you can proceed to the [installation](#zenml-helm-installation) section.

{% hint style="info" %}
**Values key rename:** The top-level `zenml` key in the Helm chart values has been renamed to `server`. Existing values files that use the old `zenml` key continue to work — the chart automatically merges both keys. We recommend using only one key; if both are present, `zenml` values take precedence for overlapping keys. The `zenml` key is deprecated and will be removed in a future release.
{% endhint %}

### Collect information from your SQL database service

Using an external MySQL-compatible database service is optional, but is recommended for production deployments. If omitted, ZenML will default to using an embedded SQLite database, which has the following limitations:

* the SQLite database is not persisted, meaning that it will be lost if the ZenML server pod is restarted or deleted
* the SQLite database does not scale horizontally, meaning that you will not be able to use more than one replica at a time for the ZenML server pod

If you decide to use an external MySQL-compatible database service, you will need to collect and prepare the following information for the Helm chart configuration:

* the hostname and port where the SQL database is reachable from the Kubernetes cluster
* the username and password that will be used to connect to the database. It is recommended that you create a dedicated database user for the ZenML server and that you restrict its privileges to only access the database that will be used by ZenML. Enforcing secure SSL connections for the user/database is also recommended. See the [MySQL documentation](https://dev.mysql.com/doc/refman/5.7/en/access-control.html) for more information on how to set up users and privileges.
* the name of the database that will be used by ZenML. The database does not have to exist prior to the deployment ( ZenML will create it on the first start). However, you need to create the database if you follow the best practice of restricting database user privileges to only access it.
* if you plan on using SSL to secure the client database connection, you may also need to prepare additional SSL certificates and keys:
  * the TLS CA certificate that was used to sign the server TLS certificate, if you're using a self-signed certificate or signed by a custom certificate authority that is not already trusted by default by most operating systems.
  * the TLS client certificate and key. This is only needed if you decide to use client certificates for your DB connection (some managed DB services support this, CloudSQL is an example).

### Collect information from your secrets management service

Using an externally managed secrets management service like those offered by Google Cloud, AWS, Azure or HashiCorp Vault is optional, but is recommended if you are already using those cloud service providers. If omitted, ZenML will default to using the SQL database to store secrets.

If you decide to use an external secrets management service, you will need to collect and prepare the following information for the Helm chart configuration (for supported back-ends only):

For the AWS secrets manager:

* the AWS region that you want to use to store your secrets
* an AWS access key ID and secret access key that provides full access to the AWS secrets manager service. You can create a dedicated IAM user for this purpose, or use an existing user with the necessary permissions. If you deploy the ZenML server in an EKS Kubernetes cluster that is already configured to use implicit authorization with an IAM role for service accounts, you can omit this step.

For the Google Cloud secrets manager:

* the Google Cloud project ID that you want to use to store your secrets
* a Google Cloud service account that has access to the secrets manager service. You can create a dedicated service account for this purpose, or use an existing service account with the necessary permissions.

For the Azure Key Vault:

* the name of the Azure Key Vault that you want to use to store your secrets
* the Azure tenant ID, client ID, and client secret associated with the Azure service principal that will be used to access the Azure Key Vault. You can create a dedicated application service principal for this purpose, or use an existing service principal with the necessary permissions. If you deploy the ZenML server in an AKS Kubernetes cluster that is already configured to use implicit authorization through the Azure-managed identity service, you can omit this step.

For the HashiCorp Vault:

* the URL of the HashiCorp Vault server
* the token that will be used to access the HashiCorp Vault server.

### Optional cluster services

It is common practice to install additional infrastructure-related services in a Kubernetes cluster to support the deployment and long-term management of applications. For example:

* an Ingress service like [nginx-ingress](https://kubernetes.github.io/ingress-nginx/deploy/) is recommended if you want to expose HTTP services to the internet. An Ingress is required if you want to use secure HTTPS for your ZenML deployment. The alternative is to use a LoadBalancer service to expose the ZenML service using plain HTTP, but this is not recommended for production.
* a [cert-manager](https://cert-manager.io/docs/installation/) is recommended if you want to generate and manage TLS certificates for your ZenML deployment. It can be used to automatically provision TLS certificates from a certificate authority (CA) of your choice, such as [Let's Encrypt](https://letsencrypt.org/). As an alternative, the ZenML Helm chart can be configured to auto-generate self-signed or you can generate the certificates yourself and provide them to the Helm chart, but this makes it more difficult to manage the certificates and you need to manually renew them when they expire.

## ZenML Helm Installation

### Configure the Helm chart

To use the Helm chart with custom values that includes path to files like the database SSL certificates, you need to pull the chart to your local directory first. You can do this with the following command:

```bash
helm pull oci://public.ecr.aws/zenml/zenml --version <VERSION> --untar
```

Next, to customize the Helm chart for your deployment, you should create a copy of the `values.yaml` file that you can find at `./zenml/values.yaml` (let’s call this `custom-values.yaml`). You’ll use this as a template to customize your configuration. Any values that you don’t override you should simply remove from your `custom-values.yaml` file to keep it clean and compatible with future Helm chart releases.

In most cases, you’ll need to change the following configuration values in `custom-values.yaml`:

* the database configuration, if you mean to use an external database:
  * the database URL, formatted as `mysql://<username>:<password>@<hostname>:<port>/<database>`
  * CA and/or client TLS certificates, if you’re using SSL to secure the connection to the database can be provided in the `database.sslCa`, `database.sslCert` and `database.sslKey` fields as either an inline value or a secret reference (in the latter case, the secret(s) must be created in the same namespace as the ZenML server before the deployment).
* the Ingress configuration, if enabled:
  * enabling TLS
  * enabling self-signed certificates
  * configuring the hostname that will be used to access the ZenML server, if different from the IP address or hostname associated with the Ingress service installed in your cluster

### Install the Helm chart

Once everything is configured, you can run the following command in the `./zenml` folder to install the Helm chart.

```
helm -n <namespace> install zenml-server . --create-namespace --values custom-values.yaml 
```

### Connect to the deployed ZenML server

Immediately after deployment, the ZenML server needs to be activated before it can be used. The activation process includes creating an initial admin user account and configuring some server settings. You can do this only by visiting the ZenML server URL in your browser and following the on-screen instructions. Connecting your local ZenML client to the server is not possible until the server is properly initialized.

The Helm chart should print out a message with the URL of the deployed ZenML server. You can use the URL to open the ZenML UI in your browser.

To connect your local client to the ZenML server, you can run:

```bash
zenml login https://zenml.example.com:8080 --no-verify-ssl
```

To disconnect from the current ZenML server and revert to using the local default database, use the following command:

```bash
zenml logout
```

## ZenML Helm Deployment Scenarios

This section covers some common Helm deployment scenarios for ZenML.

### Minimal deployment

The example below is a minimal configuration for a ZenML server deployment that uses a temporary SQLite database and a ClusterIP service that is not exposed to the internet:

```yaml
server:

  ingress:
    enabled: false
```

Once deployed, you have to use port-forwarding to access the ZenML server and to connect to it from your local machine:

```bash
kubectl -n zenml-server port-forward svc/zenml-server 8080:8080
zenml login http://localhost:8080
```

This is just a simple example only fit for testing and evaluation purposes. For production deployments, you should use an external database and an Ingress service with TLS certificates to secure and expose the ZenML server to the internet.

### Deployment with persistent local database

When using a local SQLite database with persistence enabled, you need to configure proper volume permissions. The ZenML Helm chart sets `podSecurityContext.fsGroup: 1000` by default to ensure the ZenML container (running as UID 1000) can write to the persistent volume.

Example configuration:

```yaml
server:
  database:
    persistence:
      enabled: true
      size: "10Gi"
      # storageClassName: ""  # Optional: use default storage class if not specified

# podSecurityContext.fsGroup is set to 1000 by default
# This ensures the container can write to the persistent volume
```

{% hint style="warning" %}
If you override `podSecurityContext` in your custom values, ensure that `fsGroup: 1000` is included. Without this setting, the persistent volume will be mounted with root:root ownership, causing permission errors and preventing the ZenML server from starting.
{% endhint %}

This configuration is also required when using persistent volumes for database backup dumps. See the [Database backup and recovery](#database-backup-and-recovery) section for more details.

### Basic deployment with local database

This deployment use-case still uses a local database, but it exposes the ZenML server to the internet using an Ingress service with TLS certificates generated by the cert-manager and signed by Let's Encrypt.

First, you need to install cert-manager and nginx-ingress in your Kubernetes cluster. You can use the following commands to install them with their default configuration:

```bash
helm repo add jetstack https://charts.jetstack.io
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true
helm install nginx-ingress ingress-nginx/ingress-nginx --namespace nginx-ingress --create-namespace
```

Next, you need to create a ClusterIssuer resource that will be used by cert-manager to generate TLS certificates with Let's Encrypt:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <your email address here>
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class: nginx
EOF
```

Finally, you can deploy the ZenML server with the following Helm values:

```yaml
server:

  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-staging"
    tls:
      enabled: true
      generateCerts: false
```

> **Note** This use-case exposes ZenML at the root URL path of the IP address or hostname of the Ingress service. You cannot share the same Ingress hostname and URL path for multiple applications. See the next section for a solution to this problem.

### Shared Ingress controller

If the root URL path of your Ingress controller is already in use by another application, you cannot use it for ZenML. This section presents three possible solutions to this problem.

#### Use a dedicated Ingress hostname for ZenML

If you know the IP address of the load balancer in use by your Ingress controller, you can use a service like <https://nip.io/> to create a new DNS name associated with it and expose ZenML at this new root URL path. For example, if your Ingress controller has the IP address `192.168.10.20`, you can use a DNS name like `zenml.192.168.10.20.nip.io` to expose ZenML at the root URL path `https://zenml.192.168.10.20.nip.io`.

To find the IP address of your Ingress controller, you can use a command like the following:

```bash
kubectl -n nginx-ingress get svc nginx-ingress-ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```

You can deploy the ZenML server with the following Helm values:

```yaml
server:

  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-staging"
    host: zenml.<nginx ingress IP address>.nip.io
    tls:
      enabled: true
      generateCerts: false
```

> **Note** This method does not work if your Ingress controller is behind a load balancer that uses a hostname mapped to several IP addresses instead of an IP address.

#### Use a dedicated Ingress URL path for ZenML

If you cannot use a dedicated Ingress hostname for ZenML, you can use a dedicated Ingress URL path instead. For example, you can expose ZenML at the URL path `https://<your ingress hostname>/zenml`.

To deploy the ZenML server with a dedicated Ingress URL path, you can use the following Helm values:

```yaml
server:

  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-staging"
      nginx.ingress.kubernetes.io/rewrite-target: /$1
    path: /zenml/?(.*)
    tls:
      enabled: true
      generateCerts: false
```

> **Note** This method has one current limitation: the ZenML UI does not support URL rewriting and will not work properly if you use a dedicated Ingress URL path. You can still connect your client to the ZenML server and use it to run pipelines as usual, but you will not be able to use the ZenML UI.

#### Use a DNS service to map a different hostname to the Ingress controller

This method requires you to configure a DNS service like AWS Route 53 or Google Cloud DNS to map a different hostname to the Ingress controller. For example, you can map the hostname `zenml.<subdomain>` to the Ingress controller's IP address or hostname. Then, simply use the new hostname to expose ZenML at the root URL path.

### Gateway API configuration

ZenML also supports Kubernetes Gateway API through `HTTPRoute` resources. This is useful when your platform team manages shared Gateway infrastructure and your application chart only needs to attach routes.

Use the following values pattern:

```yaml
server:
  ingress:
    enabled: false
  gateway:
    enabled: true
    annotations: {}
    gatewayRef:
      name: zenml-gateway
      namespace: gateway-infra
    sectionName: https-backend
    host: zenml.example.com
    path: /
```

Both `server.ingress.enabled` and `server.gateway.enabled` can be set to `true` simultaneously. This is useful during migration periods: the chart renders both an Ingress and an HTTPRoute, allowing traffic to flow through whichever controller DNS points to. Kubernetes controllers only act on resources they own — an ingress controller ignores HTTPRoutes and a Gateway controller ignores Ingresses — so enabling both creates no conflict.

For a full migration flow (prerequisites, rollout strategy, TLS options, DNS cutover, and rollback), see [Migrate to Gateway API](/deploying-zenml/deploying-zenml/deploy-with-helm/migrate-to-gateway-api).

### Secret Store configuration

Unless explicitly disabled or configured otherwise, the ZenML server will use the SQL database as [a secrets store backend](/deploying-zenml/deploying-zenml/secret-management) where secret values are stored. If you want to use an external secrets management service like the AWS Secrets Manager, GCP Secrets Manager, Azure Key Vault, HashiCorp Vault or even your custom Secrets Store back-end implementation instead, you need to configure it in the Helm values. Depending on where you deploy your ZenML server and how your Kubernetes cluster is configured, you will also need to provide the credentials needed to access the secrets management service API.

> **Important:** If you are updating the configuration of your ZenML Server deployment to use a different secrets store back-end or location, you should follow [the documented secrets migration strategy](/deploying-zenml/deploying-zenml/secret-management#secrets-migration-strategy) to minimize downtime and to ensure that existing secrets are also properly migrated.

{% tabs %}
{% tab title="AWS" %}
**Using the SQL database as a secrets store backend (default)**

The SQL database is used as the default location where the ZenML secrets store keeps the secret values. You only need to configure these options if you want to change the default behavior.

It is particularly recommended to enable encryption at rest for the SQL database if you plan on using it as a secrets store backend. You'll have to configure the secret key used to encrypt the secret values. If not set, encryption will not be used and passwords will be stored unencrypted in the database. This value should be set to a random string with a recommended length of at least 32 characters, e.g.:

* generate a random string with Python:

```python
from secrets import token_hex
token_hex(32)
```

* or with OpenSSL:

```shell
openssl rand -hex 32
```

* then configure it in the Helm values:

```yaml
 server:

   # ...

   # Secrets store settings. This is used to store centralized secrets.
   secretsStore:

     # The type of the secrets store
     type: sql

     # Configuration for the SQL secrets store
     sql:
       encryptionKey: 0f00e4282a3181be32c108819e8a860a429b613e470ad58531f0730afff64545
```

> **Important:** If you configure encryption for your SQL database secrets store, you should keep the `encryptionKey` value somewhere safe and secure, as it will always be required by the ZenML Server to decrypt the secrets in the database. If you lose the encryption key, you will not be able to decrypt the secrets anymore and will have to reset them.
> {% endtab %}

{% tab title="AWS" %}
**Using the AWS Secrets Manager as a secrets store backend**

The AWS Secrets Store uses the ZenML AWS Service Connector under the hood to authenticate with the AWS Secrets Manager API. This means that you can use any of the [authentication methods supported by the AWS Service Connector](https://docs.zenml.io/stacks/service-connectors/connector-types/aws-service-connector#authentication-methods) to authenticate with the AWS Secrets Manager API.

The minimum set of permissions that must be attached to the implicit or configured AWS credentials are: `secretsmanager:CreateSecret`, `secretsmanager:GetSecretValue`, `secretsmanager:DescribeSecret`, `secretsmanager:PutSecretValue`, `secretsmanager:TagResource` and `secretsmanager:DeleteSecret` and they must be associated with secrets that have a name starting with `zenml/` in the target region and account. The following IAM policy example can be used as a starting point:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ZenMLSecretsStore",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:CreateSecret",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:PutSecretValue",
                "secretsmanager:TagResource",
                "secretsmanager:DeleteSecret"
            ],
            "Resource": "arn:aws:secretsmanager:<AWS-region>:<AWS-account-id>:secret:zenml/*"
        }
    ]
}
```

Example configuration for the AWS Secrets Store:

```yaml
 server:

   # ...

   # Secrets store settings. This is used to store centralized secrets.
   secretsStore:

     # Set to false to disable the secrets store.
     enabled: true

     # The type of the secrets store
     type: aws

     # Configuration for the AWS Secrets Manager secrets store
     aws:

       # The AWS Service Connector authentication method to use.
       authMethod: secret-key

       # The AWS Service Connector configuration.
       authConfig:
        # The AWS region to use. This must be set to the region where the AWS
        # Secrets Manager service that you want to use is located.
        region: us-east-1

        # The AWS credentials to use to authenticate with the AWS Secrets
        aws_access_key_id: <your AWS access key ID>
        aws_secret_access_key: <your AWS secret access key>
```

{% endtab %}

{% tab title="GCP" %}
**Using the GCP Secrets Manager as a secrets store backend**

The GCP Secrets Store uses the ZenML GCP Service Connector under the hood to authenticate with the GCP Secrets Manager API. This means that you can use any of the [authentication methods supported by the GCP Service Connector](https://docs.zenml.io/stacks/service-connectors/connector-types/gcp-service-connector#authentication-methods) to authenticate with the GCP Secrets Manager API.

The minimum set of permissions that must be attached to the implicit or configured GCP credentials are as follows:

* `secretmanager.secrets.create` for the target GCP project (i.e. no condition on the name prefix)
* `secretmanager.secrets.get`, `secretmanager.secrets.update`, `secretmanager.versions.access`, `secretmanager.versions.add` and `secretmanager.secrets.delete` for the target GCP project and for secrets that have a name starting with `zenml-`

This can be achieved by creating two custom IAM roles and attaching them to the principal (e.g. user or service account) that will be used to access the GCP Secrets Manager API with a condition configured when attaching the second role to limit access to secrets with a name prefix of `zenml-`. The following `gcloud` CLI command examples can be used as a starting point:

```bash
gcloud iam roles create ZenMLServerSecretsStoreCreator \
  --project <your GCP project ID> \
  --title "ZenML Server Secrets Store Creator" \
  --description "Allow the ZenML Server to create new secrets" \
  --stage GA \
  --permissions "secretmanager.secrets.create"

gcloud iam roles create ZenMLServerSecretsStoreEditor \
  --project <your GCP project ID> \
  --title "ZenML Server Secrets Store Editor" \
  --description "Allow the ZenML Server to manage its secrets" \
  --stage GA \
  --permissions "secretmanager.secrets.get,secretmanager.secrets.update,secretmanager.versions.access,secretmanager.versions.add,secretmanager.secrets.delete"

gcloud projects add-iam-policy-binding <your GCP project ID> \
  --member serviceAccount:<your GCP service account email> \
  --role projects/<your GCP project ID>/roles/ZenMLServerSecretsStoreCreator \
  --condition None

# NOTE: use the GCP project NUMBER, not the project ID in the condition
gcloud projects add-iam-policy-binding <your GCP project ID> \
  --member serviceAccount:<your GCP service account email> \
  --role projects/<your GCP project ID>/roles/ZenMLServerSecretsStoreEditor \
  --condition 'title=limit_access_zenml,description="Limit access to secrets with prefix zenml-",expression=resource.name.startsWith("projects/<your GCP project NUMBER>/secrets/zenml-")'
```

Example configuration for the GCP Secrets Store:

```yaml
 server:

   # ...

   # Secrets store settings. This is used to store centralized secrets.
   secretsStore:

     # Set to false to disable the secrets store.
     enabled: true

     # The type of the secrets store
     type: gcp

     # Configuration for the GCP Secrets Manager secrets store
     gcp:

       # The GCP Service Connector authentication method to use.
       authMethod: service-account

       # The GCP Service Connector configuration.
       authConfig:

          # The GCP project ID to use. This must be set to the project ID where the
          # GCP Secrets Manager service that you want to use is located.
          project_id: my-gcp-project

          # GCP credentials JSON to use to authenticate with the GCP Secrets
          # Manager instance. 
          google_application_credentials: |
            {
              "type": "service_account",
              "project_id": "my-project",
              "private_key_id": "...",
              "private_key": "-----BEGIN PRIVATE KEY-----\n...=\n-----END PRIVATE KEY-----\n",
              "client_email": "...",
              "client_id": "...",
              "auth_uri": "https://accounts.google.com/o/oauth2/auth",
              "token_uri": "https://oauth2.googleapis.com/token",
              "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
              "client_x509_cert_url": "..."
            }

 serviceAccount:

   # If you're using workload identity, you need to annotate the service
   # account with the GCP service account name (see https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
   annotations:
     iam.gke.io/gcp-service-account: <SERVICE_ACCOUNT_NAME>@<PROJECT_NAME>.iam.gserviceaccount.com

```

{% endtab %}

{% tab title="Azure" %}
**Using the Azure Key Vault as a secrets store backend**

The Azure Secrets Store uses the ZenML Azure Service Connector under the hood to authenticate with the Azure Key Vault API. This means that you can use any of the [authentication methods supported by the Azure Service Connector](https://docs.zenml.io/stacks/service-connectors/connector-types/azure-service-connector#authentication-methods) to authenticate with the Azure Key Vault API.

Example configuration for the Azure Key Vault Secrets Store:

```yaml
 server:

   # ...

   # Secrets store settings. This is used to store centralized secrets.
   secretsStore:

     # Set to false to disable the secrets store.
     enabled: true

     # The type of the secrets store
     type: azure

     # Configuration for the Azure Key Vault secrets store
     azure:

       # The name of the Azure Key Vault. This must be set to point to the Azure
       # Key Vault instance that you want to use.
       key_vault_name:

       # The Azure Service Connector authentication method to use.
       authMethod: service-principal

       # The Azure Service Connector configuration.
       authConfig:

          # The Azure application service principal credentials to use to
          # authenticate with the Azure Key Vault API.
          client_id: <your Azure client ID>
          client_secret: <your Azure client secret>
          tenant_id: <your Azure tenant ID>
```

{% endtab %}

{% tab title="Hashicorp" %}
**Using the HashiCorp Vault as a secrets store backend**

To use the HashiCorp Vault service as a Secrets Store back-end, it must be configured in the Helm values:

```yaml
 server:

   # ...

   # Secrets store settings. This is used to store centralized secrets.
   secretsStore:

     # Set to false to disable the secrets store.
     enabled: true

     # The type of the secrets store
     type: hashicorp

     # Configuration for the HashiCorp Vault secrets store
     hashicorp:

       # The url of the HashiCorp Vault server to use
       vault_addr: https://vault.example.com
       # The token used to authenticate with the Vault server
       vault_token: <your Vault token>
       # The Vault Enterprise namespace. Not required for Vault OSS.
       vault_namespace: <your Vault namespace>
       # The mount point to use for the HashiCorp Vault secrets store. If not set, the default value of `secret` will be used.
       mount_point: <your Vault mount point>
```

{% endtab %}

{% tab title="Custom" %}
**Using a custom secrets store backend implementation**

You have the option of using [a custom implementation of the secrets store API](/deploying-zenml/deploying-zenml/secret-management) as your secrets store back-end. This must come in the form of a class derived from `zenml.zen_stores.secrets_stores.base_secrets_store.BaseSecretsStore`. This class must be importable from within the ZenML server container, which means you most likely need to build a custom container image that contains the class. Then, you can configure the Helm values to use your custom secrets store as follows:

```yaml
 server:

   # ...

   # Secrets store settings. This is used to store centralized secrets.
   secretsStore:

     # Set to false to disable the secrets store.
     enabled: true

     # The type of the secrets store
     type: custom

     # Configuration for the HashiCorp Vault secrets store
     custom:

       # The class path of the custom secrets store implementation. This should
       # point to a full Python class that extends the
       # `zenml.zen_stores.secrets_stores.base_secrets_store.BaseSecretsStore`
       # base class. The class should be importable from the container image
       # that you are using for the ZenML server.
       class_path: my.custom.secrets.store.MyCustomSecretsStore

   # Extra environment variables used to configure the custom secrets store.
   environment:
     ZENML_SECRETS_STORE_OPTION_1: value1
     ZENML_SECRETS_STORE_OPTION_2: value2

   # Extra environment variables to set in the ZenML server container that
   # should be kept secret and are used to configure the custom secrets store.
   secretEnvironment:
     ZENML_SECRETS_STORE_SECRET_OPTION_3: value3
     ZENML_SECRETS_STORE_SECRET_OPTION_4: value4

   # Extra environment variables to set in the ZenML server container that
   # are injected from external secret references. These references are also
   # applied to the DB migration job and worker deployments.
   environmentSecretKeyRefs:
      - name: ZENML_SECRETS_STORE_OPTION_5
        secretName: my-existing-secret
        secretKey: key_name_not_value5
      - name: ZENML_SECRETS_STORE_OPTION_6
        secretName: my-existing-secret
        secretKey: key_name_not_value6
```

{% endtab %}
{% endtabs %}

#### Backup secrets store

[A backup secrets store](/deploying-zenml/deploying-zenml/secret-management#backup-secrets-store) back-end may be configured for high-availability and backup purposes. or as an intermediate step in the process of [migrating secrets to a different external location or secrets manager provider](/deploying-zenml/deploying-zenml/secret-management#secrets-migration-strategy).

To configure a backup secrets store in the Helm chart, use the same approach and instructions documented for the primary secrets store, but using the `backupSecretsStore` configuration section instead of `secretsStore`, e.g.:

```yaml
 server:

   # ...

   # Backup secrets store settings. This is used as a backup for the primary
   # secrets store.
   backupSecretsStore:

     # Set to true to enable the backup secrets store.
     enabled: true

     # The type of the backup secrets store
     type: aws

     # Configuration for the AWS Secrets Manager backup secrets store
     aws:

       # The AWS Service Connector authentication method to use.
       authMethod: secret-key

       # The AWS Service Connector configuration.
       authConfig:
        # The AWS region to use. This must be set to the region where the AWS
        # Secrets Manager service that you want to use is located.
        region: us-east-1

        # The AWS credentials to use to authenticate with the AWS Secrets
        aws_access_key_id: <your AWS access key ID>
        aws_secret_access_key: <your AWS secret access key>
```

### Database backup and recovery

An automated database backup and recovery feature is enabled by default for all Helm deployments. During Helm updates, the ZenML server will automatically back up the database before upgrading it and restore it if the upgrade fails.

{% hint style="info" %}
The database backup automatically created by the ZenML server is only temporary and only used as an immediate recovery in case of database migration failures. It is not meant to be used as a long-term backup solution. If you need to back up your database for long-term storage, you should use a dedicated backup solution.
{% endhint %}

Several database backup strategies are supported, depending on where and how the backup is stored. The strategy can be configured by means of the `server.database.backupStrategy` Helm value:

* `disabled` - no backup is performed
* `in-memory` - the database schema and data are stored in memory. This is the fastest backup strategy, but the backup is not persisted across pod restarts, so no manual intervention is possible in case the automatic DB recovery fails after a failed DB migration. Adequate memory resources should be allocated to the ZenML server pod when using this backup strategy with larger databases. This is the default backup strategy.
* `database` - the database is copied to a backup database in the same database server. This requires the `backupDatabase` option to be set to the name of the backup database. This backup strategy is only supported for MySQL compatible databases and the user specified in the database URL must have permissions to manage (create, drop, and modify) the backup database in addition to the main database.
* `dump-file` - the database schema and data are dumped to a file local to the database initialization and upgrade job. Users may optionally configure a persistent volume where the dump file will be stored by setting the `backupPVStorageSize` and optionally the `backupPVStorageClass` options. If a persistent volume is not configured, the dump file will be stored in an emptyDir volume, which is not persisted. If configured, the user is responsible for deleting the resulting PVC when uninstalling the Helm release.
* `mydumper` - the database is backed up using mydumper/myloader. This requires the `mydumper` and `myloader` utilities to be installed in the ZenML server container. The `mydumperThreads`, `mydumperCompress`, `mydumperExtraArgs`, `myloaderThreads`, and `myloaderExtraArgs` options can be used to configure the backup and restore processes.
* `custom` - use a custom backup engine. This requires the `customBackupEngine` option to be set to the class path of the custom backup engine. The class should extend from the `zenml.zen_stores.migrations.backup.base_backup_engine.BaseBackupEngine` base class and be importable from the container image that you are using for the ZenML server. Arguments for the custom backup engine can be passed using the `customBackupEngineConfig` option.

> **NOTE:** You should also set the `podSecurityContext.fsGroup` option if you are using a persistent volume to store the dump file.

{% hint style="warning" %}
When running in production where database sizes are large, you should use the `mydumper` backup strategy or write your own custom backup engine. The other backup strategies are not recommended because they are inefficient and will take a long time and consume a lot of resources to handle large databases.
{% endhint %}

The following additional rules are applied concerning the creation and lifetime of the backup:

* a backup is not attempted if the database doesn't need to undergo a migration (e.g. when the ZenML server is upgraded to a new version that doesn't require a database schema change or if the ZenML version doesn't change at all).
* a backup file or database is created before every database migration attempt (i.e. during every Helm upgrade). If a backup already exists (i.e. persisted in a persistent volume or backup database), it is NOT overwritten. Instead, the existing backup is used to rollback the database to the previous state in case the migration fails again.
* the persistent backup file or database is cleaned up after the migration is completed successfully or if the database doesn't need to undergo a migration. This includes backups created by previous failed migration attempts.
* the persistent backup file or database is NOT cleaned up after a failed migration. This allows the user to manually inspect and/or apply the backup if the automatic recovery fails.

The following example shows how to configure the ZenML server to use a persistent volume to store the database dump file:

```yaml
 server:

   # ...

  database:
    url: "mysql://admin:password@my.database.org:3306/zenml"

    # Configure the database backup strategy
    backupStrategy: dump-file
    backupPVStorageSize: 1Gi

podSecurityContext:
  fsGroup: 1000 # if you're using a PVC for backup, this should necessarily be set.
```

### Observability and OpenTelemetry

You can configure server log output and OpenTelemetry export through dedicated Helm values. For example:

```yaml
server:
  logging:
    verbosity: <debug|info|warning|error|critical>  # default is info
    format: <console|json|custom_format>  # default is console
    colorsDisabled: <true|false>  # default is false
  openTelemetry:
    endpoint: http://otel-collector:4318
    serviceName: zenml-server
```

`server.logging.verbosity` sets the ZenML server log level. The legacy `server.debug` option is still supported for compatibility and forces the server log level to `debug` when set to `true`, but new deployments should use `server.logging.verbosity` instead.

`server.logging.format` can be set to `console` (default), `json`, or a valid Python `%`-style logging format string. This controls the server container stdout/stderr output, i.e. the logs that Kubernetes pod log collectors scrape. The older `ZENML_LOGGING_FORMAT` environment variable is still supported through `server.environment` as a deprecated alias but will be removed in a future version.

Setting `server.openTelemetry.endpoint` enables server OpenTelemetry instrumentation and exports traces, metrics, and logs using OTLP/HTTP. The standard `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable is also supported as a fallback through `server.environment`. Configure the base collector endpoint and ZenML appends `/v1/traces`, `/v1/metrics`, and `/v1/logs` for each signal.

You can override individual signal endpoints with the standard `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` variables, or with the matching `ZENML_SERVER_OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT` names.

If no base or per-signal endpoint is set, server OpenTelemetry instrumentation is disabled.

Standard OTLP headers, timeout, and compression variables are handled by the OpenTelemetry Python exporters. OTLP/gRPC protocol variables are not supported because the server configures OTLP/HTTP exporters directly. You can read more about the OpenTelemetry environment variables and SDK configuration [here](https://opentelemetry.io/docs/languages/sdk-configuration/).

The server logs export to the OpenTelemetry collector is independent of `ZENML_CONSOLE_LOGGING_FORMAT`: logs are exported as OTLP records with structured attributes derived from the underlying log record, not as console-formatted text or JSON.

### Custom CA Certificates

If you need to connect to services using HTTPS with certificates signed by custom Certificate Authorities (e.g., self-signed certificates), you can configure custom CA certificates. There are two ways to provide custom CA certificates:

1. Direct injection in values.yaml:

```yaml
server:
  certificates:
    customCAs:
      - name: "my-custom-ca"
        certificate: |
          -----BEGIN CERTIFICATE-----
          MIIDXTCCAkWgAwIBAgIJAJC1HiIAZAiIMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV
          ...
          -----END CERTIFICATE-----
```

2. Reference existing Kubernetes secrets:

```yaml
server:
  certificates:
    secretRefs:
      - name: "my-secret"
        key: "ca.crt"
```

The certificates will be installed in the server container, allowing it to securely connect to services using these custom CA certificates.

### HTTP Proxy Configuration

If your environment requires a proxy for external connections, you can configure it using:

```yaml
server:
  proxy:
    enabled: true
    httpProxy: "http://proxy.example.com:8080"
    httpsProxy: "http://proxy.example.com:8080"
    # Additional hostnames/domains/IPs/CIDRs to exclude from proxying
    additionalNoProxy:
      - "internal.example.com"
      - "10.0.0.0/8"
```

By default, the following hostnames/domains are excluded from proxying:

* `localhost`, `127.0.0.1`, `::1` (IPv4 and IPv6 localhost)
* `fe80::/10` (IPv6 link-local addresses)
* `.svc` and `.svc.cluster.local` (Kubernetes service DNS domains)
* The hostname from `server.serverURL` if configured
* The ingress hostname (`server.ingress.host`) if configured
* Internal service names used for communication between components

You can add additional exclusions using the `additionalNoProxy` list. The NO\_PROXY environment variable accepts:

* Hostnames (e.g., "zenml.example.com")
* Domain names with leading dot for wildcards (e.g., ".example.com")
* IPv4 addresses (e.g., "10.0.0.1")
* IPv4 ranges in CIDR notation (e.g., "10.0.0.0/8")
* IPv6 addresses (e.g., "::1")
* IPv6 ranges in CIDR notation (e.g., "fe80::/10")

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Migrate to Gateway API

Migrate ZenML Helm deployments from Ingress to Kubernetes Gateway API.

Gateway API is the Kubernetes networking successor to the legacy Ingress model. If you currently expose ZenML through `server.ingress`, this guide shows how to migrate to `server.gateway` with minimal downtime.

## Why migrate

* Gateway API is the long-term Kubernetes direction for north-south traffic.
* It provides clearer separation between shared infrastructure ownership (`Gateway`) and app routing ownership (`HTTPRoute`).
* It maps well to multi-tenant environments where platform teams manage shared gateways and application teams manage routes.

## Prerequisites

* Kubernetes 1.26+
* A Gateway API implementation with `v1` CRDs available (for example, `gateway.networking.k8s.io/v1` for `Gateway` and `HTTPRoute`; Envoy Gateway, Istio, NGINX Gateway Fabric, GKE Gateway)
* A `Gateway` resource already provisioned and reachable using the installed Gateway API `v1` CRDs
* DNS access for cutover planning

> ZenML renders `HTTPRoute` resources with `apiVersion: gateway.networking.k8s.io/v1`. Clusters that only have older Gateway API CRDs installed (for example, `v1beta1`) must upgrade those CRDs before enabling `server.gateway`.

## Migration overview

1. Install or verify your Gateway API implementation.
2. Create a shared `Gateway` (platform-managed).
3. Enable `server.gateway` in ZenML Helm values (alongside existing `server.ingress` for zero-downtime migration).
4. Deploy and verify `HTTPRoute` status and application health.
5. Perform DNS cutover (if using a new load balancer).
6. Disable `server.ingress` after stabilization.

## Helm values migration

### Step 1: Enable both (parallel period)

Both `server.ingress.enabled` and `server.gateway.enabled` can be `true` simultaneously. The chart renders both an Ingress and an HTTPRoute. Kubernetes controllers only act on resources they own — an ingress controller ignores HTTPRoutes and a Gateway controller ignores Ingresses — so enabling both creates no conflict.

```yaml
server:
  ingress:
    enabled: true              # keep Ingress active during migration
    host: zenml.example.com
  gateway:
    enabled: true              # also create HTTPRoute
    gatewayRef:
      name: zenml-gateway
      namespace: gateway-infra
    sectionName: https-backend
    host: zenml.example.com
    path: /
```

At this point, traffic still flows through the ingress controller (DNS points to its load balancer). The HTTPRoute is ready on the Gateway side but idle.

### Step 2: DNS cutover

Update DNS to point to the Gateway's load balancer. Both the Ingress and HTTPRoute exist, so the transition is seamless — the ingress controller handles traffic until DNS propagates, then the Gateway takes over.

### Step 3: Disable Ingress (cleanup)

After DNS is stable and traffic is flowing through the Gateway, disable Ingress:

```yaml
server:
  ingress:
    enabled: false
  gateway:
    enabled: true
    gatewayRef:
      name: zenml-gateway
      namespace: gateway-infra
    sectionName: https-backend
    host: zenml.example.com
    path: /
```

## Gateway implementation examples

### Envoy Gateway

Use a shared `GatewayClass` and `Gateway`, then attach ZenML `HTTPRoute` resources via `parentRefs`.

### Istio

Use Istio's Gateway API support (managed Gateway + HTTPRoute) and keep ZenML route ownership in tenant namespaces.

### NGINX Gateway Fabric

Deploy NGINX Gateway Fabric and configure a shared `Gateway` listener; point ZenML `gatewayRef` to that shared gateway.

### GKE Gateway

Create a GKE-managed `Gateway` and map your external DNS hostnames to the provisioned load balancer.

## Custom annotations

You can add custom annotations to the `HTTPRoute` resource:

```yaml
server:
  gateway:
    enabled: true
    annotations:
      my-annotation: my-value
    gatewayRef:
      name: zenml-gateway
      namespace: gateway-infra
```

This is useful for attaching Envoy-specific policies, adding metadata for service mesh integration, or custom routing labels.

## TLS options

You can terminate TLS in one of two common ways:

* **Gateway-managed TLS** (e.g. cert-manager certificate references in gateway listeners)
* **Cloud load balancer TLS termination** (e.g. ACM on AWS NLB) with HTTP forwarded to Gateway listeners

Choose the model that matches your platform standards. ZenML only needs the correct external hostname and reachable `Gateway` listener.

## Validation checklist

* `kubectl get gateway -A` shows the target gateway as accepted/programmed
* `kubectl get httproute -A` shows ZenML routes attached/accepted
* `curl https://<zenml-host>/health` returns success
* Optional: test large uploads and UI/API login flows

## DNS cutover

If migration introduces a new load balancer:

1. Lower DNS TTL (for example 60s) before cutover.
2. Update CNAME/A records to the new load balancer.
3. Monitor health, latency, and error rate.
4. Restore normal TTL after stabilization.

## Rollback

Since both Ingress and Gateway can be enabled simultaneously, rollback during the parallel period is simply a DNS change — point DNS back to the ingress controller's load balancer. No Helm changes needed.

If you've already disabled Ingress (Step 3), re-enable it:

```yaml
server:
  ingress:
    enabled: true
    host: zenml.example.com
  gateway:
    enabled: false
```

Re-deploy, repoint DNS to the ingress controller's load balancer, and verify `/health` and login.

Keep both infrastructure paths available during stabilization to ensure a low-risk rollback.


# Deploy using HuggingFace Spaces

Deploying ZenML to Huggingface Spaces.

A quick way to deploy ZenML and get started is to use [HuggingFace Spaces](https://huggingface.co/spaces). HuggingFace Spaces is a platform for hosting and sharing ML projects and workflows, and it also works to deploy ZenML. You can be up and running in minutes (for free) with a hosted ZenML server, so it's a good option if you want to try out ZenML without any infrastructure overhead.

{% hint style="info" %}
If you are planning to use HuggingFace Spaces for production use, make sure you have [persistent storage turned on](https://huggingface.co/docs/hub/en/spaces-storage) so as to prevent loss of data. See our [other deployment options](/deploying-zenml/deploying-zenml) if you want alternative options.
{% endhint %}

![ZenML on HuggingFace Spaces -- default deployment](/files/vdYScMA1WMimsfDqfX9Q)

In this diagram, you can see what the default deployment of ZenML on HuggingFace looks like.

## Deploying ZenML on HuggingFace Spaces

You can deploy ZenML on HuggingFace Spaces with just a few clicks:

[![](https://huggingface.co/datasets/huggingface/badges/raw/main/deploy-to-spaces-lg.svg)](https://huggingface.co/new-space?template=zenml/zenml)

To set up your ZenML app, you need to specify three main components: the Owner (either your personal account or an organization), a Space name, and the Visibility (a bit lower down the page). Note that the space visibility needs to be set to 'Public' if you wish to connect to the ZenML server from your local machine.

![HuggingFace Spaces SDK interface](/files/sgiYRFgLwPGdOt8CSpct)

You have the option here to select a higher-tier machine to use for your server. The advantage of selecting a paid CPU instance is that it is not subject to auto-shutdown policies and thus will stay up as long as you leave it up. In order to make use of a persistent CPU, you'll likely want to create and set up a MySQL database to connect to (see below).

To personalize your Space's appearance, such as the title, emojis, and colors, navigate to "Files and Versions" and modify the metadata in your README.md file. Full information on Spaces configuration parameters can be found on the HuggingFace [documentation reference guide](https://huggingface.co/docs/hub/spaces-config-reference).

After creating your Space, you'll notice a 'Building' status along with logs displayed on the screen. When this switches to 'Running', your Space is ready for use. If the ZenML login UI isn't visible, try refreshing the page.

In the upper-right hand corner of your space you'll see a button with three dots which, when you click on it, will offer you a menu option to "Embed this Space". (See [the HuggingFace documentation](https://huggingface.co/docs/hub/spaces-embed) for more details on this feature.) Copy the "Direct URL" shown in the box that you can now see on the screen. This should look something like this: `https://<YOUR_USERNAME>-<SPACE_NAME>.hf.space`. Open that URL and follow the instructions to initialize your ZenML server and set up an initial admin user account.

## Connecting to your ZenML Server from your local machine

Once you have your ZenML server up and running, you can connect to it from your local machine. To do this, you'll need to get your Space's 'Direct URL' (see above).

{% hint style="warning" %}
Your Space's URL will only be available and usable for connecting from your local machine if the visibility of the space is set to 'Public'.
{% endhint %}

You can use the 'Direct URL' to connect to your ZenML server from your local machine with the following CLI command (after installing ZenML, and using your custom URL instead of the placeholder):

```shell
zenml login '<YOUR_HF_SPACES_DIRECT_URL>'
```

You can also use the Direct URL in your browser to use the ZenML dashboard as a fullscreen application (i.e. without the HuggingFace Spaces wrapper around it).

## Extra configuration options

By default, the ZenML application will be configured to use an SQLite non-persistent database. If you want to use a persistent database, you can configure this by amending the `Dockerfile` to your Space's root directory. For full details on the various parameters you can change, see [our reference documentation](/deploying-zenml/deploying-zenml/deploy-with-docker#advanced-server-configuration-options) on configuring ZenML when deployed with Docker.

{% hint style="info" %}
If you are using the space just for testing and experimentation, you don't need to make any changes to the configuration. Everything will work out of the box.
{% endhint %}

You can also use an external secrets backend together with your HuggingFace Spaces as described in [our documentation](/deploying-zenml/deploying-zenml/deploy-with-docker#advanced-server-configuration-options). You should be sure to use HuggingFace's inbuilt ' Repository secrets' functionality to configure any secrets you need to use in your`Dockerfile` configuration. [See the documentation](https://huggingface.co/docs/hub/spaces-sdks-docker#secret-management) for more details on how to set this up.

{% hint style="warning" %}
If you wish to use a cloud secrets backend together with ZenML for secrets management, **you must update your password** on your ZenML Server on the Dashboard. This is because the default user created by the HuggingFace Spaces deployment process has no password assigned to it and as the Space is publicly accessible (since the Space is public) *potentially anyone could access your secrets without this extra step*. To change your password navigate to the Settings page by clicking the button in the upper right-hand corner of the Dashboard and then click 'Update Password'.
{% endhint %}

## Troubleshooting

If you are having trouble with your ZenML server on HuggingFace Spaces, you can view the logs by clicking on the "Open Logs" button at the top of the space. This will give you more context of what's happening with your server.

If you have any other issues, please feel free to reach out to us on our [Slack channel](https://zenml.io/slack/) for more support.

## Upgrading your ZenML Server on HF Spaces

The default space will use the latest version of ZenML automatically. If you want to update your version, you can simply select the 'Factory reboot' option within the 'Settings' tab of the space. Note that this will wipe any data contained within the space and so if you are not using a MySQL persistent database (as described above) you will lose any data contained within your ZenML deployment on the space. You can also configure the space to use an earlier version by updating the `Dockerfile`'s `FROM` import statement at the very top.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Deploy with custom images

Deploying ZenML with custom Docker images.

In most cases, deploying ZenML with the default `zenmlhub/zenml-server` Docker image should work just fine. However, there are some scenarios when you might need to deploy ZenML with a custom Docker image:

* You have implemented a custom artifact store for which you want to enable [artifact visualizations](/concepts/artifacts/visualizations) or [step logs](/concepts/steps_and_pipelines/logging) in your dashboard.
* You have forked the ZenML repository and want to deploy a ZenML server based on your own fork because you made changes to the server / database logic.

{% hint style="warning" %}
Deploying ZenML with custom Docker images is only possible for [Docker](/deploying-zenml/deploying-zenml/deploy-with-docker) or [Helm](/deploying-zenml/deploying-zenml/deploy-with-helm) deployments.
{% endhint %}

### Build and Push Custom ZenML Server Docker Image

Here is how you can build a custom ZenML server Docker image:

1. Set up a container registry of your choice. E.g., as an indivial developer you could create a free [Docker Hub](https://hub.docker.com/) account and then set up a free Docker Hub repository.
2. Clone ZenML (or your ZenML fork) and checkout the branch that you want to deploy, e.g., if you want to deploy ZenML version 0.41.0, run

   ```bash
   git checkout release/0.41.0
   ```
3. Copy the [ZenML base.Dockerfile](https://github.com/zenml-io/zenml/blob/main/docker/base.Dockerfile), e.g.:

   ```bash
   cp docker/base.Dockerfile docker/custom.Dockerfile
   ```
4. Modify the copied Dockerfile:

   * Add additional dependencies:

   ```bash
   RUN pip install <my_package>
   ```

   * (Forks only) install local files instead of official ZenML:

   ```bash
   RUN pip install -e .[server,secrets-aws,secrets-gcp,secrets-azure,secrets-hashicorp,s3fs,gcsfs,adlfs,connectors-aws,connectors-gcp,connectors-azure]
   ```
5. Build and push an image based on your Dockerfile:

   ```bash
   docker build -f docker/custom.Dockerfile . -t <YOUR_CONTAINER_REGISTRY>/<IMAGE_NAME>:<IMAGE_TAG> --platform linux/amd64
   docker push <YOUR_CONTAINER_REGISTRY>/<IMAGE_NAME>:<IMAGE_TAG>
   ```

{% hint style="info" %}
If you want to verify your custom image locally, you can follow the [Deploy a custom ZenML image via Docker](#deploy-a-custom-zenml-image-via-docker) section below to deploy the ZenML server locally first.
{% endhint %}

### Deploy ZenML with your custom image

Next, adjust your preferred deployment strategy to use the custom Docker image you just built.

#### Deploy a custom ZenML image via Docker

To deploy your custom image via Docker, first familiarize yourself with the general [ZenML Docker Deployment Guide](/deploying-zenml/deploying-zenml/deploy-with-docker).

To use your own image, follow the general guide step by step but replace all mentions of `zenmldocker/zenml-server` with your custom image reference `<YOUR_CONTAINER_REGISTRY>/<IMAGE_NAME>:<IMAGE_TAG>`. E.g.:

* To run the ZenML server with Docker based on your custom image, do

```bash
docker run -it -d -p 8080:8080 --name zenml <YOUR_CONTAINER_REGISTRY>/<IMAGE_NAME>:<IMAGE_TAG>
```

* To use `docker-compose`, adjust your `docker-compose.yml`:

```yaml
services:
  zenml:
    image: <YOUR_CONTAINER_REGISTRY>/<IMAGE_NAME>:<IMAGE_TAG>
```

#### Deploy a custom ZenML image via Helm

To deploy your custom image via Helm, first familiarize yourself with the general [ZenML Helm Deployment Guide](/deploying-zenml/deploying-zenml/deploy-with-helm).

To use your own image, the only thing you need to do differently is to modify the `image` section of your `values.yaml` file:

```yaml
server:
  image:
    repository: <YOUR_CONTAINER_REGISTRY>/<IMAGE_NAME>
    tag: <IMAGE_TAG>
```

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Secret management

Configuring the secrets store.

## Centralized secrets store

ZenML provides a centralized secrets management system that allows you to register and manage secrets in a secure way. The metadata of the ZenML secrets (e.g. name, ID, owner, scope etc.) is always stored in the ZenML server database, while the actual secret values are stored and managed separately, through the ZenML Secrets Store. This allows for a flexible deployment strategy that meets the security and compliance requirements of your organization.

In a local ZenML deployment, secret values are also stored in the local SQLite database. When connected to a remote ZenML server, the secret values are stored in the secrets management back-end that the server's Secrets Store is configured to use, while all access to the secrets is done through the ZenML server API.

<figure><img src="/files/iQ37lfI22hwaCo4oN5iH" alt=""><figcaption><p>Basic Secrets Store Architecture</p></figcaption></figure>

Currently, the ZenML server can be configured to use one of the following supported secrets store back-ends:

* the same SQL database that the ZenML server is using to store secrets metadata as well as other managed objects such as pipelines, stacks, etc. This is the default option.
* the AWS Secrets Manager
* the GCP Secret Manager
* the Azure Key Vault
* the HashiCorp Vault
* a custom secrets store back-end implementation is also supported

## Secret value size limits

A ZenML **secret** is a name plus a map of string keys to string values. You can reason about its size as the sum of the UTF-8 size of every key name and every value in the map, not a per-key limit. Non-Latin text usually needs more bytes per character than English in that total.

When the server uses the default SQL (server database) secrets store, ZenML enforces a 64 KiB (65,536 byte) maximum on that total, and the server rejects larger payloads.

For AWS, GCP, Azure, HashiCorp Vault, a custom store, or any other non-default back end, the effective limit is whatever that system allows for the stored payload; use the table and official links below.

| **Secrets store**               | **Approx. limit**        | **More information**                                                                                                                                               |
| ------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Default (server database)**   | **64 KiB**               | —                                                                                                                                                                  |
| **AWS Secrets Manager**         | **64 KiB**               | [Secrets Manager quotas](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_limits.html)                                                        |
| **Google Cloud Secret Manager** | **64 KiB**               | [Quotas and limits](https://cloud.google.com/secret-manager/quotas)                                                                                                |
| **Azure Key Vault**             | **25 KB**                | [About secrets (size and attributes)](https://learn.microsoft.com/en-us/azure/key-vault/secrets/about-secrets)                                                     |
| **HashiCorp Vault (KV v2)**     | **\~512 KiB to \~1 MiB** | [Vault KV v2 size limits](https://developer.hashicorp.com/vault/docs/internals/limits#versioned-key-value-store-kv-v2-secret-engine) (depends on storage back-end) |
| **Custom**                      | **Varies**               |                                                                                                                                                                    |

If the content is still too big, register another ZenML secret (a new name), split keys across secrets, or keep the file in an artifact or object store and only put a reference in a value.

## Configuration and deployment

Configuring the specific secrets store back-end that the ZenML server uses is done at deployment time. This involves deciding on one of the supported back-ends and authentication mechanisms and configuring the ZenML server with the necessary credentials to authenticate with the back-end.

The ZenML secrets store reuses the [ZenML Service Connector](https://docs.zenml.io/stacks/service-connectors/auth-management) authentication mechanisms to authenticate with the secrets store back-end. This means that the same authentication methods and configuration parameters that are supported by the available Service Connectors are also reflected in the ZenML secrets store configuration. It is recommended to practice the principle of least privilege when configuring the ZenML secrets store and to use credentials with the documented minimum required permissions to access the secrets store back-end.

The ZenML secrets store configured for the ZenML Server can be updated at any time by updating the ZenML Server configuration and redeploying the server. This allows you to easily switch between different secrets store back-ends and authentication mechanisms. However, it is recommended to follow [the documented secret store migration strategy](#secrets-migration-strategy) to minimize downtime and to ensure that existing secrets are also properly migrated, in case the location where secrets are stored in the back-end changes.

For more information on how to deploy a ZenML server and configure the secrets store back-end, refer to your deployment strategy inside the deployment guide.

## Backup secrets store

The ZenML Server deployment may be configured to optionally connect to *a second Secrets Store* to provide additional features such as high-availability, backup and disaster recovery as well as an intermediate step in the process of migrating [secrets from one secrets store location to another](#secrets-migration-strategy). For example, the primary Secrets Store may be configured to use the internal database, while the backup Secrets Store may be configured to use the AWS Secrets Manager. Or two different AWS Secrets Manager accounts or regions may be used.

{% hint style="warning" %}
Always make sure that the backup Secrets Store is configured to use a different location than the primary Secrets Store. The location can be different in terms of the Secrets Store back-end type (e.g. internal database vs. AWS Secrets Manager) or the actual location of the Secrets Store back-end (e.g. different AWS Secrets Manager account or region, GCP Secret Manager project or Azure Key Vault's vault).

Using the same location for both the primary and backup Secrets Store will not provide any additional benefits and may even result in unexpected behavior.
{% endhint %}

When a backup secrets store is in use, the ZenML Server will always attempt to read and write secret values from/to the primary Secrets Store first while ensuring to keep the backup Secrets Store in sync. If the primary Secrets Store is unreachable, if the secret values are not found there, or any otherwise unexpected error occurs, the ZenML Server falls back to reading and writing from/to the backup Secrets Store. Only if the backup Secrets Store is also unavailable, the ZenML Server will return an error.

In addition to the hidden backup operations, users can also explicitly trigger a backup operation by using the `zenml secret backup` CLI command. This command will attempt to read all secrets from the primary Secrets Store and write them to the backup Secrets Store. Similarly, the `zenml secret restore` CLI command can be used to restore secrets from the backup Secrets Store to the primary Secrets Store. These CLI commands are useful for migrating secrets from one Secrets Store to another.

## Secrets migration strategy

Sometimes you may need to change the external provider or location where secrets values are stored by the Secrets Store. The immediate implication of this is that the ZenML server will no longer be able to access existing secrets with the new configuration until they are also manually copied to the new location. Some examples of such changes include:

* switching Secrets Store back-end types (e.g. from internal SQL database to AWS Secrets Manager or Azure Key Vault)
* switching back-end locations (e.g. changing the AWS Secrets Manager account or region, GCP Secret Manager project or Azure Key Vault's vault).

In such cases, it is not sufficient to simply reconfigure and redeploy the ZenML server with the new Secrets Store configuration. This is because the ZenML server will not automatically migrate existing secrets to the new location. Instead, you should follow a specific migration strategy to ensure that existing secrets are also properly migrated to the new location with minimal, even zero downtime.

The secrets migration process makes use of the fact that [a secondary Secrets Store](#backup-secrets-store) can be configured for the ZenML server for backup purposes. This secondary Secrets Store is used as an intermediate step in the migration process. The migration process is as follows (we'll refer to the Secrets Store that is currently in use as *Secrets Store A* and the Secrets Store that will be used after the migration as *Secrets Store B*):

1. Re-configure the ZenML server to use *Secrets Store B* as the secondary Secrets Store.
2. Re-deploy the ZenML server.
3. Use the `zenml secret backup` CLI command to back up all secrets from *Secrets Store A* to *Secrets Store B*. You don't have to worry about secrets that are created or updated by users during or after this process, as they will be automatically backed up to *Secrets Store B*. If you also wish to delete secrets from *Secrets Store A* after they are successfully backed up to *Secrets Store B*, you should run `zenml secret backup --delete-secrets` instead.
4. Re-configure the ZenML server to use *Secrets Store B* as the primary Secrets Store and remove *Secrets Store A* as the secondary Secrets Store.
5. Re-deploy the ZenML server.

This migration strategy is not necessary if the actual location of the secrets values in the Secrets Store back-end does not change. For example:

* updating the credentials used to authenticate with the Secrets Store back-end before or after they expire
* switching to a different authentication method to authenticate with the same Secrets Store back-end (e.g. switching from an IAM account secret key to an IAM role in the AWS Secrets Manager)

If you are a [ZenML Pro](https://zenml.io/pro) user, you can configure your cloud backend based on your [deployment scenario](/getting-started/system-architectures).

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Custom secret stores

Learning how to develop a custom secret store.

The secrets store acts as the one-stop shop for all the secrets to which your pipeline or stack components might need access. It is responsible for storing, updating and deleting *only the secrets values* for ZenML secrets, while the ZenML secret metadata is stored in the SQL database. The secrets store interface implemented by all available secrets store back-ends is defined in the `zenml.zen_stores.secrets_stores.secrets_store_interface` core module and looks more or less like this:

```python
from abc import ABC, abstractmethod
from typing import Dict
from uuid import UUID

class SecretsStoreInterface(ABC):
    """ZenML secrets store interface.

    All ZenML secrets stores must implement the methods in this interface.
    """

    # ---------------------------------
    # Initialization and configuration
    # ---------------------------------

    @abstractmethod
    def _initialize(self) -> None:
        """Initialize the secrets store.

        This method is called immediately after the secrets store is created.
        It should be used to set up the backend (database, connection etc.).
        """

    # ---------
    # Secrets
    # ---------

    @abstractmethod
    def store_secret_values(
        self,
        secret_id: UUID,
        secret_values: Dict[str, str],
    ) -> None:
        """Store secret values for a new secret.

        Args:
            secret_id: ID of the secret.
            secret_values: Values for the secret.
        """

    @abstractmethod
    def get_secret_values(self, secret_id: UUID) -> Dict[str, str]:
        """Get the secret values for an existing secret.

        Args:
            secret_id: ID of the secret.

        Returns:
            The secret values.

        Raises:
            KeyError: if no secret values for the given ID are stored in the
                secrets store.
        """

    @abstractmethod
    def update_secret_values(
        self,
        secret_id: UUID,
        secret_values: Dict[str, str],
    ) -> None:
        """Updates secret values for an existing secret.

        Args:
            secret_id: The ID of the secret to be updated.
            secret_values: The new secret values.

        Raises:
            KeyError: if no secret values for the given ID are stored in the
                secrets store.
        """

    @abstractmethod
    def delete_secret_values(self, secret_id: UUID) -> None:
        """Deletes secret values for an existing secret.

        Args:
            secret_id: The ID of the secret.

        Raises:
            KeyError: if no secret values for the given ID are stored in the
                secrets store.
        """
```

{% hint style="info" %}
This is a slimmed-down version of the real interface which aims to highlight the abstraction layer. In order to see the full definition and get the complete docstrings, please check the [SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-zen_stores.html#zenml.zen_stores.secrets_stores) .
{% endhint %}

## Build your own custom secrets store

If you want to create your own custom secrets store implementation, you can follow the following steps:

1. Create a class that inherits from the `zenml.zen_stores.secrets_stores.base_secrets_store.BaseSecretsStore` base class and implements the `abstractmethod`s shown in the interface above. Use `SecretsStoreType.CUSTOM` as the `TYPE` value for your secrets store class.
2. If you need to provide any configuration, create a class that inherits from the `SecretsStoreConfiguration` class and add your configuration parameters there. Use that as the `CONFIG_TYPE` value for your secrets store class.
3. To configure the ZenML server to use your custom secrets store, make sure your code is available in the container image that is used to run the ZenML server. Then, use environment variables or helm chart values to configure the ZenML server to use your custom secrets store, as covered in the [deployment guide](/deploying-zenml/deploying-zenml).

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Live event streaming

Enable live event streaming on the ZenML server and consume the HTTP/SSE feed.

ZenML can send events published from inside a pipeline run to any HTTP client subscribed to that run. Use it for LLM token streaming, progress updates, live dashboards, and similar cases where you need to surface intermediate output from a running step.

This page covers operating streaming on the server: how to turn it on, which broker to pick, how to consume the SSE feed, and the wire contract clients code against. For the producer-side Python API (calling `zenml.streaming.publish()` from a step), see [Streaming Events](/concepts/steps_and_pipelines/streaming_events).

{% hint style="warning" %}
Streaming is best-effort, not persistent storage. Events are size-capped, can be dropped under load, and disappear when the broker's retention window elapses. **Once an event is lost it is gone — there is no secondary store, no replay endpoint, and no fallback.** If you need to keep something, write it as run metadata or an artifact from the step.
{% endhint %}

## Enable streaming

Streaming is off by default. The single setting that enables it is `stream_broker_implementation_source` on the server config (or `streaming.streamBrokerImplementationSource` on the Helm chart). Until that field is set, the streaming endpoints return `501 Not Implemented`, producer-side `publish()` calls are dropped without sending HTTP, and the server does not open a broker connection.

### Pick a broker

The Redis Streams broker (`zenml.zen_server.streaming.brokers.redis_streams.RedisStreamsBroker`) ships with ZenML. It requires Redis 5+ and the `redis` Python extra (`pip install 'zenml[server-streaming]'`). It namespaces stream keys by deployment ID so multiple ZenML servers can share a Redis cluster without colliding.

### Configure with Helm

```yaml
server:
  streaming:
    streamBrokerImplementationSource: zenml.zen_server.streaming.brokers.redis_streams.RedisStreamsBroker
  environment:
    ZENML_REDIS_BROKER_URL: redis://my-redis.svc.cluster.local:6379/0
```

The chart installs an SSE-only Gateway API `HTTPRoute` rule that disables Envoy's default 15-second request timeout for clients that send `Accept: text/event-stream` against the `/api/v1/runs/` tree. Browsers' `EventSource` and the ZenML server's own emitted frames match this condition. Custom clients that send a quality-list `Accept` header fall through to the default rule and are cut off at 15 seconds.

### Configure with environment variables

If you deploy without the chart, set the same field via env var:

```
ZENML_SERVER_STREAM_BROKER_IMPLEMENTATION_SOURCE=zenml.zen_server.streaming.brokers.redis_streams.RedisStreamsBroker
ZENML_REDIS_BROKER_URL=redis://...
```

Behind your own ingress, disable request timeouts and any response buffering for the SSE path. The server emits `X-Accel-Buffering: no` and `Cache-Control: no-cache, no-store, no-transform` to cover common intermediaries, but you may need to set the same on your proxy.

### Server config reference

| Field (`ServerConfiguration`)              | Helm key (`server.streaming.*`)    | Default | Notes                                                                                                                                                    |
| ------------------------------------------ | ---------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stream_broker_implementation_source`      | `streamBrokerImplementationSource` | unset   | Setting this enables streaming.                                                                                                                          |
| `streaming_heartbeat_seconds`              | `heartbeatSeconds`                 | `30.0`  | SSE heartbeat interval.                                                                                                                                  |
| `streaming_max_subscribers_per_stream`     | `maxSubscribersPerStream`          | `100`   | Maximum simultaneous subscribers per run. The 101st subscriber receives `503`.                                                                           |
| `streaming_broadcaster_idle_grace_seconds` | `broadcasterIdleGraceSeconds`      | `30.0`  | How long the server keeps a stream's broker reader running after the last subscriber disconnects, so a quick reconnect does not have to re-establish it. |

### Redis settings

Connection settings are read from the shared `ZENML_REDIS_` prefix, so the same Redis instance can be used by the streaming broker and by other ZenML components that talk to Redis. Settings specific to the streaming broker are read from `ZENML_REDIS_STREAMS_BROKER_` and override the shared values when set.

| Variable                                        | Default | Notes                                                                                                                                        |
| ----------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `ZENML_REDIS_BROKER_URL`                        | —       | `redis://...` or `rediss://...`. Required.                                                                                                   |
| `ZENML_REDIS_MAX_CONNECTIONS`                   | `10`    | Connection pool size. Increase if you expect many concurrent runs. Override per component with `ZENML_REDIS_STREAMS_BROKER_MAX_CONNECTIONS`. |
| `ZENML_REDIS_SOCKET_TIMEOUT`                    | `2.0`   | Per-call timeout in seconds.                                                                                                                 |
| `ZENML_REDIS_STREAMS_BROKER_MAX_STREAM_LENGTH`  | `10000` | Maximum entries retained per run (`XADD MAXLEN ~`).                                                                                          |
| `ZENML_REDIS_STREAMS_BROKER_STREAM_TTL_SECONDS` | `3600`  | TTL on each run's stream, refreshed on every publish.                                                                                        |

At startup, the server runs a single connectivity check against the broker. If the configured Redis URL is wrong or the host is unreachable, the server fails to boot and reports the error, instead of returning `503` on every later request.

## Consume the stream

Streams are exposed as [Server-Sent Events (SSE)](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) on:

```
GET /api/v1/runs/{pipeline_run_id}/events/stream
Accept: text/event-stream
Authorization: Bearer <token>
```

Consuming requires `READ` permission on the run (the same permission as viewing it in the dashboard). Publishing — covered on the [client SDK page](/concepts/steps_and_pipelines/streaming_events) — requires `UPDATE`.

### Browser

```javascript
const es = new EventSource(
  `/api/v1/runs/${runId}/events/stream`,
  { withCredentials: true }
);
es.addEventListener("event", (e) => console.log(JSON.parse(e.data)));
es.addEventListener("end", () => es.close());
```

`EventSource` automatically reconnects with the standard `Last-Event-ID` header, so transient drops resume after the last received event (see [Resuming after a disconnect](#resuming-after-a-disconnect)).

### Command line

```bash
curl -N -H "Accept: text/event-stream" \
  -H "Authorization: Bearer $ZENML_TOKEN" \
  "$ZENML_URL/api/v1/runs/$RUN_ID/events/stream"
```

`-N` disables curl's output buffering so frames arrive as the server writes them.

## SSE wire format

Each frame the server emits has the form:

```
id: <broker-assigned id>
event: <kind>
data: <JSON-encoded StreamEvent>

```

Reserved event names:

| `event:`                               | Meaning                                                                                                                                                                                                                                                                                                                                         |
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `event` (default) or any custom `kind` | A producer-published payload. `data` is the JSON-serialized `StreamEvent`.                                                                                                                                                                                                                                                                      |
| `end`                                  | The run has reached a terminal state. The server will close the connection.                                                                                                                                                                                                                                                                     |
| `gap`                                  | The subscriber may have missed events between the last `id` and now. Reasons: `outage` (broker reachability/reader error), `overflow` (per-subscriber queue full), `shutdown` (server is shutting down).                                                                                                                                        |
| `error`                                | A transient server-side error. The client should reconnect with `Last-Event-ID`.                                                                                                                                                                                                                                                                |
| `cursor`                               | A frame the server emits for filtered-out events and for forward-compatible unknown frame types. Carries an `id:` so `Last-Event-ID` advances. `data` is `{}` for filtered events and `{"unknown_type": "<type>"}` for frames the server didn't recognize (useful for spotting producer-vs-server version mismatches). Clients can ignore both. |

Heartbeats arrive as comment frames (`: ping\n\n`) every `streaming_heartbeat_seconds` (default 30 s) and require no client handling. Comments do not dispatch events (they will not trigger any `addEventListener` callback), which is why filtered or unknown frames use `event: cursor` instead.

### Filtering

The SSE endpoint accepts three multi-value query parameters that restrict which events are delivered. Each parameter accepts repeated values; within a parameter the values are ORed, and the parameters combine with AND. Filtered-out events still advance the server cursor via `cursor` frames — clients can reconnect with `Last-Event-ID` and will not see them replayed.

| Parameter         | Matches                                                  | Example                       |
| ----------------- | -------------------------------------------------------- | ----------------------------- |
| `kinds`           | `StreamEvent.kind`                                       | `?kinds=token&kinds=progress` |
| `step_names`      | `StreamEvent.step_name` (the invocation id of the step)  | `?step_names=summarize`       |
| `correlation_ids` | `StreamEvent.correlation_id` (producer-set sub-flow tag) | `?correlation_ids=gen-42`     |

Combined:

```
GET /api/v1/runs/{run}/events/stream?kinds=token&step_names=summarize
```

returns only `token`-kind events from the `summarize` step.

### Resuming after a disconnect

The server honors the standard SSE `Last-Event-ID` request header on reconnect. Browsers' `EventSource` sends it automatically. Other clients should track the last `id:` they received and send it back to resume:

```
GET /api/v1/runs/{run}/events/stream
Last-Event-ID: <last id you received>
```

Clients that cannot set request headers (some embedded environments) can use the `?since=<id>` query parameter as an equivalent — both specify the starting cursor. If both are sent, the header wins.

If the cursor is older than the broker's retention window, the missing events are not redelivered and the server does not signal that loss happened. The next read returns whatever is still retained.

Subscribers can also attach to a run that has already terminated. The server replays the broker's retained event history (up to the retention TTL) and then closes with an `end` event. Once the TTL elapses the history is gone and the subscribe returns just `end`.

**Lost events are unrecoverable.** Streaming is best-effort: events never leave the broker for any durable store, and ZenML keeps no secondary copy. Artifacts and run metadata persist the run's *outcomes*, not the intermediate stream. Plan accordingly:

* If you need replay, write the relevant state as an artifact or metadata entry from the step.
* If your consumer maintains UI state derived from the stream (running aggregates, scrollback), design it to tolerate gaps — drop accumulated state and re-derive from new events going forward, rather than expecting to "fetch what you missed".

## Delivery semantics

| Property      | What you get                                                                                                                                                                                                                                                                                                                                                                                                 |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Ordering      | Per-run, monotonic by broker id.                                                                                                                                                                                                                                                                                                                                                                             |
| Duplicates    | Within a single connection, each event id is delivered at most once. On reconnect with `Last-Event-ID`, the server resumes strictly after the last seen id, so events are not re-delivered. There is no producer-side retry on publish failure, so producers cannot introduce duplicates. Subscribers should still dedupe on event `id` defensively.                                                         |
| Loss          | Events can be lost via producer-side queue overflow (4 096 per process), server-side publish failure (logged, not retried), broker-side `MAXLEN` truncation, retention TTL, or per-subscriber queue overflow. Per-subscriber overflow emits a `gap: overflow` frame; the other loss modes are silent. **Lost events are not recoverable from any other source** — ZenML keeps no durable copy of the stream. |
| Retention     | `ZENML_REDIS_STREAMS_BROKER_STREAM_TTL_SECONDS` (default 1 h after the last publish).                                                                                                                                                                                                                                                                                                                        |
| Multi-replica | The broker delivers events across replicas, keyed by deployment id.                                                                                                                                                                                                                                                                                                                                          |
| Persistence   | None. Use run metadata or artifacts if you need durable storage.                                                                                                                                                                                                                                                                                                                                             |

## Limits

* Per-event payload is limited to **64 KiB** on the wire envelope.
* The producer-side queue holds up to **4 096 events per process**. When full, the oldest queued event is dropped to make room.
* The broker stream is limited per run (default 10 000 entries). Subscribers that fall too far behind will silently miss the trimmed events — there is no wire-level signal for retention loss, and trimmed events are not stored anywhere recoverable.
* The per-run subscriber limit is `streaming_max_subscribers_per_stream` (default 100). The 101st connection receives `503 Service Unavailable` with `Retry-After: 5`.

## Troubleshooting

**SSE connections drop after 15 seconds behind an ingress.** Your proxy is enforcing a request timeout. The bundled Helm chart configures the Gateway API `HTTPRoute` to disable it for SSE; if you run a custom ingress, do the same for `/api/v1/runs/.../events/stream` (or any path where the request carries `Accept: text/event-stream`).

**Subscribers report missing events on reconnect.** The subscriber is falling behind the broker's retention window. The missed events are gone — they are not stored anywhere durable. Either reduce the producer rate, raise `ZENML_REDIS_STREAMS_BROKER_MAX_STREAM_LENGTH`, or have the subscriber drop accumulated stream-derived state on every `gap` and re-derive from new events going forward.

**No events arrive.** Confirm streaming is enabled (the streaming endpoints return something other than `501`), check the consumer has `READ` on the run, and verify the producer is calling `zenml.streaming.publish()` from inside a step or pipeline context. Calls made outside such a context are dropped.

**`501 Not Implemented` on the streaming endpoints.** `stream_broker_implementation_source` is unset. Once it is configured, both the publish endpoint and the SSE endpoint become available. They return `501` together when streaming is disabled.

**Server boot fails with "Stream broker startup probe failed".** The configured broker cannot reach its backing store. For Redis, check `ZENML_REDIS_BROKER_URL`, TLS settings, and network reachability from the server pod.


# Connect

Various means of connecting to ZenML.

Once [ZenML is deployed](/deploying-zenml/deploying-zenml), there are various ways to connect to it.

## Choose how to connect

Use this quick guide to pick the right method based on your context:

| Context                                                                                   | Use                                     | Credentials                 | Docs                                                                                                                                                           |
| ----------------------------------------------------------------------------------------- | --------------------------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| You are a human using the CLI and browser                                                 | Interactive login (device flow)         | Your user session (24h/30d) | [Connect with your user](/deploying-zenml/connecting-to-zenml/connect-in-with-your-user-interactive)                                                           |
| Script/notebook needs to make quick API calls to an OSS server                            | Service account + API key               | Long‑lived API key          | [Connect with a service account](/deploying-zenml/connecting-to-zenml/connect-with-a-service-account)                                                          |
| Script/notebook needs to make quick API calls to a ZenML Pro workspace                    | ZenML Pro Personal Access Token         | Long‑lived PAT              | [Connect with a personal access token](/deploying-zenml/connecting-to-zenml/connect-with-a-pat)                                                                |
| CI/CD or long‑lived automation calling an OSS server                                      | Service account + API key               | Long‑lived API key          | [Connect with a service account](/deploying-zenml/connecting-to-zenml/connect-with-a-service-account)                                                          |
| CI/CD or long‑lived automation calling a ZenML Pro workspace                              | ZenML Pro API service account + API key | Long‑lived API key          | [Connect with a ZenML Pro service account](https://docs.zenml.io/api-reference/pro-api/getting-started#programmatic-access-with-service-accounts-and-api-keys) |
| CI/CD or long‑lived automation calling the ZenML Pro management API (`cloudapi.zenml.io`) | ZenML Pro service account + API key     | Long-lived API key          | [Connect with a ZenML Pro service account](https://docs.zenml.io/api-reference/pro-api/getting-started#programmatic-access-with-service-accounts-and-api-keys) |

{% hint style="warning" %}
Which base URL should you call?

* Workspace/OSS API: your server or workspace URL (e.g., `https://<workspace-id>.zenml.io`).
* ZenML Pro management API: `https://cloudapi.zenml.io`.

In ZenML Pro, use Personal Access Tokens or ZenML Pro service accounts and API keys. These credentials can be used for both the Workspace API and the Pro management API. See [ZenML Pro Personal Access Tokens](https://docs.zenml.io/pro/access-management/personal-access-tokens) and [ZenML Pro Service Accounts](https://docs.zenml.io/pro/access-management/service-accounts).
{% endhint %}

## Common pitfalls

* 401 Unauthorized: verify you’re using the correct base URL, the token hasn’t expired, and the header is `Authorization: Bearer <token>`.
* Automation fails after 1 hour: check the expiration date of the PAT or API key and rotate it if it has expired.
* Can’t find Run Template endpoints: they exist on the Workspace/OSS API, not on `cloudapi.zenml.io`.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# with your User (interactive)

Connect to the ZenML server using the ZenML CLI and the web based login.

You can authenticate your clients with the ZenML Server using the ZenML CLI and the web‑based login (device flow). This method is ideal for humans working locally and applies to OSS servers and ZenML Pro workspaces.

```bash
zenml login https://...
```

This command starts a browser flow to validate the device you are connecting from. You can choose whether to mark the device as trusted. If you don’t trust the device, a 24‑hour token is issued; if you do, a 30‑day token is issued.

{% hint style="warning" %}
Managing authorized devices for ZenML Pro workspaces is not yet supported in the dashboard. CLI device management is available.
{% endhint %}

To see all devices you've permitted, use the following command:

```bash
zenml authorized-device list
```

Additionally, the following command allows you to more precisely inspect one of these devices:

```bash
zenml authorized-device describe <DEVICE_ID>  
```

For increased security, you can invalidate a token using the `zenml authorized-device lock` command followed by the device ID.

```
zenml authorized-device lock <DEVICE_ID>  
```

To keep things simple, we can summarize the steps:

1. Use the `zenml login <URL>` command to start a device flow and connect to a zenml server.
2. Choose whether to trust the device when prompted.
3. Check permitted devices with `zenml authorized-device list`.
4. Invalidate a token with `zenml authorized-device lock ...`.

### Important notice

Using the ZenML CLI is a secure and comfortable way to interact with your ZenML servers. It's important to always ensure that only trusted devices are used to maintain security and privacy.

{% hint style="info" %}
Calling the ZenML Pro management API (`cloudapi.zenml.io`)? Interactive CLI login does not apply there. Use a ZenML Pro Personal Access Token or a ZenML Pro Service Account and API key instead. See [ZenML Pro API Getting Started](https://docs.zenml.io/api-reference/pro-api/getting-started).
{% endhint %}

Don't forget to manage your device trust levels regularly for optimal security. Should you feel a device trust needs to be revoked, lock the device immediately. Every token issued is a potential gateway to access your data, secrets and infrastructure.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# with your User (programmatic)

Connect to the ZenML server using a Personal Access Token.

If you are using ZenML Pro and need to call the ZenML Pro workspace API from a non-interactive environment, you also have the option of creating and using a Personal Access Token. Personal Access Tokens are scoped to your ZenML Pro user account and can be used to access all workspaces you are a member of in any organization. See the [ZenML Pro Personal Access Tokens](https://docs.zenml.io/pro/access-management/personal-access-tokens) documentation for more information.

{% hint style="warning" %}
**Personal Access Tokens are only available in ZenML Pro**

If you are using ZenML OSS and need to call the ZenML OSS API from a non-interactive environment, you can use a service account and an API key. See the [Connect with a service account](/deploying-zenml/connecting-to-zenml/connect-with-a-service-account) documentation for more information.
{% endhint %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# with a Service Account

Connect to the ZenML server using a service account and an API key.

Sometimes you may need to authenticate to a ZenML server from a non-interactive environment where the web login is not possible, like a CI/CD workload or a serverless function. In these cases, you can configure a service account and an API key and use the API key to authenticate to the ZenML server:

```bash
zenml service-account create <SERVICE_ACCOUNT_NAME>
```

This command creates a service account and an API key for it. The API key is displayed as part of the command output and cannot be retrieved later. You can then use the issued API key to connect your ZenML client to the server through one of the following methods:

* using the CLI:

```bash
# This command will prompt you to enter the API key
zenml login https://... --api-key
```

* setting the `ZENML_STORE_URL` and `ZENML_STORE_API_KEY` environment variables when you set up your ZenML client for the first time. This method is particularly useful when you are using the ZenML client in an automated CI/CD workload environment like GitHub Actions or GitLab CI or in a containerized environment like Docker or Kubernetes:

```bash
export ZENML_STORE_URL=https://...
export ZENML_STORE_API_KEY=<API_KEY>
```

{% hint style="info" %}
You don't need to run `zenml login` after setting these two environment variables and can start interacting with your server right away.
{% endhint %}

{% hint style="info" %}
Using ZenML Pro?

Use a ZenML Pro service account and API key. Set the workspace URL and your service account API key as environment variables:

```bash
export ZENML_STORE_URL=https://<your-workspace>.zenml.io
export ZENML_STORE_API_KEY=<YOUR_SERVICE_ACCOUNT_API_KEY>
# Optional for self-hosted Pro deployments:
export ZENML_PRO_API_URL=https://<your-pro-api-url>
```

You can also authenticate via CLI:

```bash
zenml login <your-workspace-name> --api-key
# You will be prompted to enter your service account API key
```

{% endhint %}

To see all the service accounts you've created and their API keys, use the following commands:

```bash
zenml service-account list
zenml service-account api-key <SERVICE_ACCOUNT_NAME> list
```

Additionally, the following command allows you to more precisely inspect one of these service accounts and an API key:

```bash
zenml service-account describe <SERVICE_ACCOUNT_NAME>
zenml service-account api-key <SERVICE_ACCOUNT_NAME> describe <API_KEY_NAME>
```

API keys don't have an expiration date. For increased security, we recommend that you regularly rotate the API keys to prevent unauthorized access to your ZenML server. You can do this with the ZenML CLI:

```bash
zenml service-account api-key <SERVICE_ACCOUNT_NAME> rotate <API_KEY_NAME>
```

Running this command will create a new API key and invalidate the old one. The new API key is displayed as part of the command output and cannot be retrieved later. You can then use the new API key to connect your ZenML client to the server just as described above.

When rotating an API key, you can also configure a retention period for the old API key. This is useful if you need to keep the old API key for a while to ensure that all your workloads have been updated to use the new API key. You can do this with the `--retain` flag. For example, to rotate an API key and keep the old one for 60 minutes, you can run the following command:

```bash
zenml service-account api-key <SERVICE_ACCOUNT_NAME> rotate <API_KEY_NAME> \
      --retain 60
```

For increased security, you can deactivate a service account or an API key using one of the following commands:

```
zenml service-account update <SERVICE_ACCOUNT_NAME> --active false
zenml service-account api-key <SERVICE_ACCOUNT_NAME> update <API_KEY_NAME> \
      --active false
```

Deactivating a service account or an API key will prevent it from being used to authenticate and has immediate effect on all workloads that use it.

To keep things simple, we can summarize the steps:

1. Use the `zenml service-account create` command to create a service account and an API key.
2. Use the `zenml login <url> --api-key` command to connect your ZenML client to the server using the API key.
3. Check configured service accounts with `zenml service-account list`.
4. Check configured API keys with `zenml service-account api-key <SERVICE_ACCOUNT_NAME> list`.
5. Regularly rotate API keys with `zenml service-account api-key <SERVICE_ACCOUNT_NAME> rotate`.
6. Deactivate service accounts or API keys with `zenml service-account update` or `zenml service-account api-key <SERVICE_ACCOUNT_NAME> update`.

## Programmatic access with API keys

You can use a service account's API key to access the ZenML server's REST API programmatically. This is particularly useful when you need to make long-term securely authenticated HTTP requests to the ZenML API endpoints. This is the recommended way to access the ZenML API programmatically when you're not using the ZenML CLI or Python client.

Accessing the OSS API with this method is thoroughly documented in the [API reference section](https://docs.zenml.io/api-reference/oss-api/getting-started#using-a-service-account-and-an-api-key). For ZenML Pro Workspace API access, use a ZenML Pro service account and API key as described in [Pro API Getting Started](https://docs.zenml.io/api-reference/pro-api/getting-started).

## Important notice

Every API key issued is a potential gateway to access your data, secrets and infrastructure. It's important to regularly rotate API keys and deactivate or delete service accounts and API keys that are no longer needed.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Manage

Learn how to upgrade your server to a new version of ZenML for the different deployment options.

The way to upgrade your ZenML server depends a lot on how you deployed it. However, there are some best practices that apply in all cases. Before you upgrade, check out the [best practices for upgrading ZenML](/deploying-zenml/upgrade-zenml-server/best-practices-upgrading-zenml) guide.

In general, upgrade your ZenML server as soon as you can once a new version is released. New versions come with a lot of improvements and fixes from which you can benefit.

{% tabs %}
{% tab title="Docker" %}
To upgrade to a new version with docker, you have to delete the existing container and then run the new version of the `zenml-server` image.

{% hint style="danger" %}
Check that your data is persisted (either on persistent storage or on an external MySQL instance) before doing this.

Optionally also perform a backup before the upgrade.
{% endhint %}

* Delete the existing ZenML container, for example like this:

  ```bash
  # find your container ID
  docker ps
  ```

  ```bash
  # stop the container
  docker stop <CONTAINER_ID>

  # remove the container
  docker rm <CONTAINER_ID>
  ```
* Deploy the version of the `zenml-server` image that you want to use. Find all versions [here](https://hub.docker.com/r/zenmldocker/zenml-server/tags).

  ```bash
  docker run -it -d -p 8080:8080 --name <CONTAINER_NAME> zenmldocker/zenml-server:<VERSION>
  ```

{% endtab %}

{% tab title="Kubernetes with Helm" %}
To upgrade your ZenML server Helm release to a new version, follow the steps below.

#### Simple in-place upgrade

If you don't need to change any configuration values, you can perform a simple in-place upgrade that reuses your existing configuration:

```bash
helm -n <namespace> upgrade zenml-server oci://public.ecr.aws/zenml/zenml --version <VERSION> --reuse-values
```

#### Upgrade with configuration changes

If you need to modify your ZenML server configuration during the upgrade, follow these steps instead:

* Extract your current configuration values to a file:

  ```bash
  helm -n <namespace> get values zenml-server > custom-values.yaml
  ```
* Make the necessary changes to your `custom-values.yaml` file (make sure they are compatible with the new version)
* Upgrade the release using your modified values file:

  ```bash
  helm -n <namespace> upgrade zenml-server oci://public.ecr.aws/zenml/zenml --version <VERSION> -f custom-values.yaml
  ```

{% hint style="info" %}
It is not recommended to change the container image tag in the Helm chart to custom values, since every Helm chart\
version is tested to work only with the default image tag. However, if you know what you're doing you can change\
the `zenml.image.tag` value in your `custom-values.yaml` file to the desired ZenML version (e.g. `0.32.0`).
{% endhint %}
{% endtab %}
{% endtabs %}

## Important Considerations After Upgrading

* **Downgrading is not supported**: Downgrading the server to an older version is not supported and can lead to unexpected behavior.
* **Client-server version alignment**: The version of the Python client that connects to the server should be kept at the same version as the server.
* **Recreate snapshots**: After upgrading your ZenML server, you need to recreate any [snapshots](/concepts/snapshots) that you were using. Snapshots are tied to specific server versions and will often not work correctly after an upgrade.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Best practices for upgrading

Simple, step-by-step guide for keeping your ZenML workspaces (servers) up to date without breaking your teams.

Upgrading ZenML doesn't have to be scary.

Whether you're using the open-source (OSS) version or ZenML Pro (where servers are called *workspaces*), this guide will help you set up a clean, testable, and stress-free upgrade process using a production + staging pattern.

1. Always have **two environments**: *production* and *staging*.
2. Mirror everything in both places.
3. Use GitOps to automate upgrades.
4. Run the right tests in staging.
5. Re-create snapshots.
6. Cut over to production once staging is green.

That's it. The rest of this chapter just fills in the details.

## ☝️ Step #1: Always Use Two Environments

Whether you're OSS or Pro:

* You should **always have two environments**:
  * **Production** — where your team builds and runs real pipelines.
  * **Staging** — used *only* to test ZenML upgrades before they hit production.

> 🏢 **ZenML Pro** users: use **two workspaces** (e.g. `prod-workspace`, `staging-workspace`)\
> 💻 **ZenML OSS** users: run **two ZenML servers** (same logic applies)

![Diagram showing "Production" and "Staging" environments side by side. Arrows show pipelines running in production, while staging is used for upgrades only.](/files/8MnjJfscJOpsUlvG4l18)

## 🧱 Step #2: Mirror Your Stacks in Both Environments

At setup time:

* For every **stack in production**, create a **mirrored stack in staging**
* Ideally, they point to **separate infra**, but can also share infra if needed

| Stack Component    | Production           | Staging                 |
| ------------------ | -------------------- | ----------------------- |
| Kubernetes cluster | `prod-k8s-cluster`   | `staging-k8s-cluster`   |
| Artifact store     | `s3://prod-bucket`   | `s3://staging-bucket`   |
| Container registry | `gcr.io/prod-images` | `gcr.io/staging-images` |

![Diagram: Mirrored stacks pointing at separate staging infra](/files/e6d83Tosy1LL4yWgWfCN)

{% hint style="info" %}

* Point staging stacks to **staging variants** of your infra (e.g., a smaller K8s cluster, a test S3 bucket).
* When you change a stack in production, immediately update the twin in staging.
  {% endhint %}

## 🛠️ Step #3: Use [GitOps](https://about.gitlab.com/topics/gitops/) to Manage Upgrades

![Diagram: GitOps](/files/MVOm35leO0ijDdsrrS7q)

Put your workspace configuration in a Git repository (Helm charts, Terraform, or the ZenML Pro API – pick your tool). Set up two long-lived branches:

* `staging` – auto-deploys to the **staging workspace**
* `main` – auto-deploys to **production**

```mermaid
flowchart LR
    dev["PR → staging branch"] --> stg["CI/CD upgrades Staging workspace"]
    stg --> tests["Run upgrade test suite"]
    tests -->|✅| merge["Merge staging ➜ main"]
    merge --> prod["CI/CD upgrades Production workspace"]
```

ZenML Pro users can call the [Workspace API](https://cloudapi.zenml.io/) from CI to bump the version. OSS users typically re-deploy the Helm chart/Docker image with the new tag.

## 🤝 Step #4: Run a test suite in staging

After upgrading staging, assume things might break — this is normal and expected.

At this point, the platform and data science / ML engineering teams should have mutually:

* Agree on a smoke test suite of pipelines or steps
* Maintain shared expectations on what counts as "upgrade success"

For example, the data science repo could contain a test suite that does the following checks:

```python
def test_artifact_loading():
    artifact = Client().get_artifact_version("xyz").load()
    assert artifact is not None

def test_simple_pipeline():
    run = run_pipeline(pipeline_name="...")
    assert run.status == "COMPLETED"
```

## 🔄 Step #5: Update all snapshots

Pipeline snapshots may now break as they have the older version of the ZenML client installed. Therefore, you would need to rebuild the snapshot and associated images.

The easiest way to do this is to re-create a snapshot using the CLI:

```shell
zenml pipeline snapshot create run.my_pipeline \
  --name upgraded-template \
  --stack staging-stack \
  --config configs/run.yaml
```

{% hint style="info" %}
Read about [how snapshots work](https://docs.zenml.io/user-guides/tutorial/trigger-pipelines-from-external-systems).
{% endhint %}

After building, execute all snapshots end-to-end as a smoke test. Ideally, your data science teams have a "smoke test" parameter in the pipeline to load mock data just for this scenario!

## 🚀 Step #6: Upgrade Production and Go Live

Once staging is ✅ :

1. Merge `staging` ➜ `main`.
2. CI upgrades the production workspace.
3. Immediately:
   * Rebuild **all snapshots** in prod
   * **Reschedule** recurring pipelines (delete old schedules, create new ones). Read more [here](https://docs.zenml.io/user-guides/tutorial/managing-scheduled-pipelines)
4. Monitor for a few hours. Done.

![From staging to production](/files/9iakUMvNflNhmBPEmv3b)

## Ops Notes (OSS only)

If you self-host the ZenML server:

* Take a **database backup** before every upgrade.
* Keep the old Docker image tag handy for rollbacks.
* Store logs from the migration job.

[ZenML Pro](http://zenml.io/pro) SaaS handles all of the above for you.

## ✅ Summary: The Upgrade Flow

```
     ┌───────────────┐
     │ Git PR to dev │
     │ → staging env │
     └──────┬────────┘
            │
            ▼
   Upgrade staging server
            │
       Run all pipelines / tests
            │
     ✔ All tests pass?
        /               \
      Yes                 No
      |                    |
Recreate snapshots        Fix
     │
Upgrade prod
     |
 Rebuild & reschedule

```

* Two workspaces keep upgrades safe.
* GitOps makes them repeatable.
* A simple pipeline test suite keeps you honest.

Upgrade with confidence 🚀.

## 🔚 Final Notes

ZenML Pro: Hosted workspaces are upgraded automatically, but you still need to test your pipelines in staging before changes hit production.

ZenML OSS: You are responsible for upgrades, backups, and reconfiguration — this guide helps you minimize downtime and bugs.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Using ZenML server in production

Learn about best practices for using ZenML server in production environments.

Setting up a ZenML server for testing is a quick process. However, most people have to move beyond so-called 'day zero' operations and in such cases, it helps to learn best practices around setting up your ZenML server in a production-ready way. This guide encapsulates all the tips and tricks we've learned ourselves and from working with people who use ZenML in production environments. Following are some of the best practices we recommend.

{% hint style="info" %}
If you are using ZenML Pro, you don't have to worry about any of these. We have got you covered!\
You can sign up for a free trial [here](https://zenml.io/pro).
{% endhint %}

## Autoscaling replicas

In production, you often have to run bigger and longer running pipelines that might strain your server's resources. It is a good idea to set up autoscaling for your ZenML server so that you don't have to worry about your pipeline runs getting interrupted or your Dashboard slowing down due to high traffic.

How you do it depends greatly on the environment in which you have deployed your ZenML server. Below are some common deployment options and how to set up autoscaling for them.

{% tabs %}
{% tab title="Kubernetes with Helm" %}
If you are using the official [ZenML Helm chart](https://artifacthub.io/packages/helm/zenml/zenml), you can take advantage of the `autoscaling.enabled` flag to enable autoscaling for your ZenML server. For example:

```yaml
autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80
```

This will create a horizontal pod autoscaler for your ZenML server that will scale the number of replicas up to 10 and down to 1 based on the CPU utilization of the pods.
{% endtab %}

{% tab title="ECS" %}
For folks using AWS, [ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) is a popular choice for running ZenML server. ECS is a container orchestration service that allows you to run and scale your containers in a managed environment.

To scale your ZenML server deployed as a service on ECS, you can follow the steps below:

* Go to the ECS console, find you service pertaining to your ZenML server and click on it.
* Click on the "Update Service" button.
* If you scroll down, you will see the "Service auto scaling - optional" section.
* Here you can enable autoscaling and set the minimum and maximum number of tasks to run for your service and also the ECS service metric to use for scaling.

![Image showing autoscaling settings for a service](/files/ct3bzHmjZMVWzTsQLKqe)
{% endtab %}

{% tab title="Cloud Run" %}
For folks on GCP, [Cloud Run](https://cloud.google.com/run) is a popular choice for running ZenML server. Cloud Run is a container orchestration service that allows you to run and scale your containers in a managed environment.

In Cloud Run, each revision is automatically scaled to the number of instances needed to handle all incoming requests, events, or CPU utilization and by default, when a revision does not receive any traffic, it is scaled in to zero instances. For production use cases, we recommend setting the minimum number of instances to at least 1 so that you have "warm" instances ready to serve incoming requests.

To scale your ZenML server deployed on Cloud Run, you can follow the steps below:

* Go to the Cloud Run console, find you service pertaining to your ZenML server and click on it.
* Click on the "Edit & Deploy new Revision" button.
* Scroll down to the "Revision auto-scaling" section.
* Here you can set the minimum and maximum number of instances to run for your service.

![Image showing autoscaling settings for a service](/files/uPBLP52KdD2md7itd0Yn)
{% endtab %}

{% tab title="Docker Compose" %}
If you use Docker Compose, you don't get autoscaling out of the box. However, you can scale your service to N number of replicas using the `scale` flag. For example:

```bash
docker compose up --scale zenml-server=N
```

This will scale your ZenML server to N replicas.
{% endtab %}
{% endtabs %}

## High connection pool values

One other way to improve the performance of your ZenML server is to increase the number of threads that your server process uses, provided that you have hardware that can support it.

You can control this by setting the `zenml.threadPoolSize` value in the ZenML Helm chart values. For example:

```yaml
server:
  threadPoolSize: 100
```

By default, it is set to 40. If you are using any other deployment option, you can set the `ZENML_SERVER_THREAD_POOL_SIZE` environment variable to the desired value.

Once this is set, you should also modify the `zenml.database.poolSize` and `zenml.database.maxOverflow` values to ensure that the ZenML server workers do not block on database connections (i.e. the sum of the pool size and max overflow should be greater than or equal to the thread pool size). If you manage your own database, ensure these values are set appropriately.

## Scaling the backing database

An important component of the ZenML server deployment is the backing database. When you start scaling your ZenML server instances, you will also need to scale your database to avoid any bottlenecks.

We would recommend starting out with a simple (single) database instance and then monitoring it to decide if it needs scaling. Some common metrics to look out for:

* CPU Utilization: If the CPU Utilization is consistently above 50%, you may need to scale your database. Some spikes in the utilization are expected but it should not be consistently high.
* Freeable Memory: It is natural for the freeable memory to go down with time as your database uses it for caching and buffering but if it drops below 100-200 MB, you may need to scale your database.

## Setting up an ingress/load balancer

Exposing your ZenML server to the internet securely and reliably is a must for production use cases. One way to do this is to set up an ingress/load balancer.

{% tabs %}
{% tab title="Kubernetes with Helm" %}
If you are using the official [ZenML Helm chart](https://artifacthub.io/packages/helm/zenml/zenml), you can take advantage of the `zenml.ingress.enabled` flag to enable ingress for your ZenML server. For example:

```yaml
server:
  ingress:
    enabled: true
    className: "nginx"
    annotations:
      # nginx.ingress.kubernetes.io/ssl-redirect: "true"
      # nginx.ingress.kubernetes.io/rewrite-target: /$1
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
      # cert-manager.io/cluster-issuer: "letsencrypt"
```

This will create an [NGINX ingress](https://github.com/kubernetes/ingress-nginx) for your ZenML service that will create a LoadBalancer on whatever cloud provider you are using.
{% endtab %}

{% tab title="ECS" %}
With ECS, you can use Application Load Balancers to evenly route traffic to your tasks running your ZenML server.

Follow the steps in the official [AWS documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html) to learn how to set this up.
{% endtab %}

{% tab title="Cloud Run" %}
With Cloud Run, you can use Cloud Load Balancing to route traffic to your service.

Follow the steps in the official [GCP documentation](https://cloud.google.com/load-balancing/docs/https/setting-up-https-serverless) to learn how to set this up.
{% endtab %}

{% tab title="Docker Compose" %}
If you are using Docker Compose, you can set up an NGINX server as a reverse proxy to route traffic to your ZenML server. Here's a [blog](https://www.docker.com/blog/how-to-use-the-official-nginx-docker-image/) that shows how to do it.
{% endtab %}
{% endtabs %}

## Monitoring

Monitoring your service is crucial to ensure that it is running smoothly and to catch any issues early before they can cause problems. Depending on the deployment option you are using, you can use different tools to monitor your service.

{% tabs %}
{% tab title="Kubernetes with Helm" %}
You can set up Prometheus and Grafana to monitor your ZenML server. We recommend using the `kube-prometheus-stack` [Helm chart from the prometheus-community](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack) to get started quickly.

Once you have deployed the chart, you can find your grafana service by searching for services in the namespace you have deployed the chart in. Port-forward it to your local machine or deploy it through an ingress.

You can now use queries like the following to monitor your ZenML server:

```
sum by(namespace) (rate(container_cpu_usage_seconds_total{namespace=~"zenml.*"}[5m]))
```

This query would give you the CPU utilization of your server pods in all namespaces that start with `zenml`. The image below shows how this query would look like in Grafana.

![Image showing CPU utilization of ZenML server pods](/files/jwJMoQuXw2UW6BQeoHLQ)
{% endtab %}

{% tab title="ECS" %}
On ECS, you can utilize the [CloudWatch integration](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html) to monitor your ZenML server.

In the "Health and metrics" section of your ECS console, you should see metrics pertaining to your ZenML service like CPU utilization and Memory utilization.

![Image showing CPU utilization ECS](/files/kKtpRbvgfosGMStIuP4b)
{% endtab %}

{% tab title="Cloud Run" %}
In Cloud Run, you can utilize the [Cloud Monitoring integration](https://cloud.google.com/run/docs/monitoring) to monitor your ZenML server.

The "Metrics" tab in the Cloud Run console will show you metrics like Container CPU utilization, Container memory utilization, and more.

![Image showing metrics in Cloud Run](/files/VkP7aIcUuUsTmjYAQeov)
{% endtab %}
{% endtabs %}

## Backups

The data in your ZenML server is critical as it contains your pipeline runs, stack configurations, and other important information. It is, therefore, recommended to have a backup strategy in place to avoid losing any data.

Some common strategies include:

* Setting up automated backups with a good retention period (say 30 days).
* Periodically exporting the data to an external storage (e.g. S3, GCS, etc.).
* Manual backups before upgrading your server to avoid any problems.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Troubleshoot your ZenML server

Troubleshooting tips for your ZenML deployment

In this document, we will go over some common issues that you might face when deploying ZenML and how to solve them.

## Viewing logs

Analyzing logs is a great way to debug issues. Depending on whether you have a Kubernetes (using Helm or `zenml deploy`) or a Docker deployment, you can view the logs in different ways.

{% tabs %}
{% tab title="Kubernetes" %}
If you are using Kubernetes, you can view the logs of the ZenML server using the following method:

* Check all pods that are running your ZenML deployment.

```bash
kubectl -n <KUBERNETES_NAMESPACE> get pods
```

* If you see that the pods aren't running, you can use the command below to get the logs for all pods at once.

```bash
kubectl -n <KUBERNETES_NAMESPACE> logs -l app.kubernetes.io/name=zenml
```

Note that the error can either be from the `zenml-db-init` container that connects to the MySQL database or from the `zenml` container that runs the server code. If the get pods command shows that the pod is failing in the `Init` state then use `zenml-db-init` as the container name, otherwise use `zenml`.

```bash
kubectl -n <KUBERNETES_NAMESPACE> logs -l app.kubernetes.io/name=zenml -c <CONTAINER_NAME>
```

{% hint style="info" %}
You can also use the `--tail` flag to limit the number of lines to show or the `--follow` flag to follow the logs in real-time.
{% endhint %}
{% endtab %}

{% tab title="Docker" %}
If you are using Docker, you can view the logs of the ZenML server using the following method:

* If you used the `zenml login --local --docker` CLI command to deploy the Docker ZenML server, you can check the logs with the command:

  ```shell
  zenml logs -f
  ```
* If you used the `docker run` command to manually deploy the Docker ZenML server, you can check the logs with the command:

  ```shell
  docker logs zenml -f
  ```
* If you used the `docker compose` command to manually deploy the Docker ZenML server, you can check the logs with the command:

  ```shell
  docker compose -p zenml logs -f
  ```

{% endtab %}
{% endtabs %}

## Fixing database connection problems

If you are using a MySQL database, you might face issues connecting to it. The logs from the `zenml-db-init` container should give you a good idea of what the problem is. Here are some common issues and how to fix them:

* If you see an error like `ERROR 1045 (28000): Access denied for user <USER> using password YES`, it means that the username or password is incorrect. Make sure that the username and password are correctly set for whatever deployment method you are using.
* If you see an error like `ERROR 2003 (HY000): Can't connect to MySQL server on <HOST> (<IP>)`, it means that the host is incorrect. Make sure that the host is correctly set for whatever deployment method you are using.

You can test the connection and the credentials by running the following command from your machine:

```bash
mysql -h <HOST> -u <USER> -p
```

{% hint style="info" %}
If you are using a Kubernetes deployment, you can use the `kubectl port-forward` command to forward the MySQL port to your local machine. This will allow you to connect to the database from your machine.
{% endhint %}

## Fixing database initialization problems

If you’ve migrated from a newer ZenML version to an older version and see errors like `Revision not found` in your `zenml-db-init` logs, one way out is to drop the database and create a new one with the same name.

* Log in to your MySQL instance.

  ```bash
  mysql -h <HOST> -u <NAME> -p
  ```
* Drop the database for the server.

  ```sql
  drop database <NAME>;
  ```
* Create the database with the same name.

  ```sql
  create database <NAME>;
  ```
* Restart the Kubernetes pods or the docker container running your server to trigger the database initialization again.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Migration guide

How to migrate your ZenML code to the newest version.

Migrations are necessary for ZenML releases that include breaking changes, which are currently all releases that increment the minor version of the release, e.g., `0.X` -> `0.Y`. Furthermore, all releases that increment the first non-zero digit of the version contain major breaking changes or paradigm shifts that are explained in separate migration guides below.

## Release Type Examples

* `0.40.2` to `0.40.3` contains *no breaking changes* and requires no migration whatsoever,
* `0.40.3` to `0.41.0` contains *minor breaking changes* that need to be taken into account when upgrading ZenML,
* `0.39.1` to `0.40.0` contains *major breaking changes* that introduce major shifts in how ZenML code is written or used.

## Major Migration Guides

The following guides contain detailed instructions on how to migrate between ZenML versions that introduced major breaking changes or paradigm shifts. The migration guides are sequential, meaning if there is more than one migration guide between your current version and the latest release, follow each guide in order.

* [Migration guide 0.13.2 → 0.20.0](/deploying-zenml/upgrade-zenml-server/migration-guide/migration-zero-twenty)
* [Migration guide 0.23.0 → 0.30.0](/deploying-zenml/upgrade-zenml-server/migration-guide/migration-zero-thirty)
* [Migration guide 0.39.1 → 0.41.0](/deploying-zenml/upgrade-zenml-server/migration-guide/migration-zero-forty)
* [Migration guide 0.58.2 → 0.60.0](/deploying-zenml/upgrade-zenml-server/migration-guide/migration-zero-sixty)

## Release Notes

For releases with minor breaking changes, e.g., `0.40.3` to `0.41.0`, check out the official [ZenML Release Notes](https://github.com/zenml-io/zenml/releases) to see which breaking changes were introduced.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Migration guide 0.13.2 → 0.20.0

How to migrate from ZenML <=0.13.2 to 0.20.0.

*Last updated: 2023-07-24*

The ZenML 0.20.0 release brings a number of big changes to its architecture and its features, some of which are not backwards compatible with previous versions. This guide walks you through these changes and offers instructions on how to migrate your existing ZenML stacks and pipelines to the new version with minimal effort and disruption to your existing workloads.

{% hint style="warning" %}
Updating to ZenML 0.20.0 needs to be followed by a migration of your existing ZenML Stacks and you may also need to make changes to your current ZenML pipeline code. Please read this guide carefully and follow the migration instructions to ensure a smooth transition.

If you have updated to ZenML 0.20.0 by mistake or are experiencing issues with the new version, you can always go back to the previous version by using `pip install zenml==0.13.2` instead of `pip install zenml` when installing ZenML manually or in your scripts.
{% endhint %}

High-level overview of the changes:

* [ZenML takes over the Metadata Store](#zenml-takes-over-the-metadata-store-role) role. All information about your ZenML Stacks, pipelines, and artifacts is tracked by ZenML itself directly. If you are currently using remote Metadata Stores (e.g. deployed in cloud) in your stacks, you will probably need to replace them with a [ZenML server deployment](https://docs.zenml.io/getting-started/deploying-zenml).
* the [new ZenML Dashboard](#the-zenml-dashboard-is-now-available) is now available with all ZenML deployments.
* [ZenML Profiles have been removed](#removal-of-profiles-and-the-local-yaml-database) in favor of ZenML Projects. You need to [manually migrate your existing ZenML Profiles](#-how-to-migrate-your-profiles) after the update.
* the [configuration of Stack Components is now decoupled from their implementation](#decoupling-stack-component-configuration-from-implementation). If you extended ZenML with custom stack component implementations, you may need to update the way they are registered in ZenML.
* the updated ZenML server provides a new and improved collaborative experience. When connected to a ZenML server, you can now [share your ZenML Stacks and Stack Components](#shared-zenml-stacks-and-stack-components) with other users. If you were previously using the ZenML Profiles or the ZenML server to share your ZenML Stacks, you should switch to the new ZenML server and Dashboard and update your existing workflows to reflect the new features.

## ZenML takes over the Metadata Store role

ZenML can now run [as a server](https://docs.zenml.io/getting-started/core-concepts#zenml-server-and-dashboard) that can be accessed via a REST API and also comes with a visual user interface (called the ZenML Dashboard). This server can be deployed in arbitrary environments (local, on-prem, via Docker, on AWS, GCP, Azure etc.) and supports user management, workspace scoping, and more.

The release introduces a series of commands to facilitate managing the lifecycle of the ZenML server and to access the pipeline and pipeline run information:

* `zenml connect / disconnect / down / up / logs / status` can be used to configure your client to connect to a ZenML server, to start a local ZenML Dashboard or to deploy a ZenML server to a cloud environment. For more information on how to use these commands, see [the ZenML deployment documentation](https://docs.zenml.io/getting-started/deploying-zenml).
* `zenml pipeline list / runs / delete` can be used to display information and about and manage your pipelines and pipeline runs.

In ZenML 0.13.2 and earlier versions, information about pipelines and pipeline runs used to be stored in a separate stack component called the Metadata Store. Starting with 0.20.0, the role of the Metadata Store is now taken over by ZenML itself. This means that the Metadata Store is no longer a separate component in the ZenML architecture, but rather a part of the ZenML core, located wherever ZenML is deployed: locally on your machine or running remotely as a server.

All metadata is now stored, tracked, and managed by ZenML itself. The Metadata Store stack component type and all its implementations have been deprecated and removed. It is no longer possible to register them or include them in ZenML stacks. This is a key architectural change in ZenML 0.20.0 that further improves usability, reproducibility and makes it possible to visualize and manage all your pipelines and pipeline runs in the new ZenML Dashboard.

The architecture changes for the local case are shown in the diagram below:

![ZenML local metadata before 0.20.0](/files/vD8OVy545Fqbxlu1y2eA) ![ZenML local metadata after 0.20.0](/files/knXSkHF4QD7mCxaR9h6I)

The architecture changes for the remote case are shown in the diagram below:

![ZenML remote metadata before 0.20.0](/files/Pn8zZtj7AvRt39XiDTEB) ![ZenML remote metadata after 0.20.0](/files/gFLplRjDfHQ3i5nfBpar)

If you're already using ZenML, aside from the above limitation, this change will impact you differently, depending on the flavor of Metadata Stores you have in your stacks:

* if you're using the default `sqlite` Metadata Store flavor in your stacks, you don't need to do anything. ZenML will automatically switch to using its local database instead of your `sqlite` Metadata Stores when you update to 0.20.0 (also see how to [migrate your stacks](#-how-to-migrate-your-profiles)).
* if you're using the `kubeflow` Metadata Store flavor *only as a way to connect to the local Kubeflow Metadata Service* (i.e. the one installed by the `kubeflow` Orchestrator in a local k3d Kubernetes cluster), you also don't need to do anything explicitly. When you [migrate your stacks](#-how-to-migrate-your-profiles) to ZenML 0.20.0, ZenML will automatically switch to using its local database.
* if you're using the `kubeflow` Metadata Store flavor to connect to a remote Kubeflow Metadata Service such as those provided by a Kubeflow installation running in AWS, Google or Azure, there is currently no equivalent in ZenML 0.20.0. You'll need to [deploy a ZenML Server](https://docs.zenml.io/getting-started/deploying-zenml) instance close to where your Kubeflow service is running (e.g. in the same cloud region).
* if you're using the `mysql` Metadata Store flavor to connect to a remote MySQL database service (e.g. a managed AWS, GCP or Azure MySQL service), you'll have to [deploy a ZenML Server](https://docs.zenml.io/getting-started/deploying-zenml) instance connected to that same database.
* if you deployed a `kubernetes` Metadata Store flavor (i.e. a MySQL database service deployed in Kubernetes), you can [deploy a ZenML Server](https://docs.zenml.io/getting-started/deploying-zenml) in the same Kubernetes cluster and connect it to that same database. However, ZenML will no longer provide the `kubernetes` Metadata Store flavor and you'll have to manage the Kubernetes MySQL database service deployment yourself going forward.

{% hint style="info" %}
The ZenML Server inherits the same limitations that the Metadata Store had prior to ZenML 0.20.0:

* it is not possible to use a local ZenML Server to track pipelines and pipeline runs that are running remotely in the cloud, unless the ZenML server is explicitly configured to be reachable from the cloud (e.g. by using a public IP address or a VPN connection).
* using a remote ZenML Server to track pipelines and pipeline runs that are running locally is possible, but can have significant performance issues due to the network latency.

It is therefore recommended that you always use a ZenML deployment that is located as close as possible to and reachable from where your pipelines and step operators are running. This will ensure the best possible performance and usability.
{% endhint %}

### 👣 How to migrate pipeline runs from your old metadata stores

{% hint style="info" %}
The `zenml pipeline runs migrate` CLI command is only available under ZenML versions \[0.21.0, 0.21.1, 0.22.0]. If you want to migrate your existing ZenML runs from `zenml<0.20.0` to `zenml>0.22.0`, please first upgrade to `zenml==0.22.0` and migrate your runs as shown below, then upgrade to the newer version.
{% endhint %}

To migrate the pipeline run information already stored in an existing metadata store to the new ZenML paradigm, you can use the `zenml pipeline runs migrate` CLI command.

1. Before upgrading ZenML, make a backup of all metadata stores you want to migrate, then upgrade ZenML.
2. Decide the ZenML deployment model that you want to follow for your projects. See the [ZenML deployment documentation](https://docs.zenml.io/getting-started/deploying-zenml) for available deployment scenarios. If you decide on using a local or remote ZenML server to manage your pipelines, make sure that you first connect your client to it by running `zenml connect`.
3. Use the `zenml pipeline runs migrate` CLI command to migrate your old pipeline runs:

* If you want to migrate from a local SQLite metadata store, you only need to pass the path to the metadata store to the command, e.g.:

```bash
zenml pipeline runs migrate PATH/TO/LOCAL/STORE/metadata.db
```

* If you would like to migrate any other store, you will need to set `--database_type=mysql` and provide the MySQL host, username, and password in addition to the database, e.g.:

```bash
zenml pipeline runs migrate DATABASE_NAME \
  --database_type=mysql \
  --mysql_host=URL/TO/MYSQL \
  --mysql_username=MYSQL_USERNAME \
  --mysql_password=MYSQL_PASSWORD
```

### 💾 The New Way (CLI Command Cheat Sheet)

**Deploy the server**

`zenml deploy --aws` (maybe don't do this :) since it spins up infrastructure on AWS…)

**Spin up a local ZenML Server**

`zenml up`

**Connect to a pre-existing server**

`zenml connect` (pass in URL / etc, or zenml connect --config + yaml file)

**List your deployed server details**

`zenml status`

## The ZenML Dashboard is now available

The new ZenML Dashboard is now bundled into the ZenML Python package and can be launched directly from Python. The source code lives in the [ZenML Dashboard repository](https://github.com/zenml-io/zenml-dashboard).

To launch it locally, simply run `zenml up` on your machine and follow the instructions:

```bash
$ zenml up
Deploying a local ZenML server with name 'local'.
Connecting ZenML to the 'local' local ZenML server (http://127.0.0.1:8237).
Updated the global store configuration.
Connected ZenML to the 'local' local ZenML server (http://127.0.0.1:8237).
The local ZenML dashboard is available at 'http://127.0.0.1:8237'. You can
connect to it using the 'default' username and an empty password.
```

The Dashboard will be available at `http://localhost:8237` by default:

![ZenML Dashboard Preview](/files/G0Q2D5r2mW3VDEuNcjWT)

For more details on other possible deployment options, see the [ZenML deployment documentation](https://docs.zenml.io/getting-started/deploying-zenml), and/or follow the [starter guide](https://docs.zenml.io/user-guides/starter-guide) to learn more.

## Removal of Profiles and the local YAML database

Prior to 0.20.0, ZenML used used a set of local YAML files to store information about the Stacks and Stack Components that were registered on your machine. In addition to that, these Stacks could be grouped together and organized under individual Profiles.

Profiles and the local YAML database have both been deprecated and removed in ZenML 0.20.0. Stack, Stack Components as well as all other information that ZenML tracks, such as Pipelines and Pipeline Runs, are now stored in a single SQL database. These entities are no longer organized into Profiles, but they can be scoped into different Projects instead.

{% hint style="warning" %}
Since the local YAML database is no longer used by ZenML 0.20.0, you will lose all the Stacks and Stack Components that you currently have configured when you update to ZenML 0.20.0. If you still want to use these Stacks, you will need to [manually migrate](#-how-to-migrate-your-profiles) them after the update.
{% endhint %}

### 👣 How to migrate your Profiles

If you're already using ZenML, you can migrate your existing Profiles to the new ZenML 0.20.0 paradigm by following these steps:

1. first, update ZenML to 0.20.0. This will automatically invalidate all your existing Profiles.
2. decide the ZenML deployment model that you want to follow for your projects. See the [ZenML deployment documentation](https://docs.zenml.io/getting-started/deploying-zenml) for available deployment scenarios. If you decide on using a local or remote ZenML server to manage your pipelines, make sure that you first connect your client to it by running `zenml connect`.
3. use the `zenml profile list` and `zenml profile migrate` CLI commands to import the Stacks and Stack Components from your Profiles into your new ZenML deployment. If you have multiple Profiles that you would like to migrate, you can either use a prefix for the names of your imported Stacks and Stack Components, or you can use a different ZenML Project for each Profile.

{% hint style="warning" %}
The ZenML Dashboard is currently limited to showing only information that is available in the `default` Project. If you wish to migrate your Profiles to a different Project, you will not be able to visualize the migrated Stacks and Stack Components in the Dashboard. This will be fixed in a future release.
{% endhint %}

Once you've migrated all your Profiles, you can delete the old YAML files.

Example of migrating a `default` profile into the `default` project:

```bash
$ zenml profile list
ZenML profiles have been deprecated and removed in this version of ZenML. All
stacks, stack components, flavors etc. are now stored and managed globally,
either in a local database or on a remote ZenML server (see the `zenml up` and
`zenml connect` commands). As an alternative to profiles, you can use projects
as a scoping mechanism for stacks, stack components and other ZenML objects.

The information stored in legacy profiles is not automatically migrated. You can
do so manually by using the `zenml profile list` and `zenml profile migrate` commands.
Found profile with 1 stacks, 3 components and 0 flavors at: /home/stefan/.config/zenml/profiles/default
Found profile with 3 stacks, 6 components and 0 flavors at: /home/stefan/.config/zenml/profiles/zenprojects
Found profile with 3 stacks, 7 components and 0 flavors at: /home/stefan/.config/zenml/profiles/zenbytes

$ zenml profile migrate /home/stefan/.config/zenml/profiles/default
No component flavors to migrate from /home/stefan/.config/zenml/profiles/default/stacks.yaml...
Migrating stack components from /home/stefan/.config/zenml/profiles/default/stacks.yaml...
Created artifact_store 'cloud_artifact_store' with flavor 's3'.
Created container_registry 'cloud_registry' with flavor 'aws'.
Created container_registry 'local_registry' with flavor 'default'.
Created model_deployer 'eks_seldon' with flavor 'seldon'.
Created orchestrator 'cloud_orchestrator' with flavor 'kubeflow'.
Created orchestrator 'kubeflow_orchestrator' with flavor 'kubeflow'.
Created secrets_manager 'aws_secret_manager' with flavor 'aws'.
Migrating stacks from /home/stefan/.config/zenml/profiles/v/stacks.yaml...
Created stack 'cloud_kubeflow_stack'.
Created stack 'local_kubeflow_stack'.

$ zenml stack list
Using the default local database.
Running with active project: 'default' (global)
┏━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┓
┃ ACTIVE │ STACK NAME           │ STACK ID                             │ SHARED │ OWNER   │ CONTAINER_REGISTRY │ ARTIFACT_STORE       │ ORCHESTRATOR          │ MODEL_DEPLOYER │ SECRETS_MANAGER    ┃
┠────────┼──────────────────────┼──────────────────────────────────────┼────────┼─────────┼────────────────────┼──────────────────────┼───────────────────────┼────────────────┼────────────────────┨
┃        │ local_kubeflow_stack │ 067cc6ee-b4da-410d-b7ed-06da4c983145 │        │ default │ local_registry     │ default              │ kubeflow_orchestrator │                │                    ┃
┠────────┼──────────────────────┼──────────────────────────────────────┼────────┼─────────┼────────────────────┼──────────────────────┼───────────────────────┼────────────────┼────────────────────┨
┃        │ cloud_kubeflow_stack │ 054f5efb-9e80-48c0-852e-5114b1165d8b │        │ default │ cloud_registry     │ cloud_artifact_store │ cloud_orchestrator    │ eks_seldon     │ aws_secret_manager ┃
┠────────┼──────────────────────┼──────────────────────────────────────┼────────┼─────────┼────────────────────┼──────────────────────┼───────────────────────┼────────────────┼────────────────────┨
┃   👉   │ default              │ fe913bb5-e631-4d4e-8c1b-936518190ebb │        │ default │                    │ default              │ default               │                │                    ┃
┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┛
```

Example of migrating a profile into the `default` project using a name prefix:

```bash
$ zenml profile migrate /home/stefan/.config/zenml/profiles/zenbytes --prefix zenbytes_
No component flavors to migrate from /home/stefan/.config/zenml/profiles/zenbytes/stacks.yaml...
Migrating stack components from /home/stefan/.config/zenml/profiles/zenbytes/stacks.yaml...
Created artifact_store 'zenbytes_s3_store' with flavor 's3'.
Created container_registry 'zenbytes_ecr_registry' with flavor 'default'.
Created experiment_tracker 'zenbytes_mlflow_tracker' with flavor 'mlflow'.
Created experiment_tracker 'zenbytes_mlflow_tracker_local' with flavor 'mlflow'.
Created model_deployer 'zenbytes_eks_seldon' with flavor 'seldon'.
Created model_deployer 'zenbytes_mlflow' with flavor 'mlflow'.
Created orchestrator 'zenbytes_eks_orchestrator' with flavor 'kubeflow'.
Created secrets_manager 'zenbytes_aws_secret_manager' with flavor 'aws'.
Migrating stacks from /home/stefan/.config/zenml/profiles/zenbytes/stacks.yaml...
Created stack 'zenbytes_aws_kubeflow_stack'.
Created stack 'zenbytes_local_with_mlflow'.

$ zenml stack list
Using the default local database.
Running with active project: 'default' (global)
┏━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┓
┃ ACTIVE │ STACK NAME           │ STACK ID             │ SHARED │ OWNER   │ ORCHESTRATOR          │ ARTIFACT_STORE    │ CONTAINER_REGISTRY   │ SECRETS_MANAGER       │ MODEL_DEPLOYER      │ EXPERIMENT_TRACKER   ┃
┠────────┼──────────────────────┼──────────────────────┼────────┼─────────┼───────────────────────┼───────────────────┼──────────────────────┼───────────────────────┼─────────────────────┼──────────────────────┨
┃        │ zenbytes_aws_kubeflo │ 9fe90f0b-2a79-47d9-8 │        │ default │ zenbytes_eks_orchestr │ zenbytes_s3_store │ zenbytes_ecr_registr │ zenbytes_aws_secret_m │ zenbytes_eks_seldon │                      ┃
┃        │ w_stack              │ f80-04e45ff02cdb     │        │         │ ator                  │                   │ y                    │ manager                │                     │                      ┃
┠────────┼──────────────────────┼──────────────────────┼────────┼─────────┼───────────────────────┼───────────────────┼──────────────────────┼───────────────────────┼─────────────────────┼──────────────────────┨
┃   👉   │ default              │ 7a587e0c-30fd-402f-a │        │ default │ default               │ default           │                      │                       │                     │                      ┃
┃        │                      │ 3a8-03651fe1458f     │        │         │                       │                   │                      │                       │                     │                      ┃
┠────────┼──────────────────────┼──────────────────────┼────────┼─────────┼───────────────────────┼───────────────────┼──────────────────────┼───────────────────────┼─────────────────────┼──────────────────────┨
┃        │ zenbytes_local_with_ │ c2acd029-8eed-4b6e-a │        │ default │ default               │ default           │                      │                       │ zenbytes_mlflow     │ zenbytes_mlflow_trac ┃
┃        │ mlflow               │ d19-91c419ce91d4     │        │         │                       │                   │                      │                       │                     │ ker                  ┃
┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┛
```

Example of migrating a profile into a new project:

```bash
$ zenml profile migrate /home/stefan/.config/zenml/profiles/zenprojects --project zenprojects
Unable to find ZenML repository in your current working directory (/home/stefan/aspyre/src/zenml) or any parent directories. If you want to use an existing repository which is in a different location, set the environment variable 'ZENML_REPOSITORY_PATH'. If you want to create a new repository, run zenml init.
Running without an active repository root.
Creating project zenprojects
Creating default stack for user 'default' in project zenprojects...
No component flavors to migrate from /home/stefan/.config/zenml/profiles/zenprojects/stacks.yaml...
Migrating stack components from /home/stefan/.config/zenml/profiles/zenprojects/stacks.yaml...
Created artifact_store 'cloud_artifact_store' with flavor 's3'.
Created container_registry 'cloud_registry' with flavor 'aws'.
Created container_registry 'local_registry' with flavor 'default'.
Created model_deployer 'eks_seldon' with flavor 'seldon'.
Created orchestrator 'cloud_orchestrator' with flavor 'kubeflow'.
Created orchestrator 'kubeflow_orchestrator' with flavor 'kubeflow'.
Created secrets_manager 'aws_secret_manager' with flavor 'aws'.
Migrating stacks from /home/stefan/.config/zenml/profiles/zenprojects/stacks.yaml...
Created stack 'cloud_kubeflow_stack'.
Created stack 'local_kubeflow_stack'.

$ zenml project set zenprojects
Currently the concept of `project` is not supported within the Dashboard. The Project functionality will be completed in the coming weeks. For the time being it is recommended to stay within the `default` 
project.
Using the default local database.
Running with active project: 'default' (global)
Set active project 'zenprojects'.

$ zenml stack list
Using the default local database.
Running with active project: 'zenprojects' (global)
The current global active stack is not part of the active project. Resetting the active stack to default.
You are running with a non-default project 'zenprojects'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future.
┏━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┓
┃ ACTIVE │ STACK NAME           │ STACK ID                             │ SHARED │ OWNER   │ ARTIFACT_STORE       │ ORCHESTRATOR          │ MODEL_DEPLOYER │ CONTAINER_REGISTRY │ SECRETS_MANAGER    ┃
┠────────┼──────────────────────┼──────────────────────────────────────┼────────┼─────────┼──────────────────────┼───────────────────────┼────────────────┼────────────────────┼────────────────────┨
┃   👉   │ default              │ 3ea77330-0c75-49c8-b046-4e971f45903a │        │ default │ default              │ default               │                │                    │                    ┃
┠────────┼──────────────────────┼──────────────────────────────────────┼────────┼─────────┼──────────────────────┼───────────────────────┼────────────────┼────────────────────┼────────────────────┨
┃        │ cloud_kubeflow_stack │ b94df4d2-5b65-4201-945a-61436c9c5384 │        │ default │ cloud_artifact_store │ cloud_orchestrator    │ eks_seldon     │ cloud_registry     │ aws_secret_manager ┃
┠────────┼──────────────────────┼──────────────────────────────────────┼────────┼─────────┼──────────────────────┼───────────────────────┼────────────────┼────────────────────┼────────────────────┨
┃        │ local_kubeflow_stack │ 8d9343ac-d405-43bd-ab9c-85637e479efe │        │ default │ default              │ kubeflow_orchestrator │                │ local_registry     │                    ┃
┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┛
```

The `zenml profile migrate` CLI command also provides command line flags for cases in which the user wants to overwrite existing components or stacks, or ignore errors.

## Decoupling Stack Component configuration from implementation

Stack components can now be registered without having the required integrations installed. As part of this change, we split all existing stack component definitions into three classes: an implementation class that defines the logic of the stack component, a config class that defines the attributes and performs input validations, and a flavor class that links implementation and config classes together. See [**component flavor models #895**](https://github.com/zenml-io/zenml/pull/895) for more details.

If you are only using stack component flavors that are shipped with the zenml Python distribution, this change has no impact on the configuration of your existing stacks. However, if you are currently using custom stack component implementations, you will need to update them to the new format. See the [documentation on writing custom stack component flavors](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/implement-a-custom-stack-component) for updated information on how to do this.

## Shared ZenML Stacks and Stack Components

With collaboration being the key part of ZenML, the 0.20.0 release puts the concepts of Users in the front and center and introduces the possibility to share stacks and stack components with other users by means of the ZenML server.

When your client is connected to a ZenML server, entities such as Stacks, Stack Components, Stack Component Flavors, Pipelines, Pipeline Runs, and artifacts are scoped to a Project and owned by the User that creates them. Only the objects that are owned by the current user used to authenticate to the ZenML server and that are part of the current project are available to the client.

Stacks and Stack Components can also be shared within the same project with other users. To share an object, either set it as shared during creation time (e.g. `zenml stack register mystack ... --share`) or afterwards (e.g. through `zenml stack share mystack`).

To differentiate between shared and private Stacks and Stack Components, these can now be addressed by name, id or the first few letters of the id in the cli. E.g. for a stack `default` with id `179ebd25-4c5b-480f-a47c-d4f04e0b6185` you can now run `zenml stack describe default` or `zenml stack describe 179` or `zenml stack describe 179ebd25-4c5b-480f-a47c-d4f04e0b6185`.

We also introduce the notion of `local` vs `non-local` stack components. Local stack components are stack components that are configured to run locally while non-local stack components are configured to run remotely or in a cloud environment. Consequently:

* stacks made up of local stack components should not be shared on a central ZenML Server, even though this is not enforced by the system.
* stacks made up of non-local stack components are only functional if they are shared through a remotely deployed ZenML Server.

Read more about shared stacks in the [production guide](https://docs.zenml.io/user-guides/production-guide/understand-stacks).

## Other changes

### The `Repository` class is now called `Client`

The `Repository` object has been renamed to `Client` to better capture its functionality. You can continue to use the `Repository` object for backwards compatibility, but it will be removed in a future release.

**How to migrate**: Rename all references to `Repository` in your code to `Client`.

### The `BaseStepConfig` class is now called `BaseParameters`

The `BaseStepConfig` object has been renamed to `BaseParameters` to better capture its functionality. You can NOT continue to use the `BaseStepConfig`.

This is part of a broader configuration rehaul which is discussed next.

**How to migrate**: Rename all references to `BaseStepConfig` in your code to `BaseParameters`.

### Configuration Rework

Alongside the architectural shift, Pipeline configuration has been completely rethought. This video gives an overview of how configuration has changed with ZenML in the post ZenML 0.20.0 world.

{% embed url="<https://www.youtube.com/embed/hI-UNV7uoNI>" %}
Configuring pipelines, steps, and stack components in ZenML
{% endembed %}

**What changed?**

ZenML pipelines and steps could previously be configured in many different ways:

* On the `@pipeline` and `@step` decorators (e.g. the `requirements` variable)
* In the `__init__` method of the pipeline and step class
* Using `@enable_xxx` decorators, e.g. `@enable_mlflow`.
* Using specialized methods like `pipeline.with_config(...)` or `step.with_return_materializer(...)`

Some of the configuration options were quite hidden, difficult to access and not tracked in any way by the ZenML metadata store.

With ZenML 0.20.0, we introduce the `BaseSettings` class, a broad class that serves as a central object to represent all runtime configuration of a pipeline run (apart from the `BaseParameters`).

Pipelines and steps now allow all configurations on their decorators as well as the `.configure(...)` method. This includes configurations for stack components that are not infrastructure-related which was previously done using the `@enable_xxx` decorators). The same configurations can also be defined in a YAML file.

Read more about this paradigm in the [new docs section about settings](https://docs.zenml.io/concepts/steps_and_pipelines/configuration).

Here is a list of changes that are the most obvious in consequence of the above code. Please note that this list is not exhaustive, and if we have missed something let us know via [Slack](https://zenml.io/slack).

**Deprecating the `enable_xxx` decorators**

With the above changes, we are deprecating the much-loved `enable_xxx` decorators, like `enable_mlflow` and `enable_wandb`.

**How to migrate**: Simply remove the decorator and pass something like this instead to step directly:

```python
@step(
    experiment_tracker="mlflow_stack_comp_name",  # name of registered component
    settings={  # settings of registered component
        "experiment_tracker.mlflow": {  # this is `category`.`flavor`, so another example is `step_operator.spark`
            "experiment_name": "name",
            "nested": False
        }
    }
)
```

**Deprecating `pipeline.with_config(...)`**

**How to migrate**: Replaced with the new `pipeline.run(config_path=...)`.

**Deprecating `step.with_return_materializer(...)`**

**How to migrate**: Simply remove the `with_return_materializer` method and pass something like this instead to step directly:

```python
@step(
  output_materializers=materializer_or_dict_of_materializers_mapped_to_outputs
)
```

**`DockerConfiguration` is now renamed to `DockerSettings`**

**How to migrate**: Rename `DockerConfiguration` to `DockerSettings` and instead of passing it in the decorator directly with `docker_configuration`, you can use:

```python
from zenml.config import DockerSettings

@step(settings={"docker": DockerSettings(...)})
def my_step() -> None:
  ...
```

With this change, all stack components (e.g. Orchestrators and Step Operators) that accepted a `docker_parent_image` as part of its Stack Configuration should now pass it through the `DockerSettings` object.

Read more [here](https://docs.zenml.io/how-to/customize-docker-builds/docker-settings-on-a-pipeline).

**`ResourceConfiguration` is now renamed to `ResourceSettings`**

**How to migrate**: Rename `ResourceConfiguration` to `ResourceSettings` and instead of passing it in the decorator directly with `resource_configuration`, you can use:

```python
from zenml.config import ResourceSettings

@step(settings={"resources": ResourceSettings(...)})
def my_step() -> None:
  ...
```

**Deprecating the `requirements` and `required_integrations` parameters**

Users used to be able to pass `requirements` and `required_integrations` directly in the `@pipeline` decorator, but now need to pass them through settings:

**How to migrate**: Simply remove the parameters and use the `DockerSettings` instead

```python
from zenml.config import DockerSettings

@step(settings={"docker": DockerSettings(requirements=[...], requirements_integrations=[...])})
def my_step() -> None:
  ...
```

Read more [here](https://docs.zenml.io/how-to/customize-docker-builds).

**A new pipeline intermediate representation**

All the aforementioned configurations as well as additional information required to run a ZenML pipelines are now combined into an intermediate representation called `PipelineDeployment`. Instead of the user-facing `BaseStep` and `BasePipeline` classes, all the ZenML orchestrators and step operators now use this intermediate representation to run pipelines and steps.

**How to migrate**: If you have written a [custom orchestrator](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/implement-a-custom-stack-component) or [step operator](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/implement-a-custom-stack-component), then you should see the new base abstractions (seen in the links). You can adjust your stack component implementations accordingly.

### `PipelineSpec` now uniquely defines pipelines

Once a pipeline has been executed, it is represented by a `PipelineSpec` that uniquely identifies it. Therefore, users are no longer able to edit a pipeline once it has been run once. There are now three options to get around this:

* Pipeline runs can be created without being associated with a pipeline explicitly: We call these `unlisted` runs. Read more about unlisted runs [here](https://docs.zenml.io/user-guides/best-practices/keep-your-dashboard-server-clean#unlisted-runs).
* Pipelines can be deleted and created again.
* Pipelines can be given unique names each time they are run to uniquely identify them.

**How to migrate**: No code changes, but rather keep in mind the behavior (e.g. in a notebook setting) when quickly [iterating over pipelines as experiments](https://docs.zenml.io/concepts/steps_and_pipelines#parameters-and-artifacts).

### New post-execution workflow

The Post-execution workflow has changed as follows:

* The `get_pipelines` and `get_pipeline` methods have been moved out of the `Repository` (i.e. the new `Client` ) class and lie directly in the post\_execution module now. To use the user has to do:

```python
from zenml.post_execution import get_pipelines, get_pipeline
```

* New methods to directly get a run have been introduced: `get_run` and `get_unlisted_runs` method has been introduced to get unlisted runs.

Usage remains largely similar. Please read the [new docs for post-execution](https://docs.zenml.io/user-guides/tutorial/fetching-pipelines) to inform yourself of what further has changed.

**How to migrate**: Replace all post-execution workflows from the paradigm of `Repository.get_pipelines` or `Repository.get_pipeline_run` to the corresponding post\_execution methods.

## 📡Future Changes

While this rehaul is big and will break previous releases, we do have some more work left to do. However we also expect this to be the last big rehaul of ZenML before our 1.0.0 release, and no other release will be so hard breaking as this one. Currently planned future breaking changes are:

* Following the metadata store, the secrets manager stack component might move out of the stack.
* ZenML `StepContext` might be deprecated.

## 🐞 Reporting Bugs

While we have tried our best to document everything that has changed, we realize that mistakes can be made and smaller changes overlooked. If this is the case, or you encounter a bug at any time, the ZenML core team and community are available around the clock on the growing [Slack community](https://zenml.io/slack).

For bug reports, please also consider submitting a [GitHub Issue](https://github.com/zenml-io/zenml/issues/new/choose).

Lastly, if the new changes have left you desiring a feature, then consider adding it to our [public feature voting board](https://zenml.io/discussion). Before doing so, do check what is already on there and consider upvoting the features you desire the most.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Migration guide 0.23.0 → 0.30.0

How to migrate from ZenML 0.20.0-0.23.0 to 0.30.0-0.39.1.

{% hint style="warning" %}
Migrating to `0.30.0` performs non-reversible database changes so downgrading to `<=0.23.0` is not possible afterwards. If you are running on an older ZenML version, please follow the [0.20.0 Migration Guide](/deploying-zenml/upgrade-zenml-server/migration-guide/migration-zero-twenty) first to prevent unexpected database migration failures.
{% endhint %}

The ZenML 0.30.0 release removed the `ml-pipelines-sdk` dependency in favor of natively storing pipeline runs and artifacts in the ZenML database. The corresponding database migration will happen automatically as soon as you run any `zenml ...` CLI command after installing the new ZenML version, e.g.:

```bash
pip install zenml==0.30.0
zenml version  # 0.30.0
```

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Migration guide 0.39.1 → 0.41.0

How to migrate your ZenML pipelines and steps from version <=0.39.1 to 0.41.0.

ZenML versions 0.40.0 to 0.41.0 introduced a new and more flexible syntax to define ZenML steps and pipelines. This page contains code samples that show you how to upgrade your steps and pipelines to the new syntax.

{% hint style="warning" %}
Newer versions of ZenML still work with pipelines and steps defined using the old syntax, but the old syntax is deprecated and will be removed in the future.
{% endhint %}

## Overview

{% tabs %}
{% tab title="Old Syntax" %}

```python
from typing import Optional

from zenml.steps import BaseParameters, Output, StepContext, step
from zenml.pipelines import pipeline

# Define a Step
class MyStepParameters(BaseParameters):
    param_1: int
    param_2: Optional[float] = None

@step
def my_step(
    params: MyStepParameters, context: StepContext,
) -> Output(int_output=int, str_output=str):
    result = int(params.param_1 * (params.param_2 or 1))
    result_uri = context.get_output_artifact_uri()
    return result, result_uri

# Run the Step separately
my_step.entrypoint()

# Define a Pipeline
@pipeline
def my_pipeline(my_step):
    my_step()

step_instance = my_step(params=MyStepParameters(param_1=17))
pipeline_instance = my_pipeline(my_step=step_instance)

# Configure and run the Pipeline
pipeline_instance.configure(enable_cache=False)
schedule = Schedule(...)
pipeline_instance.run(schedule=schedule)

# Fetch the Pipeline Run
last_run = pipeline_instance.get_runs()[0]
int_output = last_run.get_step["my_step"].outputs["int_output"].read()
```

{% endtab %}

{% tab title="New Syntax" %}

```python
from typing import Annotated, Optional, Tuple

from zenml import get_step_context, pipeline, step
from zenml.client import Client

# Define a Step
@step
def my_step(
    param_1: int, param_2: Optional[float] = None
) -> Tuple[Annotated[int, "int_output"], Annotated[str, "str_output"]]:
    result = int(param_1 * (param_2 or 1))
    result_uri = get_step_context().get_output_artifact_uri()
    return result, result_uri

# Run the Step separately
my_step()

# Define a Pipeline
@pipeline
def my_pipeline():
    my_step(param_1=17)

# Configure and run the Pipeline
my_pipeline = my_pipeline.with_options(enable_cache=False, schedule=schedule)
my_pipeline()

# Fetch the Pipeline Run
last_run = my_pipeline.last_run
int_output = last_run.steps["my_step"].outputs["int_output"].load()
```

{% endtab %}
{% endtabs %}

## Defining steps

{% tabs %}
{% tab title="Old Syntax" %}

```python
from typing import Optional
from zenml.steps import step, BaseParameters
from zenml.pipelines import pipeline

# Old: Subclass `BaseParameters` to define parameters for a step
class MyStepParameters(BaseParameters):
    param_1: int
    param_2: Optional[float] = None

@step
def my_step(params: MyStepParameters) -> None:
    ...

@pipeline
def my_pipeline(my_step):
    my_step()

step_instance = my_step(params=MyStepParameters(param_1=17))
pipeline_instance = my_pipeline(my_step=step_instance)
```

{% endtab %}

{% tab title="New Syntax" %}

```python
# New: Directly define the parameters as arguments of your step function.
# In case you still want to group your parameters in a separate class,
# you can subclass `pydantic.BaseModel` and use that as an argument of your
# step function
from zenml import pipeline, step

@step
def my_step(param_1: int, param_2: Optional[float] = None) -> None:
    ...

@pipeline
def my_pipeline():
    my_step(param_1=17)
```

{% endtab %}
{% endtabs %}

Check out [this page](https://docs.zenml.io/concepts/steps_and_pipelines#parameters-and-artifacts) for more information on how to parameterize your steps.

## Calling a step outside of a pipeline

{% tabs %}
{% tab title="Old Syntax" %}

```python
from zenml.steps import step

@step
def my_step() -> None:
    ...

my_step.entrypoint()  # Old: Call `step.entrypoint(...)`
```

{% endtab %}

{% tab title="New Syntax" %}

```python
from zenml import step

@step
def my_step() -> None:
    ...

my_step()  # New: Call the step directly `step(...)`
```

{% endtab %}
{% endtabs %}

## Defining pipelines

{% tabs %}
{% tab title="Old Syntax" %}

```python
from zenml.pipelines import pipeline

@pipeline
def my_pipeline(my_step):  # Old: steps are arguments of the pipeline function
    my_step()
```

{% endtab %}

{% tab title="New Syntax" %}

```python
from zenml import pipeline, step

@step
def my_step() -> None:
    ...

@pipeline
def my_pipeline():
    my_step()  # New: The pipeline function calls the step directly
```

{% endtab %}
{% endtabs %}

## Configuring pipelines

{% tabs %}
{% tab title="Old Syntax" %}

```python
from zenml.pipelines import pipeline
from zenml.steps import step

@step
def my_step() -> None:
    ...

@pipeline
def my_pipeline(my_step):
    my_step()

# Old: Create an instance of the pipeline and then call `pipeline_instance.configure(...)`
pipeline_instance = my_pipeline(my_step=my_step())
pipeline_instance.configure(enable_cache=False)
```

{% endtab %}

{% tab title="New Syntax" %}

```python
from zenml import pipeline, step

@step
def my_step() -> None:
    ...

@pipeline
def my_pipeline():
    my_step()

# New: Call the `with_options(...)` method on the pipeline
my_pipeline = my_pipeline.with_options(enable_cache=False)
```

{% endtab %}
{% endtabs %}

## Running pipelines

{% tabs %}
{% tab title="Old Syntax" %}

```python
from zenml.pipelines import pipeline
from zenml.steps import step

@step
def my_step() -> None:
    ...

@pipeline
def my_pipeline(my_step):
    my_step()

# Old: Create an instance of the pipeline and then call `pipeline_instance.run(...)`
pipeline_instance = my_pipeline(my_step=my_step())
pipeline_instance.run(...)
```

{% endtab %}

{% tab title="New Syntax" %}

```python
from zenml import pipeline, step

@step
def my_step() -> None:
    ...

@pipeline
def my_pipeline():
    my_step()

my_pipeline()  # New: Call the pipeline
```

{% endtab %}
{% endtabs %}

## Scheduling pipelines

{% tabs %}
{% tab title="Old Syntax" %}

```python
from zenml.pipelines import pipeline, Schedule
from zenml.steps import step

@step
def my_step() -> None:
    ...

@pipeline
def my_pipeline(my_step):
    my_step()

# Old: Create an instance of the pipeline and then call `pipeline_instance.run(schedule=...)`
schedule = Schedule(...)
pipeline_instance = my_pipeline(my_step=my_step())
pipeline_instance.run(schedule=schedule)
```

{% endtab %}

{% tab title="New Syntax" %}

```python
from zenml.pipelines import Schedule
from zenml import pipeline, step

@step
def my_step() -> None:
    ...

@pipeline
def my_pipeline():
    my_step()

# New: Set the schedule using the `pipeline.with_options(...)` method and then run it
schedule = Schedule(...)
my_pipeline = my_pipeline.with_options(schedule=schedule)
my_pipeline()
```

{% endtab %}
{% endtabs %}

Check out [this page](https://docs.zenml.io/user-guides/tutorial/managing-scheduled-pipelines) for more information on how to schedule your pipelines.

## Fetching pipelines after execution

{% tabs %}
{% tab title="Old Syntax" %}

```python
pipeline: PipelineView = zenml.post_execution.get_pipeline("first_pipeline")

last_run: PipelineRunView = pipeline.runs[0]
# OR: last_run = my_pipeline.get_runs()[0]

model_trainer_step: StepView = last_run.get_step("model_trainer")

model: ArtifactView = model_trainer_step.output
loaded_model = model.read()
```

{% endtab %}

{% tab title="New Syntax" %}

```python
pipeline: PipelineResponseModel = zenml.client.Client().get_pipeline("first_pipeline")
# OR: pipeline = pipeline_instance.model

last_run: PipelineRunResponseModel = pipeline.last_run  
# OR: last_run = pipeline.runs[0] 
# OR: last_run = pipeline.get_runs(custom_filters)[0] 
# OR: last_run = pipeline.last_successful_run

model_trainer_step: StepRunResponseModel = last_run.steps["model_trainer"]

model: ArtifactResponseModel = model_trainer_step.output
loaded_model = model.load()
```

{% endtab %}
{% endtabs %}

Check out [this page](https://docs.zenml.io/how-to/model-management-metrics/track-metrics-metadata/fetch-metadata-within-steps) for more information on how to programmatically fetch information about previous pipeline runs.

## Controlling the step execution order

{% tabs %}
{% tab title="Old Syntax" %}

```python
from zenml.pipelines import pipeline

@pipeline
def my_pipeline(step_1, step_2, step_3):
    step_1()
    step_2()
    step_3()
    step_3.after(step_1)  # Old: Use the `step.after(...)` method
    step_3.after(step_2)
```

{% endtab %}

{% tab title="New Syntax" %}

```python
from zenml import pipeline

@pipeline
def my_pipeline():
    step_1()
    step_2()
    step_3(after=["step_1", "step_2"])  # New: Pass the `after` argument when calling a step
```

{% endtab %}
{% endtabs %}

Check out [this page](https://docs.zenml.io/concepts/steps_and_pipelines/advanced_features#step-execution-order) for more information on how to control the step execution order.

## Defining steps with multiple outputs

{% tabs %}
{% tab title="Old Syntax" %}

```python
# Old: Use the `Output` class
from zenml.steps import step, Output

@step
def my_step() -> Output(int_output=int, str_output=str):
    ...
```

{% endtab %}

{% tab title="New Syntax" %}

```python
# New: Use a `Tuple` annotation and optionally assign custom output names
from typing import Annotated
from typing import Tuple
from zenml import step

# Default output names `output_0`, `output_1`
@step
def my_step() -> Tuple[int, str]:
    ...

# Custom output names
@step
def my_step() -> Tuple[
    Annotated[int, "int_output"],
    Annotated[str, "str_output"],
]:
    ...
```

{% endtab %}
{% endtabs %}

Check out [this page](https://docs.zenml.io/concepts/steps_and_pipelines#type-annotations) for more information on how to annotate your step outputs.

## Accessing run information inside steps

{% tabs %}
{% tab title="Old Syntax" %}

```python
from zenml.steps import StepContext, step
from zenml.environment import Environment

@step
def my_step(context: StepContext) -> Any:  # Old: `StepContext` class defined as arg
    env = Environment().step_environment
    output_uri = context.get_output_artifact_uri()
    step_name = env.step_name  # Old: Run info accessible via `StepEnvironment`
    ...
```

{% endtab %}

{% tab title="New Syntax" %}

```python
from zenml import get_step_context, step

@step
def my_step() -> Any:  # New: StepContext is no longer an argument of the step
    context = get_step_context()
    output_uri = context.get_output_artifact_uri()
    step_name = context.step_name  # New: StepContext now has ALL run/step info
    ...
```

{% endtab %}
{% endtabs %}

Check out [this page](https://docs.zenml.io/how-to/model-management-metrics/track-metrics-metadata/fetch-metadata-within-steps) for more information on how to fetch run information inside your steps using `get_step_context()`.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Migration guide 0.58.2 → 0.60.0

How to migrate from ZenML 0.58.2 to 0.60.0 (Pydantic 2 edition).

ZenML now uses Pydantic v2. 🥳

This upgrade comes with a set of critical updates. While your user experience mostly remains unaffected, you might see unexpected behavior due to the changes in our dependencies. Moreover, since Pydantic v2 provides a slightly stricter validation process, you might end up bumping into some validation errors which was not caught before, but it is all for the better 🙂 If you run into any other errors, please let us know either on [GitHub](https://github.com/zenml-io/zenml) or on our [Slack](https://zenml.io/slack-invite).

## Changes in some of the critical dependencies

* SQLModel is one of the core dependencies of ZenML and prior to this upgrade, we were utilizing version `0.0.8`. However, this version is relatively outdated and incompatible with Pydantic v2. Within the scope of this upgrade, we upgraded it to `0.0.18`.
* Due to the change in the SQLModel version, we also had to upgrade our SQLAlchemy dependency from V1 to v2. While this does not affect the way that you are using ZenML, if you are using SQLAlchemy in your environment, you might have to migrate your code as well. For a detailed list of changes, feel free to check [their migration guide](https://docs.sqlalchemy.org/en/20/changelog/migration_20.html).

## Changes in `pydantic`

Pydantic v2 brings a lot of new and exciting changes to the table. The core logic now uses Rust and it is much faster and more efficient in terms of performance. On top of it, the main concepts like model design, configuration, validation, or serialization now include a lot of new cool features. If you are using `pydantic` in your workflow and are interested in the new changes, you can check [the brilliant migration guide](https://docs.pydantic.dev/2.7/migration/) provided by the `pydantic` team to see the full list of changes.

## Changes in our integrations changes

Much like ZenML, `pydantic` is an important dependency in many other Python packages. That’s why conducting this upgrade helped us unlock a new version for several ZenML integration dependencies. Additionally, in some instances, we had to adapt the functionality of the integration to keep it compatible with `pydantic`. So, if you are using any of these integrations, please go through the changes.

### Airflow

As mentioned above upgrading our `pydantic` dependency meant we had to upgrade our `sqlmodel` dependency. Upgrading our `sqlmodel` dependency meant we had to upgrade our `sqlalchemy` dependency as well. Unfortunately, `apache-airflow` is still using `sqlalchemy` v1 and is incompatible with pydantic v2. As a solution, we have removed the dependencies of the `airflow` integration. Now, you can use ZenML to create your Airflow pipelines and use a separate environment to run them with Airflow. You can check the updated docs [right here](https://docs.zenml.io/stacks/orchestrators/airflow).

### AWS

Some of our integrations now require `protobuf` 4. Since our previous `sagemaker` version (`2.117.0`) did not support `protobof` 4, we could not pair it with these new integrations. Thankfully `sagemaker` started supporting `protobuf` 4 with version `2.172.0` and relaxing its dependency solved the compatibility issue.

### Evidently

The old version of our `evidently` integration was not compatible with Pydantic v2. They started supporting it starting from version `0.4.16`. As their latest version is `0.4.22`, the new dependency of the integration is limited between these two versions.

### Feast

Our previous implementation of the `feast` integration was not compatible with Pydantic v2 due to the extra `redis` dependency we were using. This extra dependency is now removed and the `feast` integration is working as intended.

### GCP

The previous version of the Kubeflow dependency (`kfp==1.8.22`) in our GCP integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the `kfp` dependency to v2 (which has no dependencies on the Pydantic library). This is why you may see some functional changes in the vertex step operator and orchestrator. If you would like to go through the changes in the `kfp` library, you can find [the migration guide here](https://www.kubeflow.org/docs/components/pipelines/v2/migration/).

### Great Expectations

Great Expectations started supporting Pydantic v2 starting from version `0.17.15` and they are closing in on their `1.0` release. Since this release might include a lot of big changes, we adjusted the dependency in our integration to `great-expectations>=0.17.15,<1.0`. We will try to keep it updated in the future once they release the `1.0` version

### Kubeflow

Similar to the GCP integration, the previous version of the kubeflow dependency (`kfp==1.8.22`) in our `kubeflow` integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the `kfp` dependency to v2 (which has no dependencies on the Pydantic library). If you would like to go through the changes in the `kfp` library, you can find [the migration guide here](https://www.kubeflow.org/docs/components/pipelines/v2/migration/). ( We also are considering adding an alternative version of this integration so our users can keep using `kfp` V1 in their environment. Stay tuned for any updates.)

### MLflow

`mlflow` is compatible with both Pydantic V1 and v2. However, due to a known issue, if you install `zenml` first and then do `zenml integration install mlflow -y`, it downgrades `pydantic` to V1. This is why we manually added the same duplicated `pydantic` requirement in the integration definition as well. Keep in mind that the `mlflow` library is still using some features of `pydantic` V1 which are deprecated. So, if the integration is installed in your environment, you might run into some deprecation warnings.

### Label Studio

While we were working on updating our `pydantic` dependency, the `label-studio-sdk` has released its 1.0 version. In this new version, `pydantic` v2 is also supported. The implementation and documentation of our Label Studio integration have been updated accordingly.

### Skypilot

With the switch to `pydantic` v2, the implementation of our `skypilot` integration mostly remained untouched. However, due to an incompatibility between the new version `pydantic` and the `azurecli`, the `skypilot[azure]` flavor can not be installed at the same time, thus our `skypilot_azure` integration is currently deactivated. We are working on fixing this issue and if you are using this integration in your workflows, we recommend staying on the previous version of ZenML until we can solve this issue.

### Tensorflow

The new version of `pydantic` creates a drift between `tensorflow` and `typing_extensions` packages and relaxing the dependencies here resolves the issue. At the same time, the upgrade to `kfp` v2 (in integrations like `kubeflow`, `tekton`, or `gcp`) bumps our `protobuf` dependency from `3.X` to `4.X`. To stay compatible with this requirement, the installed version of `tensorflow` needs to be `>=2.12.0`. While this change solves the dependency issues in most settings, we have bumped into some errors while using `tensorflow` 2.12.0 on Python 3.8 on Ubuntu. If you would like to use this integration, please consider using a higher Python version.

### Tekton

Similar to the `gcp` and `kubeflow` integrations, the old version of our `tekton` integration was not compatible with `pydantic` V1 due to its `kfp` dependency. With the switch from `kfp` V1 to v2, we have adapted our implementation to use the new version of `kfp` library and updated our documentation accordingly.

{% hint style="warning" %}
Due to all aforementioned changes, when you upgrade ZenML to 0.60.0, you might run into some dependency issues, especially if you were previously using an integration which was not supporting Pydantic v2 before. In such cases, we highly recommend setting up a fresh Python environment.
{% endhint %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Steps & Pipelines

Steps and Pipelines are the core building blocks of ZenML

Steps and Pipelines are the fundamental building blocks of ZenML. A **Step** is a reusable unit of computation, and a **Pipeline** is a directed acyclic graph (DAG) composed of steps. Together, they allow you to define, version, and execute machine learning workflows.

## The Relationship Between Steps and Pipelines

In ZenML, steps and pipelines work together in a clear hierarchy:

1. **Steps** are individual functions that perform specific tasks, like loading data, processing it, or training models
2. **Pipelines** orchestrate these steps, connecting them in a defined sequence where outputs from one step can flow as inputs to others
3. Each step produces artifacts that are tracked, versioned, and can be reused across pipeline runs

Think of a step as a single LEGO brick, and a pipeline as the complete structure you build by connecting many bricks together.

## Basic Steps

### Creating a Simple Step

A step is created by applying the `@step` decorator to a Python function:

```python
from zenml import step

@step
def load_data() -> dict:
    training_data = [[1, 2], [3, 4], [5, 6]]
    labels = [0, 1, 0]
    return {'features': training_data, 'labels': labels}
```

### Step Inputs and Outputs

Steps can take inputs and produce outputs. These can be simple types, complex data structures, or custom objects.

```python
@step
def process_data(data: dict) -> dict:
    # Input: data dictionary with features and labels
    # Process the input data
    processed_features = [feature * 2 for feature in data['features']]
    
    # Output: return processed data and statistics
    return {
        'processed_features': processed_features,
        'labels': data['labels'],
        'num_samples': len(data['features']),
        'feature_sum': sum(map(sum, data['features']))
    }
```

In this example:

* The step takes a `dict` as input containing features and labels
* It processes the features and computes some statistics
* It returns a new `dict` as output with the processed data and additional information

### Custom Output Names

You can name your step outputs using the `Annotated` type:

```python
from typing import Annotated
from typing import Tuple

@step
def divide(a: int, b: int) -> Tuple[
    Annotated[int, "quotient"],
    Annotated[int, "remainder"]
]:
    return a // b, a % b
```

By default, step outputs are named `output` for single output steps and `output_0`, `output_1`, etc. for steps with multiple outputs.

## Basic Pipelines

### Creating a Simple Pipeline

A pipeline is created by applying the `@pipeline` decorator to a Python function that composes steps together:

```python
from zenml import pipeline

@pipeline
def simple_ml_pipeline():
    dataset = load_data()
    train_model(dataset)
```

### Running Pipelines

You can run a pipeline by simply calling the function:

```python
simple_ml_pipeline()
```

The run is automatically logged to the ZenML dashboard where you can view the DAG or [Timeline view](/concepts/dashboard-features#timeline-view) and associated metadata.

{% hint style="info" %}
**Want an AI agent to help write this pipeline?** Install ZenML Agent Skills:

* `zenml-pipeline-authoring` for step/pipeline code, artifacts, config, and remote execution details
* `zenml-scoping` first if you're designing a multi-pipeline system

See [LLM tooling](/reference/llms-txt).
{% endhint %}

## End-to-End Example

Here's a simple end-to-end example that demonstrates the basic workflow:

```python
import numpy as np

from typing import Tuple

from zenml import step, pipeline

# Create steps for a simple ML workflow
@step
def get_data() -> Tuple[np.ndarray, np.ndarray]:
    # Generate some synthetic data
    X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
    y = np.array([0, 1, 0, 1])
    return X, y

@step
def process_data(data: Tuple[np.ndarray, np.ndarray]) -> Tuple[np.ndarray, np.ndarray]:
    X, y = data
    # Apply a simple transformation
    X_processed = X * 2
    return X_processed, y

@step
def train_and_evaluate(processed_data: Tuple[np.ndarray, np.ndarray]) -> float:
    X, y = processed_data
    # Simplistic "training" - just compute accuracy based on a rule
    predictions = [1 if sum(sample) > 10 else 0 for sample in X]
    accuracy = sum(p == actual for p, actual in zip(predictions, y)) / len(y)
    return accuracy

# Create a pipeline that combines these steps
@pipeline
def simple_example_pipeline():
    raw_data = get_data()
    processed_data = process_data(raw_data)
    accuracy = train_and_evaluate(processed_data)
    print(f"Model accuracy: {accuracy}")

# Run the pipeline
if __name__ == "__main__":
    simple_example_pipeline()
```

## Parameters and Artifacts

### Understanding the Difference

ZenML distinguishes between two types of inputs to steps:

1. **Artifacts**: Outputs from other steps in the same pipeline
   * These are tracked, versioned, and stored in the artifact store
   * They are passed between steps and represent data flowing through your pipeline
   * Examples: datasets, trained models, evaluation metrics
2. **Parameters**: Direct values provided when invoking a step
   * These are typically simple configuration values passed directly to the step
   * They're not tracked as separate artifacts but are recorded with the pipeline run
   * Examples: learning rates, batch sizes, model hyperparameters

This example demonstrates the difference:

```python
@pipeline
def my_pipeline():
    int_artifact = some_other_step()  # This is an artifact
    # input_1 is an artifact, input_2 is a parameter
    my_step(input_1=int_artifact, input_2=42)
```

### Parameter Types

Parameters can be:

1. **Primitive types**: `int`, `float`, `str`, `bool`
2. **Container types**: `list`, `dict`, `tuple` (containing primitives)
3. **Custom types**: As long as they can be serialized to JSON using Pydantic

Parameters that cannot be serialized to JSON should be passed as artifacts rather than parameters.

## Parameterizing Workflows

### Step Parameterization

Steps can take parameters like regular Python functions:

```python
@step
def train_model(data: dict, learning_rate: float = 0.01, epochs: int = 10) -> None:
    # Use learning_rate and epochs parameters
    print(f"Training with learning rate: {learning_rate} for {epochs} epochs")
```

### Pipeline Parameterization

Pipelines can also be parameterized, allowing values to be passed down to steps:

```python
@pipeline
def training_pipeline(dataset_name: str = "default_dataset", learning_rate: float = 0.01):
    data = load_data(dataset_name=dataset_name)
    train_model(data=data, learning_rate=learning_rate, epochs=20)
```

You can then run the pipeline with specific parameters:

```python
training_pipeline(dataset_name="custom_dataset", learning_rate=0.005)
```

## Step Type Handling & Output Management

### Type Annotations

While optional, type annotations are highly recommended and provide several benefits:

* **Artifact handling**: ZenML uses type annotations to determine how to serialize, store, and load [artifacts](/concepts/artifacts). The type information guides ZenML to select the appropriate [materializer](/concepts/artifacts/materializers) for saving and loading step outputs.
* **Type validation**: ZenML validates inputs against type annotations at runtime to catch errors early.
* **Code documentation**: Types make your code more self-documenting and easier to understand.

```python
from typing import Tuple

@step
def square_root(number: int) -> float:
    return number ** 0.5

@step
def divide(a: int, b: int) -> Tuple[int, int]:
    return a // b, a % b
```

When you specify a return type like `-> float` or `-> Tuple[int, int]`, ZenML uses this information to determine how to store the step's output in the artifact store. For instance, a step returning a pandas DataFrame with the annotation `-> pd.DataFrame` will use the pandas-specific materializer for efficient storage.

{% hint style="info" %}
If you want to enforce type annotations for all steps, set the environment variable `ZENML_ENFORCE_TYPE_ANNOTATIONS` to `True`.
{% endhint %}

### Multiple Return Values

Steps can return multiple artifacts:

```python
from typing import Tuple
from sklearn.base import ClassifierMixin
from typing import Annotated

@step
def train_classifier(X_train, y_train) -> Tuple[
    Annotated[ClassifierMixin, "model"],
    Annotated[float, "accuracy"]
]:
    model = SVC(gamma=0.001)
    model.fit(X_train, y_train)
    accuracy = model.score(X_train, y_train)
    return model, accuracy
```

ZenML uses the following convention to differentiate between a single output of type `Tuple` and multiple outputs:

* When the `return` statement is followed by a tuple literal (e.g., `return 1, 2` or `return (value_1, value_2)`), it's treated as a step with multiple outputs
* All other cases are treated as a step with a single output of type `Tuple`

## Conclusion

Steps and Pipelines provide a flexible, powerful way to build machine learning workflows in ZenML. This guide covered the basic concepts of creating steps and pipelines, managing inputs and outputs, and working with parameters.

For more advanced features, check out the [Advanced Features](/concepts/steps_and_pipelines/advanced_features) guide. For configuration using YAML files, see [Configuration with YAML](/concepts/steps_and_pipelines/yaml_configuration).

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Configuration

Configuring and customizing your pipeline runs.

ZenML provides several approaches to configure your pipelines and steps:

#### Understanding `.configure()` vs `.with_options()`

ZenML provides two primary methods to configure pipelines and steps: `.configure()` and `.with_options()`. While they accept the same parameters, they behave differently:

* **`.configure()`**: Modifies the configuration **in-place** and returns the same object.
* **`.with_options()`**: Creates a **new copy** with the applied configuration, leaving the original unchanged.

When to use each:

* Use `.with_options()` in most cases, especially inside pipeline definitions:

  ```python
  @pipeline
  def my_pipeline():
      # This creates a new configuration just for this instance
      my_step.with_options(parameters={"param": "value"})()
  ```
* Use `.configure()` only when you intentionally want to modify a step globally, and are aware that the change will affect all subsequent invocations of that step.

### Approaches to Configuration

#### Pipeline Configuration with `configure`

You can configure various aspects of a pipeline using the `configure` method:

```python
from zenml import pipeline

# Assuming MyPipeline is your pipeline function
# @pipeline
# def MyPipeline():
#     ...

# Create a pipeline
my_pipeline = MyPipeline()

# Configure the pipeline
my_pipeline.configure(
    enable_cache=False,
    enable_artifact_metadata=True,
    settings={
        "docker": {
            "parent_image": "zenml-io/zenml-cuda:latest"
        }
    }
)

# Run the pipeline
my_pipeline()
```

#### Runtime Configuration with `with_options`

You can configure a pipeline at runtime using the `with_options` method:

```python
# Configure specific step parameters
my_pipeline.with_options(steps={"trainer": {"parameters": {"learning_rate": 0.01}}})()

# Or using a YAML configuration file
my_pipeline.with_options(config_file="path_to_yaml_file")()
```

#### Step-Level Configuration

You can configure individual steps with the `@step` decorator:

```python
import tensorflow as tf
from zenml import step

@step(
    settings={
        # Custom materializer for handling output serialization
        "output_materializers": {
            "output": "zenml.materializers.tensorflow_materializer.TensorflowModelMaterializer"
        },
        # Step-specific experiment tracker settings
        "experiment_tracker.mlflow": {
            "experiment_name": "custom_experiment"
        }
    }
)
def train_model() -> tf.keras.Model:
    model = build_and_train_model()
    return model
```

#### Direct Component Assignment

If you have an experiment tracker or step operator in your active stack, you can enable them for specific steps like this:

```python
from zenml import step

@step(experiment_tracker=True, step_operator=True)
def train_model():
    # This step will use the experiment tracker and step operator of the active stack
    ...
```

If you want to make sure a step can only run with a specific experiment tracker/step operator, you can also specify the component names like this:

```python
from zenml import step

@step(experiment_tracker="mlflow_tracker", step_operator="vertex_ai")
def train_model():
    # This step will use MLflow for tracking and run on Vertex AI
    ...
```

You can combine both approaches with settings to configure the specific behavior of those components:

```python
from zenml import step

@step(step_operator=True, settings={"step_operator": {"estimator_args": {"instance_type": "m7g.medium"}}})
def my_step():
    # This step will use the step operator of the active stack with custom instance type
    ...

# Alternatively, using the step operator name and appropriate settings class:
@step(step_operator="nameofstepoperator", settings={"step_operator": SagemakerStepOperatorSettings(instance_type="m7g.medium")})
def my_step():
    # Same configuration using the settings class
    ...
```

This approach allows you to use different components for different steps in your pipeline while also customizing their runtime behavior.

{% hint style="info" %}
If your stack contains multiple step operators, `@step(step_operator=True)` uses the default step operator, which is the first attached step operator in the stack. If your stack contains multiple experiment trackers, `@step(experiment_tracker=True)` activates all attached experiment trackers for that step.
{% endhint %}

### Types of Settings

Settings in ZenML are categorized into three main types:

* **General settings** that can be used on all ZenML pipelines:
  * `DockerSettings` for container configuration
  * `ResourceSettings` for CPU, memory, and GPU allocation (on ZenML Pro, the same fields drive [resource pools](https://docs.zenml.io/pro/core-concepts/resource-pools) for workspace quotas, queuing, and preemption)
  * `DeploymentSettings` for pipeline deployment configuration - can only be set at the pipeline level
* **Stack-component-specific settings** for configuring behaviors of components in your stack:
  * These use the pattern `<COMPONENT_CATEGORY>`, `<COMPONENT_CATEGORY>.<COMPONENT_FLAVOR>`, or `<COMPONENT_CATEGORY>:<COMPONENT_NAME>` as keys
  * Use `<COMPONENT_CATEGORY>` to target the default attached component of that type
  * Use `.` to select by flavor: `<COMPONENT_CATEGORY>.<COMPONENT_FLAVOR>` only works if exactly one attached component of that flavor exists in the stack
  * Use `:` to select an exact named instance: `<COMPONENT_CATEGORY>:<COMPONENT_NAME>` targets a specific attached component by name
  * Examples include `experiment_tracker`, `experiment_tracker.wandb`, or `step_operator:vertex`

### Configuration Hierarchy

There are a few general rules when it comes to settings and configurations that are applied in multiple places. Generally the following is true:

* Configurations in code override configurations made inside of the yaml file
* Configurations at the step level override those made at the pipeline level
* In case of attributes the dictionaries are merged

```python
from zenml import pipeline, step
from zenml.config import ResourceSettings


@step
def load_data(parameter: int) -> dict:
    ...

@step(settings={"resources": ResourceSettings(gpu_count=1, memory="2GB")})
def train_model(data: dict) -> None:
    ...


@pipeline(settings={"resources": ResourceSettings(cpu_count=2, memory="1GB")}) 
def simple_ml_pipeline(parameter: int):
    ...
    
# ZenMl merges the two configurations and uses the step configuration to override 
# values defined on the pipeline level

train_model.configuration.settings["resources"]
# -> cpu_count: 2, gpu_count=1, memory="2GB"

simple_ml_pipeline.configuration.settings["resources"]
# -> cpu_count: 2, memory="1GB"
```

### Common Setting Types

#### Resource Settings

Resource settings allow you to specify the CPU, memory, and GPU requirements for your steps.

On **ZenML Pro**, those declarations are also what the **resource pool** feature uses: for eligible dynamic pipelines, the server builds resource requests from your merged `ResourceSettings` (including `pool_resources` and `preemptible`), matches them against workspace pools and policies on your stack’s orchestrator or step operator, and may queue or preempt work accordingly. For more information on how this feature works, see [ZenML Pro Resource Pools](https://docs.zenml.io/pro/core-concepts/resource-pools).

```python
from zenml.config import ResourceSettings

@step(settings={"resources": ResourceSettings(gpu_count=1, memory="2GB")})
def train_model(data: dict) -> None:
    ...

@pipeline(settings={"resources": ResourceSettings(cpu_count=2, memory="1GB")}) 
def simple_ml_pipeline(parameter: int):
    ...
```

When both pipeline and step resource settings are specified, they are merged with step settings taking precedence:

```python
# Result of merging the above configurations:
# train_model.configuration.settings["resources"]
# -> cpu_count: 2, gpu_count=1, memory="2GB"
```

{% hint style="info" %}
Note that `ResourceSettings` are not always applied by all orchestrators. The ability to enforce resource constraints depends on the specific orchestrator being used. Some orchestrators like Kubernetes fully support these settings, while others may ignore them. In order to learn more, read the [individual pages](https://docs.zenml.io/stacks/stack-components/orchestrators) of the orchestrator you are using.
{% endhint %}

Resource settings also allow you to configure scaling options - including minimum and maximum number of instances, and scaling policy - for your pipeline deployments, when used at the pipeline level:

```python
from zenml.config import ResourceSettings

@pipeline(settings={"resources": ResourceSettings(
  cpu_count=2,
  memory="4GB",
  min_replicas=0,
  max_replicas=10,
  max_concurrency=10
)}) 
def simple_llm_pipeline(parameter: int):
    ...
```

{% hint style="info" %}
Note that `ResourceSettings` are not always applied exactly as specified by all deployers. Some deployers fully support these settings, while others may adjust them automatically to match a set of predefined static values or simply ignore them. In order to learn more, read the [individual pages](https://docs.zenml.io/stacks/stack-components/deployers) of the deployer you are using.
{% endhint %}

#### Docker Settings

Docker settings allow you to customize the containerization process:

```python
@pipeline(settings={
    "docker": {
        "parent_image": "zenml-io/zenml-cuda:latest"
    }
})
def my_pipeline():
    ...
```

For more detailed information on containerization options, see the [containerization guide](/concepts/containerization).

#### Deployment Settings

Deployment settings allow you to customize the web server and ASGI application used to run your pipeline deployments. You can specify a range of options, including custom endpoints, middleware, extensions and even custom files used to serve an entire single-page application alongside your pipeline:

```python
from typing import Dict, Any
import psutil
from zenml.config import DeploymentSettings, EndpointSpec, EndpointMethod, SecureHeadersConfig
from zenml import pipeline

async def health_detailed() -> Dict[str, Any]:
    return {
        "status": "healthy",
        "cpu_percent": psutil.cpu_percent(),
        "memory_percent": psutil.virtual_memory().percent,
        "disk_percent": psutil.disk_usage("/").percent,
    }

@pipeline(settings={
    "deployment": DeploymentSettings(
      custom_endpoints=[
          EndpointSpec(
              path="/health",
              method=EndpointMethod.GET,
              handler=health_detailed,
              auth_required=False,
          ),
      ],
      secure_headers=SecureHeadersConfig(
        csp=(
            "default-src 'none'; "
            "script-src 'self' 'unsafe-inline' https://cdn.jsdelivr.net; "
            "connect-src 'self' https://cdn.jsdelivr.net; "
            "style-src 'self' 'unsafe-inline'"
        ),
      ),
      dashboard_files_path="my/custom/ui",
})
def my_pipeline():
    ...
```

For more detailed information on deployment options, see the [pipeline deployment guide](/concepts/deployment), particularly the [deployment settings](/concepts/deployment/deployment_settings) section.

### Stack Component Configuration

#### Registration-time vs Runtime Stack Component Settings

Stack components have two types of configuration:

1. **Registration-time configuration**: Static settings defined when registering a component

   ```bash
   # Example: Setting a fixed tracking URL for MLflow
   zenml experiment-tracker register mlflow_tracker --flavor=mlflow --tracking_url=http://localhost:5000
   ```
2. **Runtime settings**: Dynamic settings that can change between pipeline runs

   ```python
   # Example: Setting experiment name that changes for each run
   @step(settings={"experiment_tracker.mlflow": {"experiment_name": "custom_experiment"}})
   def my_step():
       ...
   ```

Even for runtime settings, you can set default values during registration:

```bash
# Setting a default value for "nested" setting
zenml experiment-tracker register <n> --flavor=mlflow --nested=True
```

#### Using the Right Key for Stack Component Settings

When specifying stack-component-specific settings, the key follows this pattern:

```python
# Using just the component category
@step(settings={"step_operator": {"estimator_args": {"instance_type": "m7g.medium"}}})

# Or using the component category and flavor
@step(settings={"experiment_tracker.mlflow": {"experiment_name": "custom_experiment"}})
```

If you specify just the category (e.g., `step_operator`), ZenML applies these settings to whatever flavor of component is in your stack. If the settings don't apply to that flavor, they are ignored.

### Making Configurations Flexible with Environment Variables

You can make your configurations more flexible by referencing environment variables using the placeholder syntax `${ENV_VARIABLE_NAME}`:

**In code:**

```python
from zenml import step

@step(extra={"value_from_environment": "${ENV_VAR}"})
def my_step() -> None:
    ...
```

**In configuration files:**

```yaml
extra:
  value_from_environment: ${ENV_VAR}
  combined_value: prefix_${ENV_VAR}_suffix
```

This allows you to easily adapt your pipelines to different environments without changing code.

### Using External Configuration Tools (Hydra)

For more advanced configuration management, you can use tools like [Hydra](https://hydra.cc/) to manage your pipeline parameters through composable YAML configs and CLI overrides, then pass the resolved values into ZenML.

The key idea: **Hydra decides&#x20;*****what*****&#x20;to run** (hyperparameters, model settings, data config) while **ZenML decides&#x20;*****where*****&#x20;and&#x20;*****when*** (orchestration, caching, artifact versioning). Neither needs to know about the other's internals.

For a complete working example, see the [Hydra + ZenML configuration management example](https://github.com/zenml-io/zenml/tree/main/examples/hydra_config_management/README.md).

### Autogenerate a template yaml file

If you want to generate a template yaml file of your specific pipeline, you can do so by using the `.write_run_configuration_template()` method. This will generate a yaml file with all options commented out. This way you can pick and choose the settings that are relevant to you.

```python
from zenml import pipeline
...

@pipeline(enable_cache=True) # set cache behavior at step level
def simple_ml_pipeline(parameter: int):
    dataset = load_data(parameter=parameter)
    train_model(dataset)

simple_ml_pipeline.write_run_configuration_template(path="<Insert_path_here>")
```

<details>

<summary>An example of a generated YAML configuration template</summary>

```yaml
build: Union[PipelineBuildBase, UUID, NoneType]
enable_artifact_metadata: Optional[bool]
enable_artifact_visualization: Optional[bool]
enable_cache: Optional[bool]
enable_step_logs: Optional[bool]
extra: Mapping[str, Any]
model:
  audience: Optional[str]
  description: Optional[str]
  ethics: Optional[str]
  license: Optional[str]
  limitations: Optional[str]
  name: str
  save_models_to_registry: bool
  suppress_class_validation_warnings: bool
  tags: Optional[List[str]]
  trade_offs: Optional[str]
  use_cases: Optional[str]
  version: Union[ModelStages, int, str, NoneType]
parameters: Optional[Mapping[str, Any]]
run_name: Optional[str]
schedule:
  catchup: bool
  cron_expression: Optional[str]
  end_time: Optional[datetime]
  interval_second: Optional[timedelta]
  name: Optional[str]
  run_once_start_time: Optional[datetime]
  start_time: Optional[datetime]
settings:
  docker:
    apt_packages: List[str]
    build_context_root: Optional[str]
    build_options: Mapping[str, Any]
    copy_files: bool
    copy_global_config: bool
    dockerfile: Optional[str]
    dockerignore: Optional[str]
    environment: Mapping[str, Any]
    runtime_environment: Mapping[str, Any]
    install_stack_requirements: bool
    parent_image: Optional[str]
    python_package_installer: PythonPackageInstaller
    replicate_local_python_environment: Union[List[str], PythonEnvironmentExportMethod,
     NoneType]
    required_integrations: List[str]
    requirements: Union[NoneType, str, List[str]]
    skip_build: bool
    prevent_build_reuse: bool
    allow_including_files_in_images: bool
    allow_download_from_code_repository: bool
    allow_download_from_artifact_store: bool
    target_repository: str
    user: Optional[str]
  resources:
    cpu_count: Optional[PositiveFloat]
    gpu_count: Optional[NonNegativeInt]
    memory: Optional[ConstrainedStrValue]
   deployment:
     api_url_path: str
     app_description: Union[str, NoneType]
     app_extensions: Union[List[AppExtensionSpec], NoneType]
     app_kwargs: Dict[str, Any]
     app_title: Union[str, NoneType]
     app_version: Union[str, NoneType]
     cors:
       allow_credentials: bool
       allow_headers: List[str]
       allow_methods: List[str]
       allow_origins: List[str]
     custom_endpoints: Union[List[EndpointSpec], NoneType]
     custom_middlewares: Union[List[MiddlewareSpec], NoneType]
     dashboard_files_path: Union[str, NoneType]
     deployment_app_runner_flavor: Union[Annotated[SourceOrObject, BeforeValidator,
       PlainSerializer], NoneType]
     deployment_app_runner_kwargs: Dict[str, Any]
     deployment_service_class: Union[Annotated[SourceOrObject, BeforeValidator, PlainSerializer],
       NoneType]
     deployment_service_kwargs: Dict[str, Any]
     docs_url_path: str
     health_url_path: str
     include_default_endpoints: bool
     include_default_middleware: bool
     info_url_path: str
     invoke_url_path: str
     log_level: LoggingLevels
     metrics_url_path: str
     redoc_url_path: str
     root_url_path: str
     secure_headers:
       cache: Union[bool, str]
       content: Union[bool, str]
       csp: Union[bool, str]
       hsts: Union[bool, str]
       permissions: Union[bool, str]
       referrer: Union[bool, str]
       server: Union[bool, str]
       xfo: Union[bool, str]
     shutdown_hook: Union[Annotated[SourceOrObject, BeforeValidator, PlainSerializer],
       NoneType]
     shutdown_hook_kwargs: Dict[str, Any]
     startup_hook: Union[Annotated[SourceOrObject, BeforeValidator, PlainSerializer],
       NoneType]
     startup_hook_kwargs: Dict[str, Any]
     thread_pool_size: int
     uvicorn_host: str
     uvicorn_kwargs: Dict[str, Any]
     uvicorn_port: int
     uvicorn_workers: int
steps:
  load_data:
    enable_artifact_metadata: Optional[bool]
    enable_artifact_visualization: Optional[bool]
    enable_cache: Optional[bool]
    enable_step_logs: Optional[bool]
    experiment_tracker: Optional[str]
    extra: Mapping[str, Any]
    failure_hook_source:
      attribute: Optional[str]
      module: str
      type: SourceType
    model:
      audience: Optional[str]
      description: Optional[str]
      ethics: Optional[str]
      license: Optional[str]
      limitations: Optional[str]
      name: str
      save_models_to_registry: bool
      suppress_class_validation_warnings: bool
      tags: Optional[List[str]]
      trade_offs: Optional[str]
      use_cases: Optional[str]
      version: Union[ModelStages, int, str, NoneType]
    name: Optional[str]
    outputs:
      output:
        default_materializer_source:
          attribute: Optional[str]
          module: str
          type: SourceType
      materializer_source: Optional[Tuple[Source, ...]]
    parameters: {}
    settings:
      docker:
        apt_packages: List[str]
        build_context_root: Optional[str]
        build_options: Mapping[str, Any]
        copy_files: bool
        copy_global_config: bool
        dockerfile: Optional[str]
        dockerignore: Optional[str]
        environment: Mapping[str, Any]
        runtime_environment: Mapping[str, Any]
        install_stack_requirements: bool
        parent_image: Optional[str]
        python_package_installer: PythonPackageInstaller
        replicate_local_python_environment: Union[List[str], PythonEnvironmentExportMethod,
         NoneType]
        required_integrations: List[str]
        requirements: Union[NoneType, str, List[str]]
        skip_build: bool
        prevent_build_reuse: bool
        allow_including_files_in_images: bool
        allow_download_from_code_repository: bool
        allow_download_from_artifact_store: bool
        target_repository: str
        user: Optional[str]
      resources:
        cpu_count: Optional[PositiveFloat]
        gpu_count: Optional[NonNegativeInt]
        memory: Optional[ConstrainedStrValue]
    step_operator: Optional[str]
    success_hook_source:
      attribute: Optional[str]
      module: str
      type: SourceType
  train_model:
    enable_artifact_metadata: Optional[bool]
    enable_artifact_visualization: Optional[bool]
    enable_cache: Optional[bool]
    enable_step_logs: Optional[bool]
    experiment_tracker: Optional[str]
    extra: Mapping[str, Any]
    failure_hook_source:
      attribute: Optional[str]
      module: str
      type: SourceType
    model:
      audience: Optional[str]
      description: Optional[str]
      ethics: Optional[str]
      license: Optional[str]
      limitations: Optional[str]
      name: str
      save_models_to_registry: bool
      suppress_class_validation_warnings: bool
      tags: Optional[List[str]]
      trade_offs: Optional[str]
      use_cases: Optional[str]
      version: Union[ModelStages, int, str, NoneType]
    name: Optional[str]
    outputs: {}
    parameters: {}
    settings:
      docker:
        apt_packages: List[str]
        build_context_root: Optional[str]
        build_options: Mapping[str, Any]
        copy_files: bool
        copy_global_config: bool
        dockerfile: Optional[str]
        dockerignore: Optional[str]
        environment: Mapping[str, Any]
        runtime_environment: Mapping[str, Any]
        install_stack_requirements: bool
        parent_image: Optional[str]
        python_package_installer: PythonPackageInstaller
        replicate_local_python_environment: Union[List[str], PythonEnvironmentExportMethod,
         NoneType]
        required_integrations: List[str]
        requirements: Union[NoneType, str, List[str]]
        skip_build: bool
        prevent_build_reuse: bool
        allow_including_files_in_images: bool
        allow_download_from_code_repository: bool
        allow_download_from_artifact_store: bool
        target_repository: str
        user: Optional[str]
      resources:
        cpu_count: Optional[PositiveFloat]
        gpu_count: Optional[NonNegativeInt]
        memory: Optional[ConstrainedStrValue]
    step_operator: Optional[str]
    success_hook_source:
      attribute: Optional[str]
      module: str
      type: SourceType

```

</details>

{% hint style="info" %}
When you want to configure your pipeline with a certain stack in mind, you can do so as well: `...write_run_configuration_template(stack=<Insert_stack_here>)`
{% endhint %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Scheduling

Learn how to create, update, activate, deactivate, and delete schedules for pipelines.

{% hint style="info" %}
Schedules don't work for all orchestrators. Here is a list of all supported orchestrators.
{% endhint %}

| Orchestrator                                                                         | Scheduling Support | Supported Schedule Types | Native Schedule Management |
| ------------------------------------------------------------------------------------ | ------------------ | ------------------------ | -------------------------- |
| [AirflowOrchestrator](https://docs.zenml.io/stacks/orchestrators/airflow)            | ✅                  | Cron, Interval           | ⛔️                         |
| [AzureMLOrchestrator](https://docs.zenml.io/stacks/orchestrators/azureml)            | ✅                  | Cron, Interval           | ⛔️                         |
| [DatabricksOrchestrator](https://docs.zenml.io/stacks/orchestrators/databricks)      | ✅                  | Cron only                | ⛔️                         |
| [HyperAIOrchestrator](https://docs.zenml.io/stacks/orchestrators/hyperai)            | ✅                  | Cron, One-time           | ⛔️                         |
| [KubeflowOrchestrator](https://docs.zenml.io/stacks/orchestrators/kubeflow)          | ✅                  | Cron, Interval           | ⛔️                         |
| [KubernetesOrchestrator](https://docs.zenml.io/stacks/orchestrators/kubernetes)      | ✅                  | Cron only                | ✅                          |
| [LocalOrchestrator](https://docs.zenml.io/stacks/orchestrators/local)                | ⛔️                 | N/A                      | N/A                        |
| [LocalDockerOrchestrator](https://docs.zenml.io/stacks/orchestrators/local-docker)   | ⛔️                 | N/A                      | N/A                        |
| [SagemakerOrchestrator](https://docs.zenml.io/stacks/orchestrators/sagemaker)        | ✅                  | Cron, Interval, One-time | ⛔️                         |
| [SkypilotAWSOrchestrator](https://docs.zenml.io/stacks/orchestrators/skypilot-vm)    | ⛔️                 | N/A                      | N/A                        |
| [SkypilotAzureOrchestrator](https://docs.zenml.io/stacks/orchestrators/skypilot-vm)  | ⛔️                 | N/A                      | N/A                        |
| [SkypilotGCPOrchestrator](https://docs.zenml.io/stacks/orchestrators/skypilot-vm)    | ⛔️                 | N/A                      | N/A                        |
| [SkypilotLambdaOrchestrator](https://docs.zenml.io/stacks/orchestrators/skypilot-vm) | ⛔️                 | N/A                      | N/A                        |
| [TektonOrchestrator](https://docs.zenml.io/stacks/orchestrators/tekton)              | ⛔️                 | N/A                      | N/A                        |
| [VertexOrchestrator](https://docs.zenml.io/stacks/orchestrators/vertex)              | ✅                  | Cron only                | ⛔️                         |

{% hint style="info" %}
**Native Schedule Management** means the orchestrator supports updating and deleting schedules directly through ZenML commands. When supported, commands like `zenml pipeline schedule update` and `zenml pipeline schedule delete` will automatically update/delete the schedule on the orchestrator platform (e.g., Kubernetes CronJobs). For orchestrators without this support, you'll need to manually manage schedules on the orchestrator side.
{% endhint %}

Check out [our tutorial on scheduling](https://docs.zenml.io/user-guides/tutorial/managing-scheduled-pipelines) for a practical guide on how to schedule a pipeline.

### Set a schedule

```python
from zenml.config.schedule import Schedule
from zenml import pipeline
from datetime import datetime

@pipeline()
def my_pipeline(...):
    ...

# Use cron expressions
schedule = Schedule(cron_expression="5 14 * * 3")
# or alternatively use human-readable notations
schedule = Schedule(start_time=datetime.now(), interval_second=1800)

my_pipeline = my_pipeline.with_options(schedule=schedule)
my_pipeline()
```

{% hint style="info" %}
Check out our [SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-config.html#zenml.config.schedule) to learn more about the different scheduling options.
{% endhint %}

### Update a schedule

You can update your schedule's cron expression:

```bash
zenml pipeline schedule update <SCHEDULE_NAME_OR_ID> --cron-expression='* * * * *'
```

### Activate and deactivate a schedule

You can temporarily pause a schedule without deleting it using the deactivate command, and resume it later with activate:

```bash
# Pause a schedule (stops future executions)
zenml pipeline schedule deactivate <SCHEDULE_NAME_OR_ID>

# Resume a paused schedule
zenml pipeline schedule activate <SCHEDULE_NAME_OR_ID>
```

{% hint style="info" %}
For the Kubernetes orchestrator, activate/deactivate controls the CronJob's `suspend` field - this is a native Kubernetes feature that pauses schedule execution without removing the CronJob resource.
{% endhint %}

### Delete a schedule

Deleting a schedule archives it by default (soft delete), which preserves references in historical pipeline runs that were triggered by this schedule:

```bash
# Archive a schedule (soft delete - default behavior)
zenml pipeline schedule delete <SCHEDULE_NAME_OR_ID>

# Permanently delete a schedule and remove all references (hard delete)
zenml pipeline schedule delete <SCHEDULE_NAME_OR_ID> --hard
```

{% hint style="warning" %}
Using `--hard` permanently removes the schedule and any historical references to it. Pipeline runs that were triggered by this schedule will no longer show the schedule association.
{% endhint %}

#### Post-Deletion Lifecycle

When an object is soft-deleted, it remains in the database but is no longer usable.

By default, list operations only return active (non-archived) schedules. To view archived schedules, use the following options:

Show archived schedules on the CLI:

```bash
zenml pipeline schedule list --is_archived=true
```

Show archived schedules on the SDK:

```python
from zenml.client import Client

archived_schedules = Client().list_schedules(is_archived=True)
```

#### Archived Object Naming

After archival, you may notice that the schedule name has changed. In ZenML, schedule names act as unique identifiers. To prevent naming conflicts and allow reuse of the original name, archived schedules are automatically renamed by appending a random hash to the original name.

#### Permanent Deletion (Hard Delete)

Archived objects can be permanently deleted using one of the following identifiers:

* ID
* ID prefix
* Name (updated/archived name only)

{% hint style="warning" %}
Deletion using the original (pre-archival) name will not work, since the object has been renamed. Deletion by name prefix is not supported for schedules (and most other entities), as it is considered unsafe.
{% endhint %}

### Kubernetes CronJob advanced configuration

When using the Kubernetes orchestrator, scheduled pipelines are backed by Kubernetes CronJobs. You can customize CronJob-specific behavior through `KubernetesOrchestratorSettings`:

* **`concurrency_policy`**: Controls whether concurrent job executions are allowed (`Allow`, `Forbid`, `Replace`)
* **`starting_deadline_seconds`**: If a scheduled run misses its trigger time, it can still start within this window (in seconds)

These settings are in addition to Job-level settings like `active_deadline_seconds` (runtime timeout), `ttl_seconds_after_finished` (cleanup delay), and history limits. See the [Kubernetes orchestrator docs](https://github.com/zenml-io/zenml/tree/main/docs/book/component-guide/orchestrators/kubernetes.md) for the full list and examples.

### Orchestrator support for schedule management

The functionality of these commands changes depending on whether the orchestrator supports schedule updates/deletions (see the "Native Schedule Management" column in the table above):

* **Kubernetes orchestrator**: Fully supports native schedule management. Update and delete commands will modify/remove the actual CronJob on the cluster as well as the schedule information in ZenML.
* **Other schedulable orchestrators**: Only update/delete the schedule information stored in ZenML. The actual schedule on the orchestrator remains unchanged.

If the orchestrator **does not** support native schedule management, maintaining the lifecycle of the schedule on the orchestrator side is the responsibility of the user. In these cases, we recommend the following steps:

1. Find schedule on ZenML
2. Match schedule on orchestrator side and delete
3. Delete schedule on ZenML
4. Re-run pipeline with new schedule

A concrete example can be found on the [GCP Vertex orchestrator](https://docs.zenml.io/stacks/orchestrators/vertex) docs, and this pattern can be adapted for other orchestrators as well.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Logging

Learn how to control and customize logging behavior in ZenML pipelines.

By default, ZenML uses a logging handler to capture two types of logs:

* **Pipeline run logs**: Logs collected from your ZenML client while triggering and waiting for a pipeline to run. These logs cover everything that happens client-side: building and pushing container images, triggering the pipeline, waiting for it to start, and waiting for it to finish. These logs are now stored in the artifact store, making them accessible even after the client session ends.
* **Step logs**: Logs collected from the execution of individual steps. These logs only cover what happens during the execution of a single step and originate mostly from the user-provided step code and the libraries it calls.

For step logs, users are free to use the default python logging module or print statements, and ZenML's logging handler will catch these logs and store them.

```python
import logging

from zenml import step

@step 
def my_step() -> None:
    logging.warning("`Hello`")  # You can use the regular `logging` module.
    print("World.")  # You can utilize `print` statements as well. 
```

All these logs are stored within the respective artifact store of your stack. You can visualize the pipeline run logs and step logs in the dashboard as follows:

* Local ZenML server (`zenml login --local`): Both local and remote artifact stores may be accessible
* Deployed ZenML server: Local artifact store logs won't be accessible; remote artifact store logs require [service connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/service-connectors-guide) configuration (see [remote storage guide](https://docs.zenml.io/user-guides/production-guide/remote-storage))

{% hint style="warning" %}
In order for logs to be visible in the dashboard with a deployed ZenML server, you must configure both a remote artifact store and the appropriate service connector to access it. Without this configuration, your logs won't be accessible through the dashboard.
{% endhint %}

![Displaying pipeline run logs on the dashboard](/files/9i3YH5DaDzVWJMGr1bJN) ![Displaying step logs on the dashboard](/files/fonmvqh13jBTgLQqmMq1)

## Logging Configuration

### Environment Variables and Remote Execution

For all logging configurations below, note:

* Setting environment variables on your local machine only affects local pipeline runs
* For remote pipeline runs, you must set these variables in the pipeline's execution environment using Docker settings:

```python
from zenml import pipeline
from zenml.config import DockerSettings

docker_settings = DockerSettings(environment={"ENVIRONMENT_VARIABLE": "value"})

# Either add it to the decorator
@pipeline(settings={"docker": docker_settings})
def my_pipeline() -> None:
    my_step()

# Or configure the pipelines options
my_pipeline = my_pipeline.with_options(
    settings={"docker": docker_settings}
)
```

### Enabling or Disabling Logs Storage

You can control log storage for both pipeline runs and steps:

#### Step Logs

To disable storing step logs in your artifact store:

1. Using the `enable_step_logs` parameter with step decorator:

   ```python
   from zenml import step

   @step(enable_step_logs=False)  # disables logging for this step
   def my_step() -> None:
       ...
   ```
2. Setting the `ZENML_DISABLE_STEP_LOGS_STORAGE=true` environment variable in the execution environment:

   ```python
   from zenml import pipeline
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(environment={"ZENML_DISABLE_STEP_LOGS_STORAGE": "true"})

   # Either add it to the decorator
   @pipeline(settings={"docker": docker_settings})
   def my_pipeline() -> None:
       my_step()

   # Or configure the pipelines options
   my_pipeline = my_pipeline.with_options(
       settings={"docker": docker_settings}
   )
   ```

   This environment variable takes precedence over the parameter mentioned above.

#### Pipeline Run Logs

To disable storing client-side pipeline run logs in your artifact store:

1. Using the `enable_pipeline_logs` parameter with pipeline decorator:

   ```python
   from zenml import pipeline

   @pipeline(enable_pipeline_logs=False)  # disables client-side logging for this pipeline
   def my_pipeline():
       ...
   ```
2. Using the runtime configuration:

   ```python
   # Disable pipeline logs at runtime
   my_pipeline.with_options(enable_pipeline_logs=False)
   ```
3. Setting the `ZENML_DISABLE_PIPELINE_LOGS_STORAGE=true` environment variable:

   ```python
   from zenml import pipeline
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(environment={"ZENML_DISABLE_PIPELINE_LOGS_STORAGE": "true"})

   # Either add it to the decorator
   @pipeline(settings={"docker": docker_settings})
   def my_pipeline() -> None:
       my_step()

   # Or configure the pipelines options
   my_pipeline = my_pipeline.with_options(
       settings={"docker": docker_settings}
   )
   ```

   The environment variable takes precedence over parameters set in the decorator or runtime configuration.

### Setting Logging Verbosity

Change the default logging level (`INFO`) with:

```bash
export ZENML_LOGGING_VERBOSITY=INFO
```

Options: `INFO`, `WARN`, `ERROR`, `CRITICAL`, `DEBUG`

For remote pipeline runs:

```python
from zenml import pipeline
from zenml.config import DockerSettings

docker_settings = DockerSettings(environment={"ZENML_LOGGING_VERBOSITY": "DEBUG"})

# Either add it to the decorator
@pipeline(settings={"docker": docker_settings})
def my_pipeline() -> None:
    my_step()

# Or configure the pipelines options
my_pipeline = my_pipeline.with_options(
    settings={"docker": docker_settings}
)
```

### Setting Console Logging Format

Change the console/stdout logging format with:

```bash
export ZENML_CONSOLE_LOGGING_FORMAT=console
```

Options:

* `console` (default): Human-readable console output. Client-side `INFO` logs use a compact layout, while `DEBUG` logs and server logs use a full structured text layout.
* `json`: JSON formatted console/stdout logs.
* Any other valid Python `%`-style logging format string, such as `%(asctime)s - %(message)s`, for custom console output.

```bash
export ZENML_CONSOLE_LOGGING_FORMAT='%(asctime)s %(message)s'
```

The format must use `%`-string formatting style. See the [available LogRecord attributes](https://docs.python.org/3/library/logging.html#logrecord-attributes). This only changes terminal output; stored logs keep their raw message and structured metadata.

{% hint style="warning" %}
The older `ZENML_LOGGING_FORMAT` environment variable is deprecated and will be removed in a future version. Use `ZENML_CONSOLE_LOGGING_FORMAT` instead. Existing configurations such as `ZENML_LOGGING_FORMAT='%(asctime)s %(message)s'` continue to work during the deprecation period.
{% endhint %}

The compact client console layout is:

```
<message> | <extras as JSON, if any>
[traceback and stack_info if any]
```

The full structured console layout for `DEBUG` logs is:

```
<time> | <loglevel> | <logger-name>:<function-name>:<line-number> | <message> | <extras as JSON, if any>
[traceback and stack_info if any]
```

The JSON format emits the same information as fields:

```json
{
  "timestamp": "2026-05-21 13:09:55,515",
  "level": "INFO",
  "logger": "__main__",
  "function": "loader",
  "line": 15,
  "message": "Training started",
  "dataset": "mnist",
  "epochs": 10,
  "step": "loader"
}
```

When an exception is logged, the JSON output includes an `exception` object with the exception type, message, and stack trace.

### Adding Structured Fields

ZenML uses Python's standard `logging` module. If you want to attach structured fields to a log record, use the standard `extra` argument:

```python
import logging

logger = logging.getLogger(__name__)

logger.info(
    "training.started",
    extra={"dataset": "mnist", "epochs": 10},
)
```

Avoid using Python `LogRecord` attribute names such as `name`, `message`, `levelname`, `filename`, or `lineno` as keys in `extra` because they are reserved for Python's internal use.

{% hint style="warning" %}
When a custom console format string is configured, ZenML uses that format as-is for console output. It does not append structured `extra` fields or apply ZenML's default console coloring and highlighting to that custom layout.
{% endhint %}

### Disabling Rich Traceback Output

ZenML uses [rich](https://rich.readthedocs.io/en/stable/traceback.html) for enhanced traceback display. Disable it with:

```bash
export ZENML_ENABLE_RICH_TRACEBACK=false
```

### Disabling Colorful Logging

Console logs use colors by default. Disable colorful logging with:

```bash
ZENML_LOGGING_COLORS_DISABLED=true
```

### Showing Step Names in Logs

ZenML hides step name prefixes in console logs by default. You can show them with:

```bash
ZENML_DISABLE_STEP_NAMES_IN_LOGS=false
```

When enabled, console logs include the step name as a prefix:

```
[data_loader] Loading data from source...
[data_loader] Data loaded successfully.
[model_trainer] Training model with parameters...
```

These prefixes only appear in console output, not in stored logs. You can disable the step name prefixes in console with:

```bash
ZENML_DISABLE_STEP_NAMES_IN_LOGS=true
```

## Limitations

### on Steps and pipelines

When running steps and pipelines, ZenML only captures logs emitted from the thread that executes the corresponding function. If your step code spawns additional threads or runs async code, logs from those execution contexts may not be captured.

For instance, only the log emitted directly in the step function is captured:

```python
import logging
import threading

from zenml import step

logger = logging.getLogger(__name__)


@step
def async_step() -> None:
    def _process() -> None:
        logger.info("This log is NOT captured")

    logger.info("This log is captured")
    thread = threading.Thread(target=_process)
    thread.start()
    thread.join()
```

As a workaround, you can run it under the copied `contextvars` context so ZenML can associate the log records with the running step:

```python
import contextvars
import logging
import threading

from zenml import step

logger = logging.getLogger(__name__)


@step
def async_step() -> None:
    def _process() -> None:
        logger.info("This log is now captured")

    ctx = contextvars.copy_context()
    thread = threading.Thread(target=lambda: ctx.run(_process))
    thread.start()
    thread.join()
```

### on the Dashboard

When viewing logs in the dashboard, ZenML currently loads logs **in bulk** and pagination/filtering happens on the client side. To keep the response size and server memory usage bounded (especially when logs are stored in remote artifact stores), the dashboard is limited to **500 pages** (**100 log entries per page**, i.e. **50,000 entries** total) by default.

You can adjust this limit by setting `ZENML_LOGS_MAX_ENTRIES_PER_REQUEST` in the environment when you are deploying your ZenML workspace.

Downloading logs from the dashboard will also only include up to this limit.

{% hint style="info" %}
We’re actively working on improving log loading to remove the need for this cap. We'll update the documentation as this evolves with future releases.
{% endhint %}

## Best Practices for Logging

1. **Use appropriate log levels**:
   * `DEBUG`: Detailed diagnostic information
   * `INFO`: Confirmation that things work as expected
   * `WARNING`: Something unexpected happened
   * `ERROR`: A more serious problem occurred
   * `CRITICAL`: A serious error that may prevent continued execution
2. **Include contextual information** in logs
3. **Log at decision points** to track execution flow
4. **Avoid logging sensitive information**
5. **Use structured logging** when appropriate
6. **Configure appropriate verbosity** for different environments

## See Also

* [Steps & Pipelines](/concepts/steps_and_pipelines)
* [YAML Configuration](/concepts/steps_and_pipelines/yaml_configuration)
* [Advanced Features](/concepts/steps_and_pipelines/advanced_features)

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# YAML Configuration

Learn how to configure ZenML pipelines using YAML configuration files.

ZenML provides configuration capabilities through YAML files that allow you to customize pipeline and step behavior without changing your code. This is particularly useful for separating configuration from code, experimenting with different parameters, and ensuring reproducibility.

## Basic Usage

You can apply a YAML configuration file when running a pipeline:

```python
my_pipeline.with_options(config_path="config.yaml")()
```

This allows you to change pipeline behavior without modifying your code.

### Sample Configuration File

Here's a simple example of a YAML configuration file:

```yaml
# Enable/disable features
enable_cache: False
enable_step_logs: True

# Pipeline parameters
parameters: 
  dataset_name: "my_dataset"
  learning_rate: 0.01

# Step-specific configuration
steps:
  train_model:
    parameters:
      learning_rate: 0.001  # Override the pipeline parameter for this step
    enable_cache: True      # Override the pipeline cache setting
```

### Configuration Hierarchy

ZenML follows a specific hierarchy when resolving configuration:

1. **Runtime Python code** - Highest precedence
2. **Step-level YAML configuration**

   ```yaml
   steps:
     train_model:
       parameters:
         learning_rate: 0.001  # Overrides pipeline-level setting
   ```
3. **Pipeline-level YAML configuration**

   ```yaml
   parameters:
     learning_rate: 0.01  # Lower precedence than step-level
   ```
4. **Default values in code** - Lowest precedence

This hierarchy allows you to define base configurations at the pipeline level and override them for specific steps as needed.

## Configuring Steps and Pipelines

### Pipeline and Step Parameters

You can specify parameters for pipelines and steps, similar to how you'd define them in Python code:

```yaml
# Pipeline parameters
parameters:
  dataset_name: "my_dataset"
  learning_rate: 0.01
  batch_size: 32
  epochs: 10

# Step parameters
steps:
  preprocessing:
    parameters:
      normalize: True
      fill_missing: "mean"
  
  train_model:
    parameters:
      learning_rate: 0.001  # Override the pipeline parameter
      optimizer: "adam"
```

These settings correspond directly to the parameters you'd normally pass to your pipeline and step functions.

### Enable Flags

These boolean flags control aspects of pipeline execution that were covered in the Advanced Features section:

```yaml
# Pipeline-level flags
enable_artifact_metadata: True      # Whether to collect and store metadata for artifacts
enable_artifact_visualization: True  # Whether to generate visualizations for artifacts
enable_cache: True                  # Whether to use caching for steps
enable_step_logs: True              # Whether to capture and store step logs

# Step-specific flags
steps:
  preprocessing:
    enable_cache: False             # Disable caching for this step only
  train_model:
    enable_artifact_visualization: False  # Disable visualizations for this step
```

### Run Name

Set a custom name for the pipeline run:

```yaml
run_name: "training_run_cifar10_resnet50_lr0.001"
```

{% hint style="warning" %}
**Important:** Pipeline run names must be unique within a project. If you try to run a pipeline with a name that already exists, you'll get an error. To avoid this:

1. **Use dynamic placeholders** to ensure uniqueness:

   ```yaml
   # Example 1: Use placeholders for date and time to ensure uniqueness
   run_name: "training_run_{date}_{time}"

   # Example 2: Combine placeholders with specific details for better context
   run_name: "training_run_cifar10_resnet50_lr0.001_{date}_{time}"
   ```
2. **Remove the 'run\_name' from your config** to let ZenML auto-generate unique names
3. **Change the run\_name** before rerunning the pipeline

Available placeholders: `{date}`, `{time}`, and any parameters defined in your pipeline configuration.
{% endhint %}

## Resource and Component Configuration

### Docker Settings

Configure Docker container settings for pipeline execution:

```yaml
settings:
  docker:
    # Packages to install via apt-get
    apt_packages: ["curl", "git", "libgomp1"]
    
    # Whether to copy files from current directory to the Docker image
    copy_files: True
    
    # Environment variables to set in the container
    environment:
      ZENML_LOGGING_VERBOSITY: DEBUG
      PYTHONUNBUFFERED: "1"
    
    # Parent image to use for building
    parent_image: "zenml-io/zenml-cuda:latest"
    
    # Additional Python packages to install
    requirements: ["torch==1.10.0", "transformers>=4.0.0", "pandas"]
```

### Resource Settings

Configure compute resources for pipeline or step execution:

```yaml
# Pipeline-level resource settings
settings:
  resources:
    cpu_count: 2
    gpu_count: 1
    memory: "4Gb"

# Step-specific resource settings
steps:
  train_model:
    settings:
      resources:
        cpu_count: 4
        gpu_count: 2
        memory: "16Gb"
```

### Stack Component Settings

Configure specific stack components for steps:

```yaml
steps:
  train_model:
    # Use specific named components
    experiment_tracker: "mlflow_tracker"
    step_operator: "vertex_gpu"
    
    # Component-specific settings
    settings:
      # MLflow specific configuration
      experiment_tracker.mlflow:
        experiment_name: "image_classification"
        nested: True
```

## Working with Configuration Files

### Autogenerating Template YAML Files

ZenML provides a command to generate a template configuration file:

```bash
zenml pipeline build-configuration my_pipeline > config.yaml
```

This generates a YAML file with all pipeline parameters, step parameters, and configuration options with their default values.

### Environment Variables in Configuration

You can reference environment variables in your YAML configuration:

```yaml
settings:
  docker:
    environment:
      # References an environment variable from the host system
      API_KEY: ${MY_API_KEY}
      DATABASE_URL: ${DB_CONNECTION_STRING}
```

### Using Configuration Files for Different Environments

A common pattern is to maintain different configuration files for different environments:

```
├── configs/
│   ├── dev.yaml     # Development configuration
│   ├── staging.yaml # Staging configuration
│   └── prod.yaml    # Production configuration
```

Example development configuration:

```yaml
# dev.yaml
enable_cache: False
enable_step_logs: True
parameters:
  dataset_size: "small"
settings:
  docker:
    parent_image: "zenml-io/zenml:latest"
```

Example production configuration:

```yaml
# prod.yaml
enable_cache: True
enable_step_logs: False
parameters:
  dataset_size: "full"
settings:
  docker:
    parent_image: "zenml-io/zenml-cuda:latest"
  resources:
    cpu_count: 8
    memory: "16Gb"
```

You can then specify which configuration to use:

```python
# For development
my_pipeline.with_options(config_path="configs/dev.yaml")()

# For production
my_pipeline.with_options(config_path="configs/prod.yaml")()
```

## Advanced Configuration Options

### Model Configuration

Link a pipeline to a ZenML Model:

```yaml
model:
  name: "classification_model"
  description: "Image classifier trained on the CIFAR-10 dataset"
  tags: ["computer-vision", "classification", "pytorch"]
  
  # Specific model version
  version: "1.2.3"
```

### Scheduling

Configure pipeline scheduling when using an orchestrator that supports it:

```yaml
schedule:
  # Whether to run the pipeline for past dates if schedule is missed
  catchup: false
  
  # Cron expression for scheduling (daily at midnight)
  cron_expression: "0 0 * * *"
  
  # Time to start scheduling from
  start_time: "2023-06-01T00:00:00Z"
```

## Conclusion

YAML configuration in ZenML provides a powerful way to customize pipeline behavior without changing your code. By separating configuration from implementation, you can make your ML workflows more flexible, maintainable, and reproducible.

See also:

* [Steps & Pipelines](/concepts/steps_and_pipelines) - Core building blocks
* [Advanced Features](/concepts/steps_and_pipelines/advanced_features) - Advanced pipeline features

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Source Code and Imports

Understanding source roots and source paths

When ZenML interacts with your pipeline code, it needs to understand how to locate and import your code. This page explains how ZenML determines the source root directory and how to construct source paths for referencing your Python objects.

## Source Root

The **source root** is the root directory of all your local code files.

ZenML determines the source root using the following priority:

1. **ZenML Repository**: If you're in a child directory of a [ZenML repository](https://docs.zenml.io/user-guides/best-practices/set-up-your-repository) (initialized with `zenml init`), the repository directory becomes the source root. We recommend always initializing a ZenML repository to make the source root explicit.
2. **Execution Context Fallback**: If no ZenML repository exists in your current working directory or parent directories, ZenML uses the parent directory of the Python file you're executing. For example, running `/a/b/run.py` sets the source root to `/a/b`.

{% hint style="warning" %}
If you're running in a notebook or an interactive Python environment, there will be no file that is currently executed and ZenML won't be able to automatically infer the source root. Therefore, you'll need to explicitly define the source root by initializing a ZenML repository in these cases.
{% endhint %}

## Source Paths

ZenML requires source paths in various configuration contexts. These are Python-style dotted paths that reference objects in your code.

### Common Use Cases

**Step Hook Configuration**:

```yaml
success_hook_source: <SUCCESS-HOOK-SOURCE>
```

**Pipeline Deployment via CLI**:

```bash
zenml pipeline deploy <PIPELINE-SOURCE>
```

### Path Construction

Import paths must be **relative to your source root** and follow Python import syntax.

**Example**: Consider this pipeline in `/a/b/c/run.py`:

```python
from zenml import pipeline

@pipeline
def my_pipeline():
    ...
```

The source path depends on your source root:

* Source root `/a/b/c` → `run.my_pipeline`
* Source root `/a` → `b.c.run.my_pipeline`

{% hint style="info" %}
Note that the source is not a file path, but instead its elements are separated by dots similar to how you would write import statements in Python.
{% endhint %}

## Containerized Step Execution

When running pipeline steps in containers, ZenML ensures your source root files are available in the container (either by including them in the image or downloading them at runtime).

To execute your step code, ZenML imports the Python module containing the step definition. **All imports of local code files must be relative to the source root** for this to work correctly.

{% hint style="info" %}
If you don't need all files inside your source root for step execution, see the [containerization guide](/concepts/containerization#controlling-included-files) for controlling which files are included.
{% endhint %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Execution

Step and pipeline execution.

This page explains what happens under the hood when ZenML executes steps in static and dynamic pipelines. Regardless of where or how a step executes (inline or in an isolated environment, synchronous or concurrent), ZenML applies the same core semantics: inputs are loaded via materializers, outputs are materialized as versioned artifacts, lineage/metadata and logs are recorded, caching policies are respected, and step/run status is published consistently.

## Static pipelines

In static pipelines, ZenML executes the pipeline function before running the pipeline to compile a DAG of steps, which the orchestrator then schedules according to their upstream dependencies. This pre-compilation allows ZenML to optimize execution order and validate the DAG structure before any steps run.

### Execution scenarios

![Static pipeline](/files/407VkhaTv5dwv2QXNtJL) ![Static pipeline with step operator](/files/tBMLwmYo2GN75MeEbbs0)

## Dynamic pipelines

[Dynamic pipelines](/concepts/steps_and_pipelines/dynamic_pipelines) execute the pipeline function at runtime. Each step executed inside the pipeline function can be:

* **Inline** (runs inside the orchestration environment)
* **Isolated** (runs in a separate environment via the orchestrator or a step operator)

And each step call can be:

* **Synchronous** (via `my_step(...)`): blocks until completion and returns the step output artifacts.
* **Concurrent** (via `my_step.submit(...)`): starts step execution in a separate thread and returns a future. The pipeline function resumes execution immediately.

### Execution scenarios

#### Synchronous inline

The step runs in-process inside the orchestration environment. The pipeline function blocks until the step completes.

![Dynamic pipeline, synchronous inline step](/files/ll6m2DnWzkEv2PTC5NS1)

#### Concurrent inline

The step runs in-process in a separate thread. The pipeline function continues immediately and only waits when results are consumed.

![Dynamic pipeline, concurrent inline step](/files/KSNIZgHdY2mJh35DSAFI)

#### Synchronous isolated

The step runs in a separate environment (via the orchestrator or step operator). The pipeline function blocks until the job completes.

![Dynamic pipeline, synchronous isolated step](/files/zA8p0LFAEMWqaQvwLqb5)

#### Concurrent isolated

The step runs in a separate environment (via the orchestrator or step operator). The pipeline function continues immediately and only waits when results are consumed.

![Dynamic pipeline, concurrent isolated step](/files/HCbg8tSnaOPEcphlK1dW)

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Wait for External Input

Pause a dynamic pipeline for external input and resume it after the input is resolved.

Use `zenml.wait(...)` when a dynamic pipeline needs a human or external system to provide input before it can continue.

{% hint style="info" %}
`zenml.wait(...)` only works inside [dynamic pipelines](/concepts/steps_and_pipelines/dynamic_pipelines). It does not work in static pipelines or inside a step.
{% endhint %}

## Basic pattern

When the pipeline reaches `wait(...)`, ZenML creates a wait condition and pauses the run. Once the condition is resolved, the pipeline continues from that point.

```python
from zenml import pipeline, step, wait


@step
def prepare_candidate() -> str:
    """Prepare the candidate that may be released."""
    return "model:v17"


@step
def register_release(candidate: str, release_tag: str) -> None:
    """Register the release tag chosen for the candidate."""
    print(f"Registering {candidate} as {release_tag}")


@pipeline(dynamic=True)
def release_pipeline() -> None:
    """Pause until an external system provides a release tag."""
    candidate = prepare_candidate()
    release_tag = wait(
        schema=str,
        question="Provide the release tag for this candidate.",
    )
    register_release(candidate=candidate, release_tag=release_tag)
```

## Data schemas

The `schema=` argument defines the shape of the value that must be returned when the wait condition is answered. You can use primitive types such as:

* `str`
* `int`
* `float`
* `bool`
* `list`
* `dict`

You can also use Pydantic objects for structured input:

```python
from pydantic import BaseModel
from zenml import pipeline, wait


class DeploymentConfig(BaseModel):
    """Structured deployment input returned to the waiting run."""

    environment: str
    replicas: int
    notify_slack: bool


@pipeline(dynamic=True)
def deployment_pipeline() -> None:
    """Pause until a structured deployment configuration is provided."""
    config = wait(
        schema=DeploymentConfig,
        question="Provide the deployment configuration for this run.",
    )
    print(config.environment, config.replicas, config.notify_slack)
```

## Agentic approval workflow

`wait(...)` is useful when an agentic workflow should prepare a recommendation but a human should approve the final action. The example below plans multiple agent tasks, runs them with dynamic mapping, summarizes the results, and then continues only if the approval wait condition resolves to `true`.

```python
from zenml import pipeline, step, wait


@step
def plan_agent_tasks(goal: str) -> list[dict[str, str]]:
    return [
        {"task_id": "research", "instruction": f"Research {goal}"},
        {"task_id": "draft", "instruction": f"Draft a plan for {goal}"},
        {"task_id": "risk_check", "instruction": f"Review risks for {goal}"},
    ]


@step
def execute_agent_task(task: dict[str, str]) -> dict[str, str]:
    return {"task_id": task["task_id"], "result": "completed"}


@step
def summarize_agent_work(results: list[dict[str, str]]) -> list[dict[str, str]]:
    return results


@step
def take_final_action(summary: list[dict[str, str]]) -> None:
    print(f"Acting on {len(summary)} reviewed tasks.")


@pipeline(dynamic=True)
def agentic_approval_pipeline(goal: str) -> None:
    tasks = plan_agent_tasks(goal=goal)
    results = execute_agent_task.map(task=tasks)
    summary = summarize_agent_work(results)
    approved = wait(
        schema=bool,
        question="Approve the agent recommendation and continue?",
        metadata={"goal": goal},
        name="human_approval",
    )

    if approved:
        take_final_action(summary)
```

For a runnable version that also logs metadata and returns tabular artifacts, see the [`agentic_hitl_pipeline` example](https://github.com/zenml-io/zenml/tree/main/examples/agentic_hitl_pipeline/README.md).

## Timeouts and pausing

`wait(...)` accepts a `timeout` (default: 600 seconds). When the timeout elapses without a resolution, ZenML transitions the run to `PAUSED` so the orchestration process can be torn down — picking it back up later via [resume](#resolve-and-resume).

There is one subtlety to know about for nested or concurrent dynamic pipelines: the run only pauses once all *tree-wide* work has settled.

* A run with concurrent steps (`step.submit(...)`), maps (`step.map(...)`), or child pipelines (`child(...)`, `child.submit(...)`) keeps the orchestration process up so it can monitor that work.
* While that process is up anyway, pausing the run wouldn't free any resources — the wait keeps polling and refreshing its lease even past `timeout`.
* Once the tree quiesces (concurrent steps and child runs have all finished, or themselves paused), the wait gives up and publishes `PAUSED`.

In other words, `timeout` is the earliest moment the run can pause, not a hard upper bound on how long the wait blocks. For a run with no concurrent or nested work it behaves as the simple bound it appears to be. For richer pipelines, treat it as a "stop polling once everything else is also idle" hint.

If you want the run to pause as soon as the timeout elapses regardless of sibling work, design the pipeline so the `wait(...)` call is the only in-flight work at that point — for example, by `.wait()`-ing on the relevant futures before calling `wait(...)`.

## Resolve and resume

You can resolve wait conditions either in the UI or from the CLI.

To review and resolve the pending wait conditions for a specific run interactively, use:

```bash
zenml pipeline runs wait-conditions resolve --run <RUN_ID_OR_NAME> --interactive
```

If you want to resolve a condition non-interactively, pass the result as JSON:

```bash
zenml pipeline runs wait-conditions resolve <WAIT_CONDITION_ID> \
  --resolution continue \
  --result '{"environment": "production", "replicas": 3, "notify_slack": true}'
```

In ZenML Pro, the run should start resuming automatically after the wait condition is resolved. If it remains paused, or if you are using OSS, resume it manually:

```bash
zenml pipeline runs resume <RUN_ID_OR_NAME>
```

If manual resume fails, check the error message before retrying. The common causes are concrete:

* The run is a child run. Resume the parent run ID shown in the error instead; the parent is the run that can safely continue the whole nested execution tree.
* The run is not currently `PAUSED`. A run that is still running, already completed, failed, or stopped cannot be resumed with this command.
* The run still has an active wait condition. Resolve the wait condition first, then resume the run.
* ZenML can no longer find the snapshot or stack that the paused run needs in order to continue. In that case, the run cannot be resumed until the missing reference is restored or recreated.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Advanced Features

Advanced features and capabilities of ZenML pipelines and steps

This guide covers advanced features and capabilities of ZenML pipelines and steps, allowing you to build more sophisticated machine learning workflows.

## Execution Control

### Caching

Steps are automatically cached based on their code, inputs and other factors. When a step runs, ZenML computes a hash of the inputs and checks if a previous run with the same inputs exists. If found, ZenML reuses the outputs instead of re-executing the step.

You can control caching behavior at the step level:

```python
@step(enable_cache=False)
def non_cached_step():
    pass
```

You can also configure caching at the pipeline level:

```python
@pipeline(enable_cache=False)
def my_pipeline():
    ...
```

Or modify it after definition:

```python
my_step.configure(enable_cache=False)
my_pipeline.configure(enable_cache=False)
```

For more information, check out [this page](https://docs.zenml.io/user-guides/starter-guide/cache-previous-executions).

### Running Individual Steps

You can run a single step directly:

```python
model, accuracy = train_classifier(X_train=X_train, y_train=y_train)
```

This creates a pipeline run with just that step. If you want to bypass ZenML completely and run the underlying function directly:

```python
model, accuracy = train_classifier.entrypoint(X_train=X_train, y_train=y_train)
```

You can make this the default behavior by setting the `ZENML_RUN_SINGLE_STEPS_WITHOUT_STACK` environment variable to `True`.

### Step signatures, definitions, and semantic types

A ZenML step is usually written as a normal Python function at module level:

```python
from zenml import step

@step
def train_model(dataset_path: str, *, epochs: int = 10) -> None:
    ...
```

Keyword-only arguments, like `epochs` in the example above, are supported and are treated as normal step inputs. This is useful when you want a call site to be explicit about important values. Variadic signatures such as `*args` and `**kwargs` are not valid step inputs because ZenML needs to know the step interface before the pipeline runs.

For production code, define the underlying Python functions for your steps and pipelines at module level. They are easier to import, package, test, and review that way. If you need a factory-style pattern, you can wrap those top-level functions with `@step` or `@pipeline(dynamic=True)` inside another function, but the decorated function itself should still be importable from a module.

You can also add semantic metadata to a step with `step_type`:

```python
from zenml import step
from zenml.enums import StepType

@step(step_type=StepType.LLM_CALL)
def summarize_prompt(prompt: str) -> str:
    ...
```

The current step types are `StepType.TOOL_CALL`, `StepType.LLM_CALL`, and `StepType.MEMORY_CALL`. Think of this as a label on the step run: it does not change how your Python function executes, but it gives the dashboard, DAG metadata, and downstream consumers a clearer story about what kind of work the step represents.

### Asynchronous Pipeline Execution

By default, pipelines run synchronously, with terminal logs displaying as the pipeline builds and runs. You can change this behavior to run pipelines asynchronously (in the background):

```python
from zenml import pipeline

@pipeline(settings={"orchestrator": {"synchronous": False}})
def my_pipeline():
    ...
```

Alternatively, you can configure this in a YAML config file:

```yaml
settings:
  orchestrator.<STACK_NAME>:
    synchronous: false
```

You can also configure the orchestrator to always run asynchronously by setting `synchronous=False` in its configuration.

### Async Steps

Step functions can be defined with `async def`. ZenML runs the coroutine to completion when the step executes, so whether a step body is async is invisible at the call site:

```python
from zenml import step

@step
async def fetch_data(url: str) -> bytes:
    async with httpx.AsyncClient() as client:
        response = await client.get(url)
        return response.content
```

An async step behaves like any other step: call it inside a pipeline, pass its outputs downstream, or run several concurrently with [`.submit()`](/concepts/steps_and_pipelines/dynamic_pipelines#parallel-step-execution). Submitted async steps run on separate threads, so their `await` points overlap and IO-bound work proceeds in parallel.

{% hint style="warning" %}
A blocking call inside an async body (`time.sleep`, a synchronous HTTP request, heavy CPU work) stalls that step's event loop and erases the concurrency. Use the async equivalents (`asyncio.sleep`, an async HTTP client) inside async steps.
{% endhint %}

### Step Execution Order

By default, ZenML determines step execution order based on data dependencies. When a step requires output from another step, it automatically creates a dependency.

You can explicitly control execution order with the `after` parameter:

```python
@pipeline
def my_pipeline():
    step_a_output = step_a()
    step_b_output = step_b()
    
    # step_c will only run after both step_a and step_b complete, even if
    # it doesn't use their outputs directly
    step_c(after=[step_a_output, step_b_output])
    
    # You can also specify dependencies using the step invocation ID
    step_d(after="step_c")
```

This is particularly useful for steps with side effects (like data loading or model deployment) where the data dependency is not explicit.

### Execution Modes

ZenML provides three execution modes that control how your orchestrator behaves when a step fails during pipeline execution. These modes are:

* `CONTINUE_ON_FAILURE`: The orchestrator continues executing steps that don't depend on any of the failed steps.
* `STOP_ON_FAILURE`: The orchestrator allows the running steps to complete, but prevents new steps from starting.
* `FAIL_FAST`: The orchestrator stops the run and any running steps immediately when a failure occurs.

You can configure the execution mode of your pipeline in several ways:

```python
from zenml import pipeline
from zenml.enums import ExecutionMode

# Use the decorator
@pipeline(execution_mode=ExecutionMode.CONTINUE_ON_FAILURE)
def my_pipeline():
    ...

# Use the `with_options` method
my_pipeline_with_fail_fast = my_pipeline.with_options(
    execution_mode=ExecutionMode.FAIL_FAST
)

# Use the `configure` method
my_pipeline.configure(execution_mode=ExecutionMode.STOP_ON_FAILURE)
```

{% hint style="warning" %}
In the current implementation, if you use the execution mode `STOP_ON_FAILURE`, the token that is associated with your pipeline run stays valid until its leeway runs out (defaults to 1 hour).
{% endhint %}

As an example, you can consider a pipeline with this dependency structure:

```
         ┌─► Step 2 ──► Step 5 ─┐
Step 1 ──┼─► Step 3 ──► Step 6 ─┼──► Step 8
         └─► Step 4 ──► Step 7 ─┘
```

If steps 2, 3, and 4 execute in parallel and step 2 fails:

* With `FAIL_FAST`: Step 1 finishes → Steps 2,3,4 start → Step 2 fails → Steps 3, 4 are stopped → No other steps get launched
* With `STOP_ON_FAILURE`: Step 1 finishes → Steps 2,3,4 start → Step 2 fails but Steps 3, 4 complete → Steps 5, 6, 7 are skipped
* With `CONTINUE_ON_FAILURE`: Step 1 finishes → Steps 2,3,4 start → Step 2 fails, Steps 3, 4 complete → Step 5 skipped (depends on failed Step 2), Steps 6, 7 run normally → Step 8 is skipped as well.

{% hint style="info" %}
All three execution modes are currently only supported by the `local`, `local_docker`, `kubernetes`, and `modal` orchestrator flavors. For any other orchestrator flavor, the default (and only available) behavior is `CONTINUE_ON_FAILURE`. If you would like to see any of the other orchestrators extended to support the other execution modes, reach out to us in [Slack](https://zenml.io/slack-invite).
{% endhint %}

#### Execution modes in dynamic pipelines

In [dynamic pipelines](/concepts/steps_and_pipelines/dynamic_pipelines), execution modes control what happens when a step launched asynchronously with `step.submit(...)` or `step.map(...)` fails. Dynamic pipelines default to `STOP_ON_FAILURE`, so a failing step surfaces as a failed run unless you opt into another mode.

With `CONTINUE_ON_FAILURE`, the failure of an asynchronous step no longer fails the run:

```python
from zenml import pipeline, step
from zenml.enums import ExecutionMode

@pipeline(dynamic=True, execution_mode=ExecutionMode.CONTINUE_ON_FAILURE)
def my_pipeline() -> None:
    risky_step.submit()          # if this fails, the run still completes
    independent_step.submit()    # keeps running regardless
```

The following rules apply:

* A failed asynchronous step does not fail the run. Independent asynchronous steps keep running and the run completes.
* Steps that depend on a failed asynchronous step (through its output or `after=...`) are skipped.
* Awaiting a failed step through `future.result()` or `future.wait()` still raises the step's exception, and a failure in a step that you call synchronously still fails the run.

### Step Heartbeat

Step heartbeat is a background mechanism that runs alongside step executions and performs two core functions:

* Periodically pings the ZenML server to refresh the step's heartbeat value.
* Retrieves the current pipeline and step status, and terminates the step if the pipeline has entered a stopping state.

This enables ZenML to:

* Track the liveness of a step execution and assess its health based on incoming heartbeats.
* Gracefully interrupt running steps when a pipeline is being stopped.

*Scope and current behavior*

* Heartbeats are enabled only for steps executed in isolated environments. This excludes:
  * `Inline` steps in `dynamic` pipelines.
  * Steps run via the `local` orchestrator.
  * Heartbeat is enabled by default.
* A step that becomes unhealthy automatically triggers a graceful shutdown (currently supported for the `kubernetes` orchestrator).
* When using `CONTINUE_ON_FAILURE` execution mode, heartbeat status is also used to decide whether execution tokens should be invalidated.

*Configuration*

You can configure how long a step may go without sending a heartbeat before it is considered unhealthy using the `heartbeat_healthy_threshold` step parameter. The default value currently applied is 30 minutes.

```python
from zenml import step

@step(heartbeat_healthy_threshold=30)
def my_step():
    ...
```

You can disable heartbeat on the pipeline level if you pass the following configuration parameter:

```python
from zenml import pipeline

@pipeline(enable_heartbeat=False)
def my_pipeline():
    ...
```

If you want to disable heartbeats for a *running* pipeline you can use the following ZenML store utility:

```python
from zenml.client import Client

client = Client()

client.zen_store.disable_run_heartbeat(run_id="run.id")
```

## Data & Output Management

## Type annotations

Your functions will work as ZenML steps even if you don't provide any type annotations for their inputs and outputs. However, adding type annotations to your step functions gives you lots of additional benefits:

* **Type validation of your step inputs**: ZenML makes sure that your step functions receive an object of the correct type from the upstream steps in your pipeline.
* **Better serialization**: Without type annotations, ZenML uses [Cloudpickle](https://github.com/cloudpipe/cloudpickle) to serialize your step outputs. When provided with type annotations, ZenML can choose a [materializer](https://docs.zenml.io/getting-started/core-concepts#materializers) that is best suited for the output. In case none of the builtin materializers work, you can even [write a custom materializer](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types).

{% hint style="warning" %}
ZenML provides a built-in [CloudpickleMaterializer](https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.cloudpickle_materializer) that can handle any object by saving it with [cloudpickle](https://github.com/cloudpipe/cloudpickle). However, this is not production-ready because the resulting artifacts cannot be loaded when running with a different Python version. In such cases, you should consider building a [custom Materializer](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types#custom-materializers) to save your objects in a more robust and efficient format.

Moreover, using the `CloudpickleMaterializer` could allow users to upload of any kind of object. This could be exploited to upload a malicious file, which could execute arbitrary code on the vulnerable system.
{% endhint %}

```python
from typing import Tuple
from zenml import step

@step
def square_root(number: int) -> float:
    return number ** 0.5

# To define a step with multiple outputs, use a `Tuple` type annotation
@step
def divide(a: int, b: int) -> Tuple[int, int]:
    return a // b, a % b
```

If you want to make sure you get all the benefits of type annotating your steps, you can set the environment variable `ZENML_ENFORCE_TYPE_ANNOTATIONS` to `True`. ZenML will then raise an exception in case one of the steps you're trying to run is missing a type annotation.

### Tuple vs multiple outputs

It is impossible for ZenML to detect whether you want your step to have a single output artifact of type `Tuple` or multiple output artifacts just by looking at the type annotation.

We use the following convention to differentiate between the two: When the `return` statement is followed by a tuple literal (e.g. `return 1, 2` or `return (value_1, value_2)`) we treat it as a step with multiple outputs. All other cases are treated as a step with a single output of type `Tuple`.

```python
from zenml import step
from typing import Annotated
from typing import Tuple

# Single output artifact
@step
def my_step() -> Tuple[int, int]:
    output_value = (0, 1)
    return output_value

# Single output artifact with variable length
@step
def my_step(condition) -> Tuple[int, ...]:
    if condition:
        output_value = (0, 1)
    else:
        output_value = (0, 1, 2)

    return output_value

# Single output artifact using the `Annotated` annotation
@step
def my_step() -> Annotated[Tuple[int, ...], "my_output"]:
    return 0, 1


# Multiple output artifacts
@step
def my_step() -> Tuple[int, int]:
    return 0, 1


# Not allowed: Variable length tuple annotation when using
# multiple output artifacts
@step
def my_step() -> Tuple[int, ...]:
    return 0, 1
```

## Step output names

By default, ZenML uses the output name `output` for single output steps and `output_0, output_1, ...` for steps with multiple outputs. These output names are used to display your outputs in the dashboard and [fetch them after your pipeline is finished](https://docs.zenml.io/user-guides/tutorial/fetching-pipelines).

If you want to use custom output names for your steps, use the `Annotated` type annotation:

```python
from typing import Annotated
from typing import Tuple
from zenml import step

@step
def square_root(number: int) -> Annotated[float, "custom_output_name"]:
    return number ** 0.5

@step
def divide(a: int, b: int) -> Tuple[
    Annotated[int, "quotient"],
    Annotated[int, "remainder"]
]:
    return a // b, a % b
```

{% hint style="info" %}
If you do not give your outputs custom names, the created artifacts will be named `{pipeline_name}::{step_name}::output` or `{pipeline_name}::{step_name}::output_{i}` in the dashboard. See the [documentation on artifact versioning and configuration](https://docs.zenml.io/user-guides/starter-guide/manage-artifacts) for more information.
{% endhint %}

## Workflow Patterns

### Pipeline Composition

You can compose pipelines from other pipelines to create modular, reusable workflows:

```python
@pipeline
def data_pipeline(mode: str):
    if mode == "train":
        data = training_data_loader_step()
    else:
        data = test_data_loader_step()
    
    processed_data = preprocessing_step(data)
    return processed_data

@pipeline
def training_pipeline():
    # Use another pipeline inside this pipeline
    training_data = data_pipeline(mode="train")
    model = train_model(data=training_data)
    test_data = data_pipeline(mode="test")
    evaluate_model(model=model, data=test_data)
```

Pipeline composition allows you to build complex workflows from simpler, well-tested components.

### Fan-out and Fan-in

The fan-out/fan-in pattern is a common pipeline architecture where a single step splits into multiple parallel operations (fan-out) and then consolidates the results back into a single step (fan-in). This pattern is particularly useful for parallel processing, distributed workloads, or when you need to process data through different transformations and then aggregate the results. For example, you might want to process different chunks of data in parallel and then aggregate the results:

```python
from zenml import step, get_step_context, pipeline
from zenml.client import Client


@step
def load_step() -> str:
    return "Hello from ZenML!"


@step
def process_step(input_data: str) -> str:
    return input_data


@step
def combine_step(step_prefix: str, output_name: str) -> None:
    run_name = get_step_context().pipeline_run.name
    run = Client().get_pipeline_run(run_name)

    # Fetch all results from parallel processing steps
    processed_results = {}
    for step_name, step_info in run.steps.items():
        if step_name.startswith(step_prefix):
            output = step_info.outputs[output_name][0]
            processed_results[step_info.name] = output.load()

    # Combine all results
    print(",".join([f"{k}: {v}" for k, v in processed_results.items()]))


@pipeline(enable_cache=False)
def fan_out_fan_in_pipeline(parallel_count: int) -> None:
    # Initial step (source)
    input_data = load_step()

    # Fan out: Process data in parallel branches
    after = []
    for i in range(parallel_count):
        artifact = process_step(input_data, id=f"process_{i}")
        after.append(artifact)

    # Fan in: Combine results from all parallel branches
    combine_step(step_prefix="process_", output_name="output", after=after)


fan_out_fan_in_pipeline(parallel_count=8)
```

The fan-out pattern allows for parallel processing and better resource utilization, while the fan-in pattern enables aggregation and consolidation of results. This is particularly useful for:

* Parallel data processing
* Distributed model training
* Ensemble methods
* Batch processing
* Data validation across multiple sources
* Hyperparameter tuning

Note that when implementing the fan-in step, you'll need to use the ZenML Client to query the results from previous parallel steps, as shown in the example above, and you can't pass in the result directly.

{% hint style="warning" %}
The fan-in, fan-out method has the following limitations:

1. Steps run sequentially rather than in parallel if the underlying orchestrator does not support parallel step runs (e.g. with the local orchestrator)
2. The number of steps need to be known ahead-of-time, and ZenML does not yet support the ability to dynamically create steps on the fly.
   {% endhint %}

### Dynamic Fan-out/Fan-in with Snapshots

For scenarios where you need to determine the number of parallel operations at runtime (e.g., based on database queries or dynamic data), you can use [snapshots](https://docs.zenml.io/user-guides/tutorial/trigger-pipelines-from-external-systems) to create a more flexible fan-out/fan-in pattern. This approach allows you to trigger multiple pipeline runs dynamically and then aggregate their results.

```python
from typing import List, Optional
from uuid import UUID
import time

from zenml import step, pipeline
from zenml.client import Client


@step
def load_relevant_chunks() -> List[str]:
    """Load chunk identifiers from database or other dynamic source."""
    # Example: Query database for chunk IDs
    # In practice, this could be a database query, API call, etc.
    return ["chunk_1", "chunk_2", "chunk_3", "chunk_4"]


@step
def trigger_chunk_processing(
    chunks: List[str], 
    snapshot_id: Optional[UUID] = None
) -> List[UUID]:
    """Trigger multiple pipeline runs for each chunk and wait for completion."""
    client = Client()
    
    # Use snapshot ID if provided, otherwise give the pipeline name 
    # of the pipeline you want triggered. Giving the pipeline name
    # will automatically find the latest snapshot of that pipeline.
    pipeline_name = None if snapshot_id else "chunk_processing_pipeline"
    
    # Trigger all chunk processing runs
    run_ids = []
    for chunk_id in chunks:
        run_config = {
            "steps": {
                "process_chunk": {
                    "parameters": {
                        "chunk_id": chunk_id
                    }
                }
            }
        }
        
        run = client.trigger_pipeline(
            snapshot_name_or_id=snapshot_id,
            pipeline_name_or_id=pipeline_name,
            run_configuration=run_config,
            synchronous=False  # Run asynchronously
        )
        run_ids.append(run.id)
    
    # Wait for all runs to complete
    print(f"Waiting for {len(run_ids)} chunk processing runs to complete...")
    completed_runs = set()  # Cache completed runs to avoid re-fetching
    while True:
        # Only check runs that haven't completed yet
        pending_runs = [run_id for run_id in run_ids if run_id not in completed_runs]
        
        for run_id in pending_runs:
            run = client.get_pipeline_run(run_id)
            if run.status.is_finished:
                completed_runs.add(run_id)
        
        if len(completed_runs) == len(run_ids):
            print("All chunk processing runs completed!")
            break
        
        print(f"Completed: {len(completed_runs)}/{len(run_ids)} runs")
        time.sleep(10)  # Wait 10 seconds before checking again
    
    return run_ids


@step
def aggregate_results(run_ids: List[UUID]) -> dict:
    """Aggregate results from all chunk processing runs."""
    client = Client()
    aggregated_results = {}
    failed_runs = []
    
    for run_id in run_ids:
        run = client.get_pipeline_run(run_id)
        
        # Check if run succeeded
        if run.status.value == "failed":
            failed_runs.append({
                "run_id": str(run_id),
                "status": run.status.value,
            })
            print(f"WARNING: Run {run_id} failed with status {run.status.value}")
            continue
        
        # Extract results from successful runs only
        if "process_chunk" in run.steps:
            step_run = run.steps["process_chunk"]
            # Simple assumption: process_chunk step has one output that we can load
            chunk_result = step_run.output.load()
            aggregated_results[str(run_id)] = chunk_result

    
    # Log summary of results
    total_runs = len(run_ids)
    successful_runs = len(aggregated_results)
    failed_count = len(failed_runs)
    
    print(f"Aggregation complete: {successful_runs}/{total_runs} runs successful")

    return {
        "successful_results": aggregated_results,
        "failed_runs": failed_runs,
        "summary": {
            "total_runs": total_runs,
            "successful_runs": successful_runs,
            "failed_runs": failed_count
        }
    }


@pipeline(enable_cache=False)
def fan_out_fan_in_pipeline(snapshot_id: Optional[UUID] = None):
    """Fan-out/fan-in pipeline that orchestrates dynamic chunk processing."""
    # Load chunks dynamically at runtime
    chunks = load_relevant_chunks()
    
    # Trigger chunk processing runs and wait for completion
    run_ids = trigger_chunk_processing(chunks, snapshot_id)
    
    # Aggregate results from all runs
    results = aggregate_results(run_ids)
    
    return results


# Define the chunk processing pipeline that will be triggered
@step
def process_chunk(chunk_id: Optional[str] = None) -> dict:
    """Process a single chunk of data."""
    # Simulate chunk processing
    print(f"Processing chunk: {chunk_id}")
    return {
        "chunk_id": chunk_id,
        "processed_items": 100,
        "status": "completed"
    }


@pipeline
def chunk_processing_pipeline():
    """Pipeline that processes a single chunk."""
    result = process_chunk()
    return result


# Usage example
if __name__ == "__main__":
    # First, create a snapshot for the chunk processing pipeline
    #  This would typically be done once during setup.
    #  Make sure a remote stack is set before running this
    snapshot = chunk_processing_pipeline.create_snapshot(
        name="chunk_processing",
        description="Snapshot for processing individual chunks"
    )

    # Run the fan-out/fan-in pipeline with the snapshot
    #  You can also get the snapshot ID from the dashboard
    fan_out_fan_in_pipeline(snapshot_id=snapshot.id)
```

This pattern enables dynamic scaling, true parallelism, and database-driven workflows. Key advantages include fault tolerance and separate monitoring for each chunk. Consider resource management and proper error handling when implementing.

### Custom Step Invocation IDs

When calling a ZenML step as part of your pipeline, it gets assigned a unique **invocation ID** that you can use to reference this step invocation when defining the execution order of your pipeline steps or use it to fetch information about the invocation after the pipeline has finished running.

```python
from zenml import pipeline, step

@step
def my_step() -> None:
    ...

@pipeline
def example_pipeline():
    # When calling a step for the first time inside a pipeline,
    # the invocation ID will be equal to the step name -> `my_step`.
    my_step()
    # When calling the same step again, the suffix `_2`, `_3`, ... will
    # be appended to the step name to generate a unique invocation ID.
    # For this call, the invocation ID would be `my_step_2`.
    my_step()
    # If you want to use a custom invocation ID when calling a step, you can
    # do so by passing it like this. If you pass a custom ID, it needs to be
    # unique for all the step invocations that happen as part of this pipeline.
    my_step(id="my_custom_invocation_id")
```

### Named Pipeline Runs

In the output logs of a pipeline run you will see the name of the run:

```bash
Pipeline run training_pipeline-2023_05_24-12_41_04_576473 has finished in 3.742s.
```

This name is automatically generated based on the current date and time. To change the name for a run, pass `run_name` as a parameter to the `with_options()` method:

```python
training_pipeline = training_pipeline.with_options(
    run_name="custom_pipeline_run_name"
)
training_pipeline()
```

Pipeline run names must be unique, so if you plan to run your pipelines multiple times or run them on a schedule, make sure to either compute the run name dynamically or include one of the placeholders that ZenML will replace.

{% hint style="info" %}
The substitutions for the custom placeholders like `experiment_name` can be set in:

* `@pipeline` decorator, so they are effective for all steps in this pipeline
* `pipeline.with_options` function, so they are effective for all steps in this pipeline run

Standard substitutions always available and consistent in all steps of the pipeline are:

* `{date}`: current date, e.g. `2024_11_27`
* `{time}`: current time in UTC format, e.g. `11_07_09_326492`
  {% endhint %}

```python
training_pipeline = training_pipeline.with_options(
    run_name="custom_pipeline_run_name_{experiment_name}_{date}_{time}"
)
training_pipeline()
```

## Error Handling & Reliability

### Automatic Step Retries

For steps that may encounter transient failures (like network issues or resource limitations), you can configure automatic retries:

```python
from zenml.config.retry_config import StepRetryConfig

@step(
    retry=StepRetryConfig(
        max_retries=3,  # Maximum number of retry attempts
        delay=10,       # Initial delay in seconds before first retry
        backoff=2       # Factor by which delay increases after each retry
    )
)
def unreliable_step():
    # This step might fail due to transient issues
    ...
```

It's important to note that **retries happen at the step level, not the pipeline level**. This means that ZenML will only retry individual failed steps, not the entire pipeline.

With this configuration, if the step fails, ZenML will:

1. Wait 10 seconds before the first retry
2. Wait 20 seconds (10 × 2) before the second retry
3. Wait 40 seconds (20 × 2) before the third retry
4. Fail the pipeline if all retries are exhausted

This is particularly useful for steps that interact with external services or resources.

## Monitoring & Notifications

### Pipeline and Step Hooks

Run custom code on step and pipeline lifecycle events (`on_start`, `on_end`, `on_success`, `on_failure`) to send notifications, log run details, or trigger external workflows. See the dedicated [Hooks](/concepts/steps_and_pipelines/hooks) page for the full hook surface, including signatures, accessing run information, alerter integration, the static-versus-dynamic pipeline behavior, and running custom hooks using `run_hook(...)`.

## Conclusion

These advanced features provide powerful capabilities for building sophisticated machine learning workflows in ZenML. By leveraging these features, you can create pipelines that are more robust, maintainable, and flexible.

See also:

* [Steps & Pipelines](/concepts/steps_and_pipelines) - Core building blocks
* [YAML Configuration](/concepts/steps_and_pipelines/yaml_configuration) - YAML configuration

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Hooks

Run custom code on pipeline and step lifecycle events and record hook invocations.

Hooks let you run custom code at lifecycle points of a run or a step, and record those invocations as queryable `HookInvocation` records. The same machinery powers the built-in lifecycle hooks ZenML fires for you and the public API you can call from your own code. Common uses include sending notifications on success or failure, logging run details, and triggering external workflows.

## Lifecycle hooks

Lifecycle hooks fire automatically and each fire creates one `HookInvocation` row.

| Hook         | Step scope                                    | Pipeline scope (dynamic)                   |
| ------------ | --------------------------------------------- | ------------------------------------------ |
| `on_start`   | Each execution attempt, before the step body  | Once before the run starts                 |
| `on_end`     | Each execution attempt, regardless of outcome | Once when the run reaches a terminal state |
| `on_success` | Once when the step completes successfully     | Once when the run completes successfully   |
| `on_failure` | Once when the step fails terminally           | Once when the run fails                    |
| `on_pause`   | —                                             | Once when the run pauses                   |
| `on_resume`  | —                                             | Once when a paused run resumes             |

Step-level hooks fire for both static and dynamic pipelines, uniformly.

### Static pipelines propagate, dynamic pipelines fire

The same `@pipeline(on_*=...)` kwarg has two different runtime meanings depending on whether the pipeline is dynamic.

| `@pipeline(on_*=X)` | Static pipeline                                | Dynamic pipeline                 |
| ------------------- | ---------------------------------------------- | -------------------------------- |
| `on_start`          | Propagates to each step's `on_start` default   | Fires once at the pipeline level |
| `on_success`        | Propagates to each step's `on_success` default | Fires once at the pipeline level |
| `on_failure`        | Propagates to each step's `on_failure` default | Fires once at the pipeline level |
| `on_end`            | Propagates to each step's `on_end` default     | Fires once at the pipeline level |
| `on_pause`          | Ignored                                        | Fires once at the pipeline level |
| `on_resume`         | Ignored                                        | Fires once at the pipeline level |

For a static pipeline, a pipeline-level hook is a default that every step inherits where it has not set its own. No pipeline-level hooks are run. For a dynamic pipeline, the hook fires once at the run level and produces `RUN_*` entries. It fires on every run, including each invocation of a deployed pipeline. Dynamic pipeline users who want per-step defaults wire each `@step` directly.

## Registering hooks

Pass a callable or a source string to the decorator, `.configure(...)`, or `.with_options(...)`.

```python
from zenml import step, pipeline

def notify_start():
    print("starting")

def notify_end(exception=None):
    print("finished")

@step(on_start=notify_start, on_end=notify_end)
def my_step() -> int:
    return 42

@pipeline(on_start=notify_start, on_end=notify_end)
def my_pipeline():
    my_step()

# Override at configuration time
my_step = my_step.with_options(on_failure="my_module.alert_on_failure")
```

### Hook signatures

`on_start`, `on_success`, `on_pause` and `on_resume` take no arguments. `on_failure` and `on_end` optionally take a single `BaseException` argument.

```python
from typing import Optional

def on_end(): ...
def on_end(exception: Optional[BaseException] = None): ...
```

`exception` is set only when the attempt or run failed. Details about the current step or run are available through the step or run context (see below).

### Accessing step/run information in hooks

Step-scope hooks read the current step run and pipeline run through the step context. Run-scope hooks on a dynamic pipeline fire outside any step and read the run from the run context instead, via `DynamicPipelineRunContext.get().run`.

```python
from zenml import get_step_context, step

def on_failure(exception: BaseException):
    context = get_step_context()
    print(f"Failed step: {context.step_run.name}")

@step(on_failure=on_failure)
def my_step(some_parameter: int = 1):
    raise ValueError("My exception")
```

### Sending alerts from hooks

Use the [Alerter stack component](https://docs.zenml.io/component-guide/alerters) to send notifications when a step or run fails or succeeds.

```python
from zenml import get_step_context
from zenml.client import Client

def on_failure():
    step_name = get_step_context().step_run.name
    Client().active_stack.alerter.post(f"{step_name} just failed!")
```

ZenML ships built-in alerter hooks for the common case.

```python
from zenml.hooks import alerter_success_hook, alerter_failure_hook

@step(on_failure=alerter_failure_hook, on_success=alerter_success_hook)
def my_step():
    ...
```

## Behavior notes

* **Retries.** A retried step fires one `on_start` / `on_end` pair per attempt. `on_success` and `on_failure` fire exactly once, at the terminal outcome.
* **Cache hits.** A cached step fires no step-level hooks. Pipeline-level hooks on a dynamic run still fire even when every step was cached.
* **Hook failures are swallowed.** When a lifecycle hook raises, the run or step is not aborted. The exception is captured into the `HookInvocation` record with `status=FAILED` and execution proceeds.
* **Async hooks.** Hook functions can be defined with `async def`. When the hook fires, ZenML runs the coroutine to completion and blocks until it finishes.
* **Return values are discarded.** Set `ZENML_TRACK_LIFECYCLE_HOOK_OUTPUTS=true` in the execution environment to instead materialize lifecycle hook return values as output artifacts of the invocation, following the same rules as `store_return=True` on `run_hook` (see below).

## Init and cleanup hooks

`on_init` and `on_cleanup` are pipeline setup and teardown hooks. They initialize and tear down shared run state rather than reacting to a single run or step outcome, so they follow different rules from the lifecycle hooks above. They are **not** recorded as `HookInvocation` records. When `on_init` fails, the run still records `RUN_START`, `RUN_END`, and `RUN_FAILURE`, but never a row for `on_init` itself. Find the root cause on `pipeline_run.exception_info`.

ZenML runs `on_init` **once per execution environment**, before any step body runs in that environment, and `on_cleanup` once when that environment is torn down. Where that lands depends on how the pipeline runs.

### Deployments

`on_init` runs once per deployment replica when the replica starts, and `on_cleanup` once when it shuts down. Individual invocations of the deployed pipeline reuse the initialized state and do not re-run either hook. Lifecycle hooks like `on_start` and `on_end` still fire on every invocation.

### Regular runs

For a run that is not a deployment, `on_init` runs once per execution environment:

* **Dynamic pipeline:** once in the orchestrator environment, where the pipeline function executes.
* **Any step that runs outside the orchestration environment (static or dynamic):** once ahead of that step body, the first time its environment is used. A step that shares the orchestration environment skips the hook, because the run context is already initialized there.

## Recording custom invocations

Beyond the built-in lifecycle hooks, you can record arbitrary invocations from inside a step or a dynamic pipeline function. This is useful for instrumenting third-party callbacks such as the tool and model calls of an agent framework.

### `run_hook`

Call `run_hook(func, ...)` to run a function and record the invocation in one call. The return value flows through to you.

```python
from zenml import run_hook, step

def call_tool(name: str) -> str:
    return f"result of {name}"

@step
def agent_step():
    # Records one CUSTOM HookInvocation, returns the function's result.
    result = run_hook(call_tool, "search")
```

Pass `store_return=True` to materialize the return value as an output artifact. A single unannotated return becomes one artifact named `output`. An annotated tuple return unpacks into one artifact per element.

```python
result = run_hook(call_tool, "search", store_return=True)
```

## Querying hook invocations

List the invocations recorded for a run through the client.

```python
from zenml.client import Client
from zenml.enums import HookType

invocations = Client().list_hook_invocations(
    pipeline_run_id=run.id,
    hook_type=HookType.CUSTOM,
)
for invocation in invocations.items:
    print(invocation.name, invocation.status)
```


# Dynamic Pipelines

Write dynamic pipelines

## Why Dynamic Pipelines?

Traditional ZenML pipelines require you to define the entire DAG structure at pipeline definition time. While this works well for many use cases, there are scenarios where you need more flexibility:

* **Runtime-dependent workflows**: When the number of steps or their configuration depends on data computed during pipeline execution
* **Dynamic parallelization**: When you need to spawn multiple parallel step executions based on runtime conditions
* **Conditional execution**: When the workflow structure needs to adapt based on intermediate results

Dynamic pipelines allow you to write pipelines that generate their DAG structure dynamically at runtime, giving you the power of Python's control flow (loops, conditionals) combined with ZenML's orchestration capabilities.

{% hint style="info" %}
Dynamic pipelines are powerful but easy to get wrong (e.g., `.load()` vs `.chunk()`, mapping vs submit). If you use an AI coding agent, the `zenml-pipeline-authoring` skill can guide implementation step-by-step. See [LLM tooling](/reference/llms-txt).
{% endhint %}

## Basic Example

The simplest dynamic pipeline uses regular Python control flow to determine step execution:

```python
from zenml import step, pipeline

@step
def generate_int() -> int:
    return 3

@step
def do_something(index: int) -> None:
    print(f"Processing index {index}")

@pipeline(dynamic=True)
def dynamic_pipeline() -> None:
    count = generate_int()
    # `count` is an artifact, we now load the data
    count_data = count.load()

    for idx in range(count_data):
        # This will run sequentially, like regular Python code would.
        do_something(idx)

if __name__ == "__main__":
    dynamic_pipeline()
```

In this example, the number of `do_something` steps executed depends on the value returned by `generate_int()`, which is only known at runtime.

## Key Features

### Dynamic Step Configuration

You can configure steps dynamically within your pipeline using `with_options()`:

```python
@pipeline(dynamic=True)
def dynamic_pipeline():
    some_step.with_options(enable_cache=False)()
```

This allows you to modify step behavior based on runtime conditions or data.

### Artifact name substitutions in dynamic pipelines

Dynamic pipelines support the same artifact name substitutions as regular pipelines. This matters when a dynamically generated step has outputs whose names include runtime-friendly placeholders. The substituted artifact name is still a real output that you can pass to downstream steps.

```python
from typing import Annotated

from zenml import ArtifactConfig, pipeline, step

@step(substitutions={"suffix": "validated"})
def produce() -> Annotated[int, ArtifactConfig(name="score_{suffix}")]:
    return 1

@step
def consume(score: int) -> None:
    print(score)

@pipeline(dynamic=True)
def dynamic_pipeline() -> None:
    score = produce()
    consume(score)
```

One caveat: when you use `child_pipeline.embed(...)`, the child pipeline's own configuration is not applied. That includes child-level `substitutions`; the parent run's configuration controls the steps that execute inline.

### Step inputs as parameters

Any value you pass to a step that is not the output of another step is uploaded to the artifact store as an external artifact, even a small `int`. Each upload costs a write to the artifact store and a request to the server, which adds up when a pipeline calls steps in a loop.

Set the `ZENML_PARAMETER_SIZE_THRESHOLD` environment variable to pass JSON-serializable inputs as step parameters instead, which skips the upload. The variable must be set in the environment in which the pipeline executes, not on the client that starts the run. See [this page](/concepts/environment-variables) for how to configure environment variables for pipeline execution.

| Value             | Behavior                                                                                           |
| ----------------- | -------------------------------------------------------------------------------------------------- |
| unset or `0`      | Every raw input is uploaded. This is the default.                                                  |
| a positive number | JSON-serializable inputs up to that many bytes become step parameters. Larger inputs are uploaded. |
| `-1`              | Every JSON-serializable input becomes a step parameter.                                            |

Inputs that cannot be parameters are still uploaded, so raising the threshold is safe. Wrap an input with `ExternalArtifact` to keep uploading it.

### Step Runtime Configuration

You can control where a step executes by specifying its runtime:

* **`runtime="inline"`**: The step runs in the orchestration environment (same process/container as the orchestrator)
* **`runtime="isolated"`**: The orchestrator spins up a separate step execution environment (new container/process)

```python
@step(runtime="isolated")
def some_step() -> None:
    # This step will run in its own isolated environment
    ...

@step(runtime="inline")
def another_step() -> None:
    # This step will run in the orchestration environment
    ...
```

Use `runtime="isolated"` when you need:

* Better resource isolation
* Different environment requirements
* Parallel execution (see below)

Use `runtime="inline"` when you need:

* Faster execution (no container startup overhead)
* Shared resources with the orchestrator
* Sequential execution

### Map/Reduce over collections

Dynamic pipelines support a high-level map/reduce pattern over sequence-like step outputs. This lets you fan out a step across items of a collection and then reduce the results without manually writing loops or loading data in the orchestration environment.

```python
from zenml import pipeline, step

@step
def producer() -> list[int]:
    return [1, 2, 3]

@step
def worker(value: int) -> int:
    return value * 2

@step
def reducer(values: list[int]) -> int:
    return sum(values)

@pipeline(dynamic=True, enable_cache=False)
def map_reduce():
    values = producer()
    results = worker.map(values)   # fan out over collection
    reducer(results)               # pass list of artifacts directly
```

For a complete agentic workflow that combines dynamic mapping, reduction, and a human approval gate, see the [`agentic_hitl_pipeline` example](https://github.com/zenml-io/zenml/tree/main/examples/agentic_hitl_pipeline/README.md).

Key points:

* `step.map(...)` fans out a step over sequence-like inputs. These inputs can be either
  * a single list-like output artifact (see the code sample above)
  * a list of output artifacts.
  * the output of a `.map(...)` or `.product(...)` call if the respective step only returns a single output artifact
* Steps can accept lists of artifacts directly as inputs (useful for reducers).
* You can pass the mapped output directly to a downstream step without loading in the orchestration environment.

#### Mapping semantics: map vs product

* `step.map(...)`: If multiple sequence-like inputs are provided, all must have the same length `n`. ZenML creates `n` mapped steps where the i-th step receives the i-th element from each input.
* `step.product(...)`: Creates a mapped step for each combination of elements across all input sequences (cartesian product).

Example (cartesian product):

```python
from zenml import pipeline, step

@step
def int_values() -> list[int]:
    return [1, 2]

@step
def str_values() -> list[str]:
    return ["a", "b", "c"]

@step
def do_something(a: int, b: str) -> int:
    ...

@pipeline(dynamic=True)
def cartesian_example():
    a = int_values()
    b = str_values()
    # Produces 2 * 3 = 6 mapped steps
    do_something.product(a=a, b=b)
```

#### Broadcasting inputs with unmapped(...)

If you want to pass a sequence-like artifact as a whole to each mapped invocation (i.e., avoid splitting), wrap it with `unmapped(...)`:

```python
from zenml import pipeline, step, unmapped

@step
def producer(length: int) -> list[int]:
    return [1] * length

@step
def consumer(a: int, b: list[int]) -> None:
    # `b` is the full list for every mapped call
    ...

@pipeline(dynamic=True)
def unmapped_example():
    a = producer(length=3)   # list of 3 ints
    b = producer(length=4)   # list of 4 ints
    consumer.map(a=a, b=unmapped(b))
```

#### Unpacking mapped outputs

If a mapped step returns multiple outputs, you can split them into separate lists (one per output) using `unpack()`. This returns a tuple of lists of artifact futures, aligned by mapped invocation.

```python
from zenml import pipeline, step

@step
def create_int_list() -> list[int]:
    return [1, 2]

@step
def compute(a: int) -> tuple[int, int]:
    return a * 2, a * 3

@pipeline(dynamic=True)
def map_pipeline():
    ints = create_int_list()
    results = compute.map(a=ints)  # Map over [1, 2]

    # Unpack per-output across all mapped invocations
    double, triple = results.unpack()

    # Each element is an ArtifactFuture; load to get concrete values
    doubles = [f.load() for f in double]  # [2, 4]
    triples = [f.load() for f in triple]  # [3, 6]
```

Notes:

* `results` is a future that refers to all outputs of all steps, and `unpack()` works for both `.map(...)` and `.product(...)`.
* Each list contains future objects that refer to a single artifact.

#### Manual Looping: `.chunk()` vs `.load()`

When looping over artifacts manually, you need two different operations:

| Method        | Purpose                  | When to Use                               |
| ------------- | ------------------------ | ----------------------------------------- |
| `.load()`     | Gets the **actual data** | Making decisions, filtering, control flow |
| `.chunk(idx)` | Creates a **DAG edge**   | Passing to downstream steps               |

{% hint style="info" %}
**Mental model**: `.chunk()` is for wiring (tells the orchestrator "this step depends on item X from upstream"), `.load()` is for decisions (gets values for your Python logic). You typically need both: load to iterate and decide, chunk to wire up the DAG.
{% endhint %}

```python
from zenml import pipeline, step

@step
def create_int_list() -> list[int]:
    return [1, 2, 3, 4]

@step
def compute(a: int) -> int:
    return a * 2

@pipeline(dynamic=True)
def custom_loop():
    ints = create_int_list()

    # .load() to get values for Python control flow (iteration + filtering)
    for index, value in enumerate(ints.load()):
        if value % 2 == 0:
            # .chunk() to create DAG edge (wiring to downstream step)
            chunk = ints.chunk(index=index)
            compute(chunk)
```

### Parallel Step Execution

Dynamic pipelines support true parallel execution using `step.submit()`. This method returns a `StepFuture` that you can use to wait for results or pass to downstream steps:

```python
from zenml import step, pipeline

@step
def some_step(arg: int) -> int:
    return arg * 2

@pipeline(dynamic=True)
def dynamic_pipeline():
    # Submit a step for parallel execution
    future = some_step.submit(arg=1)
    
    # Wait and get artifact response(s)
    artifact = future.result()
    
    # Wait and load artifact data
    data = future.load()
    
    # Pass the output to another step
    downstream_step(future)

    # Run multiple steps in parallel
    for idx in range(3):
        some_step.submit(arg=idx)
```

The `StepFuture` object provides several methods:

* **`result()`**: Wait for the step to complete and return the artifact response(s)
* **`load()`**: Wait for the step to complete and load the actual artifact data
* **Pass directly**: You can pass a `StepFuture` directly to downstream steps, and ZenML will automatically wait for it

{% hint style="info" %}
When using `step.submit()`, steps with `runtime="isolated"` will execute in separate containers/processes, while steps with `runtime="inline"` will execute in separate threads within the orchestration environment.
{% endhint %}

### Ordering submitted steps

A submitted step starts as soon as its inputs are available. To impose an order between steps that have no data dependency, use `after` or `start_after`:

* `after=` waits for the upstream step to **finish** before starting.
* `start_after=` waits for the upstream step to **start** before starting.

`start_after` is useful when an upstream step is long-running and you want a dependent to run alongside it rather than after it. A common case is a step that brings up a service and a second step that uses it:

```python
@pipeline(dynamic=True)
def serve_and_query():
    server = serve_model.submit()  # long-running
    # Starts once the server step is running, not when it finishes.
    query = run_inference.submit(start_after=server)
    query.wait()
```

Using `after=server` here would deadlock, since the dependent would wait for the long-running server step to finish. Both parameters accept a single future or a list, and you can combine them: `run_inference.submit(after=preprocess, start_after=server)`. `start_after` is available on `step.submit(...)`, `step.map(...)`, `step.product(...)`, and on a direct synchronous `step(...)` call, where the entrypoint blocks until the upstream has started. The upstream can be another step or a submitted child pipeline, for example `run_inference(start_after=serve_pipeline.submit())`.

`start_after` orders execution, it does not probe readiness. The dependent starts once the upstream step has launched (for isolated steps, once it is submitted to the infrastructure), which does not guarantee that whatever the upstream sets up is ready to serve. Add your own connection retries if the dependent needs to reach a service the upstream starts. A failed upstream counts as started, so a `start_after` dependent is released rather than blocked when the upstream fails. Circular `start_after` dependencies are not detected and will stall the involved steps.

### Child pipelines inside dynamic pipelines

Dynamic pipelines can call other dynamic pipelines from their `@pipeline` body. This is useful for composing larger workflows out of reusable dynamic building blocks.

Key behavior:

* Only dynamic pipelines can be called as child pipelines.
* Child pipelines run on the same stack as the parent run.
* Child pipelines can run synchronously (`child(...)`) or concurrently (`child.submit(...)`).
* Child pipeline calls are only allowed in pipeline bodies, not inside step functions.
* Child pipelines reuse the parent run's Docker image — they don't trigger a new build. The child snapshot inherits the parent's build, code reference, and code path so the child runs against the exact same image and source bundle as the parent.

Child pipeline outputs are returned as artifact references:

* `None`
* A single output artifact
* A tuple of output artifacts

These outputs can be passed directly to downstream steps.

```python
from zenml import pipeline, step

@step
def produce_number() -> int:
    return 42

@pipeline(dynamic=True)
def child_pipeline():
    return produce_number()

@step
def consume_number(value: int) -> None:
    print(value)

@pipeline(dynamic=True)
def parent_pipeline():
    child_output = child_pipeline()
    consume_number(child_output)
```

For concurrent execution, use `submit()` and wait on the future:

```python
@pipeline(dynamic=True)
def parent_pipeline_concurrent():
    future = child_pipeline.submit()
    child_output = future.result()
    consume_number(child_output)
```

### Inline child pipelines with `embed(...)`

Use `child_pipeline.embed(...)` if you want to reuse another dynamic pipeline's body without creating a child pipeline run.

```python
@pipeline(dynamic=True)
def parent_pipeline_inline():
    # Executes child steps in the parent run context
    child_output = child_pipeline.embed()
    consume_number(child_output)
```

`embed(...)` behavior:

* It executes the child pipeline entrypoint inline as part of the parent run.
* It does not create a separate child run in the dashboard.
* It is only valid inside a dynamic pipeline body.
* It is not allowed inside `@step` functions.

{% hint style="warning" %}
**Limitations of `embed(...)`.** Unlike `child_pipeline(...)` and `child_pipeline.submit(...)`, the inline form does not apply the child pipeline's own configuration. The parent run's configuration governs every step that runs inline:

* Child-level `settings`, `retry`, `enable_cache`, `enable_step_logs`, `environment`, `secrets`, `tags`, `substitutions`, `model`, and `on_init` / `on_success` / `on_failure` / `on_cleanup` hooks are ignored.
* Per-step Docker overrides on the child pipeline are also ignored — the parent's image is used for any inline isolated step.
* `depends_on` config templates declared on the child pipeline are not picked up.
* There is no failure isolation: an exception inside the inline body aborts the parent run.

If any of these matter to your use case, call the child as `child_pipeline(...)` (sync) or `child_pipeline.submit(...)` (concurrent) instead. Both create a real child run with its own configuration applied.
{% endhint %}

In short, use:

* `child_pipeline(...)` for a synchronous child run
* `child_pipeline.submit(...)` for a concurrent child run
* `child_pipeline.embed(...)` for embedded execution in the parent run

{% hint style="warning" %}
**Resume idempotency depends on submit order.** Child pipeline child runs are identified by the order of `child_pipeline(...)` / `child_pipeline.submit(...)` calls in the parent body: the first call to `my_pipeline` becomes `pipeline:my_pipeline`, the second becomes `pipeline:my_pipeline_2`, and so on. On resume, ZenML reuses an existing child run only if the same call appears in the same position. If you reorder, insert, or remove child pipeline calls before existing ones, every subsequent ID shifts and previously completed children are re-executed. Same caveat applies to step invocation IDs.
{% endhint %}

### Build, code, and Docker settings inheritance

Child runs share the parent's orchestration environment, image, and code bundle. This has two consequences worth knowing:

* **No new Docker build.** The child snapshot inherits the parent's `build`, `code_reference`, and `code_path`. The child runs against the exact same image and source bundle as the parent — there is no separate build step, and the child's code/dependencies must already be installed in the parent's image.
* **Pipeline-level Docker settings on the child are ignored.** When a child pipeline (or a child step) declares non-default `docker_settings`, those settings are silently overridden by the parent's. If you need a different image for a step inside a child pipeline, configure that step with a `step_operator` or use `runtime="isolated"` together with stack-level resource configuration on the parent.

This applies to all three call modes (`child(...)`, `child.submit(...)`, and `child.embed(...)`).

### Permissions and authentication for nested runs

Nested runs orchestrate from the parent's environment, so they share the parent's API token. The token must be scoped to the **root** run of the nesting tree — the root orchestrator can mint per-child-run tokens for any descendant. Child runs cannot mint tokens for their siblings; only descendants of the same root tree are reachable from a given parent token.

This is transparent for the default flow (the root orchestrator launches everything in the same environment). It matters if you build automation on top of `ZENML_PIPELINE_RUN_ID` tokens — those tokens give you read/update access to the run they were minted for and any of its descendants, but not to siblings or unrelated runs.

### Config Templates with `depends_on`

You can use YAML configuration files to provide default parameters for steps using the `depends_on` parameter:

```yaml
# config.yaml
steps:
  some_step:
    parameters:
      arg: 3
```

```python
# run.py
from zenml import step, pipeline

@step
def some_step(arg: int) -> None:
    print(f"arg is {arg}")

@pipeline(dynamic=True, depends_on=[some_step])
def dynamic_pipeline():
    some_step()

if __name__ == "__main__":
    dynamic_pipeline.with_options(config_path="config.yaml")()
```

The `depends_on` parameter tells ZenML which steps can be configured via the YAML file. This is particularly useful when you want to allow users to configure pipeline behavior without modifying code.

### Pass pipeline parameters when running snapshots from the server

When running a snapshot from the server (either via the UI or the SDK/Rest API), you can now pass pipeline parameters for your dynamic pipelines.

For example:

```python
from zenml.client import Client

Client().trigger_pipeline(snapshot_id=<ID>, run_configuration={"parameters": {"my_param": 3}})
```

## Limitations and Known Issues

### Execution modes

When using the `FAIL_FAST` execution mode, failure of a step does not immediately cancel other **inline** steps. Instead, they continue executing until finished. **Isolated** steps on the other hand will be shut down immediately.

### Orchestrator Support

Dynamic pipelines are currently only supported by:

| Orchestrator                                                                                        | Isolated steps | Handles orchestration environment failures |
| --------------------------------------------------------------------------------------------------- | :------------: | :----------------------------------------: |
| [LocalOrchestrator](https://docs.zenml.io/stacks/stack-components/orchestrators/local)              |        ❌       |                      ❌                     |
| [LocalDockerOrchestrator](https://docs.zenml.io/stacks/stack-components/orchestrators/local-docker) |        ❌       |                      ❌                     |
| [KubernetesOrchestrator](https://docs.zenml.io/stacks/stack-components/orchestrators/kubernetes)    |        ✅       |                      ✅                     |
| [VertexOrchestrator](https://docs.zenml.io/stacks/stack-components/orchestrators/vertex)            |        ✅       |                      ❌                     |
| [SagemakerOrchestrator](https://docs.zenml.io/stacks/stack-components/orchestrators/sagemaker)      |        ✅       |                      ❌                     |
| [AzureMLOrchestrator](https://docs.zenml.io/stacks/stack-components/orchestrators/azureml)          |        ✅       |                      ❌                     |

### Artifact Loading

When you call `.load()` on an artifact in a dynamic pipeline, it synchronously loads the data. For large artifacts or when you want to maintain parallelism, consider passing the step outputs (future or artifact) directly to downstream steps instead of loading them.

### Mapping Limitations

* Mapping is currently supported only over artifacts produced within the same pipeline run (mapping over raw data or external artifacts is not supported).
* Chunk size for mapped collection loading defaults to 1 and is not yet configurable.

## Best Practices

1. **Use `runtime="isolated"` for parallel steps**: This ensures better resource isolation and prevents interference between concurrent step executions.
2. **Handle step outputs appropriately**: If you need the data immediately, use `.load()`. If you're just passing to another step, pass the output directly.
3. **Be mindful of resource usage**: Running many steps in parallel can consume significant resources. Monitor your orchestrator's resource limits.
4. **Test incrementally**: Start with simple dynamic pipelines and gradually add complexity. Dynamic pipelines can be harder to debug than static ones.
5. **Use config templates for flexibility**: The `depends_on` feature allows you to make pipelines configurable without code changes.

## When to Use Dynamic Pipelines

Dynamic pipelines are ideal for:

* **AI agent orchestration**: Coordinating multiple autonomous agents (e.g., retrieval or reasoning agents) whose interactions or number of invocations are determined at runtime
* **Hyperparameter tuning**: Spawning multiple training runs with different configurations
* **Data processing**: Processing variable numbers of data chunks in parallel
* **Conditional workflows**: Adapting pipeline structure based on runtime data
* **Dynamic batching**: Creating batches based on available data
* **Multi-agent and collaborative AI workflows**: Building flexible, adaptive workflows where agents or LLM-driven components can be dynamically spawned, routed, or looped based on outputs, results, or user input

For most standard ML workflows, traditional static pipelines are simpler and more maintainable. Use dynamic pipelines when you specifically need runtime flexibility that static pipelines cannot provide.

## Real-World Example: Hierarchical Document Search

The [`examples/hierarchical_doc_search_agent`](https://github.com/zenml-io/zenml/tree/main/examples/hierarchical_doc_search_agent/README.md) example combines dynamic pipelines with Pydantic AI agents for intelligent document traversal. It demonstrates:

* Using `.with_options()` to pass parameters vs artifacts
* The `.chunk()` vs `.load()` pattern: chunks for wiring the DAG, loads for making traversal decisions
* Spawning steps dynamically based on AI agent decisions

Each `traverse_node` call appears as a separate step in the DAG, created at runtime based on what the agent decides to explore.

Two other examples are useful when you want to see dynamic pipelines in more specialized settings:

* [`examples/rlm_document_analysis`](https://github.com/zenml-io/zenml/tree/main/examples/rlm_document_analysis/README.md) shows a Recursive Language Model style document-analysis workflow. ZenML decides how many chunk-processing steps to create at runtime, while the LLM loop inside each chunk decides which typed search tools to use.
* [`examples/optuna_hyperparameter_tuning`](https://github.com/zenml-io/zenml/tree/main/examples/optuna_hyperparameter_tuning/README.md) combines Optuna's ask API with ZenML dynamic pipelines. Optuna decides which hyperparameters to try next; ZenML runs the trials, tracks their artifacts and metadata, and can fan the work out in parallel.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Command Steps

Run arbitrary commands as pipeline steps

A command step runs an arbitrary command as a step in your pipeline instead of a Python function. Use the `CommandStep` class to wrap any command, and add it to a pipeline like any other step:

```python
from zenml import CommandStep, pipeline

train = CommandStep(command=["python", "train.py"])
report = CommandStep(command=["bash", "-c", "echo 'done'"])

@pipeline(dynamic=True, enable_cache=False)
def my_pipeline() -> None:
    train()
    report()

if __name__ == "__main__":
    my_pipeline()
```

The command is whatever you would type in a shell, split into a list. The image just needs to contain the binary you invoke. Shell features like pipes and `&&` need an explicit shell, for example `["bash", "-c", "a | b"]`.

## Running a Python function

Instead of a command, you can pass a Python function. ZenML extracts the source code of the function and runs it with `python -c`:

```python
from zenml import CommandStep

def train() -> None:
    import json

    print(json.dumps({"status": "training"}))

train_step = CommandStep(command=train)
```

The extracted source code is the entire program, nothing around the function travels with it. This means the function must be self-contained:

* All imports must happen inside the function body. Module-level imports are not available.
* The function must not reference module-level variables, constants, or other functions.
* The function must not take any parameters, be decorated, or be defined inside another function.

ZenML rejects parameters, decorators, and references to enclosing functions when the step is created. The execution environment only needs a `python` binary on the `PATH`, ZenML does not have to be installed.

## Running without ZenML in the image (dynamic pipelines only)

A regular step runs your Python function inside the container, so the image has to contain `zenml`, your step code, and all of its dependencies. A command step does not. ZenML treats the command as a black box and never imports anything inside the container, so you can run an image that does not have `zenml` installed.

Point a command step at such an image with the existing Docker settings:

```python
from zenml import CommandStep
from zenml.config import DockerSettings

train = CommandStep(
    command=["python", "train.py"],
    step_operator="sagemaker",
    settings={
        "docker": DockerSettings(skip_build=True, parent_image="my-registry/train:latest")
    },
)
```

The image must be pullable by the execution backend and already carry your code and libraries.

## Where command steps run

A command step follows the same execution routing as any other step. It can run:

* On a **step operator**, in both static and dynamic pipelines.
* As an **isolated step** (dynamic pipelines only)
* **Locally as a subprocess** as an inline step (dynamic pipelines only)

A static pipeline without a step operator is rejected at compile time. Attach a step operator, or use a dynamic pipeline.

## Configuring a command step

A command step is a regular step. It takes the same options as any other step (step operator, settings, resources, retry, environment, secrets, and so on), either in the constructor or through `.with_options()` and `.configure()`:

```python
train = CommandStep(command=["python", "train.py"], step_operator="sagemaker")

@pipeline(dynamic=True)
def my_pipeline() -> None:
    train.with_options(environment={"EPOCHS": "10"}, secrets=["my_api_key"])()
```

Environment variables and secrets are passed to the command as environment variables. The step succeeds when the command exits with status `0` and fails on any non-zero exit. See [Configuration](/concepts/steps_and_pipelines/configuration) for the full set of options and the difference between `.with_options()` and `.configure()`.

## Multi-node distributed training

For **multi-node** distributed training, a command step is the recommended launcher: point the command at a tool that owns the worker gang (TorchX, Ray) while ZenML owns the run. It's also one option for single-node multi-GPU (for example wrapping `torchrun`), though a native step works there too. See [Train with GPUs and Accelerate](https://github.com/zenml-io/zenml/tree/main/docs/book/user-guide/tutorial/distributed-training.md) for all the patterns and worked examples.

## Limitations

* Command steps do not support inputs and outputs.
* Your code is not downloaded into the execution environment.
* `get_step_context()`, metadata, visualizations, and tags are not available inside the command.
* Logs of command steps are not tracked by ZenML. They stay in the backend's native logging (for example CloudWatch or pod logs).
* Step hooks are not allowed.
* Steps in static pipelines without a step operator are not supported.
* Functions passed as commands must be self-contained (see [Running a Python function](#running-a-python-function)).


# Streaming Events

Publish live events from inside a step to subscribed clients.

ZenML pipelines can send live events from inside a running step to any subscriber listening on the server. Use it for LLM token streaming, progress updates from long-running steps, real-time dashboards, and similar cases where you need to surface intermediate output before the step returns.

This page covers the producer-side Python API — calling `zenml.streaming.publish()` from inside a step. For enabling streaming on the server and the HTTP/SSE wire contract, see [Live event streaming](/deploying-zenml/deploying-zenml/live-event-streaming).

{% hint style="info" %}
Streaming is off by default and must be enabled on the server. Pipelines that don't call `zenml.streaming.publish()` are unaffected. Once a server returns `501 Not Implemented`, the producer self-mutes for the rest of the process and further `publish()` calls return without sending HTTP.
{% endhint %}

{% hint style="warning" %}
Streaming is best-effort, not persistent storage. Events are size-capped, can be dropped under load, and disappear when the broker's retention window elapses. **Once an event is lost it's gone — ZenML keeps no secondary copy.** If you need to keep something, write it as [run metadata](/concepts/metadata) or an [artifact](/concepts/artifacts).
{% endhint %}

## Publish from inside a step

```python
from zenml import step
from zenml.streaming import publish

@step
def my_streaming_step() -> str:
    publish({"phase": "warmup"})
    for i in range(10):
        publish({"i": i, "msg": f"working on item {i}"})
    publish({"phase": "done"})
    return "ok"
```

`publish()` reads the current pipeline run and step from the step context — inside `@step`-decorated functions you don't need to pass any handle. The call does not block; events are queued and a background thread sends them to the server in small batches.

## The `publish()` API

```python
from zenml.streaming import publish, flush

publish(
    payload: Dict[str, Any],
    *,
    kind: str = "event",
    correlation_id: Optional[str] = None,
    index: Optional[int] = None,
) -> None
```

* `payload`: any JSON-serializable dict. Each event is limited to 64 KiB on the wire envelope; the check runs inside `publish()` so oversize payloads fail locally rather than after an HTTP round-trip.
* `kind`: a free-form label that clients can filter on.
* `correlation_id` / `index`: opaque to ZenML — passed through to consumers unchanged so clients can group or order events that belong to the same logical sub-flow (for example, one `correlation_id` per LLM generation, or one per tool call). Consumers can filter by `correlation_id` on the SSE endpoint.

### Picking a `kind`

`kind` is also the SSE `event:` field on the wire — clients subscribe with `addEventListener("token", ...)`. A small set of stable names that your consumers code against is easier to maintain than ad-hoc labels. Common choices: `token` (LLM streaming), `progress` (step progress), `status` (state changes), `log` (free-form log lines).

### Grouping with `correlation_id`

When a single step emits events for multiple parallel sub-flows — for example, an agent that issues several LLM generations in parallel — use `correlation_id` to tag each event so consumers can separate them. The field is opaque to ZenML (no validation, no interpretation); the only contract is that consumers can filter on equality.

```python
publish({"text": chunk}, kind="token", correlation_id="gen-42", index=i)
```

A consumer can then subscribe with `?correlation_ids=gen-42` to receive only that generation's tokens.

## `flush()` — wait for delivery

```python
from zenml.streaming import flush

published = flush(timeout=2.0)  # True if the queue drained
```

A background worker drains the queue continuously, and `atexit` catches process termination, so most users don't need to call `flush()`. Call it when you need to be sure a specific event has reached the server before doing something else — for example, posting a "ready" event right before sending an external webhook whose target will consume the stream.

## Inside dynamic pipelines

Inside the body of a `@pipeline(dynamic=True)` (outside any `@step`), `publish()` attributes events to the pipeline run with no `step_run_id` or `step_name`. See [Dynamic Pipelines](/concepts/steps_and_pipelines/dynamic_pipelines).

Calls to `publish()` made outside any pipeline or step context (for example, module-level code, a REPL, scripts) are dropped after a `debug`-level log line — there is no run to attribute the event to.

## Producer-side limits

* The in-process queue holds up to **4 096 events** per Python process. Once full, the oldest queued event is dropped to make room — `publish()` itself never blocks.
* Each event payload is limited to **64 KiB** on the wire envelope.
* The publisher batches up to **64 events per HTTP request** by default. Override with `ZENML_STREAM_PUBLISHER_BATCH_SIZE` if your producer needs higher throughput. Keep it ≤ 1 000 (the server's batch cap); larger values cause every send to fail validation server-side.
* Publishing requires `UPDATE` permission on the run.

For the wire format, delivery semantics, the consumer protocol, and server configuration, see [Live event streaming](/deploying-zenml/deploying-zenml/live-event-streaming).


# Artifacts

Learn how ZenML manages data artifacts, tracks versioning and lineage, and enables effective data flow between steps.

Artifacts are a cornerstone of ZenML's ML pipeline management system. This guide explains what artifacts are, how they work, and how to use them effectively in your pipelines.

### Artifacts in the Pipeline Workflow

Here's how artifacts fit into the ZenML pipeline workflow:

1. A step produces data as output
2. ZenML automatically stores this output as an artifact
3. Other steps can use this artifact as input
4. ZenML tracks the relationships between artifacts and steps

This system creates a complete data lineage for every artifact in your ML workflows, enabling reproducibility and traceability.

## Basic Artifact Usage

### Creating Artifacts (Step Outputs)

Any value returned from a step becomes an artifact:

```python
from zenml import pipeline, step
import pandas as pd

@step
def create_data() -> pd.DataFrame:
    """Creates a dataframe that becomes an artifact."""
    return pd.DataFrame({
        "feature_1": [1, 2, 3],
        "feature_2": [4, 5, 6],
        "target": [10, 20, 30]
    })

@step
def create_prompt_template() -> str:
    """Creates a prompt template that becomes an artifact."""
    return """
    You are a helpful customer service agent. 
    
    Customer Query: {query}
    Previous Context: {context}
    
    Please provide a helpful response following our company guidelines.
    """
```

### Consuming Artifacts (Step Inputs)

You can use artifacts by receiving them as inputs to other steps:

```python
@step
def process_data(df: pd.DataFrame) -> pd.DataFrame:
    """Takes an artifact as input and returns a new artifact."""
    df["feature_3"] = df["feature_1"] * df["feature_2"]
    return df

@step
def test_agent_response(prompt_template: str, test_query: str) -> dict:
    """Uses a prompt template artifact to test agent responses."""
    filled_prompt = prompt_template.format(
        query=test_query, 
        context="Previous customer complained about delayed shipping"
    )
    # Your agent logic here
    response = call_llm_agent(filled_prompt)
    return {"query": test_query, "response": response, "prompt_used": filled_prompt}

@pipeline
def simple_pipeline():
    """Pipeline that creates and processes artifacts."""
    # Traditional ML artifacts
    data = create_data()  # Produces an artifact
    processed_data = process_data(data)  # Uses and produces artifacts
    
    # AI agent artifacts
    prompt = create_prompt_template()  # Produces a prompt artifact
    agent_test = test_agent_response(prompt, "Where is my order?")  # Uses prompt artifact
```

### Artifacts vs. Parameters

When calling a step, inputs can be either artifacts or parameters:

* **Artifacts** are outputs from other steps in the pipeline. They are tracked, versioned, and stored in the artifact store.
* **Parameters** are literal values provided directly to the step. They aren't stored as artifacts but are recorded with the pipeline run.

```python
import pandas as pd
from zenml import step, pipeline

@step
def train_model(data: pd.DataFrame, learning_rate: float) -> object:
    """Step with both artifact and parameter inputs."""
    # data is an artifact (output from another step)
    # learning_rate is a parameter (literal value)
    # Note: create_model would be your own model creation function
    model = create_model(learning_rate)
    model.fit(data)
    return model

@pipeline
def training_pipeline():
    # data is an artifact
    data = create_data()
    
    # data is passed as an artifact, learning_rate as a parameter
    model = train_model(data=data, learning_rate=0.01)
```

Parameters are limited to JSON-serializable values (numbers, strings, lists, dictionaries, etc.). More complex objects should be passed as artifacts.

### Accessing Artifacts After Pipeline Runs

You can access artifacts from completed runs using the ZenML Client:

```python
from zenml.client import Client

# Get a specific run
client = Client()
pipeline_run = client.get_pipeline_run("<PIPELINE_RUN_ID>")

# Get an artifact from a specific step
train_data = pipeline_run.steps["split_data"].outputs["train_data"].load()

# Use the artifact
print(train_data.shape)
```

## Working with Artifact Types

### Type Annotations

Type annotations are important when working with artifacts as they:

1. Help ZenML select the appropriate materializer for storage
2. Validate inputs and outputs at runtime
3. Document the data flow of your pipeline

```python
from typing import Tuple
import numpy as np
import pandas as pd
from zenml import step

@step
def preprocess_data(df: pd.DataFrame) -> np.ndarray:
    """Type annotation tells ZenML this returns a numpy array."""
    return df.values

@step
def split_data(data: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:
    """Type annotation tells ZenML this returns a tuple of numpy arrays."""
    split_point = len(data) // 2
    return data[:split_point], data[split_point:]
```

ZenML supports many common data types out of the box:

* Primitive types (`int`, `float`, `str`, `bool`)
* Container types (`dict`, `list`, `tuple`)
* NumPy arrays
* Pandas DataFrames
* Many ML model formats (through integrations)

### Returning Multiple Outputs

Steps can return multiple artifacts using tuples:

```python
from typing import Tuple, Annotated
import numpy as np

@step
def split_data(
    data: np.ndarray, 
    target: np.ndarray
) -> Tuple[
    Annotated[np.ndarray, "X_train"],
    Annotated[np.ndarray, "X_test"],
    Annotated[np.ndarray, "y_train"],
    Annotated[np.ndarray, "y_test"]
]:
    """Split data into training and testing sets."""
    # Implement split logic
    X_train, X_test = data[:80], data[80:]
    y_train, y_test = target[:80], target[80:]
    
    return X_train, X_test, y_train, y_test
```

ZenML differentiates between:

* A step with multiple outputs: `return a, b` or `return (a, b)`
* A step with a single tuple output: `return some_tuple`

### Naming Your Artifacts

By default, artifacts are named based on their position or variable name:

* Single outputs are named `output`
* Multiple outputs are named `output_0`, `output_1`, etc.

You can give your artifacts more meaningful names using the `Annotated` type:

```python
from typing import Tuple
from typing import Annotated
import pandas as pd
from zenml import step

@step
def split_dataset(
    df: pd.DataFrame
) -> Tuple[
    Annotated[pd.DataFrame, "train_data"],
    Annotated[pd.DataFrame, "test_data"]
]:
    """Split a dataframe into training and testing sets."""
    train = df.sample(frac=0.8, random_state=42)
    test = df.drop(train.index)
    return train, test
```

You can even use dynamic naming with placeholders:

```python
from typing import Annotated
import pandas as pd
from zenml import step, pipeline

@step
def extract_data(source: str) -> Annotated[pd.DataFrame, "{dataset_type}_data"]:
    """Extract data with a dynamically named output."""
    # Implementation...
    data = pd.DataFrame()  # Your data extraction logic here
    return data

@pipeline
def data_pipeline():
    # These will create artifacts named "train_data" and "test_data"
    train_df = extract_data.with_options(
        substitutions={"dataset_type": "train"}
    )(source="train_source")
    
    test_df = extract_data.with_options(
        substitutions={"dataset_type": "test"}
    )(source="test_source")
```

ZenML supports these placeholders:

* `{date}`: Current date (e.g., "2023\_06\_15")
* `{time}`: Current time (e.g., "14\_30\_45\_123456")
* Custom placeholders can be defined using `substitutions`

## How Artifacts Work Under the Hood

### Materializers: How Data Gets Stored

Materializers are a key concept in ZenML's artifact system. They handle:

* **Serializing data** when saving artifacts to storage
* **Deserializing data** when loading artifacts from storage
* **Generating visualizations** for the dashboard
* **Extracting metadata** for tracking and searching

When a step produces an output, ZenML automatically selects the appropriate materializer based on the data type (using type annotations). ZenML includes built-in materializers for common data types like:

* Primitive types (`int`, `float`, `str`, `bool`)
* Container types (`dict`, `list`, `tuple`)
* NumPy arrays, Pandas DataFrames and many other ML-related formats (through integrations)

Here's how materializers work in practice:

```python
from zenml import step
from sklearn.linear_model import LinearRegression

@step
def train_model(X_train, y_train) -> LinearRegression:
    """Train a model and return it as an artifact."""
    model = LinearRegression()
    model.fit(X_train, y_train)
    return model  # ZenML uses a specific materializer for scikit-learn models
```

For custom data types, you can create your own materializers. See the [Materializers](/concepts/artifacts/materializers) guide for details.

### Lineage and Caching

ZenML automatically tracks the complete lineage of each artifact:

* Which step produced it
* Which pipeline run it belongs to
* Which other artifacts it depends on
* Which steps have consumed it

This lineage tracking enables powerful caching capabilities. When you run a pipeline, ZenML checks if any steps have been run before with the same inputs, code, and configuration. If so, it reuses the cached outputs instead of rerunning the step:

```python
@pipeline
def cached_pipeline():
    # If create_data has been run before with the same code and inputs,
    # the cached artifact will be used
    data = create_data()
    
    # If process_data has been run before with the same code and inputs
    # (including the exact same data artifact), the cached output will be used
    processed_data = process_data(data)
```

## Advanced Artifact Usage

### Accessing Artifacts from Previous Runs

You can access artifacts from any previous run by name or ID:

```python
from zenml.client import Client

# Get a specific artifact version
artifact = Client().get_artifact_version("my_model", "1.0")

# Get the latest version of an artifact
latest_artifact = Client().get_artifact_version("my_model")

# Load it into memory
model = latest_artifact.load()
```

You can also access artifacts within steps:

```python
from zenml.client import Client
from zenml import step

@step
def evaluate_against_previous(model, X_test, y_test) -> float:
    """Compare current model with the previous best model."""
    client = Client()
    
    # Get the previous best model
    best_model = client.get_artifact_version("best_model")
    
    # Use it for comparison
    previous_accuracy = best_model.data.score(X_test, y_test)
    current_accuracy = model.score(X_test, y_test)
    
    return current_accuracy - previous_accuracy
```

### Cross-Pipeline Artifact Usage

You can use artifacts produced by one pipeline in another pipeline:

```python
from zenml.client import Client
from zenml import step, pipeline

@step
def use_trained_model(data: pd.DataFrame, model) -> pd.Series:
    """Use a model loaded from a previous pipeline run."""
    return pd.Series(model.predict(data))

@pipeline
def inference_pipeline():
    # Load data
    data = load_data()
    
    # Get the latest model from another pipeline
    model = Client().get_artifact_version("trained_model")
    
    # Use it for predictions
    predictions = use_trained_model(data=data, model=model)
```

This allows you to build modular pipelines that can work together as part of a larger ML system.

### Visualizing Artifacts

ZenML automatically generates visualizations for many types of artifacts, viewable in the dashboard:

```python
# You can also view visualizations in notebooks
from zenml.client import Client

artifact = Client().get_artifact_version("<ARTIFACT_NAME>")
artifact.visualize()
```

For detailed information on visualizations, see [Visualizations](/concepts/artifacts/visualizations).

### Managing Artifacts

Individual artifacts cannot be deleted directly (to prevent broken references). However, you can clean up unused artifacts:

```bash
zenml artifact prune
```

This deletes artifacts that are no longer referenced by any pipeline run. You can control this behavior with flags:

* `--only-artifact`: Only delete the physical files, keep database entries
* `--only-metadata`: Only delete database entries, keep files
* `--ignore-errors`: Continue pruning even if some artifacts can't be deleted

### Registering Existing Data as Artifacts

Sometimes, you may have data created externally (outside of ZenML pipelines) that you want to use within your ZenML workflows. Instead of reading and materializing this data within a step, you can register existing files or folders as ZenML artifacts directly.

#### Register an Existing Folder

To register a folder as a ZenML artifact:

```python
from zenml.client import Client
from zenml import register_artifact
import os
from pathlib import Path

# Path to an existing folder in your artifact store
prefix = Client().active_stack.artifact_store.path
existing_folder = os.path.join(prefix, "my_folder")

# Register it as a ZenML artifact
register_artifact(
    folder_or_file_uri=existing_folder,
    name="my_folder_artifact"
)

# Later, load the artifact
folder_path = Client().get_artifact_version("my_folder_artifact").load()
assert isinstance(folder_path, Path)
assert os.path.isdir(folder_path)
```

#### Register an Existing File

Similarly, you can register individual files:

```python
from zenml.client import Client
from zenml import register_artifact
import os
from pathlib import Path

# Path to an existing file in your artifact store
prefix = Client().active_stack.artifact_store.path
existing_file = os.path.join(prefix, "my_folder/model.pkl")

# Register it as a ZenML artifact
register_artifact(
    folder_or_file_uri=existing_file,
    name="my_model_artifact"
)

# Later, load the artifact
file_path = Client().get_artifact_version("my_model_artifact").load()
assert isinstance(file_path, Path)
assert not os.path.isdir(file_path)
```

This approach is particularly useful for:

* Integrating with external ML frameworks that save their own data
* Working with pre-existing datasets
* Registering model checkpoints created during training

When you load these artifacts, you'll receive a `pathlib.Path` pointing to a temporary location in your executing environment, ready for use as a normal local path.

#### Register Framework Checkpoints

A common use case is registering model checkpoints from training frameworks like PyTorch Lightning:

```python
import os
from uuid import uuid4
from zenml.client import Client
from zenml import register_artifact
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks import ModelCheckpoint

# Define checkpoint location in your artifact store
prefix = Client().active_stack.artifact_store.path
checkpoint_dir = os.path.join(prefix, uuid4().hex)

# Configure PyTorch Lightning trainer with checkpointing
model = YourLightningModel()
trainer = Trainer(
    default_root_dir=checkpoint_dir,
    callbacks=[
        ModelCheckpoint(
            every_n_epochs=1, 
            save_top_k=-1,  # Keep all checkpoints
            filename="checkpoint-{epoch:02d}"
        )
    ],
)

# Train the model
trainer.fit(model)

# Register all checkpoints as a ZenML artifact
register_artifact(
    folder_or_file_uri=checkpoint_dir, 
    name="lightning_checkpoints"
)

# Later, you can load the checkpoint folder
checkpoint_path = Client().get_artifact_version("lightning_checkpoints").load()
```

You can also extend the `ModelCheckpoint` callback to register each checkpoint as a separate artifact version during training. This approach enables better version control of intermediate checkpoints.

## Conclusion

Artifacts are a central part of ZenML's approach to ML pipelines. They provide:

* Automatic versioning and lineage tracking
* Efficient storage and caching
* Type-safe data handling
* Visualization capabilities
* Cross-pipeline data sharing

Whether you're working with traditional ML models, prompt templates, agent configurations, or evaluation datasets, ZenML's artifact system treats them all uniformly. This enables you to apply the same MLOps principles across your entire AI stack - from classical ML to complex multi-agent systems.

By understanding how artifacts work, you can build more effective, maintainable, and reproducible ML pipelines and AI workflows.

For more information on specific aspects of artifacts, see:

* [Materializers](/concepts/artifacts/materializers): Creating custom serializers for your data types
* [Visualizations](/concepts/artifacts/visualizations): Customizing artifact visualizations

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Materializers

Understanding and creating materializers to handle custom data types in ZenML pipelines

Materializers are a core concept in ZenML that enable the serialization, storage, and retrieval of artifacts in your ML pipelines. This guide explains how materializers work and how to create custom materializers for your specific data types.

## What Are Materializers?

A materializer is a class that defines how a particular data type is:

* **Serialized**: Converted from Python objects to a storable format
* **Saved**: Written to the artifact store
* **Loaded**: Read from the artifact store
* **Deserialized**: Converted back to Python objects
* **Visualized**: Displayed in the ZenML dashboard
* **Analyzed**: Metadata extraction for tracking and search

Materializers act as the bridge between your Python code and the underlying storage system, ensuring that any artifact can be saved, loaded, and visualized correctly, regardless of the data type.

## Built-In Materializers

ZenML includes built-in materializers for many common data types:

### Core Materializers

<table data-full-width="true"><thead><tr><th>Materializer</th><th>Handled Data Types</th><th>Storage Format</th></tr></thead><tbody><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.built_in_materializer">BuiltInMaterializer</a></td><td><code>bool</code>, <code>float</code>, <code>int</code>, <code>str</code>, <code>None</code></td><td><code>.json</code></td></tr><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.built_in_materializer">BytesInMaterializer</a></td><td><code>bytes</code></td><td><code>.txt</code></td></tr><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.built_in_materializer">BuiltInContainerMaterializer</a></td><td><code>dict</code>, <code>list</code>, <code>set</code>, <code>tuple</code></td><td>Directory</td></tr><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.numpy_materializer">NumpyMaterializer</a></td><td><code>np.ndarray</code></td><td><code>.npy</code></td></tr><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.pandas_materializer">PandasMaterializer</a></td><td><code>pd.DataFrame</code>, <code>pd.Series</code></td><td><code>.csv</code> (or <code>.gzip</code> if <code>parquet</code> is installed)</td></tr><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.pydantic_materializer">PydanticMaterializer</a></td><td><code>pydantic.BaseModel</code></td><td><code>.json</code></td></tr><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.dataclass_materializer">DataclassMaterializer</a></td><td>JSON-serializable Python <code>dataclass</code> types</td><td><code>.json</code></td></tr><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.service_materializer">ServiceMaterializer</a></td><td><code>zenml.services.service.BaseService</code></td><td><code>.json</code></td></tr><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.structured_string_materializer">StructuredStringMaterializer</a></td><td><code>zenml.types.CSVString</code>, <code>zenml.types.HTMLString</code>, <code>zenml.types.MarkdownString</code></td><td><code>.csv</code> / <code>.html</code> / <code>.md</code> (depending on type)</td></tr><tr><td><a href="https://sdkdocs.zenml.io/latest/core_code_docs/core-materializers.html#zenml.materializers.path_materializer">PathMaterializer</a></td><td><code>pathlib.Path</code></td><td><code>.tar.gz</code> (directories) or direct copy (files)</td></tr></tbody></table>

ZenML also provides a CloudpickleMaterializer that can handle any object by saving it with [cloudpickle](https://github.com/cloudpipe/cloudpickle). However, this is not production-ready because the resulting artifacts cannot be loaded when running with a different Python version. For production use, you should implement a custom materializer for your specific data types.

{% hint style="info" %}
Pydantic artifacts created by current ZenML versions are stored in `data_v2.json`. ZenML can still load older Pydantic artifacts stored as `data.json` by ZenML `<= 0.94.2`, so existing runs remain readable after an upgrade.
{% endhint %}

### Dataclass artifacts

The `DataclassMaterializer` handles JSON-serializable Python dataclasses without requiring you to write a custom materializer.

```python
from dataclasses import dataclass

from zenml import step

@dataclass
class TrainingConfig:
    learning_rate: float
    epochs: int

@step
def make_config() -> TrainingConfig:
    return TrainingConfig(learning_rate=0.01, epochs=10)
```

This works for dataclasses that Pydantic can serialize to JSON. If your dataclass contains objects such as open file handles, live model objects, database connections, or other arbitrary Python objects, use a custom materializer instead.

### Passing Files and Directories Between Steps

The `PathMaterializer` lets you pass `pathlib.Path` objects between steps. This is especially useful when working with files or directories that need to be shared across steps — for example, dataset directories, exported model files, or any file-based artifacts.

When a step returns a `Path`:

* **Directories** are compressed into a `.tar.gz` archive and uploaded to the artifact store
* **Single files** are copied directly to the artifact store

When a downstream step receives the `Path`, the materializer downloads the contents to a local temporary directory and returns a `Path` pointing to it.

```python
from pathlib import Path
from typing import Annotated

from zenml import step, pipeline


@step
def prepare_dataset(num_samples: int = 100) -> Annotated[Path, "dataset_dir"]:
    """Prepare a dataset directory with training files."""
    output_dir = Path("training_data")
    output_dir.mkdir(exist_ok=True)

    # Write training files into the directory
    (output_dir / "features.csv").write_text("feature1,feature2\n1.0,2.0\n")
    (output_dir / "labels.csv").write_text("label\n1\n")

    # ZenML will tar.gz this directory and upload it to the artifact store
    return output_dir


@step
def train_model(dataset_dir: Path) -> None:
    """Train a model using the dataset directory."""
    # dataset_dir points to a local temp directory with the extracted contents
    features = (dataset_dir / "features.csv").read_text()
    labels = (dataset_dir / "labels.csv").read_text()
    print(f"Training with features: {features}")


@pipeline
def training_pipeline():
    dataset = prepare_dataset()
    train_model(dataset)
```

This works transparently with remote orchestrators (Kubernetes, Vertex AI, etc.) where each step runs on a different pod — the artifact store acts as the shared transport layer.

{% hint style="info" %}
If you prefer the previous behavior where `Path` objects were serialized with `cloudpickle` (which only preserves the path string, not the file contents), you can disable the `PathMaterializer` by setting the environment variable `ZENML_DISABLE_PATH_MATERIALIZER=true`.
{% endhint %}

### Integration-Specific Materializers

When you install ZenML integrations, additional materializers become available:

<table data-full-width="true"><thead><tr><th width="199.5">Integration</th><th width="271">Materializer</th><th width="390">Handled Data Types</th><th>Storage Format</th></tr></thead><tbody><tr><td>bentoml</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-bentoml.html#zenml.integrations.bentoml">BentoMaterializer</a></td><td><code>bentoml.Bento</code></td><td><code>.bento</code></td></tr><tr><td>deepchecks</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-deepchecks.html#zenml.integrations.deepchecks">DeepchecksResultMateriailzer</a></td><td><code>deepchecks.CheckResult</code>, <code>deepchecks.SuiteResult</code></td><td><code>.json</code></td></tr><tr><td>evidently</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-evidently.html#zenml.integrations.evidently">EvidentlyProfileMaterializer</a></td><td><code>evidently.Profile</code></td><td><code>.json</code></td></tr><tr><td>great_expectations</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-great_expectations.html#zenml.integrations.great_expectations">GreatExpectationsMaterializer</a></td><td><code>great_expectations.ExpectationSuite</code>, <code>great_expectations.CheckpointResult</code></td><td><code>.json</code></td></tr><tr><td>huggingface</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-huggingface.html#zenml.integrations.huggingface">HFDatasetMaterializer</a></td><td><code>datasets.Dataset</code>, <code>datasets.DatasetDict</code></td><td>Directory</td></tr><tr><td>huggingface</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-huggingface.html#zenml.integrations.huggingface">HFPTModelMaterializer</a></td><td><code>transformers.PreTrainedModel</code></td><td>Directory</td></tr><tr><td>huggingface</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-huggingface.html#zenml.integrations.huggingface">HFTFModelMaterializer</a></td><td><code>transformers.TFPreTrainedModel</code></td><td>Directory</td></tr><tr><td>huggingface</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-huggingface.html#zenml.integrations.huggingface">HFTokenizerMaterializer</a></td><td><code>transformers.PreTrainedTokenizerBase</code></td><td>Directory</td></tr><tr><td>lightgbm</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-lightgbm.html#zenml.integrations.lightgbm">LightGBMBoosterMaterializer</a></td><td><code>lgbm.Booster</code></td><td><code>.txt</code></td></tr><tr><td>lightgbm</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-lightgbm.html#zenml.integrations.lightgbm">LightGBMDatasetMaterializer</a></td><td><code>lgbm.Dataset</code></td><td><code>.binary</code></td></tr><tr><td>neural_prophet</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-neural_prophet.html#zenml.integrations.neural_prophet">NeuralProphetMaterializer</a></td><td><code>NeuralProphet</code></td><td><code>.pt</code></td></tr><tr><td>pillow</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-pillow.html#zenml.integrations.pillow">PillowImageMaterializer</a></td><td><code>Pillow.Image</code></td><td><code>.PNG</code></td></tr><tr><td>polars</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-polars.html#zenml.integrations.polars">PolarsMaterializer</a></td><td><code>pl.DataFrame</code>, <code>pl.Series</code></td><td><code>.parquet</code></td></tr><tr><td>pycaret</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-pycaret.html#zenml.integrations.pycaret">PyCaretMaterializer</a></td><td>Any <code>sklearn</code>, <code>xgboost</code>, <code>lightgbm</code> or <code>catboost</code> model</td><td><code>.pkl</code></td></tr><tr><td>pytorch</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-pytorch.html#zenml.integrations.pytorch">PyTorchDataLoaderMaterializer</a></td><td><code>torch.Dataset</code>, <code>torch.DataLoader</code></td><td><code>.pt</code></td></tr><tr><td>pytorch</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-pytorch.html#zenml.integrations.pytorch">PyTorchModuleMaterializer</a></td><td><code>torch.Module</code></td><td><code>.pt</code></td></tr><tr><td>scipy</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-scipy.html#zenml.integrations.scipy">SparseMaterializer</a></td><td><code>scipy.spmatrix</code></td><td><code>.npz</code></td></tr><tr><td>spark</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-spark.html#zenml.integrations.spark">SparkDataFrameMaterializer</a></td><td><code>pyspark.DataFrame</code></td><td><code>.parquet</code></td></tr><tr><td>spark</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-spark.html#zenml.integrations.spark">SparkModelMaterializer</a></td><td><code>pyspark.Transformer</code></td><td><code>pyspark.Estimator</code></td></tr><tr><td>tensorflow</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-tensorflow.html#zenml.integrations.tensorflow">KerasMaterializer</a></td><td><code>tf.keras.Model</code></td><td>Directory</td></tr><tr><td>tensorflow</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-tensorflow.html#zenml.integrations.tensorflow">TensorflowDatasetMaterializer</a></td><td><code>tf.Dataset</code></td><td>Directory</td></tr><tr><td>whylogs</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-whylogs.html#zenml.integrations.whylogs">WhylogsMaterializer</a></td><td><code>whylogs.DatasetProfileView</code></td><td><code>.pb</code></td></tr><tr><td>xgboost</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-xgboost.html#zenml.integrations.xgboost">XgboostBoosterMaterializer</a></td><td><code>xgb.Booster</code></td><td><code>.json</code></td></tr><tr><td>xgboost</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-xgboost.html#zenml.integrations.xgboost">XgboostDMatrixMaterializer</a></td><td><code>xgb.DMatrix</code></td><td><code>.binary</code></td></tr><tr><td>jax</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-jax.html#zenml.integrations.jax">JAXArrayMaterializer</a></td><td><code>jax.Array</code></td><td><code>.npy</code></td></tr><tr><td>mlx</td><td><a href="https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-mlx.html#zenml.integrations.mlx">MLXArrayMaterializer</a></td><td><code>mlx.core.array</code></td><td><code>.npy</code></td></tr></tbody></table>

> **Note**: When using Docker-based orchestrators, you must specify the appropriate integrations in your `DockerSettings` to ensure the materializers are available inside the container.

## Creating Custom Materializers

When working with custom data types, you'll need to create materializers to handle them. Here's how:

### 1. Define Your Materializer Class

Create a new class that inherits from `BaseMaterializer`:

```python
import os
import json
from typing import Type, Any, Dict
from zenml.materializers.base_materializer import BaseMaterializer
from zenml.enums import ArtifactType, VisualizationType
from zenml.metadata.metadata_types import MetadataType

# Assume MyClass is your custom class defined elsewhere
# from mymodule import MyClass

class MyClassMaterializer(BaseMaterializer):
    """Materializer for MyClass objects."""
    
    # List the data types this materializer can handle
    ASSOCIATED_TYPES = (MyClass,)
    
    # Define what type of artifact this is (usually DATA or MODEL)
    ASSOCIATED_ARTIFACT_TYPE = ArtifactType.DATA
    
    def load(self, data_type: Type[Any]) -> MyClass:
        """Load MyClass from storage."""
        # Implementation here
        filepath = os.path.join(self.uri, "data.json")
        with self.artifact_store.open(filepath, "r") as f:
            data = json.load(f)
        
        # Create and return an instance of MyClass
        return MyClass(**data)
    
    def save(self, data: MyClass) -> None:
        """Save MyClass to storage."""
        # Implementation here
        filepath = os.path.join(self.uri, "data.json")
        with self.artifact_store.open(filepath, "w") as f:
            json.dump(data.to_dict(), f)
    
    def save_visualizations(self, data: MyClass) -> Dict[str, VisualizationType]:
        """Generate visualizations for the dashboard."""
        # Optional - generate visualizations
        vis_path = os.path.join(self.uri, "visualization.html")
        with self.artifact_store.open(vis_path, "w") as f:
            f.write(data.to_html())
        
        return {vis_path: VisualizationType.HTML}
    
    def extract_metadata(self, data: MyClass) -> Dict[str, MetadataType]:
        """Extract metadata for tracking."""
        # Optional - extract metadata
        return {
            "name": data.name,
            "created_at": data.created_at,
            "num_records": len(data.records)
        }
```

### 2. Using Your Custom Materializer

Once you've defined the materializer, you can use it in your pipeline:

```python
from zenml import step, pipeline
# from mymodule import MyClass, MyClassMaterializer

@step(output_materializers=MyClassMaterializer)
def create_my_class() -> MyClass:
    """Create an instance of MyClass."""
    return MyClass(name="test", records=[1, 2, 3])

@step
def use_my_class(my_obj: MyClass) -> None:
    """Use the MyClass instance."""
    print(f"Name: {my_obj.name}, Records: {my_obj.records}")

@pipeline
def custom_pipeline():
    data = create_my_class()
    use_my_class(data)
```

### 3. Multiple Outputs with Different Materializers

When a step has multiple outputs that need different materializers:

```python
from typing import Tuple, Annotated

@step(output_materializers={
    "obj1": MyClass1Materializer,
    "obj2": MyClass2Materializer
})
def create_objects() -> Tuple[
    Annotated[MyClass1, "obj1"],
    Annotated[MyClass2, "obj2"]
]:
    """Create instances of different classes."""
    return MyClass1(), MyClass2()
```

### 4. Registering a Materializer Globally

You can register a materializer globally to override the default materializer for a specific type:

```python
from zenml.materializers.materializer_registry import materializer_registry
from zenml.materializers.base_materializer import BaseMaterializer
import pandas as pd

# Create a custom pandas materializer
class FastPandasMaterializer(BaseMaterializer):
    # Implementation here
    ...

# Register it for pandas DataFrames globally
materializer_registry.register_and_overwrite_type(
    key=pd.DataFrame, 
    type_=FastPandasMaterializer
)
```

## Materializer Implementation Details

When implementing a custom materializer, consider these aspects:

### Handling Storage

The `self.uri` property contains the path to the directory where your artifact should be stored. Use this path to create files or subdirectories for your data.

When reading or writing files, always use `self.artifact_store.open()` rather than direct file I/O to ensure compatibility with different artifact stores (local filesystem, cloud storage, etc.).

### Visualization Support

The `save_visualizations()` method allows you to create visualizations that will be shown in the ZenML dashboard. You can return multiple visualizations of different types:

* `VisualizationType.HTML`: Embedded HTML content
* `VisualizationType.MARKDOWN`: Markdown content
* `VisualizationType.IMAGE`: Image files
* `VisualizationType.CSV`: CSV tables

**Configuring Visualizations**

Some materializers support configuration via environment variables to customize their visualization behavior. For example:

* `ZENML_PANDAS_SAMPLE_ROWS`: Controls the number of rows shown in sample visualizations created by the `PandasMaterializer`. Default is 10 rows.

### Metadata Extraction

The `extract_metadata()` method allows you to extract key information about your artifact for indexing and searching. This metadata will be displayed alongside the artifact in the dashboard.

### Temporary Files

If you need a temporary directory while processing artifacts, use the `get_temporary_directory()` helper:

```python
with self.get_temporary_directory() as temp_dir:
    # Process files in the temporary directory
    # Files will be automatically cleaned up
```

### Example: A Complete Materializer

Here's a complete example of a custom materializer for a simple class:

```python
import os
import json
from typing import Type, Any, Dict
from zenml.materializers.base_materializer import BaseMaterializer
from zenml.enums import ArtifactType

class MyObj:
    def __init__(self, name: str):
        self.name = name
    
    def to_dict(self):
        return {"name": self.name}
    
    @classmethod
    def from_dict(cls, data):
        return cls(name=data["name"])

class MyMaterializer(BaseMaterializer):
    """Materializer for MyObj objects."""
    
    ASSOCIATED_TYPES = (MyObj,)
    ASSOCIATED_ARTIFACT_TYPE = ArtifactType.DATA
    
    def load(self, data_type: Type[Any]) -> MyObj:
        """Load MyObj from storage."""
        filepath = os.path.join(self.uri, "data.json")
        with self.artifact_store.open(filepath, "r") as f:
            data = json.load(f)
        
        return MyObj.from_dict(data)
    
    def save(self, data: MyObj) -> None:
        """Save MyObj to storage."""
        filepath = os.path.join(self.uri, "data.json")
        with self.artifact_store.open(filepath, "w") as f:
            json.dump(data.to_dict(), f)

# Usage in a pipeline
@step(output_materializers=MyMaterializer)
def create_my_obj() -> MyObj:
    return MyObj(name="my_object")

@step
def use_my_obj(my_obj: MyObj) -> None:
    print(f"Object name: {my_obj.name}")

@pipeline
def my_pipeline():
    obj = create_my_obj()
    use_my_obj(obj)
```

## Unmaterialized artifacts

Whenever you pass artifacts as outputs from one pipeline step to other steps as inputs, the corresponding materializer for the respective data type defines how this artifact is first serialized and written to the artifact store, and then deserialized and read in the next step.handle-custom-data-types. However, there are instances where you might **not** want to materialize an artifact in a step, but rather use a reference to it instead. This is where skipping materialization comes in.

{% hint style="warning" %}
Skipping materialization might have unintended consequences for downstream tasks that rely on materialized artifacts. Only skip materialization if there is no other way to do what you want to do.
{% endhint %}

#### How to skip materialization

While materializers should in most cases be used to control how artifacts are returned and consumed from pipeline steps, you might sometimes need to have a completely unmaterialized artifact in a step, e.g., if you need to know the exact path to where your artifact is stored.

An unmaterialized artifact is a [`zenml.materializers.UnmaterializedArtifact`](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifacts.html#zenml.artifacts.unmaterialized_artifact). Among others, it has a property `uri` that points to the unique path in the artifact store where the artifact is persisted. One can use an unmaterialized artifact by specifying `UnmaterializedArtifact` as the type in the step:

```python
from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact
from zenml import step

@step
def my_step(my_artifact: UnmaterializedArtifact):  # rather than pd.DataFrame
    pass
```

The following shows an example of how unmaterialized artifacts can be used in the steps of a pipeline. The pipeline we define will look like this:

```shell
s1 -> s3 
s2 -> s4
```

`s1` and `s2` produce identical artifacts, however `s3` consumes materialized artifacts while `s4` consumes unmaterialized artifacts. `s4` can now use the `dict_.uri` and `list_.uri` paths directly rather than their materialized counterparts.

```python
from typing import Annotated
from typing import Dict, List, Tuple

from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact
from zenml import pipeline, step


@step
def step_1() -> Tuple[
    Annotated[Dict[str, str], "dict_"],
    Annotated[List[str], "list_"],
]:
    return {"some": "data"}, []


@step
def step_2() -> Tuple[
    Annotated[Dict[str, str], "dict_"],
    Annotated[List[str], "list_"],
]:
    return {"some": "data"}, []


@step
def step_3(dict_: Dict, list_: List) -> None:
    assert isinstance(dict_, dict)
    assert isinstance(list_, list)


@step
def step_4(
        dict_: UnmaterializedArtifact,
        list_: UnmaterializedArtifact,
) -> None:
    print(dict_.uri)
    print(list_.uri)


@pipeline
def example_pipeline():
    step_3(*step_1())
    step_4(*step_2())


example_pipeline()
```

You can see another example of using an `UnmaterializedArtifact` when triggering a [pipeline from another](/concepts/snapshots#advanced-usage-running-snapshots-from-other-pipelines).

## Best Practices

When working with materializers:

1. **Prefer structured formats** over pickle or other binary formats for better cross-environment compatibility.
2. **Test your materializer** with different artifact stores (local, S3, etc.) to ensure it works consistently.
3. **Consider versioning** if your data structure might change over time.
4. **Create visualizations** to help users understand your artifacts in the dashboard.
5. **Extract useful metadata** to make artifacts easier to find and understand.
6. **Be explicit** about materializer assignments for clarity, even if ZenML can detect them automatically.
7. **Avoid using the CloudpickleMaterializer** in production as it's not reliable across different Python versions.

## Conclusion

Materializers are a powerful part of ZenML's artifact system, enabling proper storage and handling of any data type. By creating custom materializers for your specific data structures, you ensure that your ML pipelines are robust, efficient, and can handle any data type required by your workflows.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Visualizations

Learn how to visualize the data artifacts produced by your ZenML pipelines.

Data visualization is a powerful tool for understanding your ML pipeline outputs. ZenML provides built-in capabilities to visualize artifacts, helping you gain insights into your data, model performance, and pipeline execution.

## Accessing Visualizations

ZenML automatically generates visualizations for many common data types, making it easy to inspect your artifacts without additional code.

### Dashboard Visualizations

The ZenML dashboard displays visualizations for artifacts produced by your pipeline runs:

To view visualizations in the dashboard:

1. Navigate to the **Runs** tab
2. Select a specific pipeline run
3. Click on any step to view its outputs
4. Select an artifact to view its visualizations

![ZenML Artifact Visualizations](/files/9x0wQrdBACrOzLs5xHHy)

### Notebook Visualizations

You can also display artifact visualizations in Jupyter notebooks using the `visualize()` method:

```python
from zenml.client import Client

# Get an artifact from a previous pipeline run
run = Client().get_pipeline_run("<PIPELINE_RUN_ID>")
artifact = run.steps["<STEP_NAME>"].outputs[<OUTPUT_NAME>][0]

# Display the visualization
artifact.visualize()
```

![output.visualize() Output](/files/MI9c3qihl86PZKadqlQi)

## Supported Visualization Types

ZenML supports visualizations for many common data types out of the box:

* A statistical representation of a [Pandas](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) Dataframe represented as a png image.
* Drift detection reports by [Evidently](https://docs.zenml.io/stacks/stack-components/data-validators/evidently), [Great Expectations](https://docs.zenml.io/stacks/stack-components/data-validators/great-expectations), and [whylogs](https://docs.zenml.io/stacks/stack-components/data-validators/whylogs).
* A [Hugging Face](https://zenml.io/integrations/huggingface) datasets viewer embedded as a HTML iframe.

![output.visualize() output for the Hugging Face datasets viewer](/files/7b0i2ACfMbSuIqXig0Bb)

## Creating Custom Visualizations

It is simple to associate a custom visualization with an artifact in ZenML, if the visualization is one of the supported visualization types. Currently, the following visualization types are supported:

* **HTML:** Embedded HTML visualizations such as data validation reports,
* **Image:** Visualizations of image data such as Pillow images (e.g. `PIL.Image`) or certain numeric numpy arrays,
* **CSV:** Tables, such as the pandas DataFrame `.describe()` output,
* **Markdown:** Markdown strings or pages.
* **JSON:** JSON strings or objects.

There are three ways how you can add custom visualizations to the dashboard:

* If you are already handling HTML, Markdown, CSV or JSON data in one of your steps, you can have them visualized in just a few lines of code by casting them to a [special class](#visualization-via-special-return-types) inside your step.
* If you want to automatically extract visualizations for all artifacts of a certain data type, you can define type-specific visualization logic by [building a custom materializer](#visualization-via-materializers).

### Curated Visualizations Across Resources

Curated visualizations let you surface a specific artifact visualization across multiple ZenML resources. Each curated visualization links to exactly one resource—for example, a model performance report that appears on the model detail page, or a deployment health dashboard that shows up in the deployment view.

Curated visualizations currently support the following resources:

* **Projects** – high-level dashboards and KPIs that summarize the state of a project.
* **Deployments** – monitoring pages for deployed pipelines.
* **Models** – evaluation dashboards and health views for registered models.
* **Pipelines** – reusable visual documentation attached to pipeline definitions.
* **Pipeline Runs** – detailed diagnostics for specific executions.
* **Pipeline Snapshots** – configuration/version comparisons for snapshot history.

You can create a curated visualization programmatically by linking an artifact visualization to a single resource. Provide the resource identifier and resource type directly when creating the visualization. The example below shows how to create separate visualizations for different resource types:

```python
from uuid import UUID

from zenml.client import Client
from zenml.enums import (
    CuratedVisualizationSize,
    VisualizationResourceTypes,
)

client = Client()

# Define the identifiers for the pipeline and run you want to enrich
pipeline_id = UUID("<PIPELINE_ID>")
pipeline_run_id = UUID("<PIPELINE_RUN_ID>")

# Retrieve the artifact version produced by the evaluation step
pipeline_run = client.get_pipeline_run(pipeline_run_id)
artifact_version_id = pipeline_run.output.get("evaluation_report")
artifact_version = client.get_artifact_version(artifact_version_id)
artifact_visualizations = artifact_version.visualizations or []

# Fetch the resources we want to enrich
model = client.list_models().items[0]
model_id = model.id

deployment = client.list_deployments().items[0]
deployment_id = deployment.id

project_id = client.active_project.id

pipeline_model = client.get_pipeline(pipeline_id)
pipeline_id = pipeline_model.id

pipeline_snapshot = pipeline_run.snapshot()
snapshot_id = pipeline_snapshot.id

pipeline_run_id = pipeline_run.id

# Create curated visualizations for each supported resource type
client.create_curated_visualization(
    artifact_visualization_id=artifact_visualizations[0].id,
    resource_id=model_id,
    resource_type=VisualizationResourceTypes.MODEL,
    project_id=project_id,
    display_name="Latest Model Evaluation",
)

client.create_curated_visualization(
    artifact_visualization_id=artifact_visualizations[1].id,
    resource_id=deployment_id,
    resource_type=VisualizationResourceTypes.DEPLOYMENT,
    project_id=project_id,
    display_name="Deployment Health Dashboard",
)

client.create_curated_visualization(
    artifact_visualization_id=artifact_visualizations[2].id,
    resource_id=project_id,
    resource_type=VisualizationResourceTypes.PROJECT,
    display_name="Project Overview",
)

client.create_curated_visualization(
    artifact_visualization_id=artifact_visualizations[3].id,
    resource_id=pipeline_id,
    resource_type=VisualizationResourceTypes.PIPELINE,
    project_id=project_id,
    display_name="Pipeline Summary",
)

client.create_curated_visualization(
    artifact_visualization_id=artifact_visualizations[4].id,
    resource_id=pipeline_run_id,
    resource_type=VisualizationResourceTypes.PIPELINE_RUN,
    project_id=project_id,
    display_name="Run Results",
)

client.create_curated_visualization(
    artifact_visualization_id=artifact_visualizations[5].id,
    resource_id=snapshot_id,
    resource_type=VisualizationResourceTypes.PIPELINE_SNAPSHOT,
    project_id=project_id,
    display_name="Snapshot Metrics",
)
```

After creation, the returned response includes the visualization ID. You can retrieve a specific visualization later with `Client.get_curated_visualization`:

```python
retrieved = client.get_curated_visualization(pipeline_viz.id, hydrate=True)
print(retrieved.display_name)
print(retrieved.resource.type)
print(retrieved.resource.id)
```

Curated visualizations are tied to their parent resources and automatically surface in the ZenML dashboard wherever those resources appear, so keep track of the IDs returned by `create_curated_visualization` if you need to reference them later.

#### Updating curated visualizations

Once you've created a curated visualization, you can update its display name, order, or tile size using `Client.update_curated_visualization`:

```python
from uuid import UUID

client.update_curated_visualization(
    visualization_id=UUID("<CURATED_VISUALIZATION_ID>"),
    display_name="Updated Dashboard Title",
    display_order=10,
    layout_size=CuratedVisualizationSize.HALF_WIDTH,
)
```

When a visualization is no longer relevant, you can remove it entirely:

```python
client.delete_curated_visualization(visualization_id=UUID("<CURATED_VISUALIZATION_ID>"))
```

#### Controlling display order and size

The optional `display_order` field determines how visualizations are sorted when displayed. Visualizations with lower order values appear first, while those with `None` (the default) appear at the end in creation order.

When setting display orders, consider leaving gaps between values (e.g., 10, 20, 30 instead of 1, 2, 3) to make it easier to insert new visualizations later without renumbering everything:

```python
# Leave gaps for future insertions
visualization_a = client.create_curated_visualization(
    artifact_visualization_id=artifact_visualizations[0].id,
    resource_type=VisualizationResourceTypes.PIPELINE,
    resource_id=pipeline_id,
    display_name="Model performance at a glance",
    display_order=10,  # Primary dashboard
    layout_size=CuratedVisualizationSize.HALF_WIDTH,
)

visualization_b = client.create_curated_visualization(
    artifact_visualization_id=artifact_visualizations[1].id,
    resource_type=VisualizationResourceTypes.PIPELINE,
    resource_id=pipeline_id,
    display_name="Drill-down metrics",
    display_order=20,  # Secondary metrics
    layout_size=CuratedVisualizationSize.HALF_WIDTH,  # Compact chart beside the primary tile
)

# Later, easily insert between them
visualization_c = client.create_curated_visualization(
    artifact_visualization_id=artifact_visualizations[2].id,
    resource_type=VisualizationResourceTypes.PIPELINE,
    resource_id=pipeline_id,
    display_name="Raw output preview",
    display_order=15,  # Now appears between A and B
    layout_size=CuratedVisualizationSize.FULL_WIDTH,
)
```

#### RBAC visibility

Curated visualizations respect the access permissions of the resource they're linked to. A user can only see a curated visualization if they have read access to the specific resource it targets. If a user lacks permission for the linked resource, the visualization will be hidden from their view.

For example, if you create a visualization linked to a specific deployment, only users with read access to that deployment will see the visualization. If you need the same visualization to appear in different contexts with different access controls (e.g., on both a project page and a deployment page), create separate curated visualizations for each resource. This ensures that visualizations never inadvertently expose information from resources a user shouldn't access, while giving you fine-grained control over visibility.

### Visualization via Special Return Types

If you already have HTML, Markdown, CSV or JSON data available as a string inside your step, you can simply cast them to one of the following types and return them from your step:

* `zenml.types.HTMLString` for strings in HTML format, e.g., `"<h1>Header</h1>Some text"`,
* `zenml.types.MarkdownString` for strings in Markdown format, e.g., `"# Header\nSome text"`,
* `zenml.types.CSVString` for strings in CSV format, e.g., `"a,b,c\n1,2,3"`.
* `zenml.types.JSONString` for strings in JSON format, e.g., `{"key": "value"}`.

#### Example:

```python
from zenml import step
from zenml.types import CSVString

@step
def my_step() -> CSVString:
    some_csv = "a,b,c\n1,2,3"
    return CSVString(some_csv)
```

This would create the following visualization in the dashboard:

![CSV Visualization Example](/files/GiGlKWf547HA0yCAGBKM)

{% hint style="info" %}
**Shared CSS for Consistent Visualizations**

When creating multiple HTML visualizations across your pipeline, consider using a shared CSS file to maintain consistent styling. Create a central CSS file with your design system (colors, components, layouts) and Python utilities to load it into your HTML templates. This approach eliminates code duplication, ensures visual consistency across all reports, and makes it easy to update styling across all visualizations from a single location.

You can create helper functions that return complete HTML templates with shared styles, and use CSS variables for theme management. This pattern is especially valuable for teams generating multiple HTML reports or dashboards where maintaining a professional, cohesive appearance is important.
{% endhint %}

Another example is visualizing a matplotlib plot by embedding the image in an HTML string:

```python
import matplotlib.pyplot as plt
import base64
import io

from zenml.types import HTMLString
from zenml import step, pipeline

@step
def create_matplotlib_visualization() -> HTMLString:
    """Creates a matplotlib visualization and returns it as embedded HTML."""
    # Create plot
    fig, ax = plt.subplots()
    ax.plot([1, 2, 3, 4], [1, 4, 2, 3])
    ax.set_title('Sample Plot')
    
    # Convert plot to base64 string
    buf = io.BytesIO()
    fig.savefig(buf, format='png', bbox_inches='tight', dpi=300)
    plt.close(fig)  # Clean up
    image_base64 = base64.b64encode(buf.getvalue()).decode('utf-8')
    
    # Create HTML with embedded image
    html = f'''
    <div style="text-align: center;">
        <img src="data:image/png;base64,{image_base64}" 
             style="max-width: 100%; height: auto;">
    </div>
    '''
    
    return HTMLString(html)

@pipeline
def visualization_pipeline():
    create_matplotlib_visualization()

if __name__ == "__main__":
    visualization_pipeline()
```

### Visualization via Materializers

If you want to automatically extract visualizations for all artifacts of a certain data type, you can do so by overriding the `save_visualizations()` method of the corresponding [materializer](/concepts/artifacts/materializers). Let's look at an example of how to visualize matplotlib figures in your ZenML dashboard:

#### Example: Matplotlib Figure Visualization

**1. Custom Class** First, we create a custom class to hold our matplotlib figure:

```python
from typing import Any
from pydantic import BaseModel

class MatplotlibVisualization(BaseModel):
    """Custom class to hold matplotlib figures."""
    figure: Any  # This will hold the matplotlib figure
```

**2. Materializer** Next, we create a [custom materializer](/concepts/artifacts/materializers#creating-custom-materializers) that handles this class and implements the visualization logic:

```python
import os
from typing import Dict
from zenml.materializers.base_materializer import BaseMaterializer
from zenml.enums import VisualizationType
from zenml.io import fileio

class MatplotlibMaterializer(BaseMaterializer):
    """Materializer that handles matplotlib figures."""
    ASSOCIATED_TYPES = (MatplotlibVisualization,)

    def save_visualizations(
        self, data: MatplotlibVisualization
    ) -> Dict[str, VisualizationType]:
        """Create and save visualizations for the matplotlib figure."""
        visualization_path = os.path.join(self.uri, "visualization.png")
        with fileio.open(visualization_path, 'wb') as f:
            data.figure.savefig(f, format='png', bbox_inches='tight')
        return {visualization_path: VisualizationType.IMAGE}
```

**3. Step** Finally, we create a step that returns our custom type:

```python
import matplotlib.pyplot as plt
from zenml import step

@step
def create_matplotlib_visualization() -> MatplotlibVisualization:
    """Creates a matplotlib visualization."""
    fig, ax = plt.subplots()
    ax.plot([1, 2, 3, 4], [1, 4, 2, 3])
    ax.set_title('Sample Plot')
    return MatplotlibVisualization(figure=fig)
```

{% hint style="info" %}
When you use this step in your pipeline:

1. The step creates and returns a `MatplotlibVisualization`
2. ZenML finds the `MatplotlibMaterializer` and calls `save_visualizations()`
3. The figure is saved as a PNG file in your artifact store
4. The dashboard loads and displays this PNG when you view the artifact
   {% endhint %}

For another example, see our [Hugging Face datasets materializer](https://github.com/zenml-io/zenml/blob/main/src/zenml/integrations/huggingface/materializers/huggingface_datasets_materializer.py) which visualizes datasets by embedding their preview viewer.

## Controlling Visualizations

### Access to Visualizations

In order for the visualizations to show up on the dashboard, the following must be true:

#### Configuring a Service Connector

Visualizations are usually stored alongside the artifact, in the [artifact store](https://docs.zenml.io/stacks/stack-components/artifact-stores). Therefore, if a user would like to see the visualization displayed on the ZenML dashboard, they must give access to the server to connect to the artifact store.

The [service connector](https://docs.zenml.io/stacks/service-connectors/auth-management) documentation goes deeper into the concept of service connectors and how they can be configured to give the server permission to access the artifact store. For a concrete example, see the [AWS S3](https://docs.zenml.io/stacks/stack-components/artifact-stores/s3) artifact store documentation.

{% hint style="info" %}
When using the default/local artifact store with a deployed ZenML, the server naturally does not have access to your local files. In this case, the visualizations are also not displayed on the dashboard.

Please use a service connector enabled and remote artifact store alongside a deployed ZenML to view visualizations.
{% endhint %}

#### Configuring Artifact Stores

If all visualizations of a certain pipeline run are not showing up in the dashboard, it might be that your ZenML server does not have the required dependencies or permissions to access that artifact store. See the [custom artifact store docs page](https://docs.zenml.io/stacks/stack-components/artifact-stores/custom#enabling-artifact-visualizations-with-custom-artifact-stores) for more information.

### Enabling/Disabling Visualizations

You can control whether visualizations are generated at the pipeline or step level:

```python
# Disable visualizations for a pipeline
@pipeline(enable_artifact_visualization=False)
def my_pipeline():
    ...

# Disable visualizations for a step
@step(enable_artifact_visualization=False)
def my_step():
    ...
```

You can also configure this in YAML:

```yaml
enable_artifact_visualization: False

steps:
  my_step:
    enable_artifact_visualization: True
```

## Conclusion

Visualizing artifacts is a powerful way to gain insights from your ML pipelines. ZenML's built-in visualization capabilities make it easy to understand your data and model outputs, identify issues, and communicate results.

By leveraging these visualization tools, you can better understand your ML workflows, debug problems more effectively, and make more informed decisions about your models.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Stack & Components

Understanding and working with ZenML Stacks and Stack Components

A [ZenML stack](https://docs.zenml.io/stacks) is a collection of components that together form an MLOps infrastructure to run your ML pipelines. While your pipeline code defines what happens in your ML workflow, the stack determines where and how that code runs.

Stacks provide several key benefits:

1. **Environment Flexibility**: Run the same pipeline code locally during development and in the cloud for production
2. **Infrastructure Separation**: Change your infrastructure without modifying your pipeline code
3. **Specialized Resources**: Use specialized tools for different aspects of your ML workflow
4. **Team Collaboration**: Share infrastructure configurations across your team
5. **Reproducibility**: Ensure consistent pipeline execution across different environments

### Stack Structure

Each ZenML stack must include these core components:

* **Orchestrator**: Controls how your pipeline steps are executed
* **Artifact Store**: Manages where your pipeline artifacts are stored

Stacks may also include these optional components:

* **Container Registry**: Stores Docker images for your pipeline steps
* **Deployer**: Deploys pipelines as long-running HTTP services
* **Step Operator**: Runs specific steps on specialized hardware
* **Model Deployer**: Deploys models as prediction services
* **Experiment Tracker**: Tracks metrics and parameters
* **Feature Store**: Manages ML features
* **Alerter**: Sends notifications about pipeline events
* **Annotator**: Manages data labeling workflows

Most component types appear at most once in a stack. Three component types are repeatable: **step operators**, **experiment trackers**, and **alerters**. If a stack has more than one component of one of these types, the first attached component is the default. You can still choose a non-default component by name in step or pipeline configuration, and you can change the default later with `zenml stack set-default`.

## Working with Stacks

### The Active Stack

In ZenML, you always have an active stack that's used when you run a pipeline:

```bash
# See your active stack
zenml stack describe

# Switch to a different stack
zenml stack set STACK_NAME
```

### Managing Stacks

You can create and manage stacks through the CLI:

```bash
# List all stacks
zenml stack list

# Register a new stack with minimal components
zenml stack register my-stack -a local-store -o local-orchestrator

# Register a stack with additional components
zenml stack register production-stack \
    --artifact-store s3-store \
    --orchestrator kubeflow \
    --container-registry ecr-registry \
    --experiment-tracker mlflow-tracker

# Attach multiple repeatable components. The first one becomes the default.
zenml stack register training-stack \
    --artifact-store s3-store \
    --orchestrator kubernetes \
    --step_operator gpu-step-operator \
    --step_operator cpu-step-operator \
    --experiment_tracker mlflow \
    --experiment_tracker wandb

# Promote a different attached component to be the default.
zenml stack set-default training-stack --step_operator cpu-step-operator
```

### Discovering flavor-specific configuration

Stack component flavors have different configuration fields. A local orchestrator needs very little information; a Kubernetes orchestrator needs cluster and namespace details; an S3 artifact store needs bucket information. Once you pass a flavor with `-f` / `--flavor`, the CLI can show the concrete configuration fields for that flavor:

```bash
zenml orchestrator register -f kubernetes --help
zenml artifact-store register -f s3 --help
zenml step-operator update my-runai-step-operator --help
```

The help output includes a **Flavor configuration** section. For register and update commands, pass those fields as `--name=value` arguments. This is often the fastest way to answer, "what exactly does this flavor need from me?"

Or through the Python API:

```python
from zenml.client import Client

client = Client()
# List all stacks
stacks = client.list_stacks()

# Set active stack
client.activate_stack("my-stack")
```

### Local vs. Cloud Stacks

ZenML provides two main types of stacks:

1. **Local Stack**: Uses your local machine for orchestration and storage. This is the default and requires no additional setup.
2. **Cloud Stack**: Uses cloud services for orchestration, storage, and other components. These stacks offer more scalability and features but require additional deployment and configuration.

When you start with ZenML, you're automatically using a local stack. As your ML projects grow, you'll likely want to deploy cloud stacks to handle larger workloads and collaborate with your team.

## Next Steps

Now that you understand what stacks are, you might want to:

* Learn about [deploying stacks](https://docs.zenml.io/stacks/deployment) on cloud platforms
* Understand [Service Connectors](/concepts/service_connectors) for authenticating with cloud services
* Explore how to [register existing cloud resources](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack) as ZenML stack components

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Service Connectors

Managing authentication to cloud services and resources with Service Connectors

Service Connectors provide a unified way to handle authentication between ZenML and external services like cloud providers. They are a critical part of working with cloud-based stacks and significantly simplify the authentication challenge in ML workflows.

A service connector is an entity that:

1. Stores credentials and authentication configuration
2. Provides secure access to specific resources
3. Can be shared across multiple stack components
4. Manages permissions and access scopes
5. Automatically generates and refreshes short-lived access tokens

Think of service connectors as secure bridges between your ZenML stack components and external services that abstract away the complexity of different authentication methods across cloud providers.

## Why Use Service Connectors?

### The Authentication Challenge

ML workflows typically interact with multiple cloud services (storage, compute, model registries, etc.), creating complex credential management challenges. Without service connectors, you would need to:

* Configure authentication separately for each stack component
* Handle different authentication methods for each cloud service
* Store and manage credentials manually in code or configuration files
* Update credentials in multiple places when they change
* Implement proper security practices across all credential usage
* Spend engineering time on authentication rather than ML development

<figure><img src="/files/Y2c9YVoJEInE3WVxtoQr" alt=""><figcaption><p>Service Connectors abstract away complexity and implement security best practices</p></figcaption></figure>

Service connectors solve these problems by providing a single point of authentication that can be reused across your stack components, decoupling credentials from code and configuration.

### Key Benefits

* **Centralized Authentication**: Manage all your cloud credentials in one place
* **Credential Reuse**: Configure authentication once, use it with multiple components
* **Security**: Implement security best practices with short-lived tokens, principle of least privilege, and reduced credential exposure
* **Authentication Abstraction**: Eliminate credential handling code in pipeline components while supporting multiple auth methods
* **Resource Discovery**: Easily find available resources on your cloud accounts
* **Simplified Rotation**: Update credentials in one place when they change
* **Team Sharing**: Securely share access to resources within your team
* **Multi-cloud Support**: Use the same interface across AWS, GCP, Azure and other services with consistent patterns

### Supported Cloud Providers and Services

ZenML supports connectors for major cloud providers and services:

* **AWS**: For Amazon Web Services (S3, ECR, SageMaker, etc.)
* **GCP**: For Google Cloud Platform (GCS, GCR, Vertex AI, etc.)
* **Azure**: For Microsoft Azure (Blob Storage, ACR, AzureML, etc.)
* **Kubernetes**: For Kubernetes clusters

Each connector type supports authentication methods specific to that service.

## Working with Service Connectors

### Creating and Managing Connectors

Service connectors can be created with different authentication methods depending on your cloud provider and security requirements.

![Authentication with Service Connectors](/files/UxfzkocQy2izC2TeI1qK)

Here is an example of how to register a new connector:

```bash
# Register a new connector using AWS profile
zenml service-connector register aws-dev \
    --type aws \
    --auth-method profile \
    --profile=dev-account

# GCP connector using service account
zenml service-connector register gcp-prod \
    --type gcp \
    --auth-method service-account \
    --service-account-json=/path/to/sa.json

# List all connectors
zenml service-connector list

# Verify a connector works
zenml service-connector verify aws-dev
```

The authentication happens transparently to your ML code. You don't need to handle credentials in your pipeline steps - the service connector takes care of that for you.

### Discovering Resources

A powerful feature of service connectors is resource discovery:

```bash
# List available resources through a connector
zenml service-connector list-resources aws-dev --resource-type s3-bucket
```

This helps you find existing resources when configuring stack components.

### Using Connectors with Stack Components

Connect components to services:

```bash
# Register a component with a connector
zenml artifact-store register s3-store \
    --type s3 \
    --bucket my-bucket \
    --connector aws-dev
```

## Best Practices

* **Use descriptive names** for connectors indicating their purpose or environment
* **Create separate connectors** for development, staging, and production environments
* **Apply least privilege** when configuring connector permissions and resource scopes
* **Regularly rotate credentials** for enhanced security
* **Document your connector configurations** for team knowledge sharing
* **Leverage short-lived tokens** where possible instead of long-lived credentials
* **Avoid hard-coding credentials** in your code and config files, use service connectors instead

## Code Example

When using service connectors, your pipeline code remains clean and focused on ML logic:

```python
from zenml import step

# Without service connectors
@step
def upload_model(model):
    # Need to handle authentication manually
    import boto3
    session = boto3.Session(aws_access_key_id='AKIAXXXXXXXX',
                          aws_secret_access_key='SECRET')
    s3 = session.client('s3')
    s3.upload_file(model.path, 'my-bucket', 'models/model.pkl')

# With service connectors
@step
def upload_model_with_connector(model):
    # Authentication handled by the service connector
    # No credential handling required
    from zenml.integrations.s3.artifact_stores import S3ArtifactStore
    store = S3ArtifactStore()
    store.copyfile(model.path, 'models/model.pkl')
```

## Next Steps

* Learn how to [deploy stacks](https://docs.zenml.io/stacks/deployment) using service connectors
* Explore [authentication methods](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/service-connectors-guide) for different cloud providers
* Understand how to [reference secrets in stack configuration](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/reference-secrets-in-stack-configuration)
* Read our [blog post](https://www.zenml.io/blog/how-to-simplify-authentication-in-machine-learning-pipelines-for-mlops) on how service connectors simplify authentication in ML pipelines

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Pipeline Snapshots

Create and run pipeline snapshots.

A **Pipeline Snapshot** is an immutable snapshot of your pipeline that includes the pipeline DAG, code, configuration, and container images. Snapshots can be run from the SDK, CLI, ZenML dashboard or via a REST API. Additionally, snapshots can also be [deployed](/concepts/deployment).

{% hint style="info" %}
Snapshots are the successor and replacement of ZenML run templates.
{% endhint %}

{% hint style="success" %}
Running snapshots is a [ZenML Pro](https://zenml.io/pro)-only feature.
{% endhint %}

{% hint style="info" %}
**Looking for a long-running HTTP service?** Snapshots are designed for batch-style pipeline runs triggered on demand. If you need a persistent HTTP endpoint with request/response semantics, custom endpoints, middleware, and concurrent request handling, see [Pipeline Deployments](/concepts/deployment) instead. Deployments are available in both OSS and Pro.
{% endhint %}

## Real-world Use Case

Imagine your team has built a robust training pipeline that needs to be run regularly with different parameters:

* **Data Scientists** need to experiment with new datasets and hyperparameters
* **MLOps Engineers** need to schedule regular retraining with production data
* **Stakeholders** need to trigger model training through a simple UI without coding

Without snapshots, each scenario would require:

1. Direct access to the codebase
2. Knowledge of pipeline implementation details
3. Manual pipeline configuration for each run

**Pipeline snapshots solve this problem by creating a reusable configuration** that can be executed with different parameters from any interface:

* **Through Python**: Data scientists can programmatically trigger snapshots with custom parameters

```python
  from zenml.client import Client
  
  Client().trigger_pipeline(
      snapshot_name_or_id=<NAME-OR-ID>,
      run_configuration={
          "steps": {
              "data_loader": {"parameters": {"data_path": "s3://new-data/"}},
              "model_trainer": {"parameters": {"learning_rate": 0.01}}
          }
      }
  )
```

* **Through REST API**: Your CI/CD system can trigger snapshots via API calls

```bash
  curl -X POST 'https://your-zenml-server/api/v1/pipeline_snapshots/<ID>/runs' -H 'Authorization: Bearer <TOKEN>' -d '{"run_configuration": {...}}'
```

* **Through Browser** (Pro feature): Non-technical stakeholders can run snapshots directly from the ZenML dashboard by simply filling in a form with the required parameters - no coding required!

This enables your team to standardize execution patterns while maintaining flexibility - perfect for production ML workflows that need to be triggered from various systems.

## Snapshots vs Deployments

Snapshots and [Deployments](/concepts/deployment) both let you execute pipelines beyond direct Python calls, but they serve different purposes:

* **Snapshots** launch a **batch-style pipeline run** each time they are triggered. There is no persistent service — the ZenML server creates a runner job, executes the pipeline, and the job exits. Use snapshots for scheduled retraining, CI/CD-triggered runs, or when non-engineers need to run pipelines from the dashboard.
* **Deployments** create a **long-running HTTP service** with a stable URL, request/response semantics, custom endpoints, middleware, and shared state across calls. Use deployments for real-time inference, agents, interactive APIs, or any scenario needing an always-on endpoint. Deployments are available in both OSS and Pro.

## Understanding Pipeline Snapshots

While the simplest way to execute a ZenML pipeline is to directly call your pipeline function, pipeline snapshots offer several advantages for more complex workflows:

* **Standardization**: Ensure all pipeline runs follow a consistent configuration pattern
* **Parameterization**: Easily modify inputs and settings without changing code
* **Remote Execution**: Trigger pipelines through the dashboard or API without code access
* **Team Collaboration**: Share ready-to-use pipeline configurations with team members
* **Automation**: Integrate with CI/CD systems or other automated processes

## Creating Pipeline Snapshots

You have several ways to create a snapshot in ZenML:

### Using the Python SDK

You can create a snapshot from your local code and configuration like this:

```python
from zenml import pipeline

@pipeline
def my_pipeline():
    ...

snapshot = my_pipeline.create_snapshot(name="<NAME>")
```

### Using the CLI

You can create a snapshot using the ZenML CLI, by passing the [source path](/concepts/steps_and_pipelines/sources#source-paths) of your pipeline:

```bash
zenml pipeline snapshot create <PIPELINE-SOURCE-PATH> --name=<SNAPSHOT-NAME>
```

{% hint style="warning" %}
If you later want to run this snapshot, you need to have an active **remote stack** while running this command or you can specify one with the `--stack` option.
{% endhint %}

### Using the Dashboard

To create a snapshot through the ZenML dashboard:

1. Navigate to a pipeline run
2. Click on `...` in the top right, and then on `+ New Snapshot`
3. Enter a name for the snapshot
4. Click `Create`

![Create Snapshots on the dashboard](/files/bNTQ2hBQS6eSUfVO6Dqk)

![Snapshot Details](/files/DUekeyD4QUbFJugypkr3)

## Running Pipeline Snapshots

Once you've created a snapshot, you can run it through various interfaces:

### Using the Python SDK

Run a snapshot programmatically:

```python
from zenml.client import Client

snapshot = Client().get_snapshot("<NAME-OR-ID>", ...)
config = snapshot.config_template

# [OPTIONAL] Modify the configuration if needed
config.steps["my_step"].parameters["my_param"] = new_value

Client().trigger_pipeline(
    snapshot_name_or_id=snapshot.id,
    run_configuration=config,
)
```

### Using the CLI

Run a snapshot using the CLI:

```bash
zenml pipeline snapshot run <SNAPSHOT-NAME-OR-ID>
# If you want to run the snapshot with a modified configuration, use the `--config=...` parameter
```

### Using the Dashboard

To run a snapshot from the dashboard:

1. Either click `Run a Pipeline` on the main `Pipelines` page, or navigate to a specific snapshot and click `Run Snapshot`
2. On the `Run Details` page, you can:
   * Modify the configuration using the built-in editor
   * Upload a `.yaml` configuration file
3. Click `Run` to start the pipeline run

![Run Details](/files/H2rxa3grcxXHBJruoENS)

Once you run the snapshot, a new run will be executed on the same stack as the original run.

### Using the REST API

To run a snapshot through the REST API, you need to make a series of calls:

1. First, get the pipeline ID:

```bash
curl -X 'GET' \
  '<YOUR_ZENML_SERVER_URL>/api/v1/pipelines?hydrate=false&name=<PIPELINE-NAME>' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer <YOUR-TOKEN>'
```

2. Using the pipeline ID, get the snapshot ID:

```bash
curl -X 'GET' \
  '<YOUR_ZENML_SERVER_URL>/api/v1/pipeline_snapshots?hydrate=false&logical_operator=and&page=1&size=20&pipeline_id=<PIPELINE-ID>' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer <YOUR-TOKEN>'
```

3. Finally, trigger the snapshot:

```bash
curl -X 'POST' \
  '<YOUR_ZENML_SERVER_URL>/api/v1/pipeline_snapshots/<SNAPSHOT-ID>/runs' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <YOUR-TOKEN>' \
  -d '{
  "run_configuration": { "steps": {"model_trainer": {"parameters": {"model_type": "rf"}}}}
}'
```

{% hint style="info" %}
Learn how to get a bearer token for the curl commands:

* For a ZenML OSS API: use [service accounts + API keys](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account).
* For a ZenML Pro workspace API: use [ZenML Pro Personal Access Tokens](https://docs.zenml.io/pro/access-management/personal-access-tokens) or [ZenML Pro Organization Service Accounts](https://docs.zenml.io/pro/access-management/service-accounts).
  {% endhint %}

## Deleting Pipeline Snapshots

You can delete a snapshot using the CLI:

```bash
zenml pipeline snapshot delete <SNAPSHOT-NAME-OR-ID>
```

You can also delete a snapshot using the Python SDK:

```python
from zenml.client import Client

Client().delete_snapshot(name_id_or_prefix=<SNAPSHOT-NAME-OR-ID>)
```

## Advanced Usage: Running Snapshots from Other Pipelines

You can run snapshots from within other pipelines, enabling complex workflows. There are two ways to do this:

### Method 1: Trigger by Pipeline Name (Uses Latest Snapshot)

If you want to run the latest runnable snapshot for a specific pipeline:

```python
import pandas as pd

from zenml import pipeline, step
from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact
from zenml.artifacts.utils import load_artifact
from zenml.client import Client
from zenml.config.pipeline_run_configuration import PipelineRunConfiguration

@step
def trainer(data_artifact_id: str):
    df = load_artifact(data_artifact_id)

@pipeline
def training_pipeline():
    trainer()

@step
def load_data() -> pd.DataFrame:
    # Your data loading logic here
    return pd.DataFrame()

@step
def trigger_pipeline(df: UnmaterializedArtifact):
    # By using UnmaterializedArtifact we can get the ID of the artifact
    run_config = PipelineRunConfiguration(
        steps={"trainer": {"parameters": {"data_artifact_id": df.id}}}
    )

    # This triggers the LATEST runnable snapshot for the "training_pipeline" pipeline
    Client().trigger_pipeline(pipeline_name_or_id="training_pipeline", run_configuration=run_config)

@pipeline
def loads_data_and_triggers_training():
    df = load_data()
    trigger_pipeline(df)  # Will trigger the other pipeline
```

### Method 2: Trigger by Specific Snapshot ID

If you want to run a specific snapshot (not necessarily the latest one):

```python
@step
def trigger_specific_snapshot(df: UnmaterializedArtifact):
    run_config = PipelineRunConfiguration(
        steps={"trainer": {"parameters": {"data_artifact_id": df.id}}}
    )
    
    Client().trigger_pipeline(snapshot_name_or_id=<SNAPSHOT-NAME-OR-ID>, run_configuration=run_config)
```

{% hint style="info" %}
**Key Difference**:

* `Client().trigger_pipeline("pipeline_name", ...)` uses the pipeline name and runs the **latest** snapshot for that pipeline
* `Client().trigger_pipeline(snapshot_id=<ID>, ...)` runs a **specific** snapshot by its unique ID
  {% endhint %}

The newly created pipeline run will show up in the DAG next to the step that triggered it:

![Pipeline Snapshot triggered by Step](/files/dBR3dS9SucrLoDpvOKvG)

This pattern is useful for:

* Creating pipeline dependencies
* Implementing dynamic workflow orchestration
* Building multi-stage ML pipelines where different steps require different resources
* Separating data preparation from model training

Read more about:

* [PipelineRunConfiguration](https://sdkdocs.zenml.io/latest/core_code_docs/core-config.html#zenml.config.pipeline_run_configuration)
* [trigger\_pipeline API](https://sdkdocs.zenml.io/latest/core_code_docs/core-client.html#zenml.client.Client)
* [Unmaterialized Artifacts](https://docs.zenml.io/concepts/artifacts)

## Best Practices

1. **Use descriptive names** for your snapshots to make them easily identifiable
2. **Document snapshot parameters** so other team members understand how to configure them
3. **Start with a working pipeline run** before creating a snapshot to ensure it's properly configured
4. **Test snapshots with different configurations** to verify they work as expected
5. **Use version control** for your snapshot configurations when storing them as YAML files
6. **Implement access controls** to manage who can run specific snapshots
7. **Monitor snapshot usage** to understand how your team is using them

{% hint style="warning" %}
**Important:** You need to recreate your snapshots after upgrading your ZenML server. Snapshots are tied to specific server versions and may not work correctly after an upgrade.
{% endhint %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Pipeline Deployments

Deploy pipelines as HTTP services for real-time execution

Pipeline deployment allows you to run ZenML pipelines as long-running HTTP services for real-time execution, rather than traditional batch mode execution. This enables you to invoke pipelines through HTTP requests and receive immediate responses.

## What is a Pipeline Deployment?

A pipeline deployment is a long-running HTTP server that wraps your pipeline for real-time, request-response interactions. While traditional (batch) pipeline execution (via orchestrators) is ideal for scheduled batch processing, data transformations, and offline training workflows, deployments are designed for scenarios where you need immediate responses - like serving predictions to a web app, processing user requests, or powering interactive AI agents. Deployments create persistent services that stay running and can handle multiple concurrent requests through HTTP endpoints.

When you deploy a pipeline, ZenML creates an HTTP server (called a **Deployment**) that can execute your pipeline multiple times in parallel by invoking HTTP endpoints.

## Deployments vs Snapshots

ZenML offers two distinct ways to execute pipelines beyond direct Python calls. They serve different operating models:

|                                        | **Deployments**                                    | **Snapshots**                                    |
| -------------------------------------- | -------------------------------------------------- | ------------------------------------------------ |
| **What it is**                         | A long-running HTTP service wrapping your pipeline | A batch-style pipeline run triggered on demand   |
| **Stable URL**                         | Yes — persistent endpoint you can call repeatedly  | No — each run is a one-off server-managed job    |
| **Response model**                     | Synchronous request/response with pipeline outputs | Fire-and-forget (poll for run status)            |
| **Custom endpoints, auth, middleware** | Yes — full ASGI app customization                  | No                                               |
| **Persistent state across calls**      | Yes — `on_init` hooks, shared model loading        | No                                               |
| **Best for**                           | Real-time inference, agents, interactive APIs      | Scheduled retraining, batch jobs, CI/CD triggers |
| **Availability**                       | OSS and Pro                                        | Running snapshots requires Pro                   |

{% hint style="info" %}
Use **Deployments** when you need an always-on HTTP service (inference APIs, agents, interactive workflows). Use **Snapshots** when you need to trigger batch pipeline runs from a UI, SDK, or external system without maintaining a running service. See the [Snapshots documentation](/concepts/snapshots) for more details.
{% endhint %}

## Common Use Cases

Pipeline deployments are ideal for scenarios requiring real-time, on-demand execution of ML workflows:

**Online ML Inference**: Deploy trained models as HTTP services for real-time predictions, such as fraud detection in payment systems, recommendation engines for e-commerce, or image classification APIs. Pipeline deployments handle feature preprocessing, model loading, and prediction logic while managing concurrent requests efficiently.

**LLM Agent Workflows**: Build intelligent agents that combine multiple AI capabilities like intent analysis, retrieval-augmented generation (RAG), and response synthesis. These deployments can power chatbots, customer support systems, or document analysis services that require multi-step reasoning and context retrieval. See the [Agent Outer Loop](https://github.com/zenml-io/zenml/tree/main/examples/agent_outer_loop/README.md) and [Deploying Agents](https://github.com/zenml-io/zenml/tree/main/examples/deploying_agent/README.md) examples for practical implementations.

**Real-time Data Processing**: Process streaming events or user interactions that require immediate analysis and response, such as real-time analytics dashboards, anomaly detection systems, or personalization engines.

**Multi-step Business Workflows**: Orchestrate complex processes involving multiple AI/ML components, like document processing pipelines that combine OCR, entity extraction, sentiment analysis, and classification into a single deployable service.

## Traditional Model Serving vs. Deployed Pipelines

If you're reaching for tools like Seldon or KServe, consider this: deployed pipelines give you all the core serving primitives, plus the power of a full application runtime.

* Equivalent functionality: A pipeline handles the end-to-end inference path out of the box — request validation, feature pre-processing, model loading and inference, post-processing, and response shaping.
* More flexible: Deployed pipelines are unopinionated, so you can layer in retrieval, guardrails, rules, A/B routing, canary logic, human-in-the-loop, or any custom orchestration. You're not constrained by a model-server template.
* More customizable: The deployment is a real ASGI app. Tailor endpoints, authentication, authorization, rate limiting, structured logging, tracing, correlation IDs, or SSO/OIDC — all with first-class middleware and framework-level hooks.
* More features: Serve single-page apps alongside the API. Ship admin/ops dashboards, experiment playgrounds, model cards, or customer-facing UIs from the very same deployment for tighter operational feedback loops.

This approach aligns better with production realities: inference is rarely "just call a model." There are policies, data dependencies, and integrations that need a programmable, evolvable surface. Deployed pipelines give you that without sacrificing the convenience of a managed deployer and a clean HTTP contract.

{% hint style="info" %}
Deprecation notice: ZenML is phasing out the Model Deployer stack components in favor of pipeline deployments. Pipeline deployments are the strategic direction for real-time serving: they are more dynamic, more extensible, and offer deeper integration points with your security, observability, and product requirements. Existing model deployers will continue to function during the transition period, but new investments will focus on pipeline deployments.
{% endhint %}

## How Deployments Work

To deploy a pipeline or snapshot, a **Deployer** stack component needs to be in your active stack. You can use the default stack, which has a default local deployer that will deploy the pipeline directly on your local machine as a background process:

```bash
zenml stack set default
```

or set up a new stack with a deployer in it:

```bash
zenml deployer register <DEPLOYER-NAME> --flavor=<DEPLOYER-FLAVOR>
zenml stack update -d <DEPLOYER-NAME>
```

The [**Deployer** stack component](https://docs.zenml.io/stacks/stack-components/deployers) manages the deployment of pipelines as long-running HTTP servers. It integrates with a specific infrastructure back-end like Docker, AWS App Runner, GCP Cloud Run etc., in order to implement the following functionalities:

* Creating and managing persistent containerized services
* Exposing HTTP endpoints for pipeline invocation
* Managing the lifecycle of deployments (creation, updates, deletion)
* Providing connection information and management commands

{% hint style="info" %}
The **Deployer** and **Model Deployer** represent distinct stack components with slightly overlapping responsibilities. The **Deployer** component orchestrates the deployment of arbitrary pipelines as persistent HTTP services, while the **Model Deployer** component focuses exclusively on the deployment and management of ML models for real-time inference scenarios.

The **Deployer** component can easily accommodate ML model deployment through deploying ML inference pipelines. This approach provides enhanced flexibility for implementing custom business logic and preprocessing workflows around the deployed model artifacts. Conversely, specialized **Model Deployer** integrations may offer optimized deployment strategies, superior performance characteristics, and resource utilization efficiencies that exceed the capabilities of general-purpose pipeline deployments.

When deciding which component to use, consider the trade-offs between how much control you need over the deployment process and how much you want to offload to a particular integration specialized for ML model serving.
{% endhint %}

With a **Deployer** stack component in your active stack, a pipeline or snapshot can be deployed using the ZenML CLI:

```bash
# Deploy the pipeline `weather_pipeline` in the `weather_agent` module as a
# deployment named `my_deployment`
zenml pipeline deploy weather_agent.weather_pipeline --name my_deployment

# Deploy a snapshot named `weather_agent_snapshot` as a deployment named
# `my_deployment`
zenml pipeline snapshot deploy weather_agent_snapshot --deployment my_deployment
```

To deploy a pipeline using the ZenML SDK:

```python
from zenml.pipeline import pipeline

@pipeline
def weather_agent(city: str = "Paris", temperature: float = 20) -> str:
    return process_weather(city=city, temperature=temperature)

# Deploy the pipeline `weather_agent` as a deployment named `my_deployment`
deployment = weather_agent.deploy(deployment_name="my_deployment")
print(f"Deployment URL: {deployment.url}")
```

It is also possible to deploy snapshots programmatically:

```python
from zenml.client import Client

client = Client()
snapshot = client.get_snapshot(snapshot_name_or_id="weather_agent_snapshot")
# Deploy the snapshot `weather_agent_snapshot` as a deployment named
# `my_deployment`
deployment = client.provision_deployment(
    name_id_or_prefix="my_deployment",
    snapshot_id=snapshot.id,
)
print(f"Deployment URL: {deployment.url}")
```

Once deployed, a pipeline can be invoked through the URL exposed by the deployment. Every invocation of the deployment will create a new pipeline run.

The ZenML CLI provides a convenient command to invoke a deployment:

```bash
zenml deployment invoke my_deployment --city="London" --temperature=20
```

which is the equivalent of the following HTTP request:

```bash
curl -X POST http://localhost:8000/invoke \
  -H "Content-Type: application/json" \
  -d '{"parameters": {"city": "London", "temperature": 20}}'
```

## Deployment Lifecycle

Once a Deployment is created, it is tied to the specific **Deployer** stack component that was used to provision it and can be managed independently of the active stack as a standalone entity with its own lifecycle.

A Deployment contains the following key information:

* **`name`**: Unique deployment name within the project
* **`url`**: HTTP endpoint URL where the deployment can be accessed
* **`status`**: Current deployment status. This can take one of the following values `DeploymentStatus` enum values:
  * **`RUNNING`**: The deployment is running and accepting HTTP requests
  * **`ABSENT`**: The deployment is not currently provisioned
  * **`PENDING`**: The deployment is currently undergoing some operation (e.g. being created, updated or deleted)
  * **`ERROR`**: The deployment is in an error state. When in this state, more information about the error can be found in the ZenML logs, the Deployment `metadata` field or in the Deployment logs.
  * **`UNKNOWN`**: The deployment is in an unknown state
* **`metadata`**: Deployer-specific metadata describing the deployment's operational state

### Managing Deployments

To list all the deployments managed in your project by all the available Deployers:

```bash
zenml deployment list
```

This shows a table with deployment details:

```
╭──────────────────────┬────────────────────────┬──────────────────────┬───────────────────────┬───────────┬─────────────────┬─────────────────╮
│         NAME         │ PIPELINE               │ SNAPSHOT             │ URL                   │ STATUS    │ STACK           │ OWNER           │
├──────────────────────┼────────────────────────┼──────────────────────┼───────────────────────┼───────────┼─────────────────┼─────────────────┤
│  zenpulse-endpoint   │ zenpulse_agent         │                      │ http://localhost:8000 │ ⚙ RUNNING │ aws-stack       │ hamza@zenml.io  │
├──────────────────────┼────────────────────────┼──────────────────────┼───────────────────────┼───────────┼─────────────────┼─────────────────┤
│ docker-weather-agent │ weather_agent_pipeline │ docker-weather-agent │ http://localhost:8000 │ ⚙ RUNNING │ docker-deployer │ stefan@zenml.io │
├──────────────────────┼────────────────────────┼──────────────────────┼───────────────────────┼───────────┼─────────────────┼─────────────────┤
│    weather_agent     │ weather_agent          │                      │ http://localhost:8001 │ ⚙ RUNNING │ docker-deployer │ stefan@zenml.io │
╰──────────────────────┴────────────────────────┴──────────────────────┴───────────────────────┴───────────┴─────────────────┴─────────────────╯
```

Detailed information about a specific deployment can be obtained with the following command:

```bash
zenml deployment describe weather_agent
```

This provides comprehensive deployment details, including its state and access information:

```
🚀 Deployment: weather_agent is: RUNNING ⚙

Pipeline: weather_agent
Snapshot: 0866c821-d73f-456d-a98d-9aa82f41282e
Stack: docker-deployer

📡 Connection Information:

Endpoint URL: http://localhost:8001
Swagger URL: http://localhost:8001/docs
CLI Command Example:
  zenml deployment invoke weather_agent --city="London"

cURL Example:
  curl -X POST http://localhost:8001/invoke \
    -H "Content-Type: application/json" \
    -d '{
      "parameters": {
        "city": "London"
      }
    }'

⚙️  Management Commands
╭────────────────────────────────────────────┬─────────────────────────────────────────────────────╮
│ zenml deployment logs weather_agent -f     │ Follow deployment logs in real-time                 │
│ zenml deployment describe weather_agent    │ Show detailed deployment information                │
│ zenml deployment deprovision weather_agent │ Deprovision this deployment and keep a record of it │
│ zenml deployment delete weather_agent      │ Deprovision and delete this deployment              │
╰────────────────────────────────────────────┴─────────────────────────────────────────────────────╯
```

{% hint style="info" %}
Additional information regarding the deployment can be shown with the same command:

* schema information about the deployment's input and output
* backend-specific metadata information about the deployment
* authentication information, if present
  {% endhint %}

Deploying or redeploying a pipeline or snapshot on top of an existing deployment will update the deployment in place:

```bash
# Update the existing deployment named `my_deployment` with a new pipeline
# code version
zenml pipeline deploy weather_agent.weather_pipeline --name my_deployment --update

# Update the existing deployment named `my_deployment` with a new snapshot
# named `other_weather_agent_snapshot`
zenml deployment provision my_deployment --snapshot other_weather_agent_snapshot
```

{% hint style="warning" %}
**Deployment update checks and limitations**

* Updating a deployment owned by a different user requires additional confirmation. This is to avoid unintentionally updating someone else's deployment.
* An existing deployment cannot be updated using a stack different from the one it was originally deployed with.
* A pipeline snapshot can only have one deployment running at a time. You cannot deploy the same snapshot multiple times. You either have to delete the existing deployment and deploy the snapshot again or create a different snapshot.
  {% endhint %}

Deprovisioning and deleting a deployment are two different operations. Deprovisioning a deployment keeps a record of it in the ZenML database so that it can be easily restored later if needed. Deleting a deployment completely removes it from the ZenML store:

```bash
# Deprovision the deployment named `my_deployment`
zenml deployment deprovision my_deployment

# Re-provision the deployment named `my_deployment` with the same configuration as before
zenml deployment provision my_deployment

# Deprovision and delete the deployment named `my_deployment`
zenml deployment delete my_deployment
```

{% hint style="warning" %}
**Deployer deletion**

A Deployer stack component cannot be deleted as long as there is at least one deployment managed by it that is not in an `ABSENT` state. To delete a Deployer stack component, you need to first deprovision or delete all the deployments managed by it. If some deployments are stuck in an `ERROR` state, you can use the `--force` flag to delete them without the need to deprovision them first, but be aware that this may leave some infrastructure resources orphaned.
{% endhint %}

The server logs of a deployment can be accessed with the following command:

```bash
zenml deployment logs my_deployment
```

## Deployable Pipeline Requirements

While any pipeline can technically be deployed, following these guidelines ensures practical usability:

### Pipeline Input Parameters

Pipelines should accept explicit parameters to enable dynamic invocation:

```python
@pipeline
def weather_agent(city: str = "Paris", temperature: float = 20) -> str:
    return process_weather(city=city, temperature=temperature)
```

{% hint style="info" %}
**Input Parameter Requirements:**

* All pipeline input parameters must have default values. This is a current limitation of the deployment mechanism.
* Input parameters must use JSON-serializable data types (`int`, `float`, `str`, `bool`, `list`, `dict`, `tuple`, Pydantic models). Other data types are not currently supported and will result in an error when deploying the pipeline.
* Pipeline input parameter names must match step parameter names. E.g. if the pipeline has an input parameter named `city` that is passed to a step input argument, that step argument must also be named `city`.
  {% endhint %}

When deployed, the example pipeline above can be invoked:

* with a CLI command like the following:

```bash
zenml deployment invoke my_pipeline --city=Paris --temperature=20
```

* or with an HTTP request like the following:

```bash
curl -X POST http://localhost:8000/invoke \
  -H "Content-Type: application/json" \
  -d '{"parameters": {"city": "Paris", "temperature": 20}}'
```

{% hint style="warning" %}
Pipeline input parameters behave differently when pipelines are deployed than when they are run as a batch job. When running a parameterized pipeline, its input parameters are evaluated before the pipeline run even starts and can be used to configure the structure of the pipeline DAG. When invoking a deployment, the input parameters do not have an effect on the pipeline DAG structure, so a pipeline like the following will not work as expected:

```python
@pipeline
def switcher(
    mode: str = "analyze",
    city: str = "Paris",
    topic: str = "ML",
) -> str:
    return (
        analyze(city) if mode == "analyze" else generate(topic)
    )  # this will always use the "analyze" step when deploying the pipeline
```

{% endhint %}

### Pipeline Outputs

Pipelines should return meaningful values for useful HTTP responses:

```python
@step
def process_weather(city: str, temperature: float) -> Annotated[str, "weather_analysis"]:
    return f"The weather in {city} is {temperature} degrees Celsius."

@pipeline
def weather_agent(city: str = "Paris", temperature: float = 20) -> str:
    weather_analysis = process_weather(city=city, temperature=temperature)
    return weather_analysis
```

{% hint style="info" %}
**Output Requirements:**

* Return values must be step outputs.
* Return values must be JSON-serializable (`int`, `float`, `str`, `bool`, `list`, `dict`, `tuple`, Pydantic models). Other data types are not currently supported and will result in an error when deploying the pipeline.
* The names of the step output artifacts determine the response structure (see example below)
* For clashing output names, the naming convention used to differentiate them is `<step_name>.<output_name>`
  {% endhint %}

Invoking a deployment of this pipeline will return the response below. Note how the `outputs` field contains the value returned by the `process_weather` step and the name of the output artifact is used as the key.

```json
{
    "success": true,
    "outputs": {
        "weather_analysis": "The weather in Utopia is 25 degrees Celsius"
    },
    "execution_time": 8.160255432128906,
    "metadata": {
        "deployment_id": "e0b34be2-d743-4686-a45b-c12e81627bbe",
        "deployment_name": "weather_agent",
        "snapshot_id": "0866c821-d73f-456d-a98d-9aa82f41282e",
        "snapshot_name": null,
        "pipeline_name": "weather_agent",
        "run_id": "f2e9a3a7-afa3-459e-a970-8558358cf1fb",
        "run_name": "weather_agent-2025_09_29-14_09_55_726165",
        "parameters_used": {
            "city": "Utopia",
            "temperature": 25
        }
    },
    "error": null
}
```

### Deployment Authentication

A rudimentary form of HTTP Basic authentication can be enabled for deployments by configuring one of two deployer configuration options:

* `generate_auth_key`: set to `True` to automatically generate a shared secret key for the deployment. This is not set by default.
* `auth_key`: configure the shared secret key manually.

```python
@pipeline(
    settings={
        "deployer": {
            "generate_auth_key": True,
        }
    }
)
def weather_agent(city: str = "Paris", temperature: float = 20) -> str:
    return process_weather(city=city, temperature=temperature)
```

Deploying the above pipeline automatically generates and returns a key that will be required in the `Authorization` header of HTTP requests made to the deployment:

```bash
curl -X POST http://localhost:8000/invoke \
  -H "Authorization: Bearer <GENERATED_AUTH_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"parameters": {"city": "Paris", "temperature": 20}}'
```

## Deployment Initialization, Cleanup and State

It often happens that the HTTP requests made to the same deployment share some type of initialization or cleanup or need to share the same global state or. For example:

* a machine learning model needs to be loaded in memory, initialized and then shared between all the HTTP requests made to the deployment in order to be used by the deployed pipeline to make predictions
* a database client must be initialized and shared across all the HTTP requests made to the deployment in order to read and write data

To achieve this, it is possible to configure custom initialization and cleanup hooks for the pipeline being deployed:

```python

def init_llm(model_name: str):
    # Initialize and store the LLM in memory when the deployment is started, to
    # be shared by all the HTTP requests made to the deployment
    return LLM(model_name=model_name)

def cleanup_llm(llm: LLM):
    # Cleanup the LLM when the deployment is stopped
    llm.cleanup()

@step
def process_weather(city: str, temperature: float) -> Annotated[str, "weather_analysis"]:
    step_context = get_step_context()
    # The value returned by the on_init hook is stored in the pipeline state
    llm = step_context.pipeline_state
    return generate_llm_response(llm, city, temperature)

@pipeline(
    on_init=init_llm,
    on_cleanup=cleanup_llm,
)
def weather_agent(city: str = "Paris", temperature: float = 20) -> str:
    return process_weather(city=city, temperature=temperature)

weather_agent_deployment = weather_agent.with_options(
    on_init_kwargs={"model_name": "gpt-4o"},
).deploy(deployment_name="my_deployment")
```

The following happens when the pipeline is deployed and then later invoked:

1. The on\_init hook is executed only once, when the deployment is started
2. The value returned by the on\_init hook is stored in memory in the deployment and can be accessed by pipeline steps using the `pipeline_state` property of the step context
3. The on\_cleanup hook is executed only once, when the deployment is stopped

This mechanism can be used to initialize and share global state between all the HTTP requests made to the deployment or to execute long-running initialization or cleanup operations when the deployment is started or stopped rather than on each HTTP request.

## Deployment Configuration

The deployer settings cover aspects of the pipeline deployment process and specific back-end infrastructure used to provision and manage the resources required to run the deployment servers. Independently of that, `DeploymentSettings` can be used to fully customize all aspects pertaining to the deployment ASGI application itself, including:

* HTTP endpoints
* middleware
* secure headers
* CORS settings
* mounting and serving static files to support deploying single-page applications alongside the pipeline
* for more advanced cases, even the ASGI framework (e.g. FastAPI, Django, Flask, Falcon, Quart, BlackSheep, etc.) and its configuration can be customized

Example:

```python
from zenml.config import DeploymentSettings, EndpointSpec, EndpointMethod
from zenml import pipeline

async def custom_health_check() -> Dict[str, Any]:
    from zenml.client import Client

    client = Client()
    return {
        "status": "healthy",
        "info": client.zen_store.get_store_info().model_dump(),
    }

@pipeline(settings={"deployment": DeploymentSettings(
    custom_endpoints=[
        EndpointSpec(
            path="/health",
            method=EndpointMethod.GET,
            handler=custom_health_check,
            auth_required=False,
        ),
    ],
)})
def my_pipeline():
    ...
```

For more detailed information on deployment options, see the [deployment settings guide](/concepts/deployment/deployment_settings).

## Best Practices

1. **Design for Parameters**: Structure your pipelines to accept meaningful parameters that control behavior
2. **Provide Default Values**: Ensure all parameters have sensible defaults
3. **Return Useful Data**: Design pipeline outputs to provide meaningful responses
4. **Use Type Annotations**: Leverage Pydantic models for complex parameter types
5. **Use Global Initialization and State**: Use the `on_init` and `on_cleanup` hooks along with the `pipeline_state` step context property to initialize and share global state between all the HTTP requests made to the deployment. Also use these hooks to execute long-running initialization or cleanup operations when the deployment is started or stopped rather than on each HTTP request.
6. **Handle Errors Gracefully**: Implement proper error handling in your steps
7. **Test Locally First**: Validate your deployable pipeline locally before deploying to production

## Conclusion

Pipeline deployment transforms ZenML pipelines from batch processing workflows into real-time services. By following the guidelines for deployable pipelines and understanding the deployment lifecycle, you can create robust, scalable ML services that integrate seamlessly with web applications and real-time systems.

See also:

* [Steps & Pipelines](/concepts/steps_and_pipelines) - Core building blocks
* [Deployer Stack Component](https://github.com/zenml-io/zenml/tree/main/docs/book/component-guide/deployers/README.md) - The stack component that manages the deployment of pipelines as long-running HTTP servers

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Deployment Settings

Customize the pipeline deployment ASGI application with DeploymentSettings.

### Deployment servers and ASGI apps

ZenML pipeline deployments run an ASGI application under a production-grade `uvicorn` server. This makes your pipelines callable over HTTP for online workloads like real-time ML inference, LLM agents/workflows, and even full web apps co-located with pipelines.

At runtime, three core components work together:

* the ASGI application: the HTTP surface that exposes endpoints (health, invoke, metrics, docs) and any custom routes or middleware you configure. This is powered by an ASGI framework like FastAPI, Starlette, Django, Flask, etc.
* the ASGI application factory (aka the Deployment App Runner): this component is responsible for constructing the ASGI application piece by piece based on the instructions provided by users via runtime configuration.
* the Deployment Service: the component responsible for the business logic that backs the pipeline deployment and its invocation lifecycle.

Both the Deployment App Runner and the Deployment Service are customizable at runtime, through the `DeploymentSettings` configuration mechanism. They can also be extended via inheritance to support different ASGI frameworks or to tweak existing functionality.

The `DeploymentSettings` class lets you shape both server behavior and the ASGI app composition without changing framework code. Typical reasons to customize include:

* Tight security posture: CORS controls, strict headers, authentication, API surface minimization.
* Observability: request/response logging, tracing, metrics, correlation identifiers.
* Enterprise integration: policy gateways, SSO/OIDC/OAuth, audit logging, routing and network architecture constraints.
* Product UX: single-page application (SPA) static files served alongside deployment APIs or custom docs paths.
* Performance/SRE: thread pool sizing, uvicorn worker settings, log levels, max request sizes and platform-specific fine-tuning.

All `DeploymentSettings` are pipeline-level settings. They apply to the deployment that serves the pipeline as a whole. They are not available at step-level.

### Configuration overview

You can configure `DeploymentSettings` in Python or via YAML, the same way as other settings classes. The settings can be attached to a pipeline decorator or via `with_options`. These settings are only valid at pipeline level.

#### Python configuration

Use the `DeploymentSettings` class to configure the deployment settings for your pipeline in-code

```python
from zenml import pipeline
from zenml.config import DeploymentSettings

deploy_settings = DeploymentSettings(
    app_title="Fraud Scoring Service",
    app_description=(
        "Online scoring API exposing synchronous and batch inference"
    ),
    app_version="1.2.0",
    root_url_path="",
    api_url_path="",
    docs_url_path="/docs",
    redoc_url_path="/redoc",
    invoke_url_path="/invoke",
    health_url_path="/health",
    info_url_path="/info",
    metrics_url_path="/metrics",
    cors={
        "allow_origins": ["https://app.example.com"],
        "allow_methods": ["GET", "POST", "OPTIONS"],
        "allow_headers": ["*"],
        "allow_credentials": True,
    },
    thread_pool_size=32,
    uvicorn_host="0.0.0.0",
    uvicorn_port=8080,
    uvicorn_workers=2,
)

@pipeline(settings={"deployment": deploy_settings})
def scoring_pipeline() -> None:
    ...

# Alternatively
scoring_pipeline = scoring_pipeline.with_options(
    settings={"deployment": deploy_settings}
)
```

#### YAML configuration

Define settings in a YAML configuration file for better separation of code and configuration:

```yaml
settings:
  deployment:
    app_title: Fraud Scoring Service
    app_description: >-
      Online scoring API exposing synchronous and batch inference
    app_version: "1.2.0"
    root_url_path: ""
    api_url_path: ""
    docs_url_path: "/docs"
    redoc_url_path: "/redoc"
    invoke_url_path: "/invoke"
    health_url_path: "/health"
    info_url_path: "/info"
    metrics_url_path: "/metrics"
    cors:
      allow_origins: ["https://app.example.com"]
      allow_methods: ["GET", "POST", "OPTIONS"]
      allow_headers: ["*"]
      allow_credentials: true
    thread_pool_size: 32
    uvicorn_host: 0.0.0.0
    uvicorn_port: 8080
    uvicorn_workers: 2
```

Check out [this page](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) for more information on the hierarchy and precedence of the various ways in which you can supply the settings.

### Basic customization options

`DeploymentSettings` expose the following basic customization options. The sections below provide short examples and guidance.

* application metadata and paths
* built-in endpoints and middleware toggles
* static files (SPAs) and dashboards
* CORS
* secure headers
* startup and shutdown hooks
* uvicorn server options, logging level, and thread pool size

#### Application metadata

You can set `app_title`, `app_description`, and `app_version` to be reflected in the ASGI application's metadata:

```python
from zenml.config import DeploymentSettings

settings = DeploymentSettings(
    app_title="LLM Agent Service",
    app_description=(
        "Agent endpoints for tools, state inspection, and tracing"
    ),
    app_version="0.7.0",
)
```

#### Default URL paths, endpoints and middleware

The ASGI application exposes the following built-in endpoints by default:

* documentation endpoints:
  * `/docs` - The OpenAPI documentation UI generated based on the endpoints and their signatures.
  * `/redoc` - The ReDoc documentation UI generated based on the endpoints and their signatures.
* REST API endpoints:
  * `/invoke` - The main pipeline invocation endpoint for synchronous inference.
  * `/health` - The health check endpoint.
  * `/info` - The info endpoint providing extensive information about the deployment and its service.
  * `/metrics` - Simple metrics endpoint.
* dashboard endpoints - present only if the accompanying UI is enabled:
  * `/`, `/index.html`, `/static` - Endpoints for serving the dashboard files from the `dashboard_files_path` directory.

The ASGI application includes the following built-in middleware by default:

* secure headers middleware: for setting security headers.
* CORS middleware: for handling CORS requests.

You can include or exclude these default endpoints and middleware either globally or individually by setting the `include_default_endpoints` and `include_default_middleware` settings. It is also possible to remap the built-in endpoint URL paths.

```python
from zenml.config import (
    DeploymentSettings,
    DeploymentDefaultEndpoints,
    DeploymentDefaultMiddleware,
)

settings = DeploymentSettings(
    # Include only the endpoints you need
    include_default_endpoints=(
        DeploymentDefaultEndpoints.DOCS
        | DeploymentDefaultEndpoints.INVOKE
        | DeploymentDefaultEndpoints.HEALTH
    ),
    # Customize the root URL path
    root_url_path="/pipeline",
    # Include only the middleware you need
    include_default_middleware=DeploymentDefaultMiddleware.CORS,
    # Customize the base API URL path used for all REST API endpoints
    api_url_path="/api",
    # Customize the documentation URL path
    docs_url_path="/documentation",
    # Customize the health check URL path
    health_url_path="/healthz",
)
```

With the above settings, the ASGI application will only expose the following endpoints and middleware:

* `/pipeline/documentation` - The API documentation (OpenAPI schema)
* `/pipeline/api/invoke` - The REST API pipeline invocation endpoint
* `/pipeline/api/healthz` - The REST API health check endpoint
* CORS middleware: for handling CORS requests

#### Static files (single-page applications)

Deployed pipelines can serve full single-page applications (React/Vue/Svelte) from the same origin as your inference API. This eliminates CORS/auth/routing friction and lets you ship user-facing UI components alongside your endpoints, such as:

* operator dashboards
* governance portals
* experiment browsers
* feature explorers
* custom data labeling interfaces
* model cards
* observability dashboards
* customer-facing playgrounds

Co-locating UI and API streamlines delivery (one image, one URL, one CI/CD), improves latency, and keeps telemetry and auth consistent.

To enable this, point `dashboard_files_path` to a directory containing an `index.html` and any static assets. The path must be relative to the [source root](/concepts/steps_and_pipelines/sources#source-root):

```python
settings = DeploymentSettings(
    dashboard_files_path="web/build"  # contains index.html and assets/
)
```

A rudimentary playground dashboard is included with the ZenML python package that features a simple UI useful for sending pipeline invocations and viewing the pipeline's response.

{% hint style="info" %}
When supplying your own custom dashboard, you may also need to [customize the security headers](#secure-headers) to allow the dashboard to access various resources. For example, you may want to tweak the `Content-Security-Policy` header to allow the dashboard to access external javascript libraries, images, etc.
{% endhint %}

**Jinja2 templates**

You can use a Jinja2 template to dynamically generate the `index.html` file that hosts the single-page application. This is useful if you want to dynamically generate the dashboard files based on the pipeline configuration, step configuration or stack configuration. A `service_info` variable is passed to the template that contains the service information, such as the service name, version, and description. This variable has the same structure as the `zenml.deployers.server.models.ServiceInfo` model.

Example:

```jinja2
<html>
<head>
    <title>Pipeline: {{ service_info.pipeline.pipeline_name }}</title>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="https://unpkg.com/mvp.css">
</head>
<body>
    <h1>Pipeline: {{ service_info.pipeline.pipeline_name }}</h1>
    <p>Deployment: {{ service_info.deployment.name }}</p>
</body>
</html>
```

#### CORS

Fine-tune cross-origin access:

```python
from zenml.config import DeploymentSettings, CORSConfig

settings = DeploymentSettings(
    cors=CORSConfig(
        allow_origins=["https://app.example.com", "https://admin.example.com"],
        allow_methods=["GET", "POST", "OPTIONS"],
        allow_headers=["authorization", "content-type", "x-request-id"],
        allow_credentials=True,
    )
)
```

#### Secure headers

Harden responses with strict headers. Each field supports either a boolean or string. Using `True` selects a safe default, `False` disables the header, and custom strings allow fully custom policies:

```python
from zenml.config import (
    DeploymentSettings,
    SecureHeadersConfig,
)

settings = DeploymentSettings(
    secure_headers=SecureHeadersConfig(
        server=True,  # emit default ZenML server header value
        hsts=True,    # default: 63072000; includeSubdomains
        xfo=True,     # default: SAMEORIGIN
        content=True, # default: nosniff
        csp=(
            "default-src 'none'; connect-src 'self' https://api.example.com; "
            "img-src 'self' data:; style-src 'self' 'unsafe-inline'"
        ),
        referrer=True,
        cache=True,
        permissions=True,
    )
)
```

Set any field to `False` to omit that header. Set to a string for a custom value. The defaults are strong, production-safe policies.

#### Startup and shutdown hooks

Lifecycle startup and shutdown hooks are called as part of the ASGI application's lifespan. This is an alternative to [the `on_init` and `on_cleanup` hooks that can be configured at pipeline level](/concepts/deployment#deployment-initialization-cleanup-and-state).

Common use-cases:

* Model inference
  * load models/tokenizers and warm caches (JIT/ONNX/TensorRT, HF, sklearn)
  * hydrate feature stores, connect to vector DBs (FAISS, Milvus, PGVector)
  * initialize GPU memory pools and thread/process pools
  * set global config, download artifacts from registry or object store
  * prefetch embeddings, label maps, lookup tables
  * create connection pools for databases, Redis, Kafka, SQS, Pub/Sub
* LLM agent workflows
  * initialize LLM client(s), tool registry, and router/policy engine
  * build or load RAG indexes; warm retrieval caches and prompts
  * configure rate limiting, concurrency guards, circuit breakers
  * load guardrails (PII filters, toxicity, jailbreak detection)
  * configure tracing/observability for token usage and tool calls
* Shutdown
  * flush metrics/traces/logs, close pools/clients, persist state/caches
  * graceful draining: wait for in-flight requests before teardown

Hooks can be provided as:

* A Python callable object
* A source path string to be loaded dynamically (e.g. `my_project.runtime.hooks.on_startup`)

The callable must accept an `app_runner` argument of type `BaseDeploymentAppRunner` and any additional keyword arguments. The `app_runner` argument is the application factory that is responsible for building the ASGI application. You can use it to access information such as:

* the ASGI application instance that is being built
* the deployment service instance that is being deployed
* the `DeploymentResponse` object itself, which also contains details about the snapshot, pipeline, etc.

```python
from zenml.deployers.server import BaseDeploymentAppRunner

def on_startup(app_runner: BaseDeploymentAppRunner, warm: bool = False) -> None:
    # e.g., warm model cache, connect tracer, prefetch embeddings
    ...

def on_shutdown(app_runner: BaseDeploymentAppRunner, drain_timeout_s: int = 2) -> None:
    # e.g., flush metrics, close clients
    ...

settings = DeploymentSettings(
    startup_hook=on_startup,
    shutdown_hook=on_shutdown,
    startup_hook_kwargs={"warm": True},
    shutdown_hook_kwargs={"drain_timeout_s": 2},
)
```

YAML using source strings:

```yaml
settings:
  deployment:
    startup_hook: my_project.runtime.hooks.on_startup
    shutdown_hook: my_project.runtime.hooks.on_shutdown
    startup_hook_kwargs:
      warm: true
    shutdown_hook_kwargs:
      drain_timeout_s: 2
```

#### Uvicorn and threading

Tune server runtime parameters for performance and topology:

The following settings are available for tuning the uvicorn server:

* `thread_pool_size`: the size of the thread pool for CPU-bound work offload.
* `uvicorn_host`: the host to bind the uvicorn server to.
* `uvicorn_port`: the port to bind the uvicorn server to.
* `uvicorn_workers`: the number of workers to use for the uvicorn server.
* `log_level`: the log level to use for the uvicorn server.
* `uvicorn_reload`: whether to enable auto-reload for the uvicorn server. This is useful when using [the local Deployer stack component](https://docs.zenml.io/stacks/stack-components/deployers/docker) to speed up local development by automatically restarting the server when code changes are detected. NOTE: the `uvicorn_reload` setting has no effect on changes in the pipeline configuration, step configuration or stack configuration.
* `uvicorn_kwargs`: a dictionary of keyword arguments to pass to the uvicorn server.

The following settings are available:

```python
from zenml.config import DeploymentSettings
from zenml.enums import LoggingLevels

settings = DeploymentSettings(
    thread_pool_size=64,  # CPU-bound work offload
    uvicorn_host="0.0.0.0",
    uvicorn_port=8000,
    uvicorn_workers=2,    # multi-process model
    log_level=LoggingLevels.INFO,
    uvicorn_kwargs={
        "proxy_headers": True,
        "forwarded_allow_ips": "*",
        "timeout_keep_alive": 15,
    },
)
```

### Advanced customization options

When the built-in ASGI application, endpoints and middleware are not enough, you can take customizing your deployment to the next level by providing your own implementation for endpoints, middleware and other ASGI application extensions. ZenML `DeploymentSettings` provides a flexible and extensible mechanism to inject your own custom code into the ASGI application at runtime:

* custom endpoints - to expose your own HTTP endpoints.
* custom middleware - to insert your own ASGI middleware.
* free-form ASGI application building extensions - to take full control of the ASGI application and its lifecycle for truly advanced use-cases when endpoints and middleware are not enough.

#### Custom endpoints

In production, custom endpoints are often required alongside the main pipeline invoke route. Common use-cases include:

* Online inference controls
  * model (re)load, warm-up, and cache priming
  * dynamic model/version switching and traffic shaping (A/B, canary)
  * async/batch prediction submission and job-status polling
  * feature store materialization/backfills and online/offline sync triggers
* Enterprise integration
  * authentication bootstrap (API key issuance/rotation), JWKS rotation
  * OIDC/OAuth device-code flows and SSO callback handlers
  * external system webhooks (CRM, billing, ticketing, audit sink)
* Observability and operations
  * detailed health/readiness endpoints (subsystems, dependencies)
  * metrics/traces/log shipping toggles; log level switch (INFO/DEBUG)
  * maintenance-mode enable/disable and graceful drain controls
* LLM agent serving
  * tool registry CRUD, tool execution sandboxes, guardrail toggles
  * RAG index CRUD (upsert documents, rebuild embeddings, vacuum/compact)
  * prompt template catalogs and runtime overrides
  * session memory inspection/reset, conversation export/import
* Governance and data management
  * payload redaction policy updates and capture sampling controls
  * schema/contract discovery (sample payloads, test vectors)
  * tenant provisioning, quotas/limits, and per-tenant configuration

You can configure `custom_endpoints` in `DeploymentSettings` to expose your own HTTP endpoints.

Endpoints support multiple definition modes (see code examples below):

1. Direct callable - a simple function that takes in request parameters and returns a response. Framework-specific arguments such as FastAPI's `Request`, `Response` and dependency injection patterns are supported.
2. Builder class - a callable class with a `__call__` method that is the actual endpoint callable described at 1). The builder class constructor is called by the ASGI application factory and can be leveraged to execute any global initialization logic before the endpoint is called.
3. Builder function - a function that returns the actual endpoint callable described at 1). Similar to the builder class.
4. Native framework-specific object (`native=True`). This can vary from ASGI framework to framework.

Definitions can be provided as Python objects or as loadable source path strings.

The builder class and builder function must accept an `app_runner` argument of type `BaseDeploymentAppRunner`. This is the application factory that is responsible for building the ASGI application. You can use it to access information such as:

* the ASGI application instance that is being built
* the deployment service instance that is being deployed
* the `DeploymentResponse` object itself, which also contains details about the snapshot, pipeline, etc.

The final endpoint callable can take any input arguments and return any output that are JSON-serializable or Pydantic models. The application factory will handle converting these into the appropriate schema for the ASGI application.

You can also use framework-specific request/response types (e.g. FastAPI `Request`, `Response`) or dependency injection patterns for your endpoint callable if needed. However, this will limit the portability of your endpoint to other frameworks.

The following code examples demonstrate the different definition modes for custom endpoints:

1. a custom detailed health check endpoint implemented as a direct callable

```python
from typing import Any, Callable, Dict, List
from pydantic import BaseModel
from zenml.client import Client
from zenml.config import (
    DeploymentSettings,
    EndpointSpec,
    EndpointMethod,
)
from zenml.deployers.server import BaseDeploymentAppRunner
from zenml.models import DeploymentResponse

async def health_detailed() -> Dict[str, Any]:
    import psutil

    client = Client()

    return {
        "status": "healthy",
        "cpu_percent": psutil.cpu_percent(),
        "memory_percent": psutil.virtual_memory().percent,
        "disk_percent": psutil.disk_usage("/").percent,
        "zenml": client.zen_store.get_store_info().model_dump(),
    }

settings = DeploymentSettings(
    custom_endpoints=[
        EndpointSpec(
            path="/health",
            method=EndpointMethod.GET,
            handler=health_detailed,
            auth_required=False,
        ),
    ]
)
```

2. a custom ML model inference endpoint, implemented as a builder function. Note how the builder function loads the model only once at runtime, and then reuses it for all subsequent requests.

```python
from typing import Any, Callable, Dict, List
from pydantic import BaseModel
from zenml.client import Client
from zenml.config import (
    DeploymentSettings,
    EndpointSpec,
    EndpointMethod,
)
from zenml.deployers.server import BaseDeploymentAppRunner
from zenml.models import DeploymentResponse

class PredictionRequest(BaseModel):
    features: List[float]

class PredictionResponse(BaseModel):
    prediction: float
    confidence: float

def build_predict_endpoint(
    app_runner: BaseDeploymentAppRunner,
    model_name: str,
    model_version: str,
    model_artifact: str,
) -> Callable[[PredictionRequest], PredictionResponse]:

    stored_model_version = Client().get_model_version(model_name, model_version)
    stored_model_artifact = stored_model_version.get_artifact(model_artifact)
    model = stored_model_artifact.load()

    def predict(
        request: PredictionRequest,
    ) -> PredictionResponse:
        pred = float(model.predict([request.features])[0])
        # Example: return fixed confidence if model lacks proba
        return PredictionResponse(prediction=pred, confidence=0.9)

    return predict

settings = DeploymentSettings(
    custom_endpoints=[
        EndpointSpec(
            path="/predict/custom",
            method=EndpointMethod.POST,
            handler=build_predict_endpoint,
            init_kwargs={
                "model_name": "fraud-classifier",
                "model_version": "v1",
                "model_artifact": "sklearn_model",
            },
            auth_required=True,
        ),
    ]
)
```

NOTE: a similar way to do this is to implement a proper ZenML pipeline that loads the model in the `on_init` hook and then runs pre-processing and inference steps in the pipeline.

3. a custom deployment info endpoint implemented as a builder class

```python
from typing import Any, Awaitable, Callable, Dict, List
from pydantic import BaseModel
from zenml.client import Client
from zenml.config import (
    DeploymentSettings,
    EndpointSpec,
    EndpointMethod,
)
from zenml.deployers.server import BaseDeploymentAppRunner
from zenml.models import DeploymentResponse

def build_deployment_info(app_runner: BaseDeploymentAppRunner) -> Callable[[], Awaitable[DeploymentResponse]]:
    async def endpoint() -> DeploymentResponse:
        return app_runner.deployment

    return endpoint

settings = DeploymentSettings(
    custom_endpoints=[
        EndpointSpec(
            path="/deployment",
            method=EndpointMethod.GET,
            handler=build_deployment_info,
            auth_required=True,
        ),
    ]
)
```

4. a custom model selection endpoint, implemented as a FastAPI router. This example is more involved and demonstrates how to coordinate multiple endpoints with the main pipeline invoke endpoint.

```python
# my_project.fastapi_endpoints
from __future__ import annotations

from typing import List, Optional

from fastapi import APIRouter, HTTPException, status
from pydantic import BaseModel, Field
from sklearn.base import ClassifierMixin
from zenml.client import Client
from zenml.models import ArtifactVersionResponse
from zenml.config import DeploymentSettings, EndpointSpec, EndpointMethod

model_router = APIRouter()

# Global, process-local model registry for inference
CURRENT_MODEL: Optional[Any] = None
CURRENT_MODEL_ARTIFACT: Optional[ArtifactVersionResponse] = None


class LoadModelRequest(BaseModel):
    """Request to load/replace the in-memory model version."""

    model_name: str = Field(default="fraud-classifier")
    version_name: str = Field(default="v1")
    artifact_name: str = Field(default="sklearn_model")


@model_router.post("/load", response_model=ArtifactVersionResponse)
def load_model(req: LoadModelRequest) -> ArtifactVersionResponse:
    """Load or replace the in-memory model version."""
    global CURRENT_MODEL, CURRENT_MODEL_ARTIFACT

    model_version = Client().get_model_version(
        req.model_name, req.version_name
    )
    CURRENT_MODEL_ARTIFACT = model_version.get_artifact(req.artifact_name)
    CURRENT_MODEL = CURRENT_MODEL_ARTIFACT.load()

    return CURRENT_MODEL_ARTIFACT


@model_router.get("/current", response_model=ArtifactVersionResponse)
def current_model() -> ArtifactVersionResponse:
    """Return the artifact of the currently loaded in-memory model."""

    if CURRENT_MODEL_ARTIFACT is None:
        raise HTTPException(
            status_code=status.HTTP_404_NOT_FOUND,
            detail="No model loaded. Use /model/load first.",
        )

    return CURRENT_MODEL_ARTIFACT

deploy_settings = DeploymentSettings(
    custom_endpoints=[
        EndpointSpec(
            path="/model",
            method=EndpointMethod.POST,  # method is ignored for native routers
            handler="my_project.fastapi_endpoints.model_router",
            native=True,
            auth_required=True,
        )
    ]
)
```

And here is a minimal ZenML inference pipeline that uses the globally loaded model. The prediction step reads the model from the global variable set by the FastAPI router above. You can invoke this pipeline via the built-in `/invoke` endpoint once a model has been loaded through `/model/load`.

```python
from typing import List

from pydantic import BaseModel
from zenml import pipeline, step


class InferenceRequest(BaseModel):
    features: List[float]


class InferenceResponse(BaseModel):
    prediction: float


@step
def preprocess_step(request: InferenceRequest) -> List[float]:
    # Replace with real transformations, scaling, encoding, etc.
    return request.features

@step
def predict_step(features: List[float]) -> InferenceResponse:
    """Run model inference using the globally loaded model."""

    if GLOBAL_CURRENT_MODEL is None:
        raise RuntimeError(
            "No model loaded. Call /model/load before invoking."
        )

    pred = float(GLOBAL_CURRENT_MODEL.predict([features])[0])
    return InferenceResponse(prediction=pred)


@pipeline(settings={"deployment": deploy_settings})
def inference_pipeline(request: InferenceRequest) -> InferenceResponse:
    processed = preprocess_step(request)
    return predict_step(processed)
```

#### Custom middleware

Middleware is where you enforce cross-cutting concerns consistently across every endpoint. Common use-cases include:

* Security and access control
  * API key/JWT verification, tenant extraction and context injection
  * IP allow/deny lists, basic WAF-style request filtering, mTLS header checks
  * Request body/schema validation and max body size enforcement
* Governance and privacy
  * PII detection/redaction on inputs/outputs; payload sampling/scrubbing
  * Policy enforcement (data residency, retention, consent) at request time
* Reliability and traffic shaping
  * Rate limiting, quotas, per-tenant concurrency limits
  * Idempotency keys, deduplication, retries with backoff, circuit breakers
  * Timeouts, slow-request detection, maintenance mode and graceful drain
* Observability
  * Correlation/trace IDs, OpenTelemetry spans, structured logging
  * Metrics for latency, throughput, error rates, request/response sizes
* Performance and caching
  * Response caching/ETags, compression (gzip/br), streaming/chunked responses
  * Adaptive content negotiation and serialization tuning
* LLM/agent-specific controls
  * Token accounting/limits, cost guards per tenant/user
  * Guardrails (toxicity/PII/jailbreak) and output filtering
  * Tool execution sandboxing gates and allowlists
* Data and feature enrichment
  * Feature store prefetch, user/tenant profile enrichment, AB bucketing tags

You can configure `custom_middlewares` in `DeploymentSettings` to insert your own ASGI middleware.

Middlewares support multiple definition modes (see code examples below):

1. Middleware class - a standard ASGI middleware class that implements the `__call__` method that takes the traditional `scope`, `receive` and `send` arguments. The constructor must accept an `app` argument of type `ASGIApplication` and any additional keyword arguments.
2. Middleware callable - a callable that takes all arguments in one go: `app`, `scope`, `receive` and `send`.
3. Native framework-specific middleware (`native=True`) - this can vary from ASGI framework to framework.

Definitions can be provided as Python objects or as loadable source path strings. The `order` parameter controls the insertion order in the middleware chain. Lower `order` values insert the middleware earlier in the chain.

The following code examples demonstrate the different definition modes for custom middlewares:

1. a custom middleware that adds a processing time header to every response, implemented as a middleware class:

```python
import time
from typing import Any
from asgiref.compatibility import guarantee_single_callable
from asgiref.typing import (
    ASGIApplication,
    ASGIReceiveCallable,
    ASGISendCallable,
    ASGISendEvent,
    Scope,
)
from zenml.config import DeploymentSettings, MiddlewareSpec

class RequestTimingMiddleware:
    """ASGI middleware to measure request processing time."""

    def __init__(self, app: ASGIApplication, header_name: str = "x-process-time-ms") -> None:
        self.app = guarantee_single_callable(app)
        self.header_name = header_name

    async def __call__(
        self,
        scope: Scope,
        receive: ASGIReceiveCallable,
        send: ASGISendCallable,
    ) -> None:
        if scope["type"] != "http":
            await self.app(scope, receive, send)
            return

        start_time = time.time()

        async def send_wrapper(message: ASGISendEvent) -> None:
            if message["type"] == "http.response.start":
                process_time = (time.time() - start_time) * 1000
                headers = list(message.get("headers", []))
                headers.append((self.header_name.encode(), str(process_time).encode()))
                message = {**message, "headers": headers}

            await send(message)

        await self.app(scope, receive, send_wrapper)


settings = DeploymentSettings(
    custom_middlewares=[
        MiddlewareSpec(
            middleware=RequestTimingMiddleware,
            order=10,
            init_kwargs={"header_name": "x-process-time-ms"},
        ),
    ]
)
```

2. a custom middleware that injects a correlation ID into responses (and generates one if missing), implemented as a middleware callable:

```python
import uuid
from typing import Any
from asgiref.compatibility import guarantee_single_callable
from asgiref.typing import (
    ASGIApplication,
    ASGIReceiveCallable,
    ASGISendCallable,
    ASGISendEvent,
    Scope,
)
from zenml.config import DeploymentSettings, MiddlewareSpec

async def request_id_middleware(
    app: ASGIApplication,
    scope: Scope,
    receive: ASGIReceiveCallable,
    send: ASGISendCallable,
    header_name: str = "x-request-id",
) -> None:
    """ASGI function middleware that ensures a correlation ID header exists."""

    app = guarantee_single_callable(app)

    if scope["type"] != "http":
        await app(scope, receive, send)
        return

    # Reuse existing request ID if present; otherwise generate one
    request_id = None
    for k, v in scope.get("headers", []):
        if k.decode().lower() == header_name:
            request_id = v.decode()
            break

    if not request_id:
        request_id = str(uuid.uuid4())

    async def send_wrapper(message: ASGISendEvent) -> None:
        if message["type"] == "http.response.start":
            headers = list(message.get("headers", []))
            headers.append((header_name.encode(), request_id.encode()))
            message = {**message, "headers": headers}

        await send(message)

    await app(scope, receive, send_wrapper)


settings = DeploymentSettings(
    custom_middlewares=[
        MiddlewareSpec(
            middleware=request_id_middleware,
            order=5,
            init_kwargs={"header_name": "x-request-id"},
        ),
    ]
)
```

4. a FastAPI/Starlette-native middleware that adds GZIP support, implemented as a native middleware:

```python
from starlette.middleware.gzip import GZipMiddleware
from zenml.config import DeploymentSettings, MiddlewareSpec

settings = DeploymentSettings(
    custom_middlewares=[
        MiddlewareSpec(
            middleware=GZipMiddleware,
            native=True,
            order=20,
            extra_kwargs={"minimum_size": 1024},
        ),
    ]
)
```

#### App extensions

App extensions are pluggable components that are running as part of the ASGI application factory that can install complex, possibly framework-specific structures. The following are usual scenarios for using a full-blown extension instead of endpoints/middleware:

* Advanced authentication and authorization
  * install org-wide dependencies (e.g., OAuth/OIDC auth, RBAC guards)
  * register custom exception handlers for uniform error envelopes
  * augment OpenAPI with security schemes and per-route security policies
* Multi-tenant and routing topology
  * programmatically include routers per tenant/region/version
  * mount sub-apps for internal admin vs public APIs under different prefixes
  * dynamic route rewrites/switches for blue/green or canary rollouts
* Observability and platform integration
  * wire OpenTelemetry instrumentation at the app level (tracer/meter providers)
  * register global request/response logging with redaction policies
  * expose or mount vendor-specific observability apps (e.g., Prometheus)
* LLM agent control plane
  * attach a tool registry/router and lifecycle hooks for tools
  * register guardrail handlers and policy engines across routes
  * install runtime prompt/template catalogs and index management routers
* API ergonomics and governance
  * reshape OpenAPI (tags, servers, components) and versioned docs
  * global response model wrapping, pagination conventions, error mappers
  * maintenance-mode switch and graceful-drain controls at the app level

App extensions support multiple definition modes (see code examples below):

1. Extension class - a class that implements the `BaseAppExtension` abstract class. The class constructor must accept any keyword arguments and the `install` method must accept an `app_runner` argument of type `BaseDeploymentAppRunner`.
2. Extension callable - a callable that takes the `app_runner` argument of type `BaseDeploymentAppRunner`.

Both classes and callables must take in an `app_runner` argument of type `BaseDeploymentAppRunner`. This is the application factory that is responsible for building the ASGI application. You can use it to access information such as:

* the ASGI application instance that is being built
* the deployment service instance that is being deployed
* the `DeploymentResponse` object itself, which also contains details about the snapshot, pipeline, etc.

Definitions can be provided as Python objects or as loadable source path strings.

The extensions are summoned to take part in the ASGI application building process near the end of the initialization - after the ASGI app has been built according to the deployment configuration settings.

The example below installs API key authentication at the FastAPI application level, attaches the dependency to selected routes, registers an auth error handler, and augments the OpenAPI schema with the security scheme.

```python
from __future__ import annotations

from typing import Literal, Sequence, Set

from fastapi import FastAPI, HTTPException, Request, status
from fastapi.openapi.utils import get_openapi
from fastapi.responses import JSONResponse
from fastapi.security import APIKeyHeader

from zenml.config import AppExtensionSpec, DeploymentSettings
from zenml.deployers.server.app import BaseDeploymentAppRunner
from zenml.deployers.server.extensions import BaseAppExtension


class FastAPIAuthExtension(BaseAppExtension):
    """Install API key auth and OpenAPI security on a FastAPI app."""

    def __init__(
        self,
        scheme: Literal["api_key"] = "api_key",
        header_name: str = "x-api-key",
        valid_keys: Sequence[str] | None = None,
    ) -> None:
        self.scheme = scheme
        self.header_name = header_name
        self.valid_keys: Set[str] = set(valid_keys or [])

    def install(self, app_runner: BaseDeploymentAppRunner) -> None:
        app = app_runner.asgi_app
        if not isinstance(app, FastAPI):
            raise RuntimeError("FastAPIAuthExtension requires FastAPI")

        api_key_header = APIKeyHeader(
            name=self.header_name, auto_error=True
        )

        # Find endpoints that have auth_required=True
        protected_endpoints = [
            endpoint.path
            for endpoint in app_runner.endpoints
            if endpoint.auth_required
        ]

        @app.middleware("http")
        async def api_key_guard(request: Request, call_next):
            if request.url.path in protected_endpoints:
                api_key = await api_key_header(request)
                if api_key not in self.valid_keys:
                    raise HTTPException(
                        status_code=status.HTTP_401_UNAUTHORIZED,
                        detail="Invalid or missing API key",
                    )
            return await call_next(request)

        # Auth error handler
        @app.exception_handler(HTTPException)
        async def auth_exception_handler(
            _, exc: HTTPException
        ) -> JSONResponse:
            if exc.status_code == status.HTTP_401_UNAUTHORIZED:
                return JSONResponse(
                    status_code=exc.status_code,
                    content={"detail": exc.detail},
                    headers={"WWW-Authenticate": "ApiKey"},
                )
            return JSONResponse(
                status_code=exc.status_code, content={"detail": exc.detail}
            )

        # OpenAPI security
        def custom_openapi() -> dict:
            if app.openapi_schema:
                return app.openapi_schema  # type: ignore[return-value]
            openapi_schema = get_openapi(
                title=app.title,
                version=app.version if app.version else "0.1.0",
                description=app.description,
                routes=app.routes,
            )
            components = openapi_schema.setdefault("components", {})
            security_schemes = components.setdefault("securitySchemes", {})
            security_schemes["ApiKeyAuth"] = {
                "type": "apiKey",
                "in": "header",
                "name": self.header_name,
            }
            openapi_schema["security"] = [{"ApiKeyAuth": []}]
            app.openapi_schema = openapi_schema
            return openapi_schema

        app.openapi = custom_openapi  # type: ignore[assignment]


settings = DeploymentSettings(
    app_extensions=[
        AppExtensionSpec(
            extension=(
                "my_project.extensions.FastAPIAuthExtension"
            ),
            extension_kwargs={
                "scheme": "api_key",
                "header_name": "x-api-key",
                "valid_keys": ["secret-1", "secret-2"],
            },
        )
    ]
)
```

### Implementation customizations for advanced use cases

For cases where you need deeper control over how the ASGI app is created or how the deployment logic is implemented, you can swap/extend the core components using the following `DeploymentSettings` fields:

* `deployment_app_runner_flavor` and `deployment_app_runner_kwargs` let you choose or extend the app runner that constructs and runs the ASGI app. This needs to be set to a subclass of `BaseDeploymentAppRunnerFlavor`, which is basically a descriptor of an app runner implementation that itself is a subclass of `BaseDeploymentAppRunner`.
* `deployment_service_class` and `deployment_service_kwargs` let you provide your own deployment service to customize the pipeline deployment logic. This needs to be set to a subclass of `BasePipelineDeploymentService`.

Both accept loadable sources or objects. We cover how to implement custom runner flavors and services in a dedicated guide.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Containerization

Customize Docker builds to run your pipelines in isolated, well-defined environments.

ZenML executes pipeline steps sequentially in the active Python environment when running locally. However, with remote [orchestrators](https://docs.zenml.io/stacks/orchestrators) or [step operators](https://docs.zenml.io/stacks/step-operators), ZenML builds [Docker](https://www.docker.com/) images to run your pipeline in an isolated, well-defined environment.

This page explains how ZenML's Docker build process works and how you can customize it to meet your specific requirements.

## Understanding Docker Builds in ZenML

When a pipeline is run with a remote orchestrator, a Dockerfile is dynamically generated at runtime. It is then used to build the Docker image using the image builder component of your stack. The Dockerfile consists of the following steps:

1. **Starts from a parent image** that has ZenML installed. By default, this will use the [official ZenML image](https://hub.docker.com/r/zenmldocker/zenml/) for the Python and ZenML version that you're using in the active Python environment.
2. **Installs additional pip dependencies**. ZenML automatically detects which integrations are used in your stack and installs the required dependencies.
3. **Optionally copies your source files**. Your source files need to be available inside the Docker container so ZenML can execute your step code.
4. **Sets user-defined environment variables.**

The process described above is automated by ZenML and covers most basic use cases. This page covers various ways to customize the Docker build process to fit your specific needs.

### Docker Build Process

ZenML uses the following process to decide how to build Docker images:

* **No `dockerfile` specified**: If any of the options regarding requirements, environment variables, or copying files require us to build an image, ZenML will build this image. Otherwise, the `parent_image` will be used to run the pipeline.
* **`dockerfile` specified**: ZenML will first build an image based on the specified Dockerfile. If any additional options regarding requirements, environment variables, or copying files require an image built on top of that, ZenML will build a second image. If not, the image built from the specified Dockerfile will be used to run the pipeline.

### Requirements Installation Order

Depending on the configuration of your Docker settings, requirements will be installed in the following order (each step is optional):

1. The packages installed in your local Python environment (if enabled)
2. The packages required by the stack (unless disabled by setting `install_stack_requirements=False`)
3. The packages specified via the `required_integrations`
4. The packages specified via the `requirements` attribute

For a full list of configuration options, check out [the DockerSettings object on the SDKDocs](https://sdkdocs.zenml.io/latest/core_code_docs/core-config.html#zenml.config.DockerSettings).

## Configuring Docker Settings

You can customize Docker builds for your pipelines and steps using the `DockerSettings` class:

```python
from zenml.config import DockerSettings
```

There are multiple ways to supply these settings:

### Pipeline-Level Settings

Configuring settings on a pipeline applies them to all steps of that pipeline:

```python
from zenml import pipeline, step
from zenml.config import DockerSettings

docker_settings = DockerSettings()

@step
def my_step() -> None:
    """Example step."""
    pass

# Either add it to the decorator
@pipeline(settings={"docker": docker_settings})
def my_pipeline() -> None:
    my_step()

# Or configure the pipelines options
my_pipeline = my_pipeline.with_options(
    settings={"docker": docker_settings}
)
```

### Step-Level Settings

For more fine-grained control, configure settings on individual steps. This is particularly useful when different steps have conflicting requirements or when some steps need specialized environments:

```python
from zenml import step
from zenml.config import DockerSettings

docker_settings = DockerSettings()

# Either add it to the decorator
@step(settings={"docker": docker_settings})
def my_step() -> None:
    pass

# Or configure the step options
my_step = my_step.with_options(
    settings={"docker": docker_settings}
)
```

### Using YAML Configuration

Define settings in a YAML configuration file for better separation of code and configuration:

```yaml
settings:
    docker:
        parent_image: python:3.11-slim
        apt_packages:
          - git
          - curl
        requirements:
          - tensorflow==2.8.0
          - pandas

steps:
  training_step:
    settings:
        docker:
            parent_image: pytorch/pytorch:2.2.0-cuda11.8-cudnn8-runtime
            required_integrations:
              - wandb
              - mlflow
```

Check out [this page](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) for more information on the hierarchy and precedence of the various ways in which you can supply the settings.

### Specifying Docker Build Options

You can customize the build process by specifying build options that get passed to the build method of the image builder:

```python
from zenml import pipeline
from zenml.config import DockerSettings

docker_settings = DockerSettings(
    build_config={"build_options": {"buildargs": {"MY_ARG": "value"}}}
)

@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
    ...
```

For the default local image builder with the **Docker** engine, these options are passed to the Docker build API (or to the `docker build` CLI if you enabled subprocess mode on the local image builder). With the **Podman** engine, builds always use the Podman CLI and the same option mapping as subprocess mode on Docker.

{% hint style="info" %}
If you're running your pipelines on MacOS with ARM architecture, the local Docker caching does not work unless you specify the target platform of the image:

```python
from zenml import pipeline
from zenml.config import DockerSettings

docker_settings = DockerSettings(
    build_config={"build_options": {"platform": "linux/amd64"}}
)

@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
    ...
```

{% endhint %}

## Using Custom Parent Images

### Pre-built Parent Images

To use a static parent image (e.g., with internal dependencies pre-installed):

```python
from zenml import pipeline
from zenml.config import DockerSettings

docker_settings = DockerSettings(parent_image="my_registry.io/image_name:tag")

@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
    ...
```

ZenML will use this image as the base and still perform the following steps:

1. Install additional pip dependencies
2. Copy source files (if configured)
3. Set environment variables

{% hint style="info" %}
If you're going to use a custom parent image, you need to make sure that it has Python, pip, and ZenML installed for it to work. If you need a starting point, you can take a look at the Dockerfile that ZenML uses [here](https://github.com/zenml-io/zenml/blob/main/docker/base.Dockerfile).
{% endhint %}

### Skip Build Process

To use the image directly to run your steps without including any code or installing any requirements on top of it, skip the Docker builds by setting `skip_build=True`:

```python
docker_settings = DockerSettings(
    parent_image="my_registry.io/image_name:tag",
    skip_build=True
)

@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
    ...
```

When `skip_build` is enabled, the `parent_image` will be used directly to run the steps of your pipeline without any additional Docker builds on top of it. This means that **none** of the following will happen:

* No installation of local Python environment packages
* No installation of stack requirements
* No installation of required integrations
* No installation of specified requirements
* No installation of apt packages
* No inclusion of source files in the container
* No setting of environment variables

{% hint style="warning" %}
This is an advanced feature and may cause unintended behavior when running your pipelines. If you use this, ensure your image contains everything necessary to run your pipeline:

1. Your stack requirements
2. Integration requirements
3. Project-specific requirements
4. Any system packages
5. Your project code files (unless a code repository is registered or `allow_download_from_artifact_store` is enabled)

Make sure that Python, `pip` and `zenml` are installed in your image, and that your code is in the `/app` directory set as the active working directory.

Also note that the Docker settings validator will raise an error if you set `skip_build=True` without specifying a `parent_image`. A parent image is required when skipping the build as it will be used directly to run your pipeline steps.
{% endhint %}

### Custom Dockerfiles

For greater control, you can specify a custom Dockerfile and build context:

```python
docker_settings = DockerSettings(
    dockerfile="/path/to/dockerfile",
    build_context_root="/path/to/build/context",
    parent_image_build_config={
        "build_options": {"buildargs": {"MY_ARG": "value"}},
        "dockerignore": "/path/to/.dockerignore"
    }
)

@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
    ...
```

Here is how the build process looks like with a custom Dockerfile:

* **`Dockerfile` specified**: ZenML will first build an image based on the specified `Dockerfile`. If any options regarding requirements, environment variables, or copying files require an additional image built on top of that, ZenML will build a second image. Otherwise, the image built from the specified `Dockerfile` will be used to run the pipeline.

{% hint style="info" %}
Important notes about using a custom Dockerfile:

* When you specify a custom `dockerfile`, the `parent_image` attribute will be ignored
* The image built from your Dockerfile must have ZenML installed
* If you set `build_context_root`, that directory will be used as the build context for the Docker build. If left empty, the build context will only contain the Dockerfile
* You can configure the build options by setting `parent_image_build_config` with specific build options and dockerignore settings
  {% endhint %}

## Managing Dependencies

ZenML offers several ways to specify dependencies for your Docker containers:

### Python Dependencies

By default, ZenML automatically installs all packages required by your active ZenML stack.

{% hint style="warning" %}
In future versions, if none of the `replicate_local_python_environment`, `pyproject_path` or `requirements` attributes on `DockerSettings` are specified, ZenML will try to automatically find a `requirements.txt` and `pyproject.toml` files inside your current [source root](/concepts/steps_and_pipelines/sources#source-root) and install packages from the first one it finds. You can disable this behavior by setting `disable_automatic_requirements_detection=True`. If you already want this automatic detection in current versions of ZenML, set `disable_automatic_requirements_detection=False`.
{% endhint %}

1. **Replicate Local Environment**:

   ```python
   from zenml import pipeline
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(replicate_local_python_environment=True)


   @pipeline(settings={"docker": docker_settings})
   def my_pipeline(...):
       ...
   ```

   This will run `pip freeze` to get a list of the installed packages in your local Python environment and will install them in the Docker image. This ensures that the same exact dependencies will be installed.

   <div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><p>This does not work when you have a local project installed. To install local projects, check out the <code>Install Local Projects</code> section below.</p></div>
2. **Specify a `pyproject.toml` file**:

   ```python
   from zenml import pipeline
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(pyproject_path="/path/to/pyproject.toml")

   @pipeline(settings={"docker": docker_settings})
   def my_pipeline(...):
       ...
   ```

   By default, ZenML will try to export the dependencies specified in the `pyproject.toml` by trying to run `uv export` and `poetry export`. If both of these commands do not work for your `pyproject.toml` file or you want to customize the command (for example to install certain extras), you can specify a custom command using the `pyproject_export_command` attribute. This command must output a list of requirements following the format of the [requirements file](https://pip.pypa.io/en/stable/reference/requirements-file-format/). The command can contain a `{directory}` placeholder which will be replaced with the directory in which the `pyproject.toml` file is stored.

   ```python
   from zenml import pipeline
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(pyproject_export_command=[
       "uv",
       "export",
       "--extra=train",
       "--format=requirements-txt",
       "--directory={directory}"
   ])


   @pipeline(settings={"docker": docker_settings})
   def my_pipeline(...):
       ...
   ```
3. **Specify Requirements Directly**:

   ```python
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(requirements=["torch==1.12.0", "torchvision"])
   ```
4. **Use Requirements File**:

   ```python
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(requirements="/path/to/requirements.txt")
   ```
5. **Specify ZenML Integrations**:

   ```python
   from zenml.integrations.constants import PYTORCH, EVIDENTLY
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(required_integrations=[PYTORCH, EVIDENTLY])
   ```
6. **Control Stack Requirements**: By default, ZenML installs the requirements needed by your active stack. You can disable this behavior if needed:

   ```python
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(install_stack_requirements=False)
   ```
7. **Control Deployment Requirements**: By default, if you have a Deployer stack component in your active stack, ZenML installs the requirements needed by the deployment application configured in your deployment settings. You can disable this behavior if needed:

   ```python
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(install_deployment_requirements=False)
   ```
8. **Install Local Projects**: If your code requires the installation of some local code files as a python package, you can specify a command that installs it as follows:

   ```python
   from zenml.config import DockerSettings

   docker_settings = DockerSettings(local_project_install_command="pip install . --no-deps")
   ```

   <div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><p>Installing a local python package only works if your code files are included in the Docker image, so make sure you have <code>allow_including_files_in_images=True</code> in your Docker settings. If you want to instead use the <a href="#source-code-management">code download functionality</a> to avoid building new Docker images for each pipeline run, you can follow <a href="https://github.com/zenml-io/zenml-patterns/tree/main/docker-local-pkg">this example</a>.</p></div>

Depending on the options specified in your Docker settings, ZenML installs the requirements in the following order (each step optional):

1. The packages installed in your local Python environment
2. The packages required by the stack (unless disabled by setting `install_stack_requirements=False`)
3. The packages specified via the `required_integrations`
4. The packages defined in the pyproject.toml file specified by the `pyproject_path` attribute
5. The packages specified via the `requirements` attribute

### System Packages

Specify apt packages to be installed in the Docker image:

```python
from zenml.config import DockerSettings

docker_settings = DockerSettings(apt_packages=["git", "curl", "libsm6", "libxext6"])
```

### Installation Control

Control how packages are installed:

```python
# Use custom installer arguments
docker_settings = DockerSettings(python_package_installer_args={"timeout": 1000})

# Use pip instead of uv
from zenml.config import DockerSettings, PythonPackageInstaller
docker_settings = DockerSettings(python_package_installer=PythonPackageInstaller.PIP)
# Or as a string
docker_settings = DockerSettings(python_package_installer="pip")

# Use uv (default)
docker_settings = DockerSettings(python_package_installer=PythonPackageInstaller.UV)
```

The available package installers are:

* `uv`: The default python package installer
* `pip`: An alternative python package installer

Full documentation for how `uv` works with PyTorch can be found on the Astral Docs website [here](https://docs.astral.sh/uv/guides/integration/pytorch/). It covers some of the particular gotchas and details you might need to know.

{% hint style="info" %}
If you're using `uv` and specify a custom parent image or Dockerfile that does not have an activated virtual environment, you need to pass `python_package_installer_args={"system": None}` in your DockerSettings so that `uv` installs the packages for the Python system installation. Depending on the parent image, you might also need to include `"break-system-packages": None` in the installer args as well to make it work.
{% endhint %}

To speed up repeated image builds, set `python_package_installer_cache_mount` to a BuildKit `--mount` spec. ZenML emits `RUN --mount=<value> ...` on the install steps and drops the default `--no-cache-dir` flag so the mount is actually used. The value is passed through verbatim, so any valid `--mount` spec works (e.g. `type=cache` or `type=bind`). Requires BuildKit to be enabled on the builder.

```python
# uv (default installer)
docker_settings = DockerSettings(
    python_package_installer_cache_mount="type=cache,target=/root/.cache/uv",
)

# pip
docker_settings = DockerSettings(
    python_package_installer="pip",
    python_package_installer_cache_mount="type=cache,target=/root/.cache/pip",
)
```

### Using custom python executable

To use a custom python executable, instead of a standard `python` you can use the `ZENML_CONTAINER_PYTHON_EXECUTABLE` environment variable to control the python executable being used in the entrypoint command, for example `ZENML_CONTAINER_PYTHON_EXECUTABLE=/home/user1/custom-venv/bin/python` can be used.

## Private PyPI Repositories

For packages that require authentication from private repositories:

```python
import os

docker_settings = DockerSettings(
    requirements=["my-internal-package==0.1.0"],
    environment={
        'PIP_EXTRA_INDEX_URL': f"https://{os.environ.get('PYPI_TOKEN', '')}@my-private-pypi-server.com/{os.environ.get('PYPI_USERNAME', '')}/"}
)
```

Be cautious with handling credentials. Always use secure methods to manage and distribute authentication information within your team. Consider using secrets management tools or environment variables passed securely.

## Source Code Management

You can specify how the files inside your [source root directory](/concepts/steps_and_pipelines/sources#source-root) are handled for containerized steps:

```python
docker_settings = DockerSettings(
    # Download files from code repository if available
    allow_download_from_code_repository=True,
    # If no code repository, upload code to artifact store
    allow_download_from_artifact_store=True,
    # If neither of the above, include files in the image
    allow_including_files_in_images=True
)
```

ZenML handles your source code in the following order:

1. If `allow_download_from_code_repository` is `True` and your files are inside a registered [code repository](https://docs.zenml.io/user-guides/production-guide/connect-code-repository) and the repository has no local changes, the files will be downloaded from the code repository and not included in the image.
2. If the previous option is disabled or no code repository without local changes exists for the root directory, ZenML will archive and upload your code to the artifact store if `allow_download_from_artifact_store` is `True`.
3. If both previous options were disabled or not possible, ZenML will include your files in the Docker image if `allow_including_files_in_images` is enabled. This means a new Docker image has to be built each time you modify one of your code files.

{% hint style="warning" %}
Setting all of the above attributes to `False` is not recommended and will most likely cause unintended and unanticipated behavior when running your pipelines. If you do this, you're responsible that all your files are at the correct paths in the Docker images that will be used to run your pipeline steps.
{% endhint %}

### Controlling Included Files

* When downloading files from a code repository, use a `.gitignore` file to exclude files.
* When including files in the image, use a `.dockerignore` file to exclude files and keep the image smaller:

  ```python
  # Have a file called .dockerignore in your source root directory
  # Or explicitly specify a .dockerignore file to use:
  docker_settings = DockerSettings(build_config={"dockerignore": "/path/to/.dockerignore"})
  ```

## Environment Variables

You can configure two types of environment variables:

1. Environment variables that will be set in the beginning of the Docker image building process before any python or apt packages are installed:

```python
docker_settings = DockerSettings(
    environment={
        "PYTHONUNBUFFERED": "1",
        "MODEL_DIR": "/models",
        "API_KEY": "${GLOBAL_API_KEY}"  # Reference a local environment variable
    }
)
```

2. Environment variables that will be set at the end of the Docker image building process after the python and apt packages are installed, right before the container entrypoint (useful for setting proxy environment variables for example):

```python
docker_settings = DockerSettings(
    runtime_environment={
        "HTTP_PROXY": "http://proxy.example.com:8080",
        "HTTPS_PROXY": "http://proxy.example.com:8080",
        "NO_PROXY": "localhost,127.0.0.1"
    }
)
```

Environment variables can reference other environment variables set in your client environment by using the `${VAR_NAME}` syntax. ZenML will substitute these before building the images.

## Build Reuse and Optimization

ZenML automatically reuses Docker builds when possible to save time and resources:

### What is a Pipeline Build?

A pipeline build is an encapsulation of a pipeline and the stack it was run on. It contains the Docker images that were built for the pipeline with all required dependencies from the stack, integrations and the user. Optionally, it also contains the pipeline code.

List all available builds for a pipeline:

```bash
zenml pipeline builds list --pipeline_id='startswith:ab53ca'
```

Create a build manually (useful for pre-building images):

```bash
zenml pipeline build --stack vertex-stack my_module.my_pipeline_instance
```

You can use options to specify the configuration file and the stack to use for the build. Learn more about the build function [here](https://sdkdocs.zenml.io/latest/cli.html#zenml.cli.Pipeline.build).

### Reusing Builds

By default, when you run a pipeline, ZenML will check if a build with the same pipeline and stack exists. If it does, it will reuse that build automatically. However, you can also force using a specific build by providing its ID:

```python
pipeline_instance.run(build="<build_id>")
```

You can also specify this in configuration files:

```yaml
build: your-build-id-here
```

{% hint style="warning" %}
Specifying a custom build when running a pipeline will **not run the code on your client machine** but will use the code **included in the Docker images of the build**. Even if you make local code changes, reusing a build will *always* execute the code bundled in the Docker image, rather than the local code.
{% endhint %}

### Controlling Image Repository Names

You can control where your Docker image is pushed by specifying a target repository name:

```python
from zenml.config import DockerSettings

docker_settings = DockerSettings(target_repository="my-custom-repo-name")
```

The repository name will be appended to the registry URI of your container registry stack component. For example, if your container registry URI is `gcr.io/my-project` and you set `target_repository="zenml-pipelines"`, the full image name would be `gcr.io/my-project/zenml-pipelines`.

If you don't specify a target repository, the default repository name configured in your container registry stack component settings will be used.

### Specifying Image tags

You can control the tag of the generated Docker images using the image tag option:

```python
from zenml.config import DockerSettings

docker_settings = DockerSettings(image_tag="1.0.0")
```

Keep in mind that this will be applied to all images built using the DockerSettings object. If there are multiple such images, only one of them will keep the tag while the rest will be untagged.

### Decoupling Code from Builds

To reuse Docker builds while still using your latest code changes, you need to decouple your code from the build. There are two main approaches:

#### 1. Using the Artifact Store to Upload Code

You can let ZenML use the artifact store to upload your code. This is the default behavior if no code repository is detected and the `allow_download_from_artifact_store` flag is not set to `False` in your `DockerSettings`.

#### 2. Using Code Repositories for Faster Builds

Registering a [code repository](/concepts/code-repositories) lets you avoid building images each time you run a pipeline **and** quickly iterate on your code. When running a pipeline that is part of a local code repository checkout, ZenML can instead build the Docker images without including any of your source files, and download the files inside the container before running your code.

ZenML will **automatically figure out which builds match your pipeline and reuse the appropriate build id**. Therefore, you **do not** need to explicitly pass in the build id when you have a clean repository state and a connected git repository.

{% hint style="warning" %}
In order to benefit from the advantages of having a code repository in a project, you need to make sure that **the relevant integrations are installed for your ZenML installation.**. For instance, let's assume you are working on a project with ZenML and one of your team members has already registered a corresponding code repository of type `github` for it. If you do `zenml code-repository list`, you would also be able to see this repository. However, in order to fully use this repository, you still need to install the corresponding integration for it, in this example the `github` integration.

```sh
zenml integration install github
```

{% endhint %}

#### Detecting local code repository checkouts

Once you have registered one or more code repositories, ZenML will check whether the files you use when running a pipeline are tracked inside one of those code repositories. This happens as follows:

* First, the [source root](/concepts/steps_and_pipelines/sources#source-root) is computed
* Next, ZenML checks whether this source root directory is included in a local checkout of one of the registered code repositories

#### Tracking code versions for pipeline runs

If a local code repository checkout is detected when running a pipeline, ZenML will store a reference to the current commit for the pipeline run, so you'll be able to know exactly which code was used.

Note that this reference is only tracked if your local checkout is clean (i.e. it does not contain any untracked or uncommitted files). This is to ensure that your pipeline is actually running with the exact code stored at the specific code repository commit.

{% hint style="info" %}
If you want to ignore untracked files, you can set the `ZENML_CODE_REPOSITORY_IGNORE_UNTRACKED_FILES` environment variable to `True`. When doing this, you're responsible that the files committed to the repository includes everything necessary to run your pipeline.
{% endhint %}

#### Preventing Build Reuse

There might be cases where you want to force a new build, even if a suitable existing build is available. You can do this by setting `prevent_build_reuse=True`:

```python
docker_settings = DockerSettings(prevent_build_reuse=True)
```

This is useful in scenarios like:

* When you've made changes to your image building process that aren't tracked by ZenML
* When troubleshooting issues in your Docker image
* When you want to ensure your Docker image uses the most up-to-date base images

#### Tips and Best Practices for Build Reuse

* **Clean Repository State**: The file download is only possible if the local checkout is clean (no untracked or uncommitted files) and the latest commit has been pushed to the remote repository.
* **Configuration Options**: If you want to disable or enforce downloading of files, check the [DockerSettings](https://sdkdocs.zenml.io/latest/index.html#zenml.config.DockerSettings) for available options.
* **Team Collaboration**: Using code repositories allows team members to reuse images that colleagues might have built for the same stack, enhancing collaboration efficiency.
* **Build Selection**: ZenML automatically selects matching builds, but you can override this with explicit build IDs for special cases.

## Image Build Location

By default, execution environments are created on your machine using a **local container engine** (see [Choosing the Container Engine](#choosing-the-container-engine) below). ZenML also offers [image builders](https://docs.zenml.io/stacks/image-builders), a special [stack component](https://docs.zenml.io/stacks), so you can build and push images in a different specialized *image builder environment* instead.

Note that even if you don't configure an image builder in your stack, ZenML still uses the [local image builder](https://docs.zenml.io/stacks/image-builders/local) to retain consistency across all builds. In this case, the image builder environment is the same as the [client environment](https://docs.zenml.io/user-guides/best-practices/configure-python-environments#client-environment-or-the-runner-environment).

You don't need to directly interact with any image builder in your code. As long as the image builder that you want to use is part of your active [ZenML stack](https://docs.zenml.io/user-guides/production-guide/understand-stacks), it will be used automatically by any component that needs to build container images.

### Choosing the Container Engine

ZenML uses the container engine (**Docker** or **Podman** currently) available on your machine for work that happens locally: building and pushing images with the [local image builder](https://docs.zenml.io/stacks/image-builders/local), preparing pipeline images before remote runs, and similar client-side steps. This is a global setting of your ZenML client.

**Default behavior:** if you do not configure an engine, ZenML **auto-selects**: it tries **Docker** first, then **Podman** if Docker is not available. So on a typical developer laptop with Docker Desktop, **Docker is used** without any extra configuration.

**To pin an engine**, use the environment variable **`ZENML_CONTAINER_ENGINE`** set to `docker` or `podman`.

For example, to use Podman everywhere on that machine:

```shell
export ZENML_CONTAINER_ENGINE=podman
```

Parts of ZenML that require the Docker daemon API (for example the [local Docker orchestrator](https://docs.zenml.io/stacks/orchestrators/local-docker)) still expect Docker; switching the global engine to Podman affects local image build/push and client-side image preparation, not those Docker-only features.

More detail on credentials, subprocess builds, and prerequisites is on the [local image builder](https://docs.zenml.io/stacks/image-builders/local) page.

## Container User Permissions

By default, Docker containers often run as the `root` user, which can pose security risks. ZenML allows you to specify a different user to run your containers:

```python
docker_settings = DockerSettings(user="non-root-user")
```

When you set the `user` parameter:

* The specified user will become the owner of the `/app` directory, which contains all your code
* The container entrypoint will run as this user instead of root
* This can help improve security by following the principle of least privilege

## Best Practices

1. **Use code repositories** to speed up builds and enable team collaboration. This approach is highly recommended for production environments.
2. **Keep dependencies minimal** to reduce build times. Only include packages you actually need.
3. **Use fine-grained Docker settings** at the step level for conflicting requirements. This prevents dependency conflicts and reduces image sizes.
4. **Use pre-built images** for common environments. This can significantly speed up your workflow.
5. **Configure dockerignore files** to reduce image size. Large Docker images take longer to build, push, and pull.
6. **Leverage build caching** by structuring your Dockerfiles and build processes to maximize cache hits.
7. **Use environment variables** for configuration instead of hardcoding values in your images.
8. **Test your Docker builds locally** before using them in production pipelines.
9. **Keep your repository clean** (no uncommitted changes) when running pipelines to ensure ZenML can correctly track code versions.
10. **Use metadata and labels** to help identify and manage your Docker images.
11. **Run containers as non-root users** when possible to improve security.

By following these practices, you can optimize your Docker builds in ZenML and create a more efficient workflow.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Code Repositories

Tracking your code and avoiding unnecessary Docker builds by connecting your git repo.

A code repository in ZenML refers to a remote storage location for your code. Some commonly known code repository platforms include [GitHub](https://github.com/) and [GitLab](https://gitlab.com/).

<figure><img src="/files/huFW8Yie35Ko0GI3EVNh" alt=""><figcaption><p>A visual representation of how the code repository fits into the general ZenML architecture.</p></figcaption></figure>

Connecting code repositories to ZenML solves two fundamental challenges in machine learning workflows. First, it enhances reproducibility by tracking which specific code version (commit hash) was used for each pipeline run, creating a clear audit trail between your code and its results. Second, it dramatically improves development efficiency by optimizing Docker image building. Instead of including source code in each build, ZenML builds images without the code and downloads it at runtime, eliminating the need to rebuild images after every code change. This not only speeds up individual development cycles but allows team members to share and reuse builds, saving time and computing resources across your organization.

Learn more about how code repositories optimize Docker builds [here](https://docs.zenml.io/how-to/customize-docker-builds/how-to-reuse-builds).

## Registering a code repository

If you are planning to use one of the available implementations of code repositories, first, you need to install the corresponding ZenML integration:

```
zenml integration install <INTEGRATION_NAME>
```

Afterward, code repositories can be registered using the CLI:

```shell
zenml code-repository register <NAME> --type=<TYPE> [--CODE_REPOSITORY_OPTIONS]
```

For concrete options, check out the section on the `GitHubCodeRepository`, the `GitLabCodeRepository` or how to develop and register a custom code repository implementation.

## Available implementations

ZenML comes with builtin implementations of the code repository abstraction for the `GitHub` and `GitLab` platforms, but it's also possible to use a custom code repository implementation.

### GitHub

ZenML provides built-in support for using GitHub as a code repository for your ZenML pipelines. You can register a GitHub code repository by providing the URL of the GitHub instance, the owner of the repository, the name of the repository, and a GitHub Personal Access Token (PAT) with access to the repository.

Before registering the code repository, first, you have to install the corresponding integration:

```sh
zenml integration install github
```

Afterward, you can register a GitHub code repository by running the following CLI command:

```shell
zenml code-repository register <NAME> --type=github \
--owner=<OWNER> --repository=<REPOSITORY> \
--token=<GITHUB_TOKEN>
```

where `<REPOSITORY>` is the name of the code repository you are registering, `<OWNER>` is the owner of the repository, `<NAME>` is the name of the repository and `<GITHUB_TOKEN>` is your GitHub Personal Access Token.

If you're using a self-hosted GitHub Enterprise instance, you'll need to also pass the `--api_url=<API_URL>` and `--host=<HOST>` options. `<API_URL>` should point to where the GitHub API is reachable (defaults to `https://api.github.com/`) and `<HOST>` should be the [hostname of your GitHub instance](https://docs.github.com/en/enterprise-server@3.10/admin/configuring-settings/configuring-network-settings/configuring-the-hostname-for-your-instance?learn=deploy_an_instance\&learnProduct=admin).

{% hint style="warning" %}
Please refer to the section on using secrets for stack configuration in order to securely store your GitHub\
Personal Access Token.

```shell
# Using central secrets management
zenml secret create github_secret \
    --pa_token=<GITHUB_TOKEN>
    
# Then reference the username and password
zenml code-repository register ... --token={{github_secret.pa_token}}
    ...
```

{% endhint %}

After registering the GitHub code repository, ZenML will automatically detect if your source files are being tracked by GitHub and store the commit hash for each pipeline run.

<details>

<summary>How to get a token for GitHub</summary>

1. Go to your GitHub account settings and click on [Developer settings](https://github.com/settings/tokens?type=beta).
2. Select "Personal access tokens" and click on "Generate new token".
3. Give your token a name and a description.

   ![](/files/FMnaCF1ypk5PddTM75um)
4. We recommend selecting the specific repository and then giving `contents` read-only access.

   ![](/files/ZFDT6s7RuEgOihBAGH5b)

   ![](/files/TTwVShHsiUslVa6FjvMi)
5. Click on "Generate token" and copy the token to a safe place.

   ![](/files/AILLH6AlyBvNs7Y9H9Ke)

</details>

### GitLab

ZenML also provides built-in support for using GitLab as a code repository for your ZenML pipelines. You can register a GitLab code repository by providing the URL of the GitLab project, the group of the project, the name of the project, and a GitLab Personal Access Token (PAT) with access to the project.

Before registering the code repository, first, you have to install the corresponding integration:

```sh
zenml integration install gitlab
```

Afterward, you can register a GitLab code repository by running the following CLI command:

```shell
zenml code-repository register <NAME> --type=gitlab \
--group=<GROUP> --project=<PROJECT> \
--token=<GITLAB_TOKEN>
```

where `<NAME>` is the name of the code repository you are registering, `<GROUP>` is the group of the project, `<PROJECT>` is the name of the project and `<GITLAB_TOKEN>` is your GitLab Personal Access Token.

If you're using a self-hosted GitLab instance, you'll need to also pass the `--instance_url=<INSTANCE_URL>` and `--host=<HOST>` options. `<INSTANCE_URL>` should point to your GitLab instance (defaults to `https://gitlab.com/`) and `<HOST>` should be the hostname of your GitLab instance (defaults to `gitlab.com`).

{% hint style="warning" %}
Please refer to the section on using secrets for stack configuration in order to securely store your GitLab\
Personal Access Token.

```shell
# Using central secrets management
zenml secret create gitlab_secret \
    --pa_token=<GITLAB_TOKEN>
    
# Then reference the username and password
zenml code-repository register ... --token={{gitlab_secret.pa_token}}
    ...
```

{% endhint %}

After registering the GitLab code repository, ZenML will automatically detect if your source files are being tracked by GitLab and store the commit hash for each pipeline run.

<details>

<summary>How to get a token for GitLab</summary>

1. Go to your GitLab account settings and click on Access Tokens.
2. Name the token and select the scopes that you need (e.g. `read_repository`, `read_user`, `read_api`)

   ![](/files/GdHF8dLfqMNbjHCS7Y7V)
3. Click on "Create personal access token" and copy the token to a safe place.

   ![](/files/LQGvqshPRmSicKvmDZT0)

</details>

## Developing a custom code repository

If you're using some other platform to store your code, and you still want to use a code repository in ZenML, you can implement and register a custom code repository.

First, you'll need to subclass and implement the abstract methods of the `zenml.code_repositories.BaseCodeRepository` class:

```python
from abc import ABC, abstractmethod
from typing import Optional

class BaseCodeRepository(ABC):
    """Base class for code repositories."""

    @abstractmethod
    def login(self) -> None:
        """Logs into the code repository."""

    @abstractmethod
    def download_files(
            self, commit: str, directory: str, repo_sub_directory: Optional[str]
    ) -> None:
        """Downloads files from the code repository to a local directory.

        Args:
            commit: The commit hash to download files from.
            directory: The directory to download files to.
            repo_sub_directory: The subdirectory in the repository to
                download files from.
        """

    @abstractmethod
    def get_local_context(
            self, path: str
    ) -> Optional["LocalRepositoryContext"]:
        """Gets a local repository context from a path.

        Args:
            path: The path to the local repository.

        Returns:
            The local repository context object.
        """
```

After you're finished implementing this, you can register it as follows:

```shell
# The `CODE_REPOSITORY_OPTIONS` are key-value pairs that your implementation will receive
# as configuration in its __init__ method. This will usually include stuff like the username
# and other credentials necessary to authenticate with the code repository platform.
zenml code-repository register <NAME> --type=custom --source=my_module.MyRepositoryClass \
    [--CODE_REPOSITORY_OPTIONS]
```

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Secrets

Registering and using secrets.

ZenML secrets are groupings of **key-value pairs** which are securely stored in the ZenML secrets store. Additionally, a secret always has a **name** that allows you to fetch or reference them in your pipelines and stacks. Secrets are essential for both traditional ML workflows (database credentials, model registry access) and AI agent development (LLM API keys, third-party service credentials).

## How to create a secret

{% tabs %}
{% tab title="CLI" %}
To create a secret with a name `<SECRET_NAME>` and a key-value pair, you can run the following CLI command:

```shell
zenml secret create <SECRET_NAME> \
    --<KEY_1>=<VALUE_1> \
    --<KEY_2>=<VALUE_2>

# Another option is to use the '--values' option and provide key-value pairs in either JSON or YAML format.
zenml secret create <SECRET_NAME> \
    --values='{"key1":"value2","key2":"value2"}'

# Example: Create secrets for LLM API keys
zenml secret create openai_secret \
    --api_key=sk-proj-... \
    --organization_id=org-...

zenml secret create anthropic_secret \
    --api_key=sk-ant-api03-...

# Example: Create secrets for multi-agent system credentials
zenml secret create agent_tools_secret \
    --google_search_api_key=AIza... \
    --weather_api_key=abc123 \
    --database_url=postgresql://user:pass@host/db

# Create a private secret (only you can access it)
zenml secret create my_private_secret --private \
    --api_key=secret-value
```

{% hint style="info" %}
By default, secrets are public (visible to other users based on RBAC). Use `--private` or `-p` to create a secret only you can access. See [Private and public secrets](#private-and-public-secrets) for more details.
{% endhint %}

Alternatively, you can create the secret in an interactive session (in which ZenML will query you for the secret keys and values) by passing the `--interactive/-i` parameter:

```shell
zenml secret create <SECRET_NAME> -i
```

For secret values that are too big to pass as a command line argument, or have special characters, you can also use the special `@` syntax to indicate to ZenML that the value needs to be read from a file:

```bash
zenml secret create <SECRET_NAME> \
   --key=@path/to/file.txt \
   ...
   
# Alternatively, you can utilize the '--values' option by specifying a file path containing key-value pairs in either JSON or YAML format.
zenml secret create <SECRET_NAME> \
    --values=@path/to/file.txt
```

The CLI also includes commands that can be used to list, update and delete secrets. A full guide on using the CLI to create, access, update and delete secrets is available [here](https://sdkdocs.zenml.io/latest/cli.html#zenml.cli--secrets-management).

**Interactively register missing secrets for your stack**

If you're using components with [secret references](#reference-secrets-in-stack-component-attributes-and-settings) in your stack, you need to make sure that all the referenced secrets exist. To make this process easier, you can use the following CLI command to interactively register all secrets for a stack:

```shell
zenml stack register-secrets [<STACK_NAME>]
```

{% endtab %}

{% tab title="Python SDK" %}
The ZenML client API offers a programmatic interface to create, e.g.:

```python
from zenml.client import Client

client = Client()
client.create_secret(
    name="my_secret",
    values={
        "username": "admin",
        "password": "abc123"
    }
)

# Example: Create LLM API secrets programmatically
client.create_secret(
    name="openai_secret",
    values={
        "api_key": "sk-proj-...",
        "organization_id": "org-..."
    }
)

# Create a private secret (only you can access it)
client.create_secret(
    name="my_private_secret",
    values={"api_key": "secret-value"},
    private=True,
)
```

{% hint style="info" %}
By default, secrets are public (`private=False`). Set `private=True` to create a secret only you can access. See [Private and public secrets](#private-and-public-secrets) for more details.
{% endhint %}

Other Client methods used for secrets management include `get_secret` to fetch a secret by name or id, `update_secret` to update an existing secret, `list_secrets` to query the secrets store using a variety of filtering and sorting criteria, and `delete_secret` to delete a secret. The full Client API reference is available [here](https://sdkdocs.zenml.io/latest/core_code_docs/core-client.html).
{% endtab %}
{% endtabs %}

## Size limits

A **secret** in ZenML is a name plus a set of key–value pairs. Size is usually the total UTF-8 size of all keys and values. See the table in [Secret value size limits](https://docs.zenml.io/getting-started/deploying-zenml/secret-management#secret-value-size-limits) for the limits associated with the different secrets stores. If you are over the limit, add another named secret, split keys across secrets, or keep large files outside ZenML and only store a reference in a value.

## Private and public secrets

ZenML secrets can be either **private** or **public**:

* **Private secrets** are only accessible to the user who created them. No other user can view, use, or manage a private secret, regardless of their role or permissions.
* **Public secrets** (the default) are accessible to other users based on your RBAC configuration. On ZenML Pro, access to public secrets is governed by your role-based access control settings.

{% hint style="info" %}
The `private` property takes precedence over RBAC. A private secret is **only** visible to its creator, even if RBAC would otherwise grant access to other users.
{% endhint %}

### Creating private secrets

By default, secrets are created as public (`private=False`). To create a private secret:

{% tabs %}
{% tab title="CLI" %}

```shell
# Use the --private or -p flag
zenml secret create <SECRET_NAME> --private \
    --<KEY_1>=<VALUE_1> \
    --<KEY_2>=<VALUE_2>

# Short form
zenml secret create <SECRET_NAME> -p \
    --<KEY_1>=<VALUE_1>
```

{% endtab %}

{% tab title="Python SDK" %}

```python
from zenml.client import Client

client = Client()
client.create_secret(
    name="my_private_secret",
    values={"api_key": "..."},
    private=True,  # Makes this secret private
)
```

{% endtab %}
{% endtabs %}

{% hint style="warning" %}
Currently, setting the private status is only available via the CLI and Python SDK. The dashboard UI does not yet support creating or modifying private secrets.
{% endhint %}

### Fetching secrets with the same name

Since private and public secrets exist in separate namespaces, you can have both a private and a public secret with the same name. When fetching a secret by name without specifying its visibility:

* ZenML searches **private secrets first**, then public secrets
* The first match is returned

To explicitly fetch a secret of a specific visibility:

{% tabs %}
{% tab title="CLI" %}

```shell
# Explicitly fetch a private secret
zenml secret get my_secret --private=true

# Explicitly fetch a public secret
zenml secret get my_secret --private=false
```

{% endtab %}

{% tab title="Python SDK" %}

```python
from zenml.client import Client

client = Client()

# Explicitly fetch a private secret
private_secret = client.get_secret("my_secret", private=True)

# Explicitly fetch a public secret
public_secret = client.get_secret("my_secret", private=False)
```

{% endtab %}
{% endtabs %}

### Updating secret visibility

You can change a secret's visibility after creation:

{% tabs %}
{% tab title="CLI" %}

```shell
# Make a public secret private
zenml secret update my_secret --private=true

# Make a private secret public
zenml secret update my_secret --private=false
```

{% endtab %}

{% tab title="Python SDK" %}

```python
from zenml.client import Client

client = Client()
client.update_secret("my_secret", update_private=True)  # Make private
```

{% endtab %}
{% endtabs %}

## Accessing registered secrets

### Reference secrets in stack component attributes and settings

Some of the components in your stack require you to configure them with sensitive information like passwords or tokens, so they can connect to the underlying infrastructure. Secret references allow you to configure these components in a secure way by not specifying the value directly but instead referencing a secret by providing the secret name and key. Referencing a secret for the value of any string attribute of your stack components, simply specify the attribute using the following syntax: `{{<SECRET_NAME>.<SECRET_KEY>}}`

For example:

{% tabs %}
{% tab title="CLI" %}

```shell
# Register a secret called `mlflow_secret` with key-value pairs for the
# username and password to authenticate with the MLflow tracking server

# Using central secrets management
zenml secret create mlflow_secret \
    --username=admin \
    --password=abc123
    

# Then reference the username and password in our experiment tracker component
zenml experiment-tracker register mlflow \
    --flavor=mlflow \
    --tracking_username={{mlflow_secret.username}} \
    --tracking_password={{mlflow_secret.password}} \
    ...
```

{% endtab %}
{% endtabs %}

When using secret references in your stack, ZenML will validate that all secrets and keys referenced in your stack components exist before running a pipeline. This helps us fail early so your pipeline doesn't fail after running for some time due to some missing secret.

This validation by default needs to fetch and read every secret to make sure that both the secret and the specified key-value pair exist. This can take quite some time and might fail if you don't have permission to read secrets.

You can use the environment variable `ZENML_SECRET_VALIDATION_LEVEL` to disable or control the degree to which ZenML validates your secrets:

* Setting it to `NONE` disables any validation.
* Setting it to `SECRET_EXISTS` only validates the existence of secrets. This might be useful if the machine you're running on only has permission to list secrets but not actually read their values.
* Setting it to `SECRET_AND_KEY_EXISTS` (the default) validates both the secret existence as well as the existence of the exact key-value pair.

### Fetch secret values in a step

If you are using [centralized secrets management](/concepts/secrets), you can access secrets directly from within your steps through the ZenML `Client` API. This allows you to use your secrets for querying APIs from within your step without hard-coding your access keys:

```python
from zenml import step
from zenml.client import Client
import openai

@step
def secret_loader() -> None:
    """Load the example secret from the server."""
    # Fetch the secret from ZenML.
    secret = Client().get_secret( < SECRET_NAME >)

    # `secret.secret_values` will contain a dictionary with all key-value
    # pairs within your secret.
    authenticate_to_some_api(
        username=secret.secret_values["username"],
        password=secret.secret_values["password"],
    )
    ...

@step
def run_llm_agent(prompt: str, query: str) -> str:
    """Execute an LLM agent using securely stored API keys."""
    # Fetch LLM API credentials from ZenML secrets
    openai_secret = Client().get_secret("openai_secret")
    
    # Use the API key to initialize the LLM client
@step
def run_llm_agent(prompt: str, query: str) -> str:
    """Execute an LLM agent using securely stored API keys."""
    # Fetch LLM API credentials from ZenML secrets
    openai_secret = Client().get_secret("openai_secret")
    
    # Initialize the OpenAI client with credentials
    from openai import OpenAI
    
    client = OpenAI(
        api_key=openai_secret.secret_values["api_key"],
        organization=openai_secret.secret_values["organization_id"]
    )
    
    # Execute the agent
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": query}
        ]
    )
    
    return response.choices[0].message.content
    return response.choices[0].message.content
```

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Environment Variables

Configuring environment variables.

Environment variables can be configured to be available at runtime during step execution. ZenML provides two ways to set environment variables:

1. **Plain text environment variables**: Configure key-value pairs directly
2. **Secrets as environment variables**: Use ZenML secrets where the secret values become environment variables. Check out [this page](/concepts/secrets) for more information on secret management in ZenML.

{% hint style="info" %}
If you need environment variables to be available at image built time, check out the [containerization documentation](/concepts/containerization#environment-variables) for more information.
{% endhint %}

## Configuration levels

Environment variables and secrets can be configured at different levels with increasing precedence:

1. **Stack components** - Available for all pipelines executed on stacks containing this component
2. **Stack** - Available for all pipelines executed on this stack
3. **Pipeline** - Available for all steps in this pipeline
4. **Step** - Available only for this specific step

{% hint style="info" %}
**Precedence order**: Step configuration overrides pipeline configuration, which overrides stack configuration, which overrides stack component configuration. Additionally, secrets always take precedence over direct environment variables when both are configured with the same key.
{% endhint %}

## ZenML repository directory name

By default, `zenml init` creates a `.zen` directory that marks the root of a ZenML repository. ZenML uses that marker later when it resolves local code, for example custom stack component flavors that live in your repository.

If your workspace cannot use `.zen` as the marker directory name, set `ZENML_REPOSITORY_DIRECTORY_NAME` before initializing or discovering the repository:

```bash
export ZENML_REPOSITORY_DIRECTORY_NAME=.my-zenml
zenml init
```

After that, ZenML looks for `.my-zenml` instead of `.zen`. Keep this setting consistent for everyone and every process that works with the repository; if one shell uses `.zen` and another uses `.my-zenml`, they will not agree on where the ZenML repository root is.

## Automatic environment variable injection

When executing a pipeline, ZenML automatically scans your local environment for any variables that start with the `__ZENML__` prefix and adds them to the pipeline environment. The prefix is removed during this process.

For example, if you set:

```bash
export __ZENML__MY_VAR=my_value
```

It will be available in your steps as follows:

```python
import os
from zenml import step

@step
def my_step():
    my_var = os.environ["MY_VAR"]  # "my_value"
```

## Configuring environment variables on stack components

Configure environment variables and secrets that will be available for all pipelines executed on stacks containing this component.

{% tabs %}
{% tab title="CLI" %}

```bash
# Configure environment variables
zenml orchestrator update <ORCHESTRATOR_NAME> --env <KEY>=<VALUE>
# Remove environment variables (set empty value)
zenml orchestrator update <ORCHESTRATOR_NAME> --env <KEY>=

# Attach secrets (secret values become environment variables)
zenml orchestrator update <ORCHESTRATOR_NAME> --secret <SECRET_NAME_OR_ID>
# Remove secrets
zenml orchestrator update <ORCHESTRATOR_NAME> --remove-secret <SECRET_NAME_OR_ID>
```

{% endtab %}

{% tab title="Python" %}

```python
from zenml import Client

Client().update_stack_component(
    name_id_or_prefix=<COMPONENT_NAME_OR_ID>,
    component_type=<COMPONENT_TYPE>,
    environment={
        "<KEY>": "<VALUE>",
        # Set to `None` to remove from previously configured environment
        "<KEY>": None
    },
    add_secrets=["<SECRET_NAME_OR_ID>", "<SECRET_NAME_OR_ID>"],
    remove_secrets=["<SECRET_NAME_OR_ID>"]
)
```

{% endtab %}
{% endtabs %}

## Setting environment variables on stacks

Configure environment variables and secrets for all pipelines executed on this stack.

{% tabs %}
{% tab title="CLI" %}

```bash
# Configure environment variables
zenml stack update <STACK_NAME> --env <KEY>=<VALUE>
# Remove environment variables
zenml stack update <STACK_NAME> --env <KEY>=

# Attach secrets
zenml stack update <STACK_NAME> --secret <SECRET_NAME_OR_ID>
# Remove secrets
zenml stack update <STACK_NAME> --remove-secret <SECRET_NAME_OR_ID>
```

{% endtab %}

{% tab title="Python" %}

```python
from zenml import Client

Client().update_stack(
    name_id_or_prefix=<STACK_NAME_OR_ID>,
    environment={
        "<KEY>": "<VALUE>",
        # Set to `None` to remove from previously configured environment
        "<KEY>": None
    },
    add_secrets=["<SECRET_NAME_OR_ID>"],
    remove_secrets=["<SECRET_NAME_OR_ID>"]
)
```

{% endtab %}
{% endtabs %}

## Configuring environment variables on pipelines

Configure environment variables and secrets for all steps of a pipeline. See [this page](/concepts/steps_and_pipelines/configuration) for more details on how to configure pipelines.

```python
from zenml import pipeline

# On the decorator
@pipeline(
    environment={
        "<KEY>": "<VALUE>",
        "<KEY>": "<VALUE>"
    },
    secrets=["<SECRET_NAME_OR_ID>", "<SECRET_NAME_OR_ID>"]
)
def my_pipeline():
    ...

# Using the `with_options(...)` method
my_pipeline = my_pipeline.with_options(
    environment={
        "<KEY>": "<VALUE>",
        "<KEY>": "<VALUE>"
    },
    secrets=["<SECRET_NAME_OR_ID>", "<SECRET_NAME_OR_ID>"]
)
```

## Setting environment variables on steps

Configure environment variables and secrets for individual steps. See [this page](/concepts/steps_and_pipelines/configuration) for more details on how to configure steps.

```python
from zenml import step

# On the decorator
@step(
    environment={
        "<KEY>": "<VALUE>",
        "<KEY>": "<VALUE>"
    },
    secrets=["<SECRET_NAME_OR_ID>"]
)
def my_step() -> str:
    ...

# Using the `with_options(...)` method
my_step = my_step.with_options(
    environment={
        "<KEY>": "<VALUE>",
        "<KEY>": "<VALUE>"
    },
    secrets=["<SECRET_NAME_OR_ID>", "<SECRET_NAME_OR_ID>"]
)
```

## When environment variables are set

The timing of when environment variables are set depends on the orchestrator being used:

* The [Databricks](https://github.com/zenml-io/zenml/tree/main/docs/book/component-guide/orchestrators/databricks.md) and [Lightning](https://github.com/zenml-io/zenml/tree/main/docs/book/component-guide/orchestrators/lightning.md) orchestrators will set the environment variables right before your step code is being executed
* **All other orchestrators** set environment variables already at container startup time

{% hint style="info" %}
**Environment variables from secrets** are always set right before your step code is being executed for security reasons, regardless of the orchestrator.
{% endhint %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Tags

Use tags to organize tags in ZenML.

Organizing and categorizing your machine learning artifacts and models can\
streamline your workflow and enhance discoverability. ZenML enables the use of\
tags as a flexible tool to classify and filter your ML assets.

![Tags are visible in the ZenML Dashboard](/files/kP0KwxQl8uJI4iFgNyLz)

## Tagging different entities

### Assigning tags to artifacts

You can tag artifact versions by using the `add_tags` utility function:

```python
from zenml import add_tags

add_tags(tags=["my_tag"], artifact="my_artifact_name_or_id")
```

Alternatively, you can tag an artifact by using CLI as well:

```bash
zenml artifact update my_artifact -t my_tag
```

### Assigning tags to artifact versions

In order to tag an artifact through the Python SDK, you can use either use\
the `ArtifactConfig` object:

```python
from typing import Annotated
import pandas as pd
from zenml import step, ArtifactConfig

@step
def data_loader() -> (
    Annotated[pd.DataFrame, ArtifactConfig(name="my_output", tags=["my_tag"])]
):
    ...
```

or the `add_tags` utility function:

```python
from zenml import add_tags

# Automatic tagging to an artifact version within a step execution
## A step with a single output
add_tags(tags=["my_tag"], infer_artifact=True)
## A step with multiple outputs (need to specify the output name)
add_tags(tags=["my_tag"], artifact_name="my_output", infer_artifact=True)

# Manual tagging to an artifact version (can happen in a step or outside of it)
## By specifying the artifact name and version
add_tags(tags=["my_tag"], artifact_name="my_output", artifact_version="v1")
## By specifying the artifact version ID
add_tags(tags=["my_tag"], artifact_version_id="artifact_version_uuid")
```

Moreover, you can tag an artifact version by using the CLI:

```bash
# Tag the artifact version
zenml artifact version update iris_dataset -v raw_2023 -t sklearn
```

{% hint style="info" %}
In the upcoming chapters, you will also learn how to use [an cascade tag](#cascade-tags) to tag an artifact version as well.
{% endhint %}

### Assigning tags to pipelines

Assigning tags to pipelines is only possible through the Python SDK and you can use the `add_tags` utility function:

```python
from zenml import add_tags

add_tags(tags=["my_tag"], pipeline="pipeline_name_or_id")
```

### Assigning tags to runs

To assign tags to a pipeline run in ZenML, you can use the `add_tags` utility function:

```python
from zenml import add_tags

# Manual tagging to a run
add_tags(tags=["my_tag"], run="run_name_or_id")
```

Alternatively, you can use the same function within a step without specifying any arguments, which will automatically tag the run:

```python
from zenml import step, add_tags

@step
def my_step():
    add_tags(tags=["my_tag"])
```

You can also use the pipeline decorator to tag the run:

```python
from zenml import pipeline

@pipeline(tags=["my_tag"])
def my_pipeline():
    ...
```

### Assigning tags to models and model versions

When creating a model version using the `Model` object, you can specify tags as key-value pairs that will be attached to the model version upon creation.

{% hint style="warning" %}
During pipeline run a model can be also implicitly created (if not exists), in such cases it will not get the `tags` from the `Model` class.
{% endhint %}

```python
from zenml import Model

# Create a model version with tags
model = Model(
    name="iris_classifier",
    version="1.0.0",
    tags=["experiment", "v1", "classification-task"],
)

# Use this tagged model in your steps and pipelines as needed
from zenml import pipeline

@pipeline(model=model)
def my_pipeline(...):
    ...
```

You can also assign tags when creating or updating models with the Python SDK:

```python
from zenml import Model
from zenml.client import Client

# Create or register a new model with tags
Client().create_model(
    name="iris_logistic_regression",
    tags=["classification", "iris-dataset"],
)

# Create or register a new model version also with tags
Client().create_model_version(
    model_name_or_id="iris_logistic_regression",
    name="2",
    tags=["version-1", "experiment-42"],
)
```

To add tags to existing models and their versions using the ZenML CLI, you can use the following commands:

```shell
# Tag an existing model
zenml model update iris_logistic_regression --tag "classification"

# Tag a specific model version
zenml model version update iris_logistic_regression 2 --tag "experiment3"
```

### Assigning tags to snapshots

Assigning tags to snapshots is only possible through the Python SDK and you can use the `add_tags` utility function:

```python
from zenml import add_tags

add_tags(tags=["my_tag"], snapshot=<SNAPSHOT-ID>)
```

## Advanced Usage

ZenML provides several advanced tagging features to help you better organize and manage your ML assets.

### Exclusive Tags

Exclusive tags are special tags that can be associated with only one instance of a specific entity type within a certain scope at a time. When you apply an exclusive tag to a new entity, it's automatically removed from any previous entity of the same type that had this tag. Exclusive tags can be used with:

* One pipeline run per pipeline
* One snapshot per pipeline
* One artifact version per artifact

The recommended way to create exclusive tags is using the `Tag` object:

```python
from zenml import pipeline, Tag

@pipeline(tags=["not_an_exclusive_tag", Tag(name="an_exclusive_tag", exclusive=True)])
def my_pipeline():
    ...
```

Alternatively, you can also create an exclusive tag separately and use it later:

```python
from zenml.client import Client
from zenml import pipeline

Client().create_tag(name="an_exclusive_tag", exclusive=True)

@pipeline(tags=["an_exclusive_tag"])
def my_pipeline():
    ...
```

{% hint style="warning" %}
The `exclusive` parameter belongs to the configuration of the tag and this information is stored in the backend. This means, that it will not lose its `exclusive` functionality even if it is being used without the explicit `exclusive=True` parameter in future calls.
{% endhint %}

### Cascade Tags

Cascade tags allow you to associate a tag from a pipeline with all artifact versions created during its execution.

```python
from zenml import pipeline, Tag

@pipeline(tags=["normal_tag", Tag(name="cascade_tag", cascade=True)])
def my_pipeline():
    ...
```

When this pipeline runs, the `cascade_tag` will be automatically applied to all artifact versions created during the pipeline execution.

{% hint style="warning" %}
Unlike the `exclusive` parameter, the `cascade` parameter is a runtime configuration and does not get stored with the `tag` object. This means that the tag will **not** have its `cascade` functionality if it is not used with the `cascade=True` parameter in future calls.
{% endhint %}

### Filtering

ZenML allows you to filter taggable objects using multiple tag conditions:

```python
from zenml import add_tags

from zenml.client import Client

# Add tags to a pipeline
add_tags(tags=["one", "two", "three"], pipeline="my_pipeline")

# Will return `my_pipeline`
Client().list_pipelines(tags=["contains:wo", "startswith:t", "equals:three"])

# Will not return `my_pipeline`
Client().list_pipelines(tags=["contains:wo", "startswith:t", "equals:four"])
```

The example above shows how you can use multiple tag conditions to filter an entity. In ZenML, the default logical operator is `AND`, which means that the entity will be returned only if there is at least one tag that matches all the conditions.

### Removing Tags

Similar to the `add_tags` utility function, you can use the `remove_tags` utility function to remove tags from an entity.

```python
from zenml.utils.tag_utils import remove_tags

# Remove tags from a pipeline
remove_tags(tags=["one", "two"], pipeline="my_pipeline")

# Remove tags from an artifact
remove_tags(tags=["three"], artifact="my_artifact")
```

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Metadata

Enrich your ML workflow with contextual information using ZenML metadata.

Metadata in ZenML provides critical context to your ML workflows, allowing you to track additional information about your steps, runs, artifacts, and models. This enhanced traceability helps you better understand, compare, and reproduce your experiments.

![Metadata in the dashboard](/files/SKFYtlGYEJxzTh0772on)

Metadata is any additional contextual information you want to associate with your ML workflow components. In ZenML, you can attach metadata to:

* **Steps**: Log evaluation metrics, execution details, or configuration information
* **Pipeline Runs**: Track overall run characteristics like environment variables or git information
* **Artifacts**: Document data characteristics, source information, or processing details
* **Models**: Capture evaluation results, hyperparameters, or deployment information

ZenML makes it easy to log and retrieve this information through a simple interface, and visualizes it in the dashboard for quick analysis.

## Logging Metadata

The primary way to log metadata in ZenML is through the `log_metadata` function, which allows you to attach JSON-serializable key-value pairs to various entities.

{% hint style="info" %}
Metadata supports primitive types (`str`, `int`, `float`, `bool`), collections (`list`, `dict`, `set`, `tuple`), and special ZenML types (`Uri`, `Path`, `DType`, `StorageSize`). Sets and tuples are automatically converted to lists during storage.
{% endhint %}

```python
from zenml import log_metadata

# Basic metadata logging
log_metadata(
    metadata={"accuracy": 0.95, "precision": 0.92},
    # Additional parameters to specify where to log the metadata
)
```

The `log_metadata` function is versatile and can target different entities depending on the parameters provided.

### Attaching Metadata to Steps

To log metadata for a step, you can either call `log_metadata` within the step (which automatically associates with the current step), or specify a step explicitly:

```python
from zenml import step, log_metadata

# Method 1: Within a step (automatically associates with current step)
@step
def train_model_step(data):
    model = train_model(data)
    accuracy = evaluate_model(model, data)
    
    # Log metrics directly within the step
    log_metadata(
        metadata={"evaluation_metrics": {"accuracy": accuracy}}
    )
    
    return model

# Method 2: Targeting a specific step after execution
log_metadata(
    metadata={"post_analysis": {"feature_importance": [0.2, 0.5, 0.3]}},
    step_name="train_model_step",
    run_id_name_or_prefix="my_run_id"
)

# Alternative: Using step_id
log_metadata(
    metadata={"post_analysis": {"feature_importance": [0.2, 0.5, 0.3]}},
    step_id="step_uuid"
)
```

### Attaching Metadata to Pipeline Runs

You can log metadata for an entire pipeline run, either from within a step during execution or manually after the run:

```python
from zenml import get_step_context, pipeline, step, log_metadata

# Method 1: Within a step (logs to the current run)
@step
def log_run_info_step():
    context = get_step_context()

    # Get some runtime information
    git_commit = get_git_hash()
    environment = get_env_info()
    
    # Log to the current pipeline run
    log_metadata(
        metadata={
            "git_info": {"commit": git_commit},
            "environment": environment
        },
        run_id_name_or_prefix=context.pipeline_run.id,
    )

# Method 2: Manually targeting a specific run
log_metadata(
    metadata={"post_run_analysis": {"total_training_time": 350}},
    run_id_name_or_prefix="my_run_id"
)
```

When logging from within a step to the pipeline run, the metadata key will have the pattern `step_name::metadata_key`, allowing multiple steps to use the same metadata key.

### Attaching Metadata to Artifacts

Artifacts are the data objects produced by pipeline steps. You can log metadata for these artifacts to provide more context about the data:

```python
from zenml import step, log_metadata
from zenml.metadata.metadata_types import StorageSize

# Method 1: Within a step for an output artifact
@step
def process_data_step(raw_data):
    processed_data = transform(raw_data)
    
    # Log metadata for the output artifact (when step has single output)
    log_metadata(
        metadata={
            "data_stats": {
                "row_count": len(processed_data),
                "columns": list(processed_data.columns),
                "storage_size": StorageSize(processed_data.memory_usage().sum())
            }
        },
        infer_artifact=True  # Automatically target the output artifact
    )
    
    return processed_data

# Method 2: For a step with multiple outputs
@step
def split_data_step(data):
    train, test = split_data(data)
    
    # Log metadata for specific output by name
    log_metadata(
        metadata={"split_info": {"train_size": len(train)}},
        artifact_name="output_0",  # Name of the specific output
        infer_artifact=True
    )
    
    return train, test

# Method 3: Explicitly target an artifact by name and version
log_metadata(
    metadata={"validation_results": {"distribution_shift": 0.03}},
    artifact_name="processed_data",
    artifact_version="20230615"
)

# Method 4: Target by artifact version ID
log_metadata(
    metadata={"validation_results": {"distribution_shift": 0.03}},
    artifact_version_id="artifact_uuid"
)
```

### Attaching Metadata to Models

Models in ZenML represent a higher-level concept that can encapsulate multiple artifacts and steps. Logging metadata for models helps track performance and other important information:

```python
from zenml import step, log_metadata

# Method 1: Within a step that produces a model
@step
def train_model_step(data):
    model = train_model(data)
    metrics = evaluate_model(model, data)
    
    # Log metadata to the model
    log_metadata(
        metadata={
            "evaluation_metrics": metrics,
            "hyperparameters": model.get_params()
        },
        infer_model=True  # Automatically target the model associated with this step
    )
    
    return model

# Method 2: Explicitly target a model by name and version
log_metadata(
    metadata={"deployment_info": {"endpoint": "api.example.com/model"}},
    model_name="fraud_detector",
    model_version="1.0.0"
)

# Method 3: Target by model version ID
log_metadata(
    metadata={"deployment_info": {"endpoint": "api.example.com/model"}},
    model_version_id="model_version_uuid"
)
```

## Bulk Metadata Logging

The `log_metadata` function does not support logging the same metadata for multiple entities simultaneously. To achieve this, you can use the `bulk_log_metadata` function:

```python
from zenml.models import (
    ArtifactVersionIdentifier,
    ModelVersionIdentifier,
    PipelineRunIdentifier,
    StepRunIdentifier,
)
from zenml import bulk_log_metadata

bulk_log_metadata(
    metadata={"python_version": "3.11", "environment": "macosx"},
    pipeline_runs=[
        PipelineRunIdentifier(id="<run_id>"),
        PipelineRunIdentifier(name="run name")
    ],
    step_runs=[
        StepRunIdentifier(id="<step_run_id>"),
        StepRunIdentifier(name="<step_name>", run=PipelineRunIdentifier(id="<run_id>"))
    ],
    artifact_versions=[
        ArtifactVersionIdentifier(id="<artifact_version_id>"),
        ArtifactVersionIdentifier(name="artifact_name", version="artifact_version")
    ],
    model_versions=[
        ModelVersionIdentifier(id="<model_version_id>"),
        ModelVersionIdentifier(name="model_name", version="model_version")
    ]
)

```

Note that the `bulk_log_metadata` function has a slightly different signature compared to `log_metadata`. You can use the Identifier class objects to specify any parameter combination that uniquely identifies an object:

* VersionedIdentifiers
  * ArtifactVersionIdentifier & ModelVersionIdentifier
  * Specify either an id or a combination of name and version.
* PipelineRunIdentifier
  * Specify an id, name, or prefix.
* StepRunIdentifier
  * Specify an id or a combination of name and a pipeline run identifier.

Similar to the `log_metadata` function, if you are calling `bulk_log_metadata` from within a step, you can use the infer options to automatically log metadata for the step’s model version or artifacts:

```python
from zenml import bulk_log_metadata, step

@step()
def get_train_test_datasets():
    
    train_dataset, test_dataset = get_datasets()

    bulk_log_metadata(
        metadata={"python_version": "3.11", "environment": "macosx"},
        infer_models=True,
        infer_artifacts=True
    )
    
    return train_dataset, test_dataset
```

Keep in mind that when using the `infer_artifacts` option, the `bulk_log_metadata` function logs metadata to all output artifacts of the step. When logging metadata, you may need the option to use `infer` options in combination with identifier references. For instance, you may want to log metadata to a step's outputs but also to its inputs. The `bulk_log_metadata` function enables you to use both options in one go:

```python
from zenml import bulk_log_metadata, get_step_context, step
from zenml.models import ArtifactVersionIdentifier


def calculate_metrics(model, test_dataset):
    ...


def summarize_metrics(metrics_report):
    ...


@step
def model_evaluation(test_dataset, model):
    metrics_report = calculate_metrics(model, test_dataset)

    slim_metrics_version = summarize_metrics(metrics_report)

    bulk_log_metadata(
        metadata=slim_metrics_version,
        infer_artifacts=True,  # log metadata for outputs
        artifact_versions=[
            ArtifactVersionIdentifier(id=get_step_context().inputs["model"].id)
        ]  # log metadata for the model input
    )

    return metrics_report
```

### Performance improvements hints

Both `log_metadata` and `bulk_log_metadata` internally use parameters such as name and version to resolve the actual IDs of entities. For example, when you provide an artifact's name and version, the function performs an additional lookup to resolve the artifact version ID.

To improve performance, prefer using the entity's ID directly instead of its name, version, or other identifiers whenever possible.

### Using the client directly

If the `log_metadata` or `bulk_log_metadata` functions are too restrictive for your use case, you can use the ZenML Client directly to create run metadata for resources:

```python
from zenml.client import Client
from zenml.enums import MetadataResourceTypes
from zenml.models import RunMetadataResource

client = Client()

client.create_run_metadata(
    metadata={"python": "3.11"},
    resources=[
        RunMetadataResource(id="<step_run_id>", type=MetadataResourceTypes.STEP_RUN),
        RunMetadataResource(id="<run_id>", type=MetadataResourceTypes.PIPELINE_RUN),
        RunMetadataResource(id="<artifact_version_id>", type=MetadataResourceTypes.ARTIFACT_VERSION),
        RunMetadataResource(id="<model_version_id>", type=MetadataResourceTypes.MODEL_VERSION)
    ]
)
```

## Special Metadata Types

ZenML includes several special metadata types that provide standardized ways to represent common metadata:

```python
from zenml import log_metadata
from zenml.metadata.metadata_types import StorageSize, DType, Uri, Path

log_metadata(
    metadata={
        "dataset_source": Uri("gs://my-bucket/datasets/source.csv"),  # External URI
        "preprocessing_script": Path("/scripts/preprocess.py"),  # File path
        "column_types": {
            "age": DType("int"),  # Data type
            "income": DType("float"),
            "score": DType("int")
        },
        "processed_data_size": StorageSize(2500000)  # Size in bytes
    },
    infer_artifact=True
)
```

These special types ensure metadata is logged in a consistent and interpretable manner, and they receive special treatment in the ZenML dashboard.

## Organizing Metadata in the Dashboard

To improve visualization in the ZenML dashboard, you can group metadata into logical sections by passing a dictionary of dictionaries:

```python
from zenml import log_metadata
from zenml.metadata.metadata_types import StorageSize

log_metadata(
    metadata={
        "model_metrics": {  # First card in the dashboard
            "accuracy": 0.95,
            "precision": 0.92,
            "recall": 0.90
        },
        "data_details": {   # Second card in the dashboard
            "dataset_size": StorageSize(1500000),
            "feature_columns": ["age", "income", "score"]
        }
    },
    artifact_name="my_artifact",
    artifact_version="version",
)
```

In the ZenML dashboard, "model\_metrics" and "data\_details" will appear as separate cards, each containing their respective key-value pairs, making it easier to navigate and interpret the metadata.

## Visualizing and Comparing Metadata (Pro)

Once you've logged metadata in your runs, you can use ZenML's Experiment Comparison tool to analyze and compare metrics across different run.

{% hint style="success" %}
The metadata comparison tool is a [ZenML Pro](https://zenml.io/pro)-only feature.
{% endhint %}

[![Experiment Comparison Introduction Video](/files/6CET5XJP4nFenycKnYIV)](https://www.loom.com/share/693b2d829600492da7cd429766aeba6a?sid=7182e55b-31e9-4b38-a3be-07c989dbea32)

### Comparison Views

The Experiment Comparison tool offers two complementary views for analyzing your pipeline metadata:

1. **Table View**: Compare metadata across runs with automatic change tracking

![Table View](/files/LkcSgOFHKik0q59eG9Yc)

2. **Parallel Coordinates Plot**: Visualize relationships between different metrics

![Parallel Coordinates](/files/0A5cYm7He2tIJCOwGcey)

The tool lets you compare up to 20 pipeline runs simultaneously and supports any numerical metadata (`float` or `int`) that you've logged in your pipelines.

## Fetching Metadata

### Retrieving Metadata Programmatically

Once metadata has been logged, you can retrieve it using the ZenML Client:

```python
from zenml.client import Client

client = Client()

# Get metadata from a step
step = client.get_pipeline_run("pipeline_run_id").steps["step_name"]
step_metadata = step.run_metadata["metadata_key"]

# Get metadata from a run
run = client.get_pipeline_run("pipeline_run_id")
run_metadata = run.run_metadata["metadata_key"]

# Get metadata from an artifact
artifact = client.get_artifact_version("artifact_name", "version")
artifact_metadata = artifact.run_metadata["metadata_key"]

# Get metadata from a model
model = client.get_model_version("model_name", "version")
model_metadata = model.run_metadata["metadata_key"]
```

{% hint style="info" %}
When fetching metadata using a specific key, the returned value will always reflect the latest entry for that key.
{% endhint %}

### Accessing Context Within Steps

The `StepContext` object is your handle to the *current* pipeline/step run while a step executes. Use it to read run/step information, inspect upstream input metadata, and work with step outputs: URIs, materializers, run metadata, and tags.

It is available:

* Inside functions decorated with `@step` (during execution, not composition time).
* Inside step hooks like `on_failure` / `on_success` (see [Hooks](/concepts/steps_and_pipelines/hooks)).
* Inside materializers triggered by a step’s `save` / `load`.
* Calling `get_step_context()` elsewhere raises `RuntimeError`.

Getting the context is done via `get_step_context()`:

```python
from zenml import step, get_step_context

@step
def trainer(param: int = 1):
    ctx = get_step_context()
    print("run:",  ctx.pipeline_run.name, ctx.pipeline_run.id)
    print("step:", ctx.step_run.name,   ctx.step_run.id)
    print("params:", ctx.step_run.config.parameters)
```

This exposes the following properties:

* `ctx.pipeline` → the `PipelineResponse` for this run (convenience; may raise if the run has no pipeline object).
* `ctx.pipeline_run` → `PipelineRunResponse` (id, name, status, timestamps, etc.).
* `ctx.step_run` → `StepRunResponse` (name, parameters via `ctx.step_run.config.parameters`, status).
* `ctx.model` → the configured `Model` (resolved from step or pipeline); raises if none configured.
* `ctx.inputs` → `{input_name: StepRunInputResponse}`; use `...["x"].run_metadata` to read upstream metadata.
* `ctx.step_name` → convenience name string.

### Working with outputs

For a single-output step you can omit `output_name`. For multi-output steps you **must** pass it (unnamed outputs are called `output_1`, `output_2`, …).

* `get_output_artifact_uri(output_name=None) -> str` – where the output artifact lives (write side files, etc.).
* `get_output_materializer(output_name=None, *, custom_materializer_class=None, data_type=None) -> BaseMaterializer` – get an initialized materializer; pass `data_type` to select from `Union[...]` materializers or `custom_materializer_class` to override.
* `add_output_metadata(metadata, output_name=None)` / `get_output_metadata(output_name=None)` – set/read run metadata for the output. Values provided via `ArtifactConfig(..., run_metadata=...)` on the return annotation are merged with runtime values.
* `add_output_tags(tags, output_name=None)` / `get_output_tags(output_name=None)` / `remove_output_tags(tags, output_name=None)` – manage tags for the produced artifact version. Configured tags via `ArtifactConfig(..., tags=...)` are unioned with runtime tags; duplicates are de‑duplicated in the final artifact.

Minimal example:

```python
from typing import Annotated, Tuple
from zenml import step, get_step_context, log_metadata
from zenml.artifacts.artifact_config import ArtifactConfig

@step
def produce(name: str) -> Tuple[
    Annotated[
        str,
        ArtifactConfig(
            name="custom_name",
            run_metadata={"config_metadata": "bar"},
            tags=["config_tags"],
        ),
    ],
    str,
]:
    ctx = get_step_context()
    # Attach metadata and tags to the named (or default) output
    ctx.add_output_metadata({"m": 1}, output_name=name)
    ctx.add_output_tags(["t1", "t1"], output_name=name)  # duplicates ok
    return "a", "b"
```

#### Reading upstream metadata via `inputs`

```python
from zenml import step, get_step_context, log_metadata

@step
def upstream() -> int:
    log_metadata({"quality": "ok"}, infer_artifact=True)
    return 42

@step
def downstream(x: int) -> None:
    md = get_step_context().inputs["x"].run_metadata
    assert md["quality"] == "ok"
```

#### Hooks and materializers (advanced)

```python
from zenml import step, get_step_context
from zenml.materializers.base_materializer import BaseMaterializer

def on_failure(exc: BaseException):
    c = get_step_context()
    print("Failed step:", c.step_run.name, "-", type(exc).__name__)

class ExampleMaterializer(BaseMaterializer):
    def save(self, data):
        # Context is available while the step triggers materialization
        data.meta = get_step_context().pipeline.name
        super().save(data)

@step(on_failure=on_failure)
def my_step():
    raise ValueError("boom")
```

**Common errors to expect.**

* `RuntimeError` if `get_step_context()` is called outside a running step.
* `StepContextError` for output helpers when:
  * The step has no outputs,
  * You omit `output_name` on a multi‑output step,
  * You reference an unknown `output_name`.

See the [full SDK docs for `StepContext`](https://sdkdocs.zenml.io/latest/core_code_docs/core-steps.html#zenml.steps.StepContext) for a concise reference to this object.

### Accessing Context During Pipeline Composition

During pipeline composition, you can access the pipeline configuration using the `PipelineContext`:

```python
from zenml import pipeline, get_pipeline_context

@pipeline(
    extra={
        "model_configs": [
            ("sklearn.tree", "DecisionTreeClassifier"),
            ("sklearn.ensemble", "RandomForestClassifier"),
        ]
    }
)
def my_pipeline():
    # Get the pipeline context
    context = get_pipeline_context()
    
    # Access the configuration
    model_configs = context.extra["model_configs"]
    
    # Use the configuration to dynamically create steps
    for i, (model_package, model_class) in enumerate(model_configs):
        train_model(
            model_package=model_package,
            model_class=model_class,
            id=f"train_model_{i}"
        )
```

## Best Practices

To make the most of ZenML's metadata capabilities:

1. **Use consistent keys**: Define standard metadata keys for your organization to ensure consistency
2. **Group related metadata**: Use nested dictionaries to create logical groupings in the dashboard
3. **Leverage special types**: Use ZenML's special metadata types for standardized representation
4. **Log relevant information**: Focus on metadata that aids reproducibility, understanding, and decision-making
5. **Consider automation**: Set up automatic metadata logging for standard metrics and information
6. **Combine with tags**: Use metadata alongside tags for a comprehensive organization system

## Conclusion

Metadata in ZenML provides a powerful way to enhance your ML workflows with contextual information. By tracking additional details about your steps, runs, artifacts, and models, you can gain deeper insights into your experiments, make more informed decisions, and ensure reproducibility of your ML pipelines.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Models

Managing ML models throughout their lifecycle with ZenML

Machine learning models and AI agent configurations are at the heart of any ML workflow and AI system. ZenML provides comprehensive model management capabilities through its Model Control Plane, allowing you to track, version, promote, and share both traditional ML models and AI agent systems across your pipelines.

{% hint style="info" %}
The ZenML Model Control Plane is a [ZenML Pro](https://zenml.io/pro) feature. While the Python functions for creating and interacting with models are available in the open-source version, the visual dashboard for exploring and managing models is only available in ZenML Pro. Please [sign up here](https://zenml.io/pro) to get access to the full model management experience.
{% endhint %}

This guide covers all aspects of working with models in ZenML, from basic concepts to advanced usage patterns.

## Understanding Models in ZenML

### What is a ZenML Model?

A ZenML Model is an entity that groups together related resources:

* Pipelines that train, evaluate, or deploy the model or agent system
* Artifacts like datasets, model weights, predictions, prompt templates, and agent configurations
* Metadata including metrics, parameters, evaluation results, and business information

Think of a ZenML Model as a container that organizes all the components related to a specific ML use case, business problem, or AI agent system. This extends beyond just model weights or agent prompts - it represents the entire ML product or intelligent system.

{% hint style="info" %}
A ZenML Model is different from a "technical model" (the actual ML model files with weights and parameters) or "agent configuration" (prompt templates, tool definitions, etc.). These technical artifacts are just components that can be associated with a ZenML Model, alongside training data, predictions, evaluation results, and other resources.
{% endhint %}

### The Model Control Plane

The Model Control Plane is ZenML's unified interface for managing models throughout their lifecycle. It allows you to:

* Register and version models
* Associate pipelines and artifacts with models
* Track lineage and dependencies
* Manage model promotions through stages (staging, production, etc.)
* Exchange data between pipelines using models

{% hint style="info" %}
While all Model Control Plane functionality is accessible programmatically through the Python SDK in both OSS and Pro versions, the visual dashboard shown below is only available in ZenML Pro.
{% endhint %}

![Model Control Plane Overview in ZenML Pro Dashboard](/files/a6LfearAC5ZjPuNamO94)

## Working with Models

### Registering a Model

You can register models in several ways:

#### Using the Python SDK

```python
from zenml import Model
from zenml.client import Client

Client().create_model(
    name="customer_service_agent",
    license="MIT",
    description="Multi-agent system for customer service automation",
    tags=["agent", "customer-service", "llm", "rag"],
)
```

#### Using the CLI

```bash
zenml model register customer_service_agent --license="MIT" --description="Multi-agent customer service system"
```

#### Using a Pipeline

The most common approach is to register a model implicitly as part of a pipeline:

```python
from zenml import pipeline, Model

@pipeline(
    model=Model(
        name="iris_classifier",
        description="Classification model for the Iris dataset",
        tags=["classification", "sklearn"]
    )
)
def training_pipeline():
    # Pipeline implementation...
```

### Model Versioning

Each time you run a pipeline with a model configuration, a new model version is created. You can:

#### Explicitly Name Versions

```python
from zenml import Model, pipeline

@pipeline(
    model=Model(
        name="iris_classifier", 
        version="1.0.5"
    )
)
def training_pipeline():
    # Pipeline implementation...
```

#### Use Templated Naming

```python
from zenml import Model, pipeline

@pipeline(
    model=Model(
        name="iris_classifier", 
        version="run-{run.id[:8]}"
    )
)
def training_pipeline():
    # Pipeline implementation...
```

### Linking Artifacts to Models

Artifacts produced during pipeline runs can be linked to models to establish lineage and enable reuse:

```python
from zenml import step, Model
from zenml.artifacts.utils import save_artifact
import pandas as pd
from typing import Annotated
from zenml.artifacts.artifact_config import ArtifactConfig
from sklearn.base import ClassifierMixin
from sklearn.ensemble import RandomForestClassifier

# Example: Agent configuration step linking artifacts
@step(model=Model(name="CustomerServiceAgent", version="2.1.0"))
def configure_agent(
    knowledge_base: pd.DataFrame,
    evaluation_results: dict
) -> Annotated[dict, ArtifactConfig("agent_config")]:
    # Create agent configuration based on knowledge base and evaluations
    agent_config = {
        "prompt_template": generate_prompt_from_kb(knowledge_base),
        "tools": ["search", "database_query", "escalation"],
        "performance_threshold": evaluation_results["min_accuracy"],
        "model_params": {"temperature": 0.7, "max_tokens": 500}
    }
    
    # Save intermediate prompt variants
    for variant in ["concise", "detailed", "empathetic"]:
        prompt_variant = generate_prompt_variant(knowledge_base, variant)
        save_artifact(
            f"prompt_template_{variant}", 
            prompt_variant,
            is_model_artifact=True,
        )
    
    return agent_config
```

### Model Promotion

Model stages represent the progression of models through their lifecycle. ZenML supports the following stages:

* `staging`: Ready for final validation before production
* `production`: Currently deployed in a production environment
* `latest`: The most recent version (virtual stage)
* `archived`: No longer in use

You can promote models to different stages:

```python
from zenml import Model
from zenml.enums import ModelStages

# Promote a specific model version to production
model = Model(name="iris_classifier", version="1.2.3")
model.set_stage(stage=ModelStages.PRODUCTION)

# Find latest model and promote to staging
latest_model = Model(name="iris_classifier", version=ModelStages.LATEST)
latest_model.set_stage(stage=ModelStages.STAGING)
```

## Using Models Across Pipelines

One of the most powerful features of ZenML's Model Control Plane is the ability to share artifacts between pipelines through models.

### Pattern: Model-Mediated Artifact Exchange

This pattern allows pipelines to exchange data without knowing the specific artifact IDs:

```python
from typing import Annotated
from zenml import step, get_pipeline_context, pipeline, Model
from zenml.enums import ModelStages
import pandas as pd
from sklearn.base import ClassifierMixin

@step
def predict(
    model: ClassifierMixin,
    data: pd.DataFrame,
) -> Annotated[pd.Series, "predictions"]:
    """Make predictions using a trained model."""
    predictions = pd.Series(model.predict(data))
    return predictions

@pipeline(
    model=Model(
        name="iris_classifier",
        # Reference the production version
        version=ModelStages.PRODUCTION,
    ),
)
def inference_pipeline():
    """Run inference using the production model."""
    # Get the model from the pipeline context
    model = get_pipeline_context().model
    
    # Load inference data (you'd need to implement this function)
    inference_data = load_data()
    
    # Run prediction using the trained model artifact
    predict(
        model=model.get_model_artifact("trained_model"),
        data=inference_data,
    )
```

This pattern enables clean separation between training and inference pipelines while maintaining a clear relationship between them.

## Tracking Metrics and Metadata

ZenML allows you to attach metadata to models, which is crucial for tracking performance, understanding training conditions, and making promotion decisions.

{% hint style="info" %}
While metadata tracking is available in both OSS and Pro versions through the Python SDK, visualizing and exploring model metrics through a dashboard interface is only available in ZenML Pro.
{% endhint %}

### Logging Model Metadata

```python
from zenml import step, log_metadata, get_step_context

@step
def evaluate_model(model, test_data):
    """Evaluate the model and log metrics."""
    predictions = model.predict(test_data)
    
    # Note: You'd need to implement these metric calculation functions
    accuracy = calculate_accuracy(predictions, test_data.target)
    precision = calculate_precision(predictions, test_data.target)
    recall = calculate_recall(predictions, test_data.target)
    
    # Log metrics to the model
    log_metadata(
        metadata={
            "evaluation_metrics": {
                "accuracy": accuracy,
                "precision": precision,
                "recall": recall
            }
        },
        infer_model=True,  # Attaches to the model in the current step context
    )

# Example: Evaluate agent and log metrics
@step
def evaluate_agent(agent_config, test_queries):
    """Evaluate the agent and log performance metrics."""
    responses = []
    for query in test_queries:
        response = agent_config.process_query(query)
        responses.append(response)
    
    # Note: You'd need to implement these agent evaluation functions
    response_quality = calculate_response_quality(responses, test_queries)
    response_time = calculate_avg_response_time(responses)
    user_satisfaction = calculate_satisfaction_score(responses)
    tool_usage_efficiency = calculate_tool_efficiency(agent_config.tools)
    
    # Log agent performance metrics to the model
    log_metadata(
        metadata={
            "agent_evaluation": {
                "response_quality": response_quality,
                "avg_response_time_ms": response_time,
                "user_satisfaction_score": user_satisfaction,
                "tool_efficiency": tool_usage_efficiency,
                "total_queries_evaluated": len(test_queries)
            },
            "agent_configuration": {
                "prompt_template_version": agent_config.prompt_version,
                "tools_enabled": agent_config.tools,
                "model_temperature": agent_config.temperature
            }
        },
        infer_model=True,  # Attaches to the agent model in the current step context
    )
```

### Fetching Model Metadata

You can retrieve logged metadata for analysis or decision-making:

```python
from zenml.client import Client

# Get a specific model version
model = Client().get_model_version("iris_classifier", "1.2.3")

# Access metadata
metrics = model.run_metadata["evaluation_metrics"].value
print(f"Model accuracy: {metrics['accuracy']}")
```

## Deleting Models

When a model is no longer needed, you can delete it or specific versions:

### Deleting All Versions of a Model

```python
from zenml.client import Client

# Using the Python SDK
Client().delete_model("iris_classifier")

# Or using the CLI
# zenml model delete iris_classifier
```

### Deleting a Specific Version

```python
from zenml.client import Client

# Using the Python SDK
Client().delete_model_version("model_version_id")

# Or using the CLI
# zenml model version delete <MODEL_VERSION_NAME>
```

## Best Practices

* **Consistent Naming**: Use consistent naming conventions for models and versions
* **Rich Metadata**: Log comprehensive metadata to provide context for each model version
* **Promotion Strategy**: Develop a clear strategy for promoting models through stages
* **Model Association**: Associate pipelines with models to maintain lineage and enable artifact sharing
* **Versioning Strategy**: Choose between explicit versioning and template-based versioning based on your needs

## Conclusion

The Model Control Plane in ZenML provides a comprehensive solution for managing both traditional ML models and AI agent systems throughout their lifecycle. By properly registering, versioning, linking artifacts, and tracking metadata, you can create a transparent and reproducible workflow for your ML projects and AI agent development.

{% hint style="info" %}
**OSS vs Pro Feature Summary:**

* **ZenML OSS:** Includes all the programmatic (Python SDK) model features described in this guide
* **ZenML Pro:** Adds visual model dashboard, advanced model exploration, comprehensive metrics visualization, and integrated model lineage views
  {% endhint %}

Whether you're working on a simple classification model, a complex production ML system, or a sophisticated multi-agent AI application, ZenML's unified model management capabilities help you organize your resources and maintain clarity across your entire AI development lifecycle.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Dashboard

Explore the features and capabilities of the ZenML dashboard

The ZenML dashboard serves as a visual control center for your ML operations, offering intuitive interfaces to navigate pipelines, artifacts, models, and metadata. This guide offers a comprehensive overview of the dashboard's features, helping you leverage its full potential for monitoring, managing, and optimizing your machine learning workflows.

### Open Source Dashboard

The open-source version of ZenML includes a robust set of dashboard features that provide significant value for individual practitioners and teams.

#### Pipeline Visualization Options

ZenML offers two complementary ways to visualize pipeline executions: the **DAG View** and the **Timeline View**. Each is optimized for different aspects of pipeline analysis, helping you understand both the structure and performance of your workflows.

**DAG View**

The DAG (Directed Acyclic Graph) view displays your pipeline as a network graph, showing how data flows between steps. It explicitly visualizes parallel branches, artifact connections, and the overall architecture of your workflow.

![Pipeline DAG visualization](/files/OQhcvqYRpTLl9voPtqOh)

This view is best for understanding pipeline architecture, tracing data lineage, and debugging dependency issues. While comprehensive, it can become visually dense in pipelines with a very large number of steps.

**Timeline View**

The Timeline View offers a Gantt chart-style visualization where each step is represented by a horizontal bar whose length corresponds to its execution duration. This view excels at performance analysis, making it easy to spot bottlenecks and understand the runtime characteristics of your pipeline.

![Pipeline Timeline View](/files/0CCBVUWqcx7FZd8DZoci)

This view is ideal for performance optimization, identifying bottlenecks, and monitoring pipeline efficiency, especially for large pipelines. For pipelines with a high number of steps (e.g., over 100), ZenML automatically defaults to the Timeline View to ensure a responsive and clear user experience.

These views are complementary and work best when used together. The DAG view helps you understand **what** your pipeline does and **how** it's structured, while the Timeline view shows you **when** things happen and **where** to focus optimization efforts.

**Use the DAG View when you need to:**

* Understand how data flows through your pipeline.
* Debug issues related to step dependencies.
* Explain the pipeline architecture to stakeholders.
* Verify that parallel execution paths are configured correctly.

**Use the Timeline View when you need to:**

* Identify performance bottlenecks.
* Optimize pipeline execution time.
* Compare execution duration across steps.
* Get a quick overview of which steps dominate runtime.

#### Pipeline Run Management

The dashboard maintains a comprehensive history of pipeline runs, allowing you to:

* Browse through previous executions
* Compare configurations across runs
* Track changes in pipeline structure over time
* Filter runs by status, name, or other attributes

![Pipeline run history](/files/rFS6uORQpK6cOu2G15JB)

#### Artifact Visualization

The dashboard provides built-in visualization capabilities for artifacts produced during pipeline execution.

**Automatic Data Type Visualizations**

Common data types receive automatic visualizations, including:

* Pandas DataFrames displayed as interactive tables
* NumPy arrays rendered as appropriate charts or heatmaps
* Images shown directly in the browser
* Text data formatted for readability

![Artifact visualization](/files/dlbx71qMTdh4BQJImTmR)

**Artifact Lineage Tracking**

The dashboard shows how artifacts are connected across pipeline steps, enabling you to:

* Trace data transformations through your pipeline
* Understand how intermediate outputs contribute to final results
* Verify data flow through complex workflows

#### Step Execution Details

**Logs and Outputs**

Access detailed logs for each step execution directly in the dashboard:

* View standard output and error logs
* Monitor execution progress
* Troubleshoot errors with full context
* Search through logs to identify specific events

![Step logs](/files/wEMtiNo4yEr01Z2n6err)

**Runtime Metrics**

Monitor runtime performance metrics for each step:

* Execution duration
* Resource utilization patterns
* Start and end timestamps
* Cache hit/miss information

#### Stack and Component Management

The dashboard provides a visual interface for managing your ZenML infrastructure through stacks and components. This graphical approach to MLOps infrastructure management simplifies what would otherwise require complex CLI commands or code.

**Stack Creation and Configuration**

Creating ML infrastructure stacks through the dashboard is intuitive and visual. The interface guides you through selecting compatible components and configuring their settings. You can see the entire stack architecture at a glance, making it easier to understand the relationships between different infrastructure pieces.

![Stack management](/files/fZ3scRpD75z1NtGRH0Wg)

When building a stack, the dashboard helps you browse available components by category and suggests compatible options. Once created, stacks can be shared with team members, enabling consistent infrastructure across your organization.

**Component Registration**

The dashboard streamlines the process of registering individual components like orchestrators, artifact stores, and container registries. Instead of writing configuration code, you can use form-based interfaces to set up each component.

The UI helps connect components to appropriate service connectors and validates settings before saving. This visual approach to component management reduces configuration errors and simplifies the setup process, especially for team members who may not be familiar with the underlying infrastructure details.

![Component registration](/files/6c10i3XNesrQC442jqEW)

#### Integration-Specific Visualizations

The dashboard supports specialized visualizations for outputs from popular integrations:

* Evidently reports as interactive HTML
* Great Expectations validation results with detailed insights
* WhyLogs profile visualizations
* Confusion matrices and classification reports
* Custom visualization components for specialized data types

![Integration visualizations](/files/9WSxjWVB5XzV7IKLWSKd)

### ZenML Pro Dashboard

ZenML Pro extends the open-source dashboard with additional capabilities for collaboration, governance, and workflow management. For a broader overview of ZenML Pro features, see [ZenML Pro](https://www.zenml.io/pro). For a side-by-side comparison of ZenML OSS vs ZenML Pro, see [ZenML Open Source vs Pro](https://www.zenml.io/open-source-vs-pro).

This page focuses on Pro features that surface in the dashboard UI. ZenML Pro also includes platform capabilities that are not specific to the dashboard.

#### Projects and Access Management

ZenML Pro introduces organizations, workspaces, teams, and projects. **Projects** are the main way to organize related pipelines, runs, artifacts, and models and to separate concerns across teams or environments. In the Pro dashboard, you can create and manage projects, switch project context, and view assets scoped to the active project.

**Organization and Team Structure**

* **Organizations**: Top-level entities containing users, teams, and workspaces
* **Teams**: Groups of users with assigned roles for simplified permission management
* **Workspaces**: Isolated ZenML deployments with separate resources
* **Projects**: Logical subdivisions for organizing related ML assets

![Organization structure](/files/rbq2nevQuigAt2kFScuI)

**Role Management**

The dashboard provides intuitive interfaces for managing roles at different levels:

* **Organization roles**: Admin, Manager, Viewer, Billing Admin, Member
* **Workspace roles**: Admin, Developer, Contributor, Viewer, Stack Admin
* **Project roles**: Admin, Developer, Contributor, Viewer
* **Custom roles**: Create roles with fine-grained permissions

**Access Control UI**

The dashboard makes it easy to:

* Configure user and team permissions
* Manage resource sharing
* Implement least-privilege access policies
* Review and audit access rights
* Visualize permission hierarchies

#### Pipeline Snapshots

A **pipeline snapshot** is an immutable, runnable representation of a pipeline that captures its structure and configuration. Snapshots enable teams to run pipelines from the dashboard without direct access to the codebase and help standardize execution across different users and environments.

To learn how to create and run snapshots from the dashboard, see [Pipeline Snapshots](/concepts/snapshots).

#### Artifact Control Plane

ZenML Pro provides a sophisticated artifact control plane that enhances your ability to manage and understand data flowing through your pipelines.

**Comprehensive Metadata Management**

The Pro dashboard transforms how you interact with pipeline and model metadata through its powerful exploration tools. When examining ML workflows, metadata provides crucial context about performance metrics, parameters, and execution details.

With the dashboard, you can browse the full set of metadata attributes and apply filters to focus on specific metrics. The interface tracks historical changes to these values, making it easy to understand how your models evolve over time.

Customizable metadata views adapt to different analysis needs, whether you're comparing accuracy across runs or examining resource utilization patterns. This metadata visualization integrates seamlessly with artifact lineage tracking, creating a complete picture of your ML workflow from inputs to outputs.

```python
from zenml import step, log_metadata, get_step_context

@step
def evaluate():
    # Log metrics that will be visualized in the dashboard
    log_metadata(
        metadata={
            "accuracy": 0.95,
            "precision": 0.92,
            "recall": 0.91,
            "f1_score": 0.93
        }
    )
```

#### Model Control Plane

The Model Control Plane provides centralized model management capabilities designed for production ML workflows that allows you to have:

* Clear visualization of model version history
* Detailed comparisons between versions
* Performance metrics for each version
* Linkage to generating pipelines and input artifacts

![Model version management](/files/Pa7RcztaCe0uLdvxEBev)

Additionally, the Pro dashboard allows you to manage model lifecycle stages:

* Move models between stages (latest, staging, production, archived)
* Track transition history and approvals
* Configure automated promotion rules
* Monitor model status across environments

#### Experiment Comparison Tools

ZenML Pro offers powerful tools for comparing experiments and understanding the relationships between different runs.

**Table View Comparisons**

Compare metadata, configurations, and outcomes across runs:

* Side-by-side comparison of metrics
* Highlight differences between runs
* Sort and filter by any attribute
* Export comparison data for further analysis

![Experiment comparison table](/files/yP0URIp6jOb4yaUlMQUh)

**Parallel Coordinates Visualization**

Understand complex relationships between parameters and outcomes:

* Visualize multiple dimensions simultaneously
* Identify patterns and correlations
* Filter runs interactively
* Focus on specific parameter ranges

![Parallel coordinates visualization](/files/ZPOhesVKxUdb7dOCeDAw)

{% hint style="info" %}
For the complete ZenML Pro feature set (including capabilities that are not dashboard-specific), see [ZenML Pro](https://www.zenml.io/pro) and [ZenML Open Source vs Pro](https://www.zenml.io/open-source-vs-pro).
{% endhint %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Templates

Create and run templates in ZenML to standardize execution.

{% hint style="warning" %}
Run templates have been replaced by [pipeline snapshots](/concepts/snapshots).
{% endhint %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Community & content

All possible ways for our community to get in touch with ZenML.

The ZenML team and community have put together a list of references that can be used to get in touch with the development team of ZenML and develop a deeper understanding of the framework.

### Slack Channel: Get help from the community

The ZenML [Slack channel](https://zenml.io/slack) is the main gathering point for the community. Not only is it the best place to get in touch with the core team of ZenML, but it is also a great way to discuss new ideas and share your ZenML projects with the community. If you have a question, there is a high chance someone else might have already answered it on Slack!

### Social Media: Bite-sized updates

We are active on LinkedIn (linkedin.com/company/zenml/) and Twitter / X (@zenml\_io), where we post bite-sized updates on releases, events, and MLOps in general. Follow us to interact and stay up to date! We would appreciate it if you could comment on and share our posts so more people can benefit from our work at ZenML!

### YouTube Channel: Video tutorials, workshops, and more

Our [YouTube channel](https://www.youtube.com/c/ZenML) features a growing set of videos that take you through the entire framework. Go here if you are a visual learner, and follow along with some tutorials.

### Public roadmap

The feedback from our community plays a significant role in the development of ZenML. That's why we have a [public roadmap](https://zenml.io/roadmap) that serves as a bridge between our users and our development team. If you have ideas regarding any new features or want to prioritize one over the other, feel free to share your thoughts here or vote on existing ideas.

### Blog

On our [Blog](https://zenml.io/blog/) page, you can find various articles written by our team. We use it as a platform to share our thoughts and explain the implementation process of our tool, its new features, and the thought process behind them.

### Podcast

We also have a [Podcast](https://podcast.zenml.io/) series that brings you interviews and discussions with industry leaders, top technology professionals, and others. We discuss the latest developments in machine learning, deep learning, and artificial intelligence, with a particular focus on MLOps, or how trained models are used in production.

### Newsletter

You can also subscribe to our [Newsletter](https://zenml.io/newsletter-signup), where we share what we learn as we develop open-source tooling for production machine learning. You will also get all the exciting news about ZenML in general.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Environment Variables

How to control ZenML behavior with environmental variables.

There are a few pre-defined environmental variables that can be used to control the behavior of ZenML. See the list below with default values and options:

## Logging verbosity

```bash
export ZENML_LOGGING_VERBOSITY=INFO
```

Choose from `INFO`, `WARN`, `ERROR`, `CRITICAL`, `DEBUG`.

## Console logging format

```bash
export ZENML_CONSOLE_LOGGING_FORMAT=console
```

Choose from `console` (default), `json`, or any valid Python `%`-style logging format string such as `%(asctime)s - %(message)s` for custom console output. The `console` format uses a default compact layout for client-side `INFO+` logs and full structured text layout for `DEBUG` and server logs. The `json` format emits JSON formatted console/stdout logs.

This controls terminal log formatting only; stored logs keep their raw message and structured metadata. The older `ZENML_LOGGING_FORMAT` environment variable is still supported as a deprecated alias.

See [this page](https://docs.zenml.io/concepts/steps_and_pipelines/logging) for more information.

## Disable step logs

Usually, ZenML [stores step logs in the artifact store](https://docs.zenml.io/concepts/steps_and_pipelines/logging), but this can sometimes cause performance bottlenecks, especially if the code utilizes progress bars.

If you want to configure whether logged output from steps is stored or not, set the `ZENML_DISABLE_STEP_LOGS_STORAGE` environment variable to `true`. Note that this will mean that logs from your steps will no longer be stored and thus won't be visible on the dashboard anymore.

```bash
export ZENML_DISABLE_STEP_LOGS_STORAGE=true
```

## File copy chunk size

When copying files between different filesystems (e.g. uploading or downloading artifacts, models or code archives to/from a remote artifact store), ZenML streams the file in chunks to keep memory usage bounded. The default chunk size is 8 MiB and can be adjusted:

```bash
export ZENML_FILEIO_COPY_CHUNK_SIZE=16777216  # 16 MiB
```

Non-positive values are ignored and the default is used instead.

## ZenML repository path

To configure where ZenML will install and look for its repository, set the environment variable `ZENML_REPOSITORY_PATH`.

```bash
export ZENML_REPOSITORY_PATH=/path/to/somewhere
```

## Analytics

Please see [our full page](/reference/global-settings#usage-analytics) on what analytics are tracked and how you can opt out, but the quick summary is that you can set this to `false` if you want to opt out of analytics.

```bash
export ZENML_ANALYTICS_OPT_IN=false
```

## Debug mode

Setting to `true` switches to developer mode. This redirects all ZenML analytics events to the development ZenML analytics server instead of the official ZenML analytics server. Should not be used in production environments.

```bash
export ZENML_DEBUG=true
```

## Active stack

Setting the `ZENML_ACTIVE_STACK_ID` to a specific UUID will make the corresponding stack the active stack:

```bash
export ZENML_ACTIVE_STACK_ID=<UUID-OF-YOUR-STACK>
```

## Prevent pipeline execution

When `true`, this prevents a pipeline from executing:

```bash
export ZENML_PREVENT_PIPELINE_EXECUTION=false
```

## Disable rich traceback

Set to `false` to disable the [`rich` traceback](https://rich.readthedocs.io/en/stable/traceback.html):

```bash
export ZENML_ENABLE_RICH_TRACEBACK=true
```

## Disable colorful logging

Console logs use colors by default. If you wish to disable colorful logging, set the following environment variable:

```bash
ZENML_LOGGING_COLORS_DISABLED=true
```

Note that setting this on the client environment (e.g. your local machine which runs the pipeline) will automatically disable colorful logging on remote orchestrators. If you wish to disable it locally, but turn on for remote orchestrators, you can set the `ZENML_LOGGING_COLORS_DISABLED` environment variable in your orchestrator's environment as follows:

```python
docker_settings = DockerSettings(environment={"ZENML_LOGGING_COLORS_DISABLED": "false"})

# Either add it to the decorator
@pipeline(settings={"docker": docker_settings})
def my_pipeline() -> None:
    my_step()

# Or configure the pipelines options
my_pipeline = my_pipeline.with_options(
    settings={"docker": docker_settings}
)
```

## Server OpenTelemetry export

Set `ZENML_SERVER_OTEL_EXPORTER_OTLP_ENDPOINT` to export ZenML server traces, metrics, and logs to an OpenTelemetry-compatible backend using OTLP/HTTP. The standard `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable is also supported as a fallback:

```bash
export ZENML_SERVER_OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
# OR
# export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
```

The endpoint should be the base OTLP/HTTP endpoint. ZenML appends `/v1/traces`, `/v1/metrics`, and `/v1/logs` for each signal. Standard per-signal OTLP/HTTP endpoint variables are also supported and take precedence:

```bash
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4318/v1/traces
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://otel-collector:4318/v1/metrics
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otel-collector:4318/v1/logs
```

You can use the matching ZenML-specific names instead: `ZENML_SERVER_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `ZENML_SERVER_OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, and `ZENML_SERVER_OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`.

If no base or per-signal endpoint is set, server OpenTelemetry instrumentation is disabled.

Each signal is enabled by default when the endpoint is configured. You can disable individual signals with:

```bash
export ZENML_SERVER_OTEL_TRACES_ENABLED=false
export ZENML_SERVER_OTEL_METRICS_ENABLED=false
export ZENML_SERVER_OTEL_LOGS_ENABLED=false
```

You can also customize the service name reported in OpenTelemetry resource attributes. The standard `OTEL_SERVICE_NAME` environment variable is supported as well as a fallback. If unset, it defaults to `zenml-server` for self-hosted deployments, or to the ZenML Pro workspace name for cloud deployments.

```bash
export ZENML_SERVER_OTEL_SERVICE_NAME=zenml-server
# OR
# export OTEL_SERVICE_NAME=zenml-server
```

Standard OTLP headers, timeout, compression, trace sampler, and resource attribute environment variables are handled by the OpenTelemetry Python SDK and OTLP/HTTP exporters where supported. OTLP/gRPC protocol environment variables are not supported because ZenML configures OTLP/HTTP exporters directly. You can read more about the OpenTelemetry environment variables and SDK configuration [here](https://opentelemetry.io/docs/languages/sdk-configuration/).

## Disable stack validation

If you wish to disable stack validation, set the following environment variable:

```bash
ZENML_SKIP_STACK_VALIDATION=true
```

## Ignore untracked code repository files

When using [code repositories](https://docs.zenml.io/concepts/code-repositories), ZenML will by default require the local checkout to have no uncommitted or untracked files in order to use the code repository to track the commit and download files. If you want to ignore untracked files, you can set the `ZENML_CODE_REPOSITORY_IGNORE_UNTRACKED_FILES` environment variable to `True`. When doing this, you're responsible that the files committed to the repository includes everything necessary to run your pipeline.

## ZenML global config path

To set the path to the global config file, used by ZenML to manage and store the state for a number of settings, set the environment variable as follows:

```bash
export ZENML_CONFIG_PATH=/path/to/somewhere
```

## CLI output formatting

### Default output format

Set the default output format for all CLI list commands:

```bash
export ZENML_DEFAULT_OUTPUT=json
```

Choose from `table` (default), `json`, `yaml`, `csv`, or `tsv`. This applies to commands like `zenml stack list`, `zenml pipeline list`, etc.

### Terminal width override

Override the automatic terminal width detection for table rendering:

```bash
export ZENML_CLI_COLUMN_WIDTH=120
```

This is useful when running ZenML in CI/CD environments or when you want to control table formatting regardless of your terminal size.

## Server configuration

For more information on server configuration, see the [ZenML Server documentation](/deploying-zenml/deploying-zenml/deploy-with-docker#zenml-server-configuration-options) for more, especially the section entitled "ZenML server configuration options".

## Client configuration

Setting the `ZENML_STORE_URL`, `ZENML_STORE_API_KEY` and `ZENML_ACTIVE_PROJECT_ID` environment variables automatically connects your ZenML Client to the specified server for a specific project. This method is particularly useful when you are using the ZenML client in an automated CI/CD workload environment like GitHub Actions or GitLab CI or in a containerized environment like Docker or Kubernetes:

```bash
export ZENML_STORE_URL=https://...
export ZENML_STORE_API_KEY=<API_KEY>
export ZENML_ACTIVE_PROJECT_ID=<PROJECT_ID>
```

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# LLM Tooling

LLM tooling for ZenML - MCP servers, llms.txt, and Agent Skills

ZenML provides multiple ways to enhance your AI-assisted development workflow:

* **MCP servers** for real-time doc queries and server interaction
* **llms.txt** for grounding LLMs with ZenML documentation
* **Agent Skills** for guided implementation of ZenML features

## About llms.txt

The llms.txt file format was proposed by [llmstxt.org](https://llmstxt.org/) as a standard way to provide information to help LLMs answer questions about a product/website. From their website:

> We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content. This file offers brief background information, guidance, and links to detailed markdown files. llms.txt markdown is human and LLM readable, but is also in a precise format allowing fixed processing methods (i.e. classical programming techniques such as parsers and regex).

## ZenML's llms.txt

ZenML's documentation is now made available to LLMs at the following link:

```
https://docs.zenml.io/llms.txt
```

This file contains a comprehensive summary of the ZenML documentation (containing links and descriptions) that LLMs can use to answer questions about ZenML's features, functionality, and usage.

## How to use the llms.txt file

When working with LLMs (like ChatGPT, Claude, or others), you can use this file to help the model provide more accurate answers about ZenML:

* Point the LLM to the `docs.zenml.io/llms.txt` URL when asking questions about ZenML
* While prompting, instruct the LLM to only provide answers based on information contained in the file to avoid hallucinations
* For best results, use models with sufficient context window to process the entire file

## Use llms-full.txt for complete documentation context

The llms-full.txt file contains the entire ZenML documentation in a single, concatenated markdown file optimized for LLMs. Use it when you want to load all docs as context at once (for example, a one-shot grounding pass) rather than querying individual pages. Access it here: <https://docs.zenml.io/llms-full.txt>. For interactive, selective queries from your IDE, the built-in MCP server is still the recommended option.

## Use the built-in GitBook MCP server (recommended)

ZenML docs are also exposed through a native GitBook MCP server that IDE agents can query in real time.

* Endpoint: <https://docs.zenml.io/\\~gitbook/mcp>

### Quick setup

#### Claude Code (VS Code)

Run the following command in your terminal to add the server:

```bash
claude mcp add zenmldocs --transport http https://docs.zenml.io/~gitbook/mcp
```

#### Cursor

Add the server via Cursor's JSON settings (Settings → search "MCP" → Configure via JSON):

```json
{
  "mcpServers": {
    "zenmldocs": {
      "transport": {
        "type": "http",
        "url": "https://docs.zenml.io/~gitbook/mcp"
      }
    }
  }
}
```

### Why use it

* Live doc queries directly from your IDE agent
* Syntax-aware, source-of-truth answers with fewer hallucinations
* Faster feature discovery across guides, APIs, and examples

The MCP server indexes the latest released documentation, not the develop branch.

{% hint style="info" %}
**Looking to chat with your ZenML server data?** ZenML also provides its own MCP server that connects directly to your ZenML server, allowing you to query pipelines, analyze runs, and trigger executions through natural language. See the [MCP Chat with Server guide](https://docs.zenml.io/user-guides/best-practices/mcp-chat-with-server) for setup instructions.
{% endhint %}

Prefer the native GitBook MCP server above for the best experience; if you prefer working directly with llms.txt or need alternative workflows, the following tools are helpful:

To use the llms.txt file in partnership with an MCP client, you can use the following tools:

* [GitMCP](https://gitmcp.io/) - A way to quickly create an MCP server for a github repository (e.g. for `zenml-io/zenml`)
* [mcp-llms](https://github.com/parlance-labs/mcp-llms.txt/) - This shows how to use an MCP server to iteratively explore the llms.txt file with your MCP client
* [mcp-llms-txt-explorer](https://github.com/thedaviddias/mcp-llms-txt-explorer) - A tool to help you explore and discover websites that have llms.txt files

## ZenML Agent Skills

Agent Skills are modular capabilities that help AI coding agents perform specific tasks. ZenML publishes skills through a plugin marketplace that works with many popular agentic coding tools.

### Supported tools

ZenML skills work with tools that support the Agent Skills format:

| Tool                                                      | Type                    | Skills support             |
| --------------------------------------------------------- | ----------------------- | -------------------------- |
| [Claude Code](https://code.claude.com/)                   | Anthropic's CLI agent   | Native plugin marketplace  |
| [OpenAI Codex CLI](https://github.com/openai/codex)       | OpenAI's terminal agent | Native skills support      |
| [GitHub Copilot](https://github.com/features/copilot)     | IDE coding assistant    | Agent Skills integration   |
| [OpenCode](https://github.com/opencode-ai/opencode)       | Open source AI agent    | Native skills support      |
| [Amp](https://amp.dev)                                    | AI coding assistant     | Agent Skills integration   |
| [Cursor](https://cursor.sh)                               | AI-powered IDE          | Via settings configuration |
| [Gemini CLI](https://github.com/google-gemini/gemini-cli) | Google's terminal agent | Skills support             |

### Installing ZenML skills

#### Claude Code

```bash
# Add the ZenML marketplace (one-time setup)
/plugin marketplace add zenml-io/skills

# Install any available skill (repeat for each skill you want)
/plugin install zenml-scoping@zenml
/plugin install zenml-pipeline-authoring@zenml
/plugin install zenml-quick-wins@zenml
```

#### OpenAI Codex CLI

```bash
# Add the ZenML marketplace
codex plugin add zenml-io/skills

# Install skills
codex plugin install zenml-scoping@zenml
codex plugin install zenml-pipeline-authoring@zenml
codex plugin install zenml-quick-wins@zenml
```

### Available skills

| Skill                      | What it's for                                                                                                                            | When to use it                                                                            |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `zenml-scoping`            | Structured interview to turn a big ML/LLM idea into a realistic multi-pipeline ZenML architecture                                        | When requirements are fuzzy, ambitious, or likely "too much in one pipeline"              |
| `zenml-pipeline-authoring` | Hands-on guidance to write/refactor ZenML steps/pipelines, config, Docker settings, materializers, metadata, secrets, and visualizations | When you're actively implementing a pipeline (or debugging "works local, fails on cloud") |
| `zenml-quick-wins`         | Repo/stack-aware recommendations of high-impact ZenML improvements, then guided implementation                                           | When you already have a project and want best-practice upgrades fast                      |

#### `zenml-scoping`

Scopes and decomposes ML workflow ideas into realistic ZenML pipeline architectures through an in-depth interview process. Produces a `pipeline_architecture.md` spec you can implement incrementally.

**Use when:**

* You have an "end-to-end" idea (ingest → train → deploy → monitor → retrain) and need a sane MVP
* You're unsure what should be one pipeline vs multiple pipelines
* You want explicit cross-pipeline data flow (e.g., via the Model Control Plane)

**What it does:**

1. Interviews you to capture goals, data sources, and operational needs
2. Classifies components (pipeline vs deployed pipeline vs not-a-pipeline)
3. Produces a decomposed architecture with an MVP recommendation (`pipeline_architecture.md`)

**Example prompts:**

```
Use zenml-scoping to turn my idea into a realistic ZenML pipeline architecture and write pipeline_architecture.md.

Use zenml-scoping to decide whether this should be one pipeline or multiple pipelines, and propose the MVP.
```

#### `zenml-pipeline-authoring`

Authors ZenML pipelines with steps, artifacts, Docker settings, materializers, metadata logging, secrets management, YAML configuration, and custom visualizations.

**Use when:**

* You want to write or refactor ZenML `@step` / `@pipeline` code with correct artifact flow and typing
* You're implementing dynamic pipelines (`@pipeline(dynamic=True)`)
* You're moving from local to remote execution (Kubernetes / Vertex / SageMaker) and hit portability issues

**What it does:**

1. Clarifies requirements (static vs dynamic, local vs remote, data sources, custom types)
2. Guides step/pipeline structure and artifact flow
3. Helps with configuration (YAML), Docker settings, materializers, metadata, and secrets

**Example prompts:**

```
Use zenml-pipeline-authoring to implement the MVP pipeline described in pipeline_architecture.md.

Use zenml-pipeline-authoring to refactor this pipeline so it works on Kubernetes/Vertex/SageMaker (fix artifact flow + Docker settings).
```

#### `zenml-quick-wins`

Guides you through discovering and implementing high-impact ZenML features. The skill investigates your current setup, recommends priorities based on your stack, and helps implement improvements interactively.

**Use when:**

* You want to improve your ZenML setup
* You're looking for MLOps best practices to adopt
* You need help with features like experiment tracking, alerting, scheduling, or model governance

**What it does:**

1. **Investigate** - Analyzes your stack configuration and codebase
2. **Recommend** - Prioritizes quick wins based on your current setup
3. **Implement** - Helps you apply selected improvements
4. **Verify** - Confirms the implementation works

**Example prompts:**

```
Use zenml-quick-wins to analyze this repo and recommend the top 3 quick wins.

Implement metadata logging and tags across my pipelines.

Set up Slack alerts for pipeline failures.
```

See the [Quick Wins guide](https://github.com/zenml-io/zenml/tree/main/docs/book/user-guide/best-practices/quick-wins.md) for the full catalog of improvements this skill can help implement.

### Coming soon

We're developing additional skills to help with common ZenML workflows:

* **Stack setup** - Guided stack component configuration
* **Debugging** - Investigating pipeline failures and performance issues
* **Migration** - Migrating from other MLOps platforms and orchestrators to ZenML

### Combining MCP + Skills

For the best AI-assisted ZenML development experience, combine:

1. **GitBook MCP server** (`https://docs.zenml.io/~gitbook/mcp`) - For doc-grounded answers
2. **ZenML server MCP** ([setup guide](https://github.com/zenml-io/zenml/tree/main/docs/book/user-guide/best-practices/mcp-chat-with-server.md)) - For querying your live pipelines, runs, and stacks
3. **Agent Skills** - For guided implementation of features

This gives your AI assistant access to documentation, your actual ZenML data, and structured workflows for making changes.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# FAQ

Find answers to the most frequently asked questions about ZenML.

This page addresses common questions about ZenML, including general information about the project and how to accomplish specific tasks.

## About ZenML

#### Why did you build ZenML?

We built it because we scratched our own itch while deploying multiple machine-learning models in production over the past three years. Our team struggled to find a simple yet production-ready solution whilst developing large-scale ML pipelines. We built a solution for it that we are now proud to share with all of you! Read more about this backstory [on our blog here](https://blog.zenml.io/why-zenml/).

#### Is ZenML just another orchestrator like Airflow, Kubeflow, Flyte, etc?

Not really! An orchestrator in MLOps is the system component that is responsible for executing and managing the execution of an ML pipeline. ZenML is a framework that allows you to run your pipelines on whatever orchestrator you like, and we coordinate with all the other parts of an ML system in production. There are [standard orchestrators](https://docs.zenml.io/stacks/orchestrators) that ZenML supports out-of-the-box, but you are encouraged to [write your own orchestrator](https://docs.zenml.io/stacks/orchestrators/custom) in order to gain more control as to exactly how your pipelines are executed!

#### Can I use the tool `X`? How does the tool `Y` integrate with ZenML?

Take a look at our [documentation](https://docs.zenml.io) (in particular the [component guide](https://docs.zenml.io/stacks)), which contains instructions and sample code to support each integration that ZenML supports out of the box. You can also check out [our integration test code](https://github.com/zenml-io/zenml/tree/main/tests/integration/examples/README.md) to see active examples of many of our integrations in action.

The ZenML team and community are constantly working to include more tools and integrations to the above list (check out the [roadmap](https://zenml.io/roadmap) for more details).

Most importantly, ZenML is extensible, and we encourage you to use it with whatever other tools you require as part of your ML process and system(s). Check out [our documentation on how to get started](/) with extending ZenML to learn more!

#### Which license does ZenML use?

ZenML is distributed under the terms of the Apache License Version 2.0. A complete version of the license is available in the [LICENSE.md](https://github.com/zenml-io/zenml/blob/main/LICENSE) in this repository. Any contribution made to this project will be licensed under the Apache License Version 2.0.

## Platform Support

#### Do you support Windows?

ZenML officially supports Windows if you're using WSL. Much of ZenML will also work on Windows outside a WSL environment, but we don't officially support it, and some features don't work (notably anything that requires spinning up a server process).

#### Do you support Macs running on Apple Silicon?

Yes, ZenML does support Macs running on Apple Silicon. You just need to make sure that you set the following environment variable:

```bash
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
```

This is a known issue with how forking works on Macs running on Apple Silicon, and it will enable you to use ZenML and the server. This environment variable is needed if you are working with a local server on your Mac, but if you're just using ZenML as a client / CLI and connecting to a deployed server, then you don't need to set it.

## Common Use Cases and How-To's

#### How do I contribute to ZenML's open-source codebase?

We develop ZenML together with our community! To get involved, the best way to get started is to select any issue from the [`good-first-issue` label](https://github.com/zenml-io/zenml/labels/good%20first%20issue).

Please read [our Contribution Guide](https://github.com/zenml-io/zenml/blob/main/CONTRIBUTING.md) for more information. For small features and bug fixes, please open a pull request as described in the guide. For anything bigger, it is worth [posting a message in Slack](https://zenml.io/slack/) or [creating an issue](https://github.com/zenml-io/zenml/issues/new/choose) so we can best discuss and support your plans.

#### How do I add custom components to ZenML?

Please start by [reading the general documentation page](https://docs.zenml.io/stacks/contribute/custom-stack-component) on implementing a custom stack component, which offers some general advice on what you'll need to do.

From there, each of the custom stack component types has a dedicated section about adding your own custom components. For example, to add a custom orchestrator, you would [visit this page](https://docs.zenml.io/stacks/orchestrators/custom).

#### How do I mitigate dependency clashes with ZenML?

Check out [our dedicated documentation page](https://docs.zenml.io/user-guides/best-practices/configure-python-environments) on some ways you can try to solve these dependency and versioning issues.

#### How do I deploy cloud infrastructure and/or MLOps stacks?

ZenML is designed to be stack-agnostic, so you can use it with any cloud infrastructure or MLOps stack. Each of the documentation pages for stack components explain how to deploy these components on the most popular cloud providers.

#### How do I deploy ZenML on my internal company cluster?

Read [the documentation on self-hosted ZenML deployments](/deploying-zenml/deploying-zenml), in which several options are presented.

#### How do I implement hyperparameter tuning?

[Our dedicated documentation guide](https://github.com/zenml-io/zenml/tree/main/docs/book/user-guide/tutorial/hyper-parameter-tuning.md) on implementing this is the place to learn more.

#### How do I reset things when something goes wrong?

To reset your ZenML client, you can run `zenml clean` which will wipe your local metadata database and reset your client. Note that this is a destructive action, so feel free to [reach out to us on Slack](https://zenml.io/slack/) before doing this if you are unsure.

#### How do I create dynamic pipelines and steps?

Please read our [general information on how to compose steps + pipelines together](https://docs.zenml.io/user-guides/starter-guide/create-an-ml-pipeline) to start with. You might also find the code examples in [our guide to implementing hyperparameter tuning](https://docs.zenml.io/user-guides/tutorial/hyper-parameter-tuning) which is related to this topic.

#### How do I use templates and starter code with ZenML?

[Project templates](https://docs.zenml.io/user-guides/best-practices/project-templates) allow you to get going quickly with ZenML. We recommend the Starter template (`starter`) for most use cases, which gives you a basic scaffold and structure around which you can write your own code. You can also build templates for others inside a Git repository and use them with ZenML's templates functionality.

#### How do I upgrade my ZenML client and/or server?

Upgrading your ZenML client package is as simple as running `pip install --upgrade zenml` in your terminal. For upgrading your ZenML server, please refer to [the dedicated documentation section](/deploying-zenml/upgrade-zenml-server), which covers most of the ways you might do this as well as common troubleshooting steps.

#### How do I use a specific stack component?

For information on how to use a specific stack component, please refer to [the component guide](https://docs.zenml.io/stacks), which contains all our tips and advice on how to use each integration and component with ZenML.

## Community and Support

#### How can I speak with the community?

The first point of contact should be [our Slack group](https://zenml.io/slack/). Ask your questions about bugs or specific use cases, and someone from the core team will respond.

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Global settings

Understanding the global settings of your ZenML installation.

The information about the global settings of ZenML on a machine is kept in a folder commonly referred to as the **ZenML Global Config Directory** or the **ZenML Config Path**. The location of this folder depends on the operating system type and the current system user, but is usually located in the following locations:

* Linux: `~/.config/zenml`
* Mac: `~/Library/Application Support/zenml`
* Windows: `C:\Users\%USERNAME%\AppData\Local\zenml`

The default location may be overridden by setting the `ZENML_CONFIG_PATH` environment variable to a custom value. The current location of the global config directory used on a system can be retrieved by running the following commands:

```shell
# The output will tell you something like this:
# Using configuration from: '/home/stefan/.config/zenml'
zenml status

python -c 'from zenml.utils.io_utils import get_global_config_directory; print(get_global_config_directory())'
```

{% hint style="warning" %}
Manually altering or deleting the files and folders stored under the ZenML global config directory is not recommended, as this can break the internal consistency of the ZenML configuration. As an alternative, ZenML provides CLI commands that can be used to manage the information stored there:

* `zenml analytics` - manage the analytics settings
* `zenml clean` - to be used only in case of emergency, to bring the ZenML configuration back to its default factory state
* `zenml downgrade` - downgrade the ZenML version in the global configuration to match the version of the ZenML package installed in the current environment. Read more about this in the [ZenML Version Mismatch](#version-mismatch-downgrading) section.
  {% endhint %}

The first time that ZenML is run on a machine, it creates the global config directory and initializes the default configuration in it, along with a default Stack:

```
Initializing the ZenML global configuration version to 0.13.2
Creating default user 'default' ...
Creating default stack for user 'default'...
The active stack is not set. Setting the active stack to the default stack.
Using the default store for the global config.
Unable to find ZenML repository in your current working directory (/tmp/folder) or any parent directories. If you want to use an existing repository which is in a different location, set the environment variable 'ZENML_REPOSITORY_PATH'. If you want to create a new repository, run zenml init.
Running without an active repository root.
Using the default local database.
┏━━━━━━━━┯━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━┓
┃ ACTIVE │ STACK NAME │ SHARED │ OWNER   │ ARTIFACT_STORE │ ORCHESTRATOR ┃
┠────────┼────────────┼────────┼─────────┼────────────────┼──────────────┨
┃   👉   │ default    │ ❌     │ default │ default        │ default      ┃
┗━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
```

{% hint style="info" %}
The output can be customized with an `--output` (json, yaml, csv, tsv, table) option and a `--columns` selection. See [environment variables](/reference/environment-variables#cli-output-formatting) for more details.
{% endhint %}

The following is an example of the layout of the global config directory immediately after initialization:

```
/home/stefan/.config/zenml   <- Global Config Directory
├── config.yaml              <- Global Configuration Settings
└── local_stores             <- Every Stack component that stores information 
    |                           locally will have its own subdirectory here.              
    ├── a1a0d3d0-d552-4a80-be09-67e5e29be8ee   <- e.g. Local Store path for the 
    |                                             `default` local Artifact Store                                           
    └── default_zen_store
        |
        └── zenml.db         <- SQLite database where ZenML data (stacks, 
                                components, etc) are stored by default.
```

As shown above, the global config directory stores the following information:

1. The `config.yaml` file stores the global configuration settings: the unique ZenML client ID, the active database configuration, the analytics-related options, and the active Stack. This is an example of the `config.yaml` file contents immediately after initialization:

   ```yaml
   active_stack_id: ...
   analytics_opt_in: true
   store:
     database: ...
     url: ...
     username: ...
     ...
   user_id: d980f13e-05d1-4765-92d2-1dc7eb7addb7
   version: 0.13.2
   ```
2. The `local_stores` directory is where some "local" flavors of stack components, such as the local artifact store or a local MLFlow experiment tracker, persist data locally. Every local stack component will have its own subdirectory here named after the stack component's unique UUID. One notable example is the local artifact store flavor that, when part of the active stack, stores all the artifacts generated by pipeline runs in the designated local directory.
3. The `zenml.db` in the `default_zen_store` directory is the default SQLite database where ZenML stores all information about the stacks, stack components, custom stack component flavors, etc.

In addition to the above, you may also find the following files and folders under the global config directory, depending on what you do with ZenML:

* `kubeflow` - this is where the Kubeflow orchestrators that are part of a stack store some of their configuration and logs.

## Usage analytics

In order to help us better understand how the community uses ZenML, the pip package reports **anonymized** usage statistics. You can always opt out by using the CLI command:

```bash
zenml analytics opt-out
```

#### Why does ZenML collect analytics? <a href="#motivation" id="motivation"></a>

In addition to the community at large, **ZenML** is created and maintained by a startup based in Munich, Germany, called [ZenML GmbH](https://zenml.io). We're a team of techies that love MLOps and want to build tools that fellow developers would love to use in their daily work. [This is us](https://zenml.io/company#CompanyTeam) if you want to put faces to the names!

However, in order to improve **ZenML** and understand how it is being used, we need to use analytics to have an overview of how it is used 'in the wild'. This not only helps us find bugs but also helps us prioritize features and commands that might be useful in future releases. If we did not have this information, all we really get is pip download statistics and chatting with people directly, which, while being valuable, is not enough to seriously better the tool as a whole.

#### How does ZenML collect these statistics? <a href="#implementation" id="implementation"></a>

We use [Segment](https://segment.com) as the data aggregation library for all our analytics. However, before any events get sent to [Segment](https://segment.com), they first go through a central ZenML analytics server. This added layer allows us to put various countermeasures to incidents such as getting spammed with events and enables us to have a more optimized tracking process.

The client code is entirely visible and can be seen in the [`analytics`](https://github.com/zenml-io/zenml/tree/main/src/zenml/analytics/README.md) module of our main repository.

#### If I share my email, will you spam me?

No, we won't. Our sole purpose of contacting you will be to ask for feedback (e.g. in the shape of a user interview). These interviews help the core team understand usage better and prioritize feature requests. If you have any concerns about data privacy and the usage of personal information, please [contact us](mailto:support@zenml.io), and we will try to alleviate any concerns as soon as possible.

## Version mismatch (downgrading)

If you've recently downgraded your ZenML version to an earlier release or installed a newer version on a different environment on the same machine, you might encounter an error message when running ZenML that says:

```shell
`The ZenML global configuration version (%s) is higher than the version of ZenML 
currently being used (%s).`
```

We generally recommend using the latest ZenML version. However, there might be cases where you need to match the global configuration version with the version of ZenML installed in the current environment. To do this, run the following command:

```shell
zenml downgrade
```

{% hint style="warning" %}
Note that downgrading the ZenML version may cause unexpected behavior, such as model schema validation failures or even data loss. In such cases, you may need to purge the local database and re-initialize the global configuration to bring it back to its default factory state. To do this, run the following command:

```shell
zenml clean
```

{% endhint %}

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Legacy docs

All legacy docs of ZenML

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td>0.96.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.96.2/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.96.2/</a></td></tr><tr><td>0.96.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.96.1/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.96.1/</a></td></tr><tr><td>0.96.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.96.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.96.0/</a></td></tr><tr><td>0.95.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.95.1/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.95.1/</a></td></tr><tr><td>0.95.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.95.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.95.0/</a></td></tr><tr><td>0.94.6</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.6/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.6/</a></td></tr><tr><td>0.94.5</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.5/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.5/</a></td></tr><tr><td>0.94.4</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.4/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.4/</a></td></tr><tr><td>0.94.3</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.3/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.3/</a></td></tr><tr><td>0.94.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.2/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.2/</a></td></tr><tr><td>0.94.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.1/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.1/</a></td></tr><tr><td>0.94.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.94.0/</a></td></tr><tr><td>0.93.3</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.93.3/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.93.3/</a></td></tr><tr><td>0.93.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.93.2/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.93.2/</a></td></tr><tr><td>0.93.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.93.1/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.93.1/</a></td></tr><tr><td>0.93.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.93.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.93.0/</a></td></tr><tr><td>0.92.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.92.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.92.0/</a></td></tr><tr><td>0.91.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.91.2/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.91.2/</a></td></tr><tr><td>0.91.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.91.1/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.91.1/</a></td></tr><tr><td>0.91.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.91.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.91.0/</a></td></tr><tr><td>0.90.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.90.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.90.0/</a></td></tr><tr><td>0.85.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.85.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.85.0/</a></td></tr><tr><td>0.84.3</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.84.3/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.84.3/</a></td></tr><tr><td>0.84.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.84.2/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.84.2/</a></td></tr><tr><td>0.84.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.84.1/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.84.1/</a></td></tr><tr><td>0.84.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.84.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.84.0/</a></td></tr><tr><td>0.83.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.83.1/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.83.1/</a></td></tr><tr><td>0.83.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.83.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.83.0/</a></td></tr><tr><td>0.82.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.82.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.82.0/</a></td></tr><tr><td>0.81.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.81.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.81.0/</a></td></tr><tr><td>0.80.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.80.2/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.80.2/</a></td></tr><tr><td>0.80.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.80.1/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.80.1/</a></td></tr><tr><td>0.80.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.80.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.80.0/</a></td></tr><tr><td>0.75.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.75.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.75.0/</a></td></tr><tr><td>0.74.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.74.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.74.0/</a></td></tr><tr><td>0.73.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.73.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.73.0/</a></td></tr><tr><td>0.72.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.72.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.72.0</a></td></tr><tr><td>0.71.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/0.71.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/0.71.0</a></td></tr><tr><td>0.70.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/0.70.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/0.70.0</a></td></tr><tr><td>0.68.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/0.68.1">https://zenml-io.gitbook.io/zenml-legacy-documentation/0.68.1</a></td></tr><tr><td>0.68.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.68.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.68.0/</a></td></tr><tr><td>0.67.0</td><td></td><td></td><td><a href="/spaces/IAT17KUXNgudlhTyCPn4">/spaces/IAT17KUXNgudlhTyCPn4</a></td></tr><tr><td>0.66.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/0.66.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/0.66.0</a></td></tr><tr><td>0.65.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.65.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.65.0/</a></td></tr><tr><td>0.64.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.64.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.64.0/</a></td></tr><tr><td>0.63.0</td><td></td><td></td><td><a href="/spaces/N7uiChp9LXuqoNrLj3U3">/spaces/N7uiChp9LXuqoNrLj3U3</a></td></tr><tr><td>0.62.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.62.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.62.0/</a></td></tr><tr><td>0.61.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.61.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.61.0</a></td></tr><tr><td>0.60.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.60.0/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.60.0/</a></td></tr><tr><td>0.58.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.58.2/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.58.2/</a></td></tr><tr><td>0.58.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.58.1/">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.58.1/</a></td></tr><tr><td>0.58.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.58.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.58.0</a></td></tr><tr><td>0.57.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.57.1">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.57.1</a></td></tr><tr><td>0.57.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.57.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.57.0</a></td></tr><tr><td>0.56.4</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.56.4">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.56.4</a></td></tr><tr><td>0.56.3</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.56.3">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.56.3</a></td></tr><tr><td>0.56.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.56.2">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.56.2</a></td></tr><tr><td>0.56.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.56.1">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.56.1</a></td></tr><tr><td>0.55.5</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.5">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.5</a></td></tr><tr><td>0.55.4</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.4">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.4</a></td></tr><tr><td>0.55.3</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.3">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.3</a></td></tr><tr><td>0.55.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.2">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.2</a></td></tr><tr><td>0.55.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.1">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.1</a></td></tr><tr><td>0.55.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.55.0</a></td></tr><tr><td>0.54.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.54.1">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.54.1</a></td></tr><tr><td>0.54.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.54.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.54.0</a></td></tr><tr><td>0.53.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.53.1">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.53.1</a></td></tr><tr><td>0.53.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.53.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.53.0</a></td></tr><tr><td>0.52.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.52.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.52.0</a></td></tr><tr><td>0.51.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.51.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.51.0</a></td></tr><tr><td>0.50.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.50.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.50.0</a></td></tr><tr><td>0.47.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.47.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.47.0-legacy</a></td></tr><tr><td>0.46.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.46.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.46.1-legacy</a></td></tr><tr><td>0.46.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.46.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.46.0-legacy</a></td></tr><tr><td>0.45.6</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.6-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.6-legacy</a></td></tr><tr><td>0.45.5</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.5-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.5-legacy</a></td></tr><tr><td>0.45.4</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.4-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.4-legacy</a></td></tr><tr><td>0.45.3</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.3-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.3-legacy</a></td></tr><tr><td>0.45.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.2-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.45.2-legacy</a></td></tr><tr><td>0.44.4</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.44.4-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.44.4-legacy</a></td></tr><tr><td>0.44.3</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.44.3-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.44.3-legacy</a></td></tr><tr><td>0.44.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.44.2-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.44.2-legacy</a></td></tr><tr><td>0.44.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.44.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.44.1-legacy</a></td></tr><tr><td>0.43.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.43.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.43.1-legacy</a></td></tr><tr><td>0.43.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.43.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.43.0-legacy</a></td></tr><tr><td>0.42.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.42.2-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.42.2-legacy</a></td></tr><tr><td>0.42.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.42.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.42.1-legacy</a></td></tr><tr><td>0.42.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.42.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.42.0-legacy</a></td></tr><tr><td>0.41.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.41.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.41.0-legacy</a></td></tr><tr><td>0.40.3</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.40.3-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.40.3-legacy</a></td></tr><tr><td>0.40.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.40.2-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.40.2-legacy</a></td></tr><tr><td>0.40.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.40.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.40.1-legacy</a></td></tr><tr><td>0.40.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.40.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.40.0-legacy</a></td></tr><tr><td>0.39.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.39.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.39.1-legacy</a></td></tr><tr><td>0.39.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.39.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.39.0-legacy</a></td></tr><tr><td>0.38.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.38.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.38.0-legacy</a></td></tr><tr><td>0.37.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.37.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.37.0-legacy</a></td></tr><tr><td>0.36.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.36.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.36.1-legacy</a></td></tr><tr><td>0.36.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.36.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.36.0-legacy</a></td></tr><tr><td>0.35.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.35.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.35.1-legacy</a></td></tr><tr><td>0.35.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.35.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.35.0-legacy</a></td></tr><tr><td>0.34.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.34.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.34.0-legacy</a></td></tr><tr><td>0.33.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.33.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.33.0-legacy</a></td></tr><tr><td>0.32.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.32.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.32.1-legacy</a></td></tr><tr><td>0.32.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.32.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.32.0-legacy</a></td></tr><tr><td>0.31.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.31.1-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.31.1-legacy</a></td></tr><tr><td>0.31.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.31.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.31.0-legacy</a></td></tr><tr><td>0.30.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.30.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.30.0-legacy</a></td></tr><tr><td>0.20.5</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.5-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.5-legacy</a></td></tr><tr><td>0.20.4</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.4-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.4-legacy</a></td></tr><tr><td>0.20.3</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.3-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.3-legacy</a></td></tr><tr><td>0.20.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.2-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.2-legacy</a></td></tr><tr><td>0.20.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.0-legacy">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.20.0-legacy</a></td></tr><tr><td>0.13.2</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.13.2">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.13.2</a></td></tr><tr><td>0.13.1</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.13.1">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.13.1</a></td></tr><tr><td>0.13.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.13.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.13.0</a></td></tr><tr><td>0.12.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.12.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.12.0</a></td></tr><tr><td>0.11.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.11.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.11.0</a></td></tr><tr><td>0.10.0</td><td></td><td></td><td><a href="https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.10.0">https://zenml-io.gitbook.io/zenml-legacy-documentation/v/0.10.0</a></td></tr></tbody></table>

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>


# Welcome to Kitaru

Replay-based evals for AI agents, in Python and TypeScript. Test your next change against what your agent already did in production.

Your agent has already been tested thousands of times in production. Most of that evidence is sitting in a trace store as something you can read but not run. Kitaru makes it runnable: it records or imports each run as a **session**, then **replays** it against your real code, with the recording answering for the world the original run saw. Change the prompt, swap the model, or point replay at the fix in your working tree, and see what improved and what broke before it ships.

{% hint style="success" %}
**Who it's for:** Teams with an agent in front of real users, where regression testing today means re-running a few samples and eyeballing the output. Kitaru replaces that with [evaluators](/kitaru/core-concepts/evaluators), [cohorts](/kitaru/core-concepts/cohorts), and [experiments](/kitaru/core-concepts/experiments) over your actual traffic. If you're prototyping and haven't shipped, it will feel like more machinery than you need.
{% endhint %}

**Frameworks:** adapters ship for [PydanticAI](/kitaru/adapters/pydantic-ai), [LangGraph](/kitaru/adapters/langgraph), and the [OpenAI Agents SDK](/kitaru/adapters/openai-agents) in Python, and for [Mastra](/kitaru/adapters/mastra) and the [Vercel AI SDK](/kitaru/adapters/vercel-ai) in TypeScript. Other frameworks still work: [import your traces](/kitaru/import-your-traces/import-your-traces) with the built-in Langfuse, LangSmith, Braintrust, Logfire, Arize Phoenix, or JSONL importers; [write a custom importer](/kitaru/import-your-traces/custom-importer), usually about a page of Python; or [build a small adapter](/kitaru/adapters/custom), where the recording API is two client calls.

{% hint style="info" %}
Kitaru has both a Python and a TypeScript SDK, and both talk to the same server. The CLI ships with the Python package.
{% endhint %}

**Kitaru is built to be driven by agents.** The MCP server gives Claude Code, Codex, Cursor, and other coding assistants bounded Kitaru operations. The agent skills teach the procedures, and the CLI speaks JSON when a shell command is the right tool. You bring the judgment; your assistant handles the investigation work. [Set up your coding agent](/kitaru/getting-started/setup) takes a few minutes.

Kitaru is open source (Apache 2.0) and self-hosted, from the team behind [ZenML](https://zenml.io): ZenML is for ML pipelines, Kitaru is for agents.

## The loop

* **Record.** Wrap your agent or import your traces (both shown below). Either way, runs land as [sessions](/kitaru/core-concepts/agents-and-sessions).
* **Replay.** [Re-execute a session](/kitaru/core-concepts/replay) against your real code. Tool calls are answered from the recording, so nothing touches real systems. An unchanged replay gives you the faithful baseline. Then fork it with a different model, a new prompt, or your working tree's code.
* **Improve.** This is where your judgment enters. In an [investigation](/kitaru/core-concepts/investigations), your coding assistant authors the review, walks you through the evidence, asks the questions Kitaru needs answered, and pins your answers to the exact trace as annotations. Those judgments calibrate the [evaluators](/kitaru/core-concepts/evaluators) that evaluate both sides; [cohorts](/kitaru/core-concepts/cohorts) freeze the population; [experiments](/kitaru/core-concepts/experiments) replay a cohort against a change and show what improved and what regressed. The cohort that caught a failure becomes the regression gate that keeps it caught.

In daily work, that loop becomes five steps: **observe** a recorded behavior, **judge** what should have happened, **define** the behavior to test, **replay** the changed agent, and **compare** the evidence. Recording gives you the raw material; observe, judge, and define turn human judgment into durable criteria; replay and compare close the loop. The [Quickstart](/kitaru/getting-started/quickstart) walks all five. To try it in a controlled environment, ask your assistant for the `kitaru-guided-tour` skill, which runs the loop on the public [`kitaru-template`](https://github.com/zenml-io/kitaru-template), or follow the [complete returns-agent tutorial](/kitaru/guides/returns-agent) manually.

## Do I have to run it in production?

No. There are two ways to get sessions, and they end in the same place:

* **Import the history you already have.** If your agent logs to Langfuse or anything else you can export from, import it. Nothing in your production path changes: your trace store stays your system of record, and Kitaru gets a runnable copy.
* **Record with an adapter.** Wrap the agent once, no rewrite, and every run becomes a session wherever the agent runs: production, staging, or your laptop.

{% tabs %}
{% tab title="Import traces" %}

```bash
kitaru session import langfuse-export.jsonl \
  --importer kitaru/langfuse@latest \
  --agent support-agent@latest --wait
```

{% endtab %}

{% tab title="Record with an adapter" %}

```python
from pydantic_ai import Agent
from kitaru_pydantic_ai import KitaruAgent

agent = Agent(
    "openai:gpt-5.4", name="support-agent", system_prompt="You resolve support tickets."
)


@agent.tool_plain
def refund_payment(order_id: str) -> str:
    return payments.refund(order_id)  # your real API


support = KitaruAgent(agent, agent_id=AGENT_ID)
support.run_sync("Refund order #4821, the card reader double-charged me.")
```

{% endtab %}
{% endtabs %}

Replays, imports, and evaluations run offline on [workers](/kitaru/core-concepts/workers) in your environment. None of that touches your production traffic. An adapter does run inside your agent's process to record; if you do not want Kitaru near production, the import path never gets close to it.

## Built to sit in your stack

* **Self-hosted.** One FastAPI + Postgres server on your infrastructure. Your traces and credentials don't leave your systems.
* **Beside your observability, not instead of it.** Langfuse, LangSmith, Braintrust, Logfire, and Arize Phoenix remain where you watch production. Kitaru is where you re-run it.
* **Choose how you drive it:** the `kitaru` CLI, Python SDK, TypeScript SDK, and your [coding agent](/kitaru/getting-started/setup). Kitaru observes your production agents; your coding assistant is how you talk to Kitaru.

**Questions, bugs, feedback?** Join the [Slack community](https://kitaru.ai/slack), report bugs at [kitaru.ai/help](https://kitaru.ai/help) (it goes straight to GitHub issues), or email <support@kitaru.ai>. All three reach a human.

## Next steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Installation</strong></td><td>SDK, CLI, a local server, and a login.</td><td><a href="/pages/cDb4N92M787W6Uf33vjO">/pages/cDb4N92M787W6Uf33vjO</a></td></tr><tr><td><strong>Quickstart</strong></td><td>Understand the five-step method before running commands.</td><td><a href="/pages/knQ03wCkSkWfSSXcMRor">/pages/knQ03wCkSkWfSSXcMRor</a></td></tr><tr><td><strong>Kitaru template</strong></td><td>Prepare a ready PydanticAI agent and checked-in Langfuse traces.</td><td><a href="https://github.com/zenml-io/kitaru-template">https://github.com/zenml-io/kitaru-template</a></td></tr><tr><td><strong>Complete tutorial</strong></td><td>Investigate and replay the template's synthetic returns agent.</td><td><a href="/pages/R1lYNNZKYCdTyMZbtsck">/pages/R1lYNNZKYCdTyMZbtsck</a></td></tr><tr><td><strong>Import your traces</strong></td><td>Start from the history you already have.</td><td><a href="/pages/o74tFfkKFSg3Ubasut52">/pages/o74tFfkKFSg3Ubasut52</a></td></tr><tr><td><strong>Core Concepts</strong></td><td>Sessions, replay, evaluators, cohorts, experiments.</td><td><a href="/pages/qw8hIFEbl4taSEvy4SNP">/pages/qw8hIFEbl4taSEvy4SNP</a></td></tr><tr><td><strong>Build a regression suite</strong></td><td>Production traffic as your test suite.</td><td><a href="/pages/ktnQrV1qibhxXwZEZKDn">/pages/ktnQrV1qibhxXwZEZKDn</a></td></tr><tr><td><strong>Deploy Kitaru</strong></td><td>Self-host for your team.</td><td><a href="/pages/RQBXLDtvODELNqqDX86n">/pages/RQBXLDtvODELNqqDX86n</a></td></tr></tbody></table>


# Installation

Install the Kitaru SDKs and CLI, start a local server, and log in, for Python and TypeScript.

Kitaru is three installable pieces:

* the **SDK + CLI** in your project
* a **server** your team shares (self-hosted, one per team)
* **workers** that execute replays and evaluations in your environment

For a first session on one machine, all three run locally.

The Kitaru CLI, server, and workers require **Python 3.11 or newer**. TypeScript agents use Node **22.22 or newer in the Node 22 release line** and connect to the same server.

The server stores everything in **PostgreSQL**, its only stateful dependency. You do not install it by hand for local use: `kitaru login --local` provisions the server and Postgres together in Docker. A [self-hosted deployment](/kitaru/getting-started/deploy) brings its own Postgres. Workers are plain processes (`kitaru worker start`) that run wherever your agent's environment lives; for containerized fleets, the published `zenmldocker/kitaru-worker` image works out of the box (see [Workers in production](/kitaru/running-in-production/workers)).

## Install the Python SDK and CLI

{% tabs %}
{% tab title="uv (recommended)" %}

```bash
uv add "kitaru[cli,worker]" kitaru-pydantic-ai
```

{% endtab %}

{% tab title="pip" %}

```bash
pip install "kitaru[cli,worker]" kitaru-pydantic-ai
```

{% endtab %}
{% endtabs %}

| Extra    | What it adds                                                                               |
| -------- | ------------------------------------------------------------------------------------------ |
| `cli`    | The `kitaru` command, the full loop: import, evaluate, cohorts, experiments, workers, jobs |
| `worker` | Run a worker in this environment (`kitaru worker start`)                                   |
| `server` | Run the Kitaru server itself from this package                                             |
| `mcp`    | The `kitaru-mcp` server for [coding assistants](/kitaru/getting-started/setup)             |
| `otel`   | OpenTelemetry export from the server                                                       |

The plain `kitaru` package is the SDK alone (the async client and the API models), which is all a production service needs to record sessions.

Adapters are **not** extras. Each ships as its own distribution, so you install the one your framework needs alongside Kitaru:

| Framework                                                                    | Install                |
| ---------------------------------------------------------------------------- | ---------------------- |
| [PydanticAI](/kitaru/adapters/pydantic-ai)                                   | `kitaru-pydantic-ai`   |
| [LangGraph](/kitaru/adapters/langgraph) (also LangChain agents, Deep Agents) | `kitaru-langgraph`     |
| [OpenAI Agents SDK](/kitaru/adapters/openai-agents)                          | `kitaru-openai-agents` |

## Install the TypeScript SDK and adapters

`@zenml-io/kitaru` is the framework-neutral TypeScript SDK: it creates and inspects Kitaru resources, records sessions, submits evaluations and experiments, and waits for exact jobs. The Python `kitaru` command remains the CLI for login and worker operations; there is no separate TypeScript CLI.

{% hint style="info" %}
The TypeScript packages require Node `>=22.22.0 <23` and are versioned and released together.
{% endhint %}

Install the adapter in the Node project that runs your agent:

{% tabs %}
{% tab title="Mastra" %}

```bash
pnpm add @zenml-io/kitaru-mastra @mastra/core@1.51.0
```

See the [Mastra adapter](/kitaru/adapters/mastra) for the wrapper, replay behavior, and supported boundary.
{% endtab %}

{% tab title="Vercel AI SDK" %}

```bash
pnpm add @zenml-io/kitaru-vercel-ai ai@7.0.65
```

See the [Vercel AI SDK adapter](/kitaru/adapters/vercel-ai) for Agent and `generateText` recording, replay behavior, and the supported boundary.
{% endtab %}

{% tab title="Build an adapter" %}

```bash
pnpm add @zenml-io/kitaru
```

The core package provides the TypeScript client and adapter primitives. It does not provide a framework-neutral agent or streaming abstraction.
{% endtab %}
{% endtabs %}

The Node agent still needs a reachable Kitaru server. Install the Python CLI and worker separately when you want to run the full loop locally, or connect the agent to your team's deployed server and workers.

No adapter for your framework? You are not blocked: [import your traces instead, or build a project-local adapter with the adapter-builder skill](/kitaru/adapters/custom).

## Install the agent skills

Do this now rather than later. Kitaru is a loop with real judgment calls in it: which sessions to review, when a behavior is worth freezing into a cohort, whether a replay result supports shipping. The [agent skills](/kitaru/getting-started/setup) teach your coding assistant how to make them with you:

{% tabs %}
{% tab title="Any skill-aware host" %}

```bash
npx skills add zenml-io/kitaru-skills
```

{% endtab %}

{% tab title="Claude Code plugin" %}

```
/plugin marketplace add zenml-io/kitaru-skills
/plugin install kitaru@kitaru
```

{% endtab %}
{% endtabs %}

`kitaru-investigation` is the front door: point your assistant at it and it will walk you from the traces you have to a reviewed cohort, choosing the review batch and stopping at checkpoints you can resume from. The others cover [replay experiments](/kitaru/adapters/adapters), [building an adapter](/kitaru/adapters/custom), and building an importer.

Pair them with the [MCP server](/kitaru/getting-started/setup) (`kitaru[mcp]`) so the assistant has bounded operations to go with the method. `kitaru` with no arguments tells you whether the skills are installed.

## Start a local server

The server is FastAPI + Postgres, and the CLI can run both for you. All it needs is [Docker](https://docs.docker.com/get-started/get-docker/) with the [Compose v2 plugin](https://docs.docker.com/compose/install/):

```bash
kitaru login --local
```

This provisions a server and PostgreSQL pinned to your installed Kitaru version, waits for `http://localhost:8000` to become healthy, selects it as your active server, and opens it in your browser. The lifecycle is three commands:

```bash
kitaru local logs            # inspect (add --service server --follow)
kitaru logout                # stop the containers; the database persists
kitaru logout --volumes      # stop and delete the database (a clean reset)
```

After upgrading the `kitaru` package, upgrade the local server to match with `kitaru login --local --upgrade`; a plain login deliberately never replaces the server image. Prefer to manage Docker yourself, or need a shared deployment with your own Postgres, real auth, and TLS? See [Docker](/kitaru/getting-started/deploy/docker) and [Deploy Kitaru](/kitaru/getting-started/deploy).

## Connect

`kitaru login --local` already connected you; `kitaru status` confirms it.

Against a shared server, log in with `kitaru login <url>`. For non-interactive use (CI, production services), create an API key and set two environment variables that the SDK, the CLI, and workers all read:

```bash
export KITARU_API_URL="https://kitaru.your-team.example"
export KITARU_API_KEY="KITKEY_..."
```

See [Authentication & API keys](/kitaru/running-in-production/authentication) for how keys are issued and managed.

Node applications can also reuse a developer's selected CLI login without exporting its token; see the [TypeScript SDK](/kitaru/get-help/sdks). Use dedicated API keys or worker task tokens for CI and production rather than copying a developer credential store.

## Verify

```bash
kitaru version
kitaru doctor
```

`kitaru doctor` checks the connection and reports what it finds.

## Next steps

Read the [Quickstart](/kitaru/getting-started/quickstart) to understand Kitaru's five-step method. For a controlled hands-on path, prepare the public [`kitaru-template`](https://github.com/zenml-io/kitaru-template) and continue with the [complete returns-agent tutorial](/kitaru/guides/returns-agent). If you already collect traces elsewhere, start with [Import your traces](/kitaru/import-your-traces/import-your-traces).


# Deploy Kitaru

Self-host Kitaru for your team: one server, your Postgres, workers in the environments where your agents live.

A Kitaru deployment is deliberately small:

* **The server** is one FastAPI service on Postgres. It stores agents, sessions, cohorts, evaluators, experiments, and replays, and serves the REST API the SDK, CLI, and workers speak. It executes no user code.
* **Workers** are processes you run wherever your agents' code and credentials live. All execution (replays, imports, evaluations) happens there. See [Workers in production](/kitaru/running-in-production/workers).
* **Postgres** is the only stateful dependency. Your database, your backups.

This shape is the data-privacy story: traces are stored on your server, parsed and replayed on your workers. Nothing needs to leave your systems.

## Setting up

1. [Docker](/kitaru/getting-started/deploy/docker): Compose for a single host, or the published server image against your managed Postgres. Start here. On Kubernetes, use the [Helm chart](/kitaru/getting-started/deploy/helm).
2. Create [accounts and API keys](/kitaru/running-in-production/authentication) for your team and your CI (Python client today; CLI verbs are on the way).
3. Start [workers](/kitaru/running-in-production/workers) in each environment agents run in.
4. Store provider credentials the server should manage as [secrets](/kitaru/running-in-production/secrets), and set client defaults via [configuration](/kitaru/get-help/configuration).

Steps 2 to 4 are covered in **Running in production**, alongside worker sizing, authentication, secrets and configuration. Come back to them once a server is up. For a first look, `kitaru login --local` in [Installation](/kitaru/getting-started/installation) is faster than any of this.


# Docker

Run the Kitaru server with Docker, using Compose for one host or the server container against your own Postgres.

The server is one container plus Postgres. Compose runs both on a single host; for anything bigger, run the server container against a managed Postgres and keep the same environment variables.

## CLI-managed local deployment

For one local deployment per user, let the CLI own the lifecycle:

```bash
kitaru login --local
```

Requires [Docker](https://docs.docker.com/get-started/get-docker/) with the [Compose v2 plugin](https://docs.docker.com/compose/install/). The CLI runs the version-matched `zenmldocker/kitaru-server` image with PostgreSQL kept private to the Compose network, stores generated runtime secrets in the Kitaru configuration directory, and opens `http://localhost:8000` once healthy. Existing images are reused without an automatic pull; `kitaru login --local --upgrade` is the explicit upgrade path, and `KITARU_LOCAL_IMAGE` points source builds at a locally built image. `kitaru local logs` inspects it; `kitaru logout` stops it (add `--volumes` to delete the database).

The rest of this page covers manually managed deployments, which are separate from the CLI-owned one.

## Docker Compose

The repository ships a Compose file that builds the server and starts Postgres beside it:

```bash
git clone https://github.com/zenml-io/kitaru.git
cd kitaru
docker compose up -d
curl http://localhost:8000/health
```

The shipped Compose file runs with `KITARU_SERVER_AUTH_SCHEME: none`, which is fine on your laptop but not for a shared server. For a team deployment, set the auth scheme to `local` and provide real keys (see below and [Authentication](/kitaru/running-in-production/authentication)).

## Configuration

The server is configured entirely through `KITARU_SERVER_*` environment variables. The ones every deployment should set:

| Variable                                                               | Meaning                                                                        |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| `KITARU_SERVER_DB_HOST` / `DB_PORT` / `DB_USER` / `DB_PWD` / `DB_NAME` | Postgres connection, or one `KITARU_SERVER_DATABASE_URL` instead               |
| `KITARU_SERVER_AUTH_SCHEME`                                            | `none` (open, dev only) or `local` (accounts + API keys)                       |
| `KITARU_SERVER_JWT_SIGNING_KEY`                                        | Secret for login tokens; set a long random value                               |
| `KITARU_SERVER_SECRET_ENCRYPTION_KEY`                                  | Key encrypting stored [secrets](/kitaru/running-in-production/secrets) at rest |
| `KITARU_SERVER_DEFAULT_ACCOUNT_PASSWORD`                               | Bootstrap password for the `default` account                                   |
| `KITARU_SERVER_SERVER_URL`                                             | The externally reachable URL clients use                                       |

Operational knobs with sensible defaults; raise or lower them deliberately:

| Variable                                          | Default | Meaning                                                           |
| ------------------------------------------------- | ------- | ----------------------------------------------------------------- |
| `KITARU_SERVER_MAX_BLOB_SIZE_BYTES`               | 100 MiB | Upload cap for trace exports and plugin code                      |
| `KITARU_SERVER_TASK_HEARTBEAT_TIMEOUT_SECONDS`    | 60      | How long a silent worker holds a task before it's requeued        |
| `KITARU_SERVER_TASK_RETRY_LIMIT`                  | 3       | Attempts before a stale task is abandoned                         |
| `KITARU_SERVER_EVALUATOR_TASK_TIMEOUT_SECONDS`    | 300     | Per-evaluator process timeout                                     |
| `KITARU_SERVER_IMPORTER_TASK_TIMEOUT_SECONDS`     | 600     | Per-import process timeout                                        |
| `KITARU_SERVER_EVALUATION_PAIR_LIMIT`             | 100     | Max (session × evaluator) pairs per batch request                 |
| `KITARU_SERVER_IDEMPOTENCY_KEY_RETENTION_SECONDS` | 900     | How long a stored response stays replayable for a retried request |
| `KITARU_SERVER_LOG_LEVEL`                         | `INFO`  | Server logging                                                    |

Database migrations run automatically at startup (`KITARU_SERVER_SKIP_DB_MIGRATION=true` disables that when you manage migrations yourself).

## The published image

For anything beyond a laptop, use the published server image instead of building from source:

```bash
docker run -d -p 8000:8000 \
  -e KITARU_SERVER_DB_HOST=your-postgres-host \
  -e KITARU_SERVER_DB_USER=... -e KITARU_SERVER_DB_PWD=... \
  -e KITARU_SERVER_AUTH_SCHEME=local \
  -e KITARU_SERVER_JWT_SIGNING_KEY=... \
  -e KITARU_SERVER_SECRET_ENCRYPTION_KEY=... \
  zenmldocker/kitaru-server:latest
```

Any container runtime works: the server listens on port 8000, runs as a non-root user, and all state lives in Postgres. Put TLS in front with your usual ingress or reverse proxy, and scale horizontally if needed, since the server is stateless between requests. On Kubernetes, use the [Helm chart](/kitaru/getting-started/deploy/helm), which wraps this same image with migrations, ingress, and secrets handled.

Workers are deployed separately, in the environments your agents live in. See [Workers in production](/kitaru/running-in-production/workers).

## First login

```bash
kitaru login https://kitaru.internal.example.com
kitaru status
```

Then create accounts and API keys for the team: [Authentication & API keys](/kitaru/running-in-production/authentication).


# Helm

Deploy the Kitaru server on Kubernetes with the first-party Helm chart, covering external Postgres, automatic migrations, ingress, and TLS.

The repository ships a first-party chart under `helm/` that deploys the Kitaru server on Kubernetes: a server Deployment (with optional autoscaling), a Service, ingress or Gateway API routing, and a database migration Job that runs before each install and upgrade so the server never starts against an unmigrated schema.

The chart deploys **the server only**. Postgres is yours to provide (managed Postgres is the expected shape), and [workers](/kitaru/running-in-production/workers) deploy separately in the environments your agents run in.

```bash
helm install kitaru oci://public.ecr.aws/zenml/kitaru \
  --namespace kitaru --create-namespace \
  --values my-values.yaml
```

## The values that matter

A minimal production `my-values.yaml`:

```yaml
server:
  serverURL: https://kitaru.internal.example.com

  database:
    host: your-postgres-host
    username: kitaru
    passwordSecretRef:
      name: kitaru-db
      key: password
    sslMode: require

  auth:
    authScheme: local
    defaultAccount:
      passwordSecretRef:
        name: kitaru-bootstrap
        key: password

ingress:
  enabled: true
  host: kitaru.internal.example.com
```

The chart mirrors the same `KITARU_SERVER_*` configuration surface as the [Docker deployment](/kitaru/getting-started/deploy/docker): every server setting has a values path, secrets can be inline for a quick start or `secretRef`s for real deployments, and database TLS supports `disable` through `verify-full` with custom CA bundles.

The image is the published `zenmldocker/kitaru-server`, tagged to match the chart version by default; pin `server.image.tag` explicitly if you want upgrades to be deliberate.

## Operational notes

* **Migrations** run as a Helm hook Job before the server pods roll, so an upgrade that needs a schema change can't race its own pods. If the migration fails, the release fails and the previous version keeps running.
* **Scaling**: the server is stateless between requests; enable the HPA block or set replicas directly. All state is in Postgres.
* **Routing**: classic Ingress (nginx by default) and Gateway API HTTPRoute are both supported; enable exactly one.

After install, point your team at it:

```bash
kitaru login https://kitaru.internal.example.com
kitaru status
```

Then create [accounts and API keys](/kitaru/running-in-production/authentication) and start [workers](/kitaru/running-in-production/workers) where your agents live.


# Set up your coding agent

Install the MCP server and agent skills, then drive the whole Kitaru loop from Claude Code, Codex, or Cursor.

Kitaru observes your production agents; your coding assistant is how you talk to Kitaru. The whole loop is scriptable, and two installable pieces let the assistant drive it without improvising:

* The **MCP server** gives it typed, bounded Kitaru operations, with capability modes for actions that create, change, or delete state.
* The **agent skills** give it the workflow: which sessions are worth reviewing, when a behavior is clear enough to freeze into a cohort, and what a replay result does and does not prove.

Skills and MCP work together: the skills say how to work, and the server bounds what can be touched.

## Install the MCP server

Assistants that speak MCP, such as Claude Code and Cursor, get typed tools instead of relying on shell commands for every operation:

```bash
uv add "kitaru[mcp]"
```

Then register it with your assistant (`.mcp.json` for Claude Code):

```json
{
  "mcpServers": {
    "kitaru": {
      "command": "uv",
      "args": ["run", "kitaru-mcp", "--server", "http://localhost:8000", "--mode", "standard"]
    }
  }
}
```

Installing with uv puts the `kitaru-mcp` executable inside your project's virtual environment. Your assistant starts the server as a plain subprocess and does not activate that environment first, so a bare `kitaru-mcp` is often missing from `PATH`. Going through `uv run` gives the assistant the right environment.

{% hint style="warning" %}
Two settings trip people up:

* The `--server` URL must match the server you are logged into. `http://localhost:8000` is only right after `kitaru login --local`; on a managed or self-hosted workspace, use your workspace URL. `kitaru status` shows the URL it resolved and whether your credential works there. The MCP server does not follow the CLI's current selection, and a mismatch usually looks like an empty workspace.
* The default mode is `read-only`, which leaves an assistant mid-investigation with nothing it can write. `--mode standard` lets it build cohorts and start runs; read-only is still a sensible place to start, as long as you expect that.
  {% endhint %}

The server needs an explicit target: `--server URL`, `KITARU_MCP_SERVER`, or `KITARU_API_URL`, in that order. Startup fails if none selects a server. Credentials come from `KITARU_API_KEY` or the stored credential for that URL (a task-scoped `KITARU_API_TOKEN` is deliberately ignored). Restart `kitaru-mcp` after changing the target or an environment-provided API key.

## Capability modes and tools

Tools are gated by a **capability mode**, either `read-only` (the default), `standard`, or `destructive`, set with `--mode` or `KITARU_MCP_MODE`. Tools above the current mode are never registered, so the assistant does not see them:

| Tool                        | Mode        | What it does                                                                                                                              |
| --------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `kitaru_registry_read`      | read-only   | Read agents, cohorts, experiments, importers, evaluators, and their versions; list and filter tags; list workers or get one by exact UUID |
| `kitaru_activity_read`      | read-only   | Read sessions, replays, evaluations, runs, jobs, and their children                                                                       |
| `kitaru_review_read`        | read-only   | Read [investigations and annotations](/kitaru/core-concepts/investigations)                                                               |
| `kitaru_cohorts_manage`     | standard    | Create or update cohorts and cohort versions                                                                                              |
| `kitaru_experiments_manage` | standard    | Create or update experiments                                                                                                              |
| `kitaru_session_import`     | standard    | Import sessions from an already-uploaded blob                                                                                             |
| `kitaru_review_manage`      | standard    | Manage investigations and annotations; create or rename tags and link them to resources                                                   |
| `kitaru_workflow_start`     | standard    | Start a session evaluation or experiment run, return immediately                                                                          |
| `kitaru_evaluators_manage`  | standard    | Create or update evaluators from an existing blob or pinned package                                                                       |
| `kitaru_workflow_cancel`    | destructive | Cancel a job or experiment run                                                                                                            |
| `kitaru_delete`             | destructive | Delete a cohort, experiment, investigation, annotation, evaluator, version, run, or tag; unlink an exact tag-resource tuple               |

Start assistants in `read-only`, move to `standard` when you want them building cohorts and starting runs, and reserve `destructive` for sessions where you are watching closely.

Tag operations follow the same split. In `read-only`, `kitaru_registry_read` can list tags and filter them by name. Existing filtered registry or activity reads can then find sessions, agent versions, cohort versions, cohorts, experiments, and experiment runs carrying that tag. The MCP server cannot enumerate a tag's links directly. In `standard`, `kitaru_review_manage` supports `create_tag`, `update_tag`, and `link_tag`. In `destructive`, `kitaru_delete` can unlink one exact `(tag, resource type, resource id)` tuple or delete the tag. Deleting a tag also deletes every link that points from it.

Worker inspection is read-only by design. Use `kitaru_registry_read` with `kind: "worker"` to list workers, or `operation: "get_worker"` with an exact worker UUID. The returned `live` and `last_seen_at` fields report recent heartbeat observations; they do not guarantee that a worker will claim a particular task. Worker registration, task assignment, credentials, and lifecycle control remain outside MCP.

`kitaru_review_manage` accepts `pending`, `in_progress`, or `completed` when updating an investigation. This does not bypass server transition rules: for example, the server can still reject moving a completed investigation back to pending. A linked session's verdict remains a separate field and does not accept `pending`.

## Install the agent skills

Skills ship separately from Kitaru as Markdown procedures in [`zenml-io/kitaru-skills`](https://github.com/zenml-io/kitaru-skills). A skill does not start another service or process; your assistant reads the document and follows its procedure with the tools already available in the host.

{% hint style="info" %}
**Want to see a Kitaru skill in action before installing it?** Watch the 26-minute [guided tour](https://youtu.be/aYLfzXEr2Rk). It follows the `kitaru-guided-tour` skill from a prepared session review through a deterministic evaluator, frozen cohort, replay experiment, and comparison.
{% endhint %}

{% tabs %}
{% tab title="Any skill-aware host" %}

```bash
npx skills add zenml-io/kitaru-skills
```

{% endtab %}

{% tab title="Claude Code plugin" %}

```
/plugin marketplace add zenml-io/kitaru-skills
/plugin install kitaru@kitaru
```

{% endtab %}
{% endtabs %}

If your host supports neither, copy the skill directory you want into wherever it reads skills from.

**Verify:** run `kitaru` with no arguments. It searches project and user locations, plus the Claude marketplace, for installed Kitaru skills and prints the installation command if it finds none. Machine-readable output reports the same under a `skills` key, so an assistant can check its own setup before it starts.

## The investigation skill

`kitaru-investigation` is the front door for your own agent, and it reflects the product design: **you do not author investigations, your assistant does**. It maps your sessions, generates a baseline [investigation](/kitaru/core-concepts/investigations), and interviews you against the trace. Your job is answering. Use it when you have one surprising session, or a larger population you want to sample before defining a failure category.

It picks one of two entry paths from what you already have:

| You have                           | The skill does                                                                                             |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| A specific session that went wrong | Reads it fully, then builds a small worklist of related sessions and at least one counterexample           |
| A population but no clear failure  | Builds a diverse sample, normally 15–30 sessions, with a random subset alongside coverage-based selections |

It begins by surveying the selected sessions, then examines relevant ones in detail. If the review identifies a useful set of cases, it can help you create a [cohort](/kitaru/core-concepts/cohorts) version for later replays. It can also select an installed evaluator that matches your criterion, and writes a new one only if none fit.

**You assign the human labels.** The assistant selects, summarizes, and organizes evidence, but an [annotation](/kitaru/core-concepts/investigations) should record your judgment rather than the assistant's suggestion. Observed behavior stays separate from expected behavior: the procedure distinguishes the agent's actions, dependency behavior, and product requirements instead of treating every unexpected outcome as an agent failure.

Before creating remote state or using worker or model compute, the skill explains the operation and asks for confirmation where required. You must confirm cohort membership explicitly. If a required payload, permission, or worker is missing, the skill records a checkpoint so the investigation can resume later. Open observations come before proposed failure categories, which helps keep the first review batch from inheriting a bad taxonomy.

## The other skills

| Skill                                                                      | Use it when                                                                                                                                                                 |
| -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kitaru-guided-tour`                                                       | First contact with no agent of your own: a value-first tour on the public template, from a prepared three-session review to an evaluator and one approved replay experiment |
| `kitaru-investigation`                                                     | Reviewing sessions, recording evidence, and creating a cohort from confirmed cases                                                                                          |
| [`kitaru-replay-experiment`](/kitaru/guides/replay-and-overrides)          | Testing one candidate change against an accepted cohort with pinned evaluators, and reading whether the evidence improved, regressed, traded off, or stayed inconclusive    |
| [`kitaru-adapter-builder`](/kitaru/adapters/adapters)                      | Building a Python or TypeScript [adapter](/kitaru/adapters/adapters) for a framework that Kitaru does not support yet, with explicit recording and replay capabilities      |
| [`kitaru-importer-builder`](/kitaru/import-your-traces/importing-sessions) | Building and locally validating an importer for an unsupported provider export; registration requires separate approval                                                     |

The replay skill stops short of the deployment decision: it reports what the evidence supports and leaves the call to you. The two builder skills default to finishing on your machine, and register or upload only when you ask for each step.

## Skills, MCP, and the CLI

Skills define the procedure and identify decisions that require human judgment. The MCP server provides bounded Kitaru operations and gates destructive ones. Skills fall back to the structured CLI for operations MCP does not cover, such as uploading a local file or waiting for a job. You can also follow every procedure manually with the CLI.

None of the three executes your agent on the Kitaru server. Replays run on a [worker](/kitaru/core-concepts/workers) you control, in the environment you configured for it. Guardrails worth setting:

* Give the assistant a **read-mostly posture**: creating evaluators and starting evaluations is cheap and reversible, and deleting cohorts or experiments is not. Over MCP that's the capability mode; review deletes yourself.
* Keep a worker running under *your* control. The assistant creating a replay doesn't execute anything; your worker does. That separation is the safety property; preserve it.
* Watch tool policies in assistant-written replays: insist on `history` + `on_miss="fail"` defaults for anything with side effects, same as you would in review. See [Tool policies](/kitaru/guides/tool-policies).

## The other surfaces

Give the assistant the connection:

```bash
export KITARU_API_URL="http://localhost:8000"
export KITARU_API_KEY="KITKEY_..."
```

* **CLI:** the full journey has commands: `kitaru session import`, `kitaru replay create`, `kitaru session evaluate`, `kitaru cohort create`, `kitaru experiment run start`, plus registration, workers, and jobs. Commands take `--output json`, so assistant-driven invocations parse cleanly.
* **Python client:** `KitaruAPIClient()` reaches everything, including single-session replays. Your assistant writes the same snippets these docs show.
* **REST:** the server's OpenAPI schema at `/docs` on your server, when the assistant wants the raw contract.

## Prompts that work

The loop compresses well into assistant tasks. Some starting points, ready to paste:

```
Use kitaru-investigation to investigate this agent and help me test one meaningful improvement. Assume I am new to Kitaru. Show me the recorded evidence before asking for a judgment, and ask before creating resources, changing code, or starting paid replay.
```

New to Kitaru with no agent or traces of your own yet? Start with the tour instead:

```
Use kitaru-guided-tour to walk me through Kitaru on the returns-agent template. I am new; explain each step as we go, and ask before anything paid or live.
```

```
The last run of support-agent failed. Fetch the most recent failed session and its nodes with the Kitaru client, and tell me which tool call went wrong.
```

```
Replay session <id> unchanged with the refund-check evaluator and a baseline history tool policy. When it completes, compare evaluations and cost against the baseline and summarize.
```

```
Here are five things our support lead says a good refund reply does: <criteria>. Write a Kitaru evaluator that checks them, test it offline with kitaru evaluator test, and register it as refund-quality.
```

```
Take every session where refund-quality failed, freeze them into a cohort called refund-hard-cases, and start an experiment that replays them with the system prompt in prompts/support_v2.txt.
```

Each is a bounded task with a verifiable artifact at the end: a session, an evaluator version, or an experiment run. That shape gives both you and the assistant something concrete to inspect.


# Quickstart

From an agent in production to your first replay-backed decision, driven by your coding assistant: every step is a prompt first, a command second.

**You probably already have an agent in production.** It serves real users. Sometimes it does the wrong thing. When that happens, the usual workflow is to read the trace, tweak a prompt, and hope the fix holds. This page gives you a better loop: bring the agent's runs into Kitaru, judge one bad behavior, and test a fix against recorded evidence instead of a fresh demo prompt.

You do not need to memorize commands to start. Kitaru is built for your coding assistant to drive: you ask, it operates Kitaru through the MCP server and the agent skills, and you keep the judgment calls. Every step below starts as a prompt; the equivalent command is there when you want to run it yourself.

{% hint style="info" %}
**Want to see the complete loop before setting anything up?** Watch the 26-minute [Kitaru guided tour](https://youtu.be/aYLfzXEr2Rk). It starts with this Quickstart, then uses the `kitaru-guided-tour` skill to inspect recorded sessions, collect human judgments, define an evaluator and cohort, and test one improvement.

**No agent in production yet?** When you are ready to try it yourself, ask your assistant for the guided tour. The skill clones the public [`kitaru-template`](https://github.com/zenml-io/kitaru-template), prepares a three-session review for you to judge, turns one accepted finding into an evaluator without a paid model call, and ends with one approved replay experiment. Prefer to see every command yourself? The [returns-agent tutorial](/kitaru/guides/returns-agent) walks the same ground manually.
{% endhint %}

Before starting, [install Kitaru and log in](/kitaru/getting-started/installation), then [set up your coding agent](/kitaru/getting-started/setup): the MCP server gives it bounded Kitaru operations, and the skills teach it the procedures.

## First: get your runs into Kitaru

Nothing else works until your agent's runs land in Kitaru as [sessions](/kitaru/core-concepts/agents-and-sessions). You have two ways in, and both can start with a prompt:

{% tabs %}
{% tab title="Import the traces you already have" %}

```
Here is an export of our agent's traces from Langfuse: langfuse-export.jsonl. Register the agent in Kitaru as support-agent, import the export, tag the sessions imported-baseline, and tell me what landed and what was skipped.
```

Prefer to do it by hand? It is two commands:

```bash
kitaru agent register support-agent --command "python support.py"
kitaru session import langfuse-export.jsonl \
  --importer kitaru/langfuse@latest \
  --agent support-agent@latest --tag imported-baseline --wait
```

See [Import your traces](/kitaru/import-your-traces/import-your-traces) for the full walkthrough, and the [Langfuse](/kitaru/import-your-traces/import-langfuse-traces), [LangSmith](/kitaru/import-your-traces/import-langsmith-traces), [Braintrust](/kitaru/import-your-traces/import-braintrust-traces), [Logfire](/kitaru/import-your-traces/import-logfire-traces), and [Arize Phoenix](/kitaru/import-your-traces/import-phoenix-traces) guides for each provider's contract.
{% endtab %}

{% tab title="Record with an adapter" %}

```
Add the Kitaru adapter to our PydanticAI agent so every run is recorded as a session. Register the agent as support-agent first and wire its agent id into the wrapper. Don't change any agent behavior.
```

The wrapper it adds is one line around the agent you already have:

```python
from pydantic_ai import Agent
from kitaru_pydantic_ai import KitaruAgent

agent = Agent("openai:gpt-5.4", name="support-agent")
support = KitaruAgent(agent, agent_id=AGENT_ID)
support.run_sync("Refund order #4821, the card reader double-charged me.")
```

See the [adapter overview](/kitaru/adapters/adapters) for your framework.
{% endtab %}
{% endtabs %}

Which one? Both, eventually:

* **Import is the fastest start.** Your history becomes reviewable today, with no code change and nothing new in production.
* **You will want the adapter anyway.** Replays and experiments re-run *your agent's code*; the adapter is what answers its tool calls from the recording. Import your backlog now, add the adapter with your next deploy.

## Then: let your assistant drive the loop

The whole method fits in one ask. `kitaru-investigation` is the skill that runs it with you:

```
Use kitaru-investigation to investigate this agent and help me test one meaningful improvement. Assume I am new to Kitaru. Show me the recorded evidence before asking for a judgment, and ask before creating resources, changing code, or starting paid replay.
```

The assistant selects sessions, walks the review, drafts the evaluator, and runs the experiment. You supply the domain judgments and approve consequential actions. These five steps are the record → replay → improve loop in working form: recording got you the sessions above; observing, judging, and defining turn evidence into criteria; replaying and comparing close the loop. The example below uses a support agent that refunds, replaces, or escalates return requests, and each step includes the prompt you would use to drive that step by itself.

{% stepper %}
{% step %}

### Observe a recorded behavior

```
Run the deterministic evaluators over support-agent's recent sessions, show me which ones look worst and why, and walk me through the worst one node by node.
```

Observation starts wide: scan the history before you stare at one trace. Kitaru ships ten [deterministic evaluators](/kitaru/guides/deterministic-evaluations), covering session diagnostics, tool health, trajectory signals, timing, and LLM-call signals, that read stored sessions without running the agent or calling a model. The sweep is cheap and repeatable, and the failures, retries, and tool errors it surfaces tell you which sessions deserve a human look. One surfaced session contains this path:

| Session node        | Result                                          |
| ------------------- | ----------------------------------------------- |
| Customer request    | The customer asks for a high-value refund.      |
| `lookup_order`      | The order exists; amount and category returned. |
| `get_return_policy` | No usable approval rule comes back.             |
| `issue_refund`      | The tool accepts the refund.                    |
| Agent response      | The agent says the refund was issued.           |

Each model call, tool call, and result is a **session node**. The `issue_refund` node matters because it proves the action occurred; the final message alone only tells you what the agent claimed. At this point Kitaru has preserved the behavior, not judged it.
{% endstep %}

{% step %}

### Judge what should have happened

```
Open an investigation on this session. I will give the verdicts; record each one as an annotation pinned to the exact nodes that support it.
```

This is the interview. Your assistant has already mapped your sessions and built a worklist: related failures plus at least one counterexample. Now it creates an [**investigation**](/kitaru/core-concepts/investigations) and asks you, against the evidence on screen, the questions Kitaru needs answered. Not "write down your eval criteria," but "given this policy lookup that returned nothing and this refund that was accepted anyway, was escalation required?"

The expert answers:

> When the agent cannot establish whether approval is required, it should escalate instead of issuing the refund.

Each answer is stored as an **annotation** pinned to the exact nodes that support it, and the conclusion becomes the session's verdict. Statistics can surface an unusual trace, but they cannot infer your business policy. The judgment you record here is the ground truth the next three steps use.
{% endstep %}

{% step %}

### Define the behavior to test

```
Turn my accepted judgment into a deterministic evaluator, and freeze the reviewed cases, including at least one counterexample, into a cohort.
```

The accepted judgment becomes a reusable [**evaluator**](/kitaru/core-concepts/evaluators). One bad case is not enough, so the review also keeps a counterexample:

| Reviewed case                  | Expected behavior         | Role                                     |
| ------------------------------ | ------------------------- | ---------------------------------------- |
| Approval cannot be established | Escalate without a refund | **Target:** what should change.          |
| Valid low-risk refund          | Issue the refund          | **Counterexample:** what must not break. |

Both are frozen into a [**cohort**](/kitaru/core-concepts/cohorts) version. The target catches a change that does not fix the failure; the counterexample catches a blunt fix such as "never issue refunds."
{% endstep %}

{% step %}

### Replay the changed agent

```
Register my working tree as a new version of support-agent and replay the cohort against it. Answer every tool call from the recorded history and fail on any missing result.
```

Kitaru [**replays**](/kitaru/core-concepts/replay) the frozen cohort against the candidate inside an [**experiment**](/kitaru/core-concepts/experiments): each replay starts from the recorded input and produces a new session.

{% hint style="warning" %}
Re-running an agent can re-run its tools, so every tool call needs a policy: **recorded history** (answer from the recording; the default for side effects), **static results**, **passthrough** (live call, only for intentionally safe tools), or **fail on a missing result**. Replay never means repeating production side effects. Insist on the recorded-history default in assistant-written replays.
{% endhint %}
{% endstep %}

{% step %}

### Compare the evidence

```
Compare evaluations between the baseline and the candidate across the cohort, and tell me what improved, what regressed, and what is inconclusive.
```

The same evaluator version checks the original and replayed sessions:

| Reviewed case                  | Original              | Candidate             | Conclusion                     |
| ------------------------------ | --------------------- | --------------------- | ------------------------------ |
| Approval cannot be established | Refund accepted, fail | Escalation, pass      | The reviewed failure improved. |
| Valid low-risk refund          | Refund accepted, pass | Refund accepted, pass | The counterexample held.       |

Four honest outcomes stay available: **improved**, **regressed**, **trade-off**, and **inconclusive**. Inconclusive is still useful: it names the missing evidence or execution control before you trust the change. The deployment decision stays with you.
{% endstep %}
{% endstepper %}

The five steps form a loop, not a one-time pipeline: a replay can expose a new failure, which becomes the next observation to review.

{% hint style="info" %}
**Every step also has a manual form.** The CLI covers the whole loop with `--output json`, and the [Python and TypeScript SDKs](/kitaru/get-help/sdks) reach everything. The [guides](/kitaru/guides/replay-and-overrides) and the [returns-agent tutorial](/kitaru/guides/returns-agent) teach the manual path so you can see each object and boundary for yourself.
{% endhint %}

## Glossary

| Term                            | Plain meaning in this example                                                    |
| ------------------------------- | -------------------------------------------------------------------------------- |
| **Agent / agent version**       | The support agent, and one immutable run specification for it.                   |
| **Session / session node**      | One complete run, and one event inside it such as `issue_refund`.                |
| **Investigation / annotation**  | The organized human review, and a verdict pinned to exact evidence.              |
| **Evaluator / evaluation**      | The reusable behavior check, and its result on one session.                      |
| **Cohort / cohort version**     | A named test population, and one frozen membership list.                         |
| **Replay**                      | A new run of candidate code from a recorded input under an explicit tool policy. |
| **Experiment / experiment run** | The reusable replay-and-measurement definition, and one execution of it.         |

You do not need to memorize these before starting; each one preserves a step of the reasoning, and your assistant knows them already.

## Where to go next

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Set up your coding agent</strong></td><td>The MCP server and skills that make all of this one ask away.</td><td><a href="/pages/UPvRjygShZEG0jzCwMXc">/pages/UPvRjygShZEG0jzCwMXc</a></td></tr><tr><td><strong>Import your traces</strong></td><td>Bring in Langfuse, LangSmith, Braintrust, Logfire, Arize Phoenix, or Kitaru JSONL data.</td><td><a href="/pages/o74tFfkKFSg3Ubasut52">/pages/o74tFfkKFSg3Ubasut52</a></td></tr><tr><td><strong>Kitaru template</strong></td><td>Prepare the synthetic PydanticAI agent and checked-in Langfuse traces.</td><td><a href="https://github.com/zenml-io/kitaru-template">https://github.com/zenml-io/kitaru-template</a></td></tr><tr><td><strong>Complete tutorial</strong></td><td>Run the five-step method manually from the prepared template.</td><td><a href="/pages/R1lYNNZKYCdTyMZbtsck">/pages/R1lYNNZKYCdTyMZbtsck</a></td></tr><tr><td><strong>Core concepts</strong></td><td>Read precise references for each Kitaru resource.</td><td><a href="/pages/qw8hIFEbl4taSEvy4SNP">/pages/qw8hIFEbl4taSEvy4SNP</a></td></tr></tbody></table>


# Overview

A handful of nouns and one verb, and how sessions, replays, evaluators, cohorts, experiments, investigations, and workers fit together.

Kitaru's object model is small. Every piece exists to serve one loop: **record → replay → improve**.

* Your production agent leaves [**sessions**](/kitaru/core-concepts/agents-and-sessions), recordings of every model call, tool call, and decision, either recorded live by an [adapter](/kitaru/adapters/adapters) or [imported](/kitaru/import-your-traces/import-your-traces) from the traces you already collect.
* [**Investigations**](/kitaru/core-concepts/investigations) are where your judgment enters. Your coding assistant maps the sessions, builds a review worklist, interviews you against the evidence, and pins your answers as **annotations** on exact trace locations. They are the ground truth evaluators are calibrated against and cohorts are justified by.
* [**Replay**](/kitaru/core-concepts/replay) re-executes a session against your real code. Unchanged, it reproduces the original and gives you the faithful baseline. Forked with one thing different, such as a model, prompt, or code change, it answers a counterfactual you can trust.
* [**Evaluators**](/kitaru/core-concepts/evaluators) evaluate sessions and write evaluations, which are typed, versioned verdicts. Human labels land in the same table.
* [**Cohorts**](/kitaru/core-concepts/cohorts) freeze a population of sessions into immutable versions, so results stay comparable.
* [**Experiments**](/kitaru/core-concepts/experiments) replay a cohort against a change and evaluate both sides, showing what improved and what regressed before you ship.
* [**Workers**](/kitaru/core-concepts/workers) execute all of it in your environment. The server coordinates; your infrastructure runs the code and holds the data.

The short version: traces tell you what happened; Kitaru re-runs it. A trace you can only read is a transcript. A session is a recording your test bench can execute, which is what turns production's past into your test suite.

## How the pieces reference each other

An **agent** is the identity everything attaches to; an **agent version** pins the code, as a run spec a worker can execute. A **session** belongs to an agent and optionally a version. A **cohort version** pins session ids. An **experiment** pins the change: override, tool policy, and evaluators. An **experiment run** pins a cohort version and an agent version, then fans out one **replay** per session. Every replay produces a new session, and **evaluations** land on sessions from either side, which is why comparing a baseline to a fork means reading two sets of rows.

Nothing is recomputed behind your back, and nothing is mutable where it matters. Cohort versions, agent versions, and evaluator versions are frozen at creation, so any number you read can be traced to the code, population, and criteria that produced it.

## Where to start

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Agents &#x26; Sessions</strong></td><td>The identity and the recording.</td><td><a href="/pages/HxcE6J2lKzl3hFUJebqn">/pages/HxcE6J2lKzl3hFUJebqn</a></td></tr><tr><td><strong>Investigations &#x26; Annotations</strong></td><td>The interview: your judgment, pinned to exact evidence.</td><td><a href="/pages/lQJsrF7YukdI9mexUMBN">/pages/lQJsrF7YukdI9mexUMBN</a></td></tr><tr><td><strong>Replay</strong></td><td>Baselines, forks, overrides, and tool policies.</td><td><a href="/pages/q8ejBRbyMBEUPEy2CDmB">/pages/q8ejBRbyMBEUPEy2CDmB</a></td></tr><tr><td><strong>Evaluators &#x26; Evaluations</strong></td><td>Evaluating sessions, human labels, calibration.</td><td><a href="/pages/Z1S2gF2qHRPaojtqKgPx">/pages/Z1S2gF2qHRPaojtqKgPx</a></td></tr><tr><td><strong>Cohorts</strong></td><td>Immutable populations for comparable results.</td><td><a href="/pages/NpZ14Rqa7IcqlrO1wwDh">/pages/NpZ14Rqa7IcqlrO1wwDh</a></td></tr><tr><td><strong>Experiments</strong></td><td>A change, replayed and evaluated at population scale.</td><td><a href="/pages/bJu6Tc6vNBJti75P23Eg">/pages/bJu6Tc6vNBJti75P23Eg</a></td></tr><tr><td><strong>Workers</strong></td><td>Execution in your environment.</td><td><a href="/pages/2uchH0kpn87pgDAO1ol0">/pages/2uchH0kpn87pgDAO1ol0</a></td></tr><tr><td><strong>Under the Hood</strong></td><td>Server, workers, tasks, and blobs: the machinery.</td><td><a href="/pages/kghgTz6zjyaZAOR6CYOw">/pages/kghgTz6zjyaZAOR6CYOw</a></td></tr></tbody></table>


# Agents & Sessions

The agent is the identity, the session is the recording of every model call, tool call, and decision a run left behind.

Most traces are transcripts: you read them. A Kitaru **session** is a recording you can run. It holds everything one agent run did, including every model call, tool call, and decision, in order, with inputs and outputs. That is what [replay](/kitaru/core-concepts/replay) needs to re-execute the run against your real code.

Two nouns carry the whole data model:

* An **agent** is the stable identity your runs attach to. You register it once, and every session, cohort, and experiment references it.
* A **session** is one recorded run of that agent. Sessions arrive three ways: recorded live by an adapter, [imported](/kitaru/import-your-traces/import-your-traces) from your existing traces, or produced by a replay. All three are the same object with a different `origin`: `recorded`, `imported`, or `replay`.

## Agents and agent versions

Register an agent with the CLI:

```bash
kitaru agent register support-agent \
  --command "python support.py" \
  --description "Resolves support tickets"
```

This creates the agent and its first **agent version** in one step. A version pins what "the agent" meant at a point in time: a run spec (the command that starts your agent, its working directory, environment, secrets, and timeout) plus optional capability metadata (tools, MCP servers, skills). The run spec is what a [worker](/kitaru/core-concepts/workers) executes when a replay or experiment re-runs the agent in your environment.

Versions are server-numbered (1, 2, 3, …); `--display-version` attaches your own label, such as a semver, a git SHA, or a branch name:

```bash
kitaru agent version register support-agent \
  --command "python support.py" \
  --display-version "pr-1284"
```

Register a new version when the code changes. An [experiment](/kitaru/core-concepts/experiments) is precisely "replay this cohort on that agent version and see what moved."

## What a session records

A session carries its top-level `inputs`, `outputs`, `status` (`in_progress` / `completed` / `failed`), timing, and rolled-up totals: `cost`, `tokens` (input / output / cached / reasoning), `llm_call_count`, and `tool_call_count`.

The step-by-step recording lives in the session's **nodes**: an ordered tree with one node per event.

| Node type       | What it records                                                                                         |
| --------------- | ------------------------------------------------------------------------------------------------------- |
| `llm_call`      | Requested and resolved model, inputs and outputs, token usage, cost, model params                       |
| `tool_call`     | Tool name, arguments, result, plus the cache key replay uses to answer the same call from the recording |
| `subagent_call` | A delegated run by a sub-agent                                                                          |
| `span`          | Any other grouping the adapter or importer wants to preserve                                            |

Adapters record nodes automatically. The [PydanticAI adapter](/kitaru/adapters/pydantic-ai) batches them to the server as the run progresses; importers write the same structure from your existing traces. There is one shape, so replay and evaluators never care where a session came from.

## One session is one end-to-end run

This is the most important thing to get right when you bring your own traces, and the easiest to get wrong.

A session is **the whole run, from the request that started it to the answer that ended it**, including every model call, tool call and sub-agent hop in between. It is not one model call, and it is not one span. Replay re-executes a session from the top, so a session that holds half a run can only ever reproduce half a run, and a cohort of them measures nothing you care about.

Adapters get this for free: the wrapper opens the session when your agent is invoked and closes it when the call returns. Importing needs a decision from you, because observability tools do not agree on what a trace is:

* Some emit **one trace per run**, which maps to one session directly. Nothing to do.
* Many emit **one trace per conversation turn**, so a five-turn support conversation arrives as five traces. If that is one run in your product, those five traces are one session.
* Some emit **one trace per model call**, which almost never matches a session on its own.

You do not have to reshape the export yourself. Importers group related traces into one session using the provider's own conversation or session identifier, and `--join-on` names the field to group on when the identity lives somewhere else. See [Join provider traces into sessions](/kitaru/import-your-traces/importing-sessions). When no identifier is present, each trace becomes its own session, which is the safe default but rarely the one you want for multi-turn agents.

So the question before importing is not "what does my tool call a trace" but **"what does my product call one run"**. Then make the import produce that. If the answer is "it depends on how we configured tracing", resolve that upstream if you can: consistent session identity in your traces is what makes cohorts, experiments, and regression suites mean the same thing every time.

If you are joining a format no importer understands, do the joining in your [custom importer](/kitaru/import-your-traces/importing-sessions) rather than after the fact. Sessions are not merged once they land.

## Reading sessions back

The Python client is async; every resource follows the same `list` / `iter` / `get` pattern:

```python
import asyncio
from kitaru.client import KitaruAPIClient
from kitaru.api_models.v1.session import SessionListParams
from kitaru.api_models.v1.session_node import SessionNodeListParams


async def main() -> None:
    client = KitaruAPIClient()  # KITARU_API_URL, KITARU_API_KEY
    page = await client.sessions.list(SessionListParams())
    for session in page.items:
        print(session.id, session.origin, session.status, session.cost)

    nodes = await client.sessions.list_nodes(
        page.items[0].id, SessionNodeListParams(include_payloads=True)
    )
    for node in nodes.items:
        print(node.index, node.node_type, node.name)


asyncio.run(main())
```

Node payloads (inputs, outputs) are returned only when you ask (`include_payloads=True`); listings stay cheap by default. The CLI mirrors both reads:

```bash
kitaru session list --agent support-agent --origin recorded
kitaru session nodes <session-id> --include-payloads
```

Sessions attach to the rest of the system by reference: a [cohort version](/kitaru/core-concepts/cohorts) pins a set of session ids, an [evaluation](/kitaru/core-concepts/evaluators) row evaluates one session, and a [replay](/kitaru/core-concepts/replay) points at its baseline session and produces a result session. **Tags** group resources ad hoc before they graduate into a cohort or another durable structure. A tag can link to a session, cohort, cohort version, agent version, experiment, or experiment run. Apply one to a whole import with `kitaru session import --tag ...`, then select on it anywhere that resource supports a `tag` filter, such as `kitaru session evaluate --tag ...`.

The native MCP server can list and filter tags, use existing filtered reads to rediscover tagged resources, and create, rename, link, unlink, or delete tags according to its capability mode. It cannot enumerate every link belonging to a tag. Deleting a tag removes all of its resource links; it does not delete the linked resources.

## Where sessions come from

* **Recorded:** wrap your agent with an adapter and run it as usual. See the [adapter overview](/kitaru/adapters/adapters).
* **Imported:** bring the traces you already collect. Langfuse stays your system of record; Kitaru gets a runnable copy. See [Import your traces](/kitaru/import-your-traces/import-your-traces).
* **Replay:** every replay produces a new session with `origin: replay`, evaluated by the same evaluators as any other session. See [Replay](/kitaru/core-concepts/replay).


# Investigations & Annotations

Where your judgment enters the system: your coding assistant maps the evidence, interviews you in context, and pins your answers to exact trace evidence as annotations.

Every evaluation system hits the same wall: where do the criteria come from? You probably never wrote them down. The people who judge the agent, your support leads and domain experts, do it every day in Slack threads and ticket comments, and most of those corrections disappear.

Investigations are how Kitaru keeps them. An **investigation** organizes a review of recorded sessions: which sessions to inspect, in what order, what question each one raises, and what the reviewer concluded. By design, **a coding agent authors it, not you**. The LLM's job is to draft a useful investigation: pick the sessions worth your time, phrase the questions, and point at the evidence. Your job is the part no model can do: answer. An **annotation** is one answer, stored as JSON and pinned to the exact evidence that supports it: a session, a node inside it, a path inside a payload, even a character range. Together they are the ground truth everything downstream is calibrated against. Replay can tell you what a change did; only your recorded judgment can say whether it got better.

## The interview

[Set up your coding agent](/kitaru/getting-started/setup), then use the `kitaru-investigation` skill to run the review as an interview. You do not have to write questions or pick sessions; the assistant does that work because a well-chosen worklist and clear questions are a good use of an LLM. Answering those questions is not.

1. **It maps the world first.** From one surprising failure, the assistant reads the session fully and builds a small worklist of related sessions plus at least one counterexample. From a vague "something is off," it samples a diverse population, normally 15 to 30 sessions, random picks alongside coverage-based ones.
2. **It creates the investigation**, with a question for each session and highlights that point you at the evidence: the policy lookup that returned nothing, the refund that was accepted anyway.
3. **It asks you, in context.** Not "write down your evaluation criteria" in the abstract, but "given this recorded policy result and this accepted refund, was escalation required?" Questions are asked against the trace, where you can answer them. This gives Kitaru the missing judgment one concrete case at a time.
4. **Your answers become annotations; your conclusions become verdicts.** Each reviewed session ends `acceptable`, `problematic`, or `uncertain`. The assistant selects, summarizes, and organizes the evidence; the judgment it records is yours, never its own suggestion.

Two design choices keep the interview honest. Open observations come before proposed failure categories, so an early taxonomy does not bias what you look at. Observed behavior also stays separate from expected behavior: the procedure distinguishes the agent's actions, dependency behavior, and product requirements instead of labeling every surprise an agent failure.

## What the answers are for

Annotations are labels with addresses. Everything that gates a change is calibrated against them:

* **Evaluators** are checked against them: run the evaluator over the reviewed sessions and [compare its evaluations with the human answers](/kitaru/guides/write-an-evaluator) before the evaluator judges anything on its own.
* **Cohorts** are justified by them: the sessions confirmed `problematic` become the [cohort](/kitaru/core-concepts/cohorts) a regression experiment replays, and the annotation trail explains why that cohort exists.
* **The next review** builds on them: verdicts and answers stay queryable, so a later investigation starts from what is already known instead of re-litigating it.

An evaluator that gates a deploy should be able to show the human judgments it was calibrated against. Annotations are those judgments.

## What an investigation contains

Everything below is what the assistant creates on your behalf during the interview. The CLI is the escape hatch and the audit surface: use it to inspect what was built, script a review, or construct an investigation by hand when you want full control.

An investigation belongs to one agent. It contains linked sessions, each with a `position` that determines the review order.

Questions belong to individual linked sessions rather than to the investigation as a whole, so the review can ask different questions about different runs. Each question has a `key`, unique within its session, and display text such as `refund_justified="Was the refund justified?"`. A question can include highlights; each highlight has a selector and a description that point the reviewer at relevant evidence.

The reviewer gives each linked session a verdict of `acceptable`, `problematic`, or `uncertain`. A session remains incomplete until it has a verdict; the investigation reports progress through `completed_sessions` and `total_sessions`, and tracks its own `status` as `pending`, `in_progress`, or `completed`.

```bash
kitaru investigation create refund-complaints --agent support-agent \
  --description "Week-32 refund complaints from the support queue" \
  --session <session-id> \
  --session-question <session-id>:refund_justified="Was the refund justified?"

kitaru investigation session list <investigation-id>
kitaru investigation session verdict <investigation-id> <session-id> problematic
```

Questions and highlights use the form `SESSION:KEY`, and the session must also appear in a `--session` argument. Highlights accept a JSON array with the selector inline:

```bash
kitaru investigation create refund-complaints --agent support-agent \
  --session <session-id> \
  --session-question <session-id>:tone="Did the tone stay professional?" \
  --session-highlights <session-id>:tone='[{"selector": {"node_id": "<node-id>"}, "description": "Reply after the refund was refused"}]'
```

## Annotations: answers with an address

Every answer is an **annotation**, which stores a JSON value against a session. A **selector** attaches it to more specific evidence: a node (`node_id`), an RFC 6901 JSON pointer into the node or session response (`path`), or a character range within the resolved string (`span`, which requires a `path`). Investigation highlights use the same selector format.

An answer to an investigation question uses `investigation_session_id` and `question_key`, and Kitaru stores both on the resulting annotation. A manual annotation uses only `session_id` and can be added to any session, inside an investigation or not. Queries can tell the two apart because only question answers populate `investigation_session_id` and `question_key`.

```bash
# an answer to a question
kitaru annotation create --investigation-session <id> \
  --question-key refund_justified --value 'false'

# a standalone label, pinned to where it happened
kitaru annotation create --session <id> \
  --selector '{"node_id": "<node-id>", "path": "/output/text"}' \
  --value '{"issue": "tone", "severity": "high"}'
```

`value` can contain any JSON: a boolean answer, a rating, a rubric object. Kitaru does not impose a schema; use a consistent shape if you plan to compare annotations or calibrate an evaluator against them. Annotations can be listed, fetched, updated (`--value` only), and deleted.

## Working through a review

A review normally uses three operations:

```bash
kitaru investigation session list <investigation-id>  # what's queued, in position order
kitaru annotation create --investigation-session <id> \
  --question-key refund_justified --value 'false'     # answer, with evidence
kitaru investigation session verdict <investigation-id> <session-id> problematic
```

Answers and verdicts are separate: answers record a value per question, the verdict records the conclusion about the session as a whole, and `completed_sessions` counts only sessions with a verdict. A session can have answers and still be incomplete.

Over [MCP](/kitaru/getting-started/setup), `kitaru_review_read` and `kitaru_review_manage` let a coding assistant read the review queue, answer questions, and create annotations. A human still decides which sessions to review and what verdict to assign. Before creating remote state or using worker or model compute, the skill explains the operation and asks for confirmation. If a required payload, permission, or worker is missing, it records a checkpoint so the interview can resume later. The client mirrors the surface: `client.investigations.*` and `client.annotations.*`.


# Replay

Re-run a recorded session against your real code, unchanged for a faithful baseline or forked with one thing different.

Replay is the verb the whole product hangs on. A [session](/kitaru/core-concepts/agents-and-sessions) is a recording; a **replay** re-executes it. Your agent's real code runs again, and the recording answers for the world the original run saw. With a `history` [tool policy](#tool-policies), tool calls are served from the recorded session, so nothing touches your real systems.

The discipline comes first: **replay unchanged before you change anything.** An unchanged replay that reproduces the original is your faithful baseline. Fork from that baseline with exactly one thing different (a model, a prompt, a code change) and the diff you read is your change, not replay noise.

## What a replay is

A replay names a **baseline session**, the **agent version** to run (by default, the version the baseline was recorded with), an optional **override**, a **tool policy**, and at least one [evaluator](/kitaru/core-concepts/evaluators). The server turns it into a job; a [worker](/kitaru/core-concepts/workers) in your environment starts your agent from its run spec, feeding it the baseline's inputs. The re-run records a fresh session (`origin: replay`), and the evaluators evaluate it as soon as it completes.

For a one-off replay, the CLI exposes the same create, list, and get flow:

```bash
kitaru replay create <baseline-session-id> \
  --evaluator refund-check@1 \
  --tool-policy '{"default":{"type":"history","scope":"baseline","on_miss":"fail"}}' \
  --evaluate-baselines --output json
kitaru replay list --output json
kitaru replay get <replay-id> --output json
```

Creation returns immediately with the replay and its job. Use `kitaru job watch <job-id>` to follow it, `kitaru job get <job-id> --tasks` to inspect task failures, or `kitaru job cancel <job-id>` to request cancellation.

{% hint style="warning" %}
\`kitaru replay create\` is not idempotent. If the command fails after the server accepted it, retrying can create another replay and job. Run \`kitaru replay list\` and check for the first replay before retrying. Omitting \`--tool-policy\` uses the server default, which may execute live tools.
{% endhint %}

```python
import asyncio
from kitaru.client import KitaruAPIClient
from kitaru.api_models.v1.replay import ReplayCreateRequest
from kitaru.api_models.v1.replay_config import (
    EvaluatorConfig,
    HistoryConfig,
    ToolPolicy,
)


async def main() -> None:
    client = KitaruAPIClient()
    replay = await client.replays.create(
        ReplayCreateRequest(
            baseline_session_id=BASELINE_ID,
            evaluators=[EvaluatorConfig(evaluator="refund-check")],
            tool_policy=ToolPolicy(
                default=HistoryConfig(scope="baseline", on_miss="fail")
            ),
            evaluate_baselines=True,
        )
    )
    print(replay.id, replay.job_id, replay.status)


asyncio.run(main())
```

`evaluate_baselines=True` evaluates the baseline session with the same evaluators, so the comparison you want, baseline evaluations next to replay evaluations, exists as soon as the replay settles. Watch the job with `kitaru job watch <job-id>`, then read `result_session_id` off the replay.

A replay moves `pending → evaluating → completed` (or `failed` / `canceled`). Its output is intentionally plain: the result session plus its evaluation rows. You compare baseline and result by reading both sessions' evaluations, cost, and tokens. See [Replay a failure and fork it](/kitaru/guides/replay-and-overrides) for the full loop.

## Forking: the override

There is no separate fork operation in the API; a "fork" is a replay that carries an `override`. The word is shorthand for that, the way "baseline" is shorthand for a replay without one. Both are the same call.

An override changes one thing about the re-run and leaves everything else alone:

```python
from kitaru.api_models.v1.replay_config import ReplayOverride

override = ReplayOverride(
    model={"openai:gpt-5.4": "openai:gpt-5-nano"},  # or just "openai:gpt-5-nano"
    # system_prompt="...",   # replace the system prompt
    # prompt="...",          # replace the user prompt
    # model_params={"temperature": 0.0},
)
```

* `model` swaps the model on every matching model call: a plain string replaces all of them, a map replaces old with new per model.
* `system_prompt` and `prompt` rewrite the run's inputs before the agent starts.
* `model_params` adjusts sampling parameters at the adapter level.

Replays re-run the agent **from the top**. There is no partial, mid-run cut point: the recording answers the world's side of the conversation, and your agent recomputes its own side in full. That is what makes a fork trustworthy: the whole decision path is real.

## Tool policies

The tool policy decides what happens when the re-running agent calls a tool. The default answers per tool name, with one fallback for everything else:

| Policy        | What a tool call gets                                                                                                                                                                                      |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `history`     | The recorded result for the same call, matched by tool name and arguments, from the baseline (or a wider scope). `on_miss` decides what an unrecorded call does: `fail`, `passthrough`, or `error_result`. |
| `static`      | A canned result you define per case, with exact or subset argument matching.                                                                                                                               |
| `passthrough` | The real tool, live. This is the current server default when you set no policy.                                                                                                                            |
| `llm`         | A model answers the tool call in-distribution. The API accepts it, but adapter support varies; PydanticAI, Mastra, and Vercel AI SDK currently reject it.                                                  |

For the "nothing touches real systems" guarantee, set `default=HistoryConfig(scope="baseline", on_miss="fail")`. Recorded calls are answered from the recording and anything novel stops the replay instead of hitting production. The full matrix, including per-tool overrides and history scopes, is in [Tool policies](/kitaru/guides/tool-policies).

## Scale: cohorts and experiments

One replay answers a question about one session. The same machinery applied to a [cohort](/kitaru/core-concepts/cohorts) of sessions, with the change expressed as an [experiment](/kitaru/core-concepts/experiments), answers the question that matters before you ship: *what does this change do to last week's production traffic?* That is the [regression suite](/kitaru/guides/regression-suite).


# Evaluators & Evaluations

Evaluators turn sessions into evaluations: the rows they write. Numbers average, words count, free text gets read.

Replay tells you what a change *did*; evaluators tell you whether it *helped*. An **evaluator** is a small piece of your code that reads one [session](/kitaru/core-concepts/agents-and-sessions), node by node, and writes one or more **evaluations**: named, typed verdicts that Kitaru stores against the session.

Because evaluators run against recorded sessions, they evaluate baselines, replays, and imported traces identically. The same evaluator you run over today's production traffic runs over the fork you're thinking about shipping.

## The evaluator contract

An evaluator is a callable (a single Python file or an installable package) that receives the full session and returns results:

```python
"""refund_check.py: did the agent issue the refund?"""

from kitaru.task.evaluator import EvaluationResult, SessionView


def evaluate(session: SessionView, **params) -> EvaluationResult:
    refund_calls = [
        node
        for node in session.nodes
        if node.node_type == "tool_call" and node.tool_name == "refund_payment"
    ]
    return EvaluationResult(
        name="refund_issued",
        score=bool(refund_calls),
        passed=bool(refund_calls),
        explanation=f"{len(refund_calls)} refund tool call(s) in the session",
    )
```

`SessionView` gives you the session and all its nodes with payloads. Return one `EvaluationResult` or a list; each becomes one stored evaluation row. `params` are per-run knobs you pass when you attach the evaluator to a replay or experiment.

Scaffold, exercise, and register it with the CLI:

```bash
kitaru evaluator scaffold refund-check          # writes refund_check_evaluator.py
kitaru evaluator test refund_check_evaluator.py --entrypoint evaluate
kitaru evaluator register refund-check \
  --script refund_check_evaluator.py --entrypoint evaluate
```

Evaluators are versioned like agents: registering again with `kitaru evaluator version register` creates version 2, and every stored evaluation remembers exactly which evaluator version wrote it. An LLM judge follows the same contract by calling a model inside `evaluate`. The walkthrough is in [Write an evaluator](/kitaru/guides/write-an-evaluator).

A suite of evaluators comes **built in**, registered at server startup under the `kitaru/` namespace: three cheap signals (`kitaru/cost`, `kitaru/latency`, `kitaru/tool-call-patterns`) plus ten deterministic checks over the recording itself, from `kitaru/output-contract` and `kitaru/tool-health` to `kitaru/timing-profile` and `kitaru/workflow-conformance`. None of them make model calls; they are the triage layer, available as `kitaru/cost@latest` before you have written anything.

## The evaluation row

One evaluation is one named result for one session. The data type is derived from what you set, never declared:

| You set                     | Stored type   | How to read a batch of them    |
| --------------------------- | ------------- | ------------------------------ |
| `score=0.87`                | `float`       | numbers average                |
| `score=True`                | `bool`        | pass rates count               |
| `value="escalated"`         | `str`         | free text gets read            |
| `score=0.9, value="polite"` | `categorical` | labels count, transitions diff |

`passed` is an independent optional verdict, based on a threshold you decided in the evaluator rather than something derived from `score`. `explanation` says why, which is the part you read when a regression gate goes red.

## Human labels are evaluations too

There is no separate labeling system. A human verdict is an evaluation written directly onto the session:

```python
from kitaru.api_models.v1.evaluation import EvaluationResult
from kitaru.api_models.v1.session import SessionEvaluationsRequest

await client.sessions.merge_evaluations(
    session_id,
    SessionEvaluationsRequest(
        evaluations=[
            EvaluationResult(
                name="human_quality",
                score=True,
                explanation="Correct refund, good tone",
            ),
        ]
    ),
)
```

Manual evaluations upsert by name: re-sending `human_quality` overwrites the earlier verdict. Rows written by evaluator runs carry their evaluator version and task; manual rows carry neither, which is how you tell them apart. Comparing your evaluator's column against the human column on the same sessions is how you calibrate the evaluator before you let it gate anything. The human column usually comes out of [the interview](/kitaru/core-concepts/investigations): your coding assistant authors the investigation, and your answers land as annotations to calibrate against.

## Running evaluators in batch

Evaluate existing sessions without replaying anything. From the CLI, select by IDs, by tag, by agent, by cohort version, by filter, or everything:

```bash
kitaru session evaluate --tag imported-baseline \
  --evaluator refund-check@latest --evaluator kitaru/cost@latest \
  --wait
```

Exactly one selection is required: explicit session IDs (arguments or `--sessions-file`), `--tag`, `--agent`, `--cohort`, `--filter`, or `--all`. An empty match is an error, not a silent no-op. The client form:

```python
from kitaru.api_models.v1.evaluation import EvaluationBatchCreateRequest
from kitaru.api_models.v1.replay_config import EvaluatorConfig

job = await client.evaluations.create(
    EvaluationBatchCreateRequest(
        input_session_ids=session_ids,
        evaluators=[EvaluatorConfig(evaluator="refund-check")],
    )
)
```

Each (session, evaluator) pair runs as its own task on a [worker](/kitaru/core-concepts/workers) (in your environment, next to your credentials), and one failed pair never cancels the rest. Read results back with `client.evaluations.list(...)`, filtered by session.

Evaluators are also how [replays](/kitaru/core-concepts/replay) and [experiments](/kitaru/core-concepts/experiments) get their numbers: both require at least one evaluator, so a re-run is evaluated the moment it lands.


# Cohorts

Named, versioned sets of sessions; the population an experiment replays, frozen so results stay comparable.

One session answers "what happened on this run." A **cohort** answers questions about a population: last week's production traffic, every run that touched refunds, the twelve sessions where the agent got it wrong. A cohort is a named set of sessions belonging to one agent, and it is the unit an [experiment](/kitaru/core-concepts/experiments) replays.

## Versions are immutable

A cohort is a namespace. Membership lives on **cohort versions**, and a version's member list never changes after creation. To add or remove sessions, create a new version as a delta on the latest one:

```python
import asyncio
from kitaru.client import KitaruAPIClient
from kitaru.api_models.v1.cohort import CohortCreateRequest
from kitaru.api_models.v1.cohort_version import CohortVersionCreateRequest


async def main() -> None:
    client = KitaruAPIClient()

    cohort = await client.cohorts.create(
        CohortCreateRequest(name="refund-regression", agent_id=AGENT_ID)
    )
    version = await client.cohorts.create_version(
        cohort.id,
        CohortVersionCreateRequest(
            add_session_ids=failing_session_ids,
            display_version="week-32",
        ),
    )
    print(version.version, version.session_count)


asyncio.run(main())
```

On the CLI, `cohort create` can snapshot a selection into version 1 at the same time, by explicit IDs, a tag, a filter, or another cohort version:

```bash
kitaru cohort create refund-regression --agent support-agent \
  --tag imported-baseline --display-version week-32
```

Later versions are membership deltas:

```bash
kitaru cohort version create refund-regression \
  --add-session <id> --remove-session <id> --display-version week-33
```

The first version starts from an empty list; each later version is the previous list minus `remove_session_ids` plus `add_session_ids`. The delta applies to the latest version by default. To branch from an exact earlier version in the CLI, pass its UUID with `--baseline`:

```bash
kitaru cohort version create refund-regression \
  --baseline <cohort-version-id> \
  --add-session <id> --display-version alternative-week-33
```

In the Python client and REST request, the same field is named `baseline_id`. Versions are server-numbered, `display_version` carries whatever you call the snapshot, and versions can be tagged and filtered by tag like sessions.

Immutability is the point. When an experiment run reports "12 of 14 sessions improved," that claim stays checkable because cohort version 3 will always contain exactly those 14 sessions. Re-running the experiment on the same version is an apples-to-apples comparison; adding this week's failures is a new version, and the numbers say which version they came from.

## The lifecycle of a good cohort

The pattern that pays off:

1. **Triage:** a bad run surfaces (a complaint, an alert, an eyeball). You [replay it](/kitaru/core-concepts/replay), understand it, fix it.
2. **Collect the population:** collect the runs like it into a cohort version. `client.sessions.list(...)` with filters, or tags you have been applying along the way, gives you the ids.
3. **Gate on it:** the [experiment](/kitaru/core-concepts/experiments) that verified your fix against that cohort becomes the regression suite that keeps the failure fixed. The cohort that caught the bug is the gate that keeps it caught.

The full workflow, including CI wiring, is in [Build a regression suite from production](/kitaru/guides/regression-suite).


# Experiments

A named change, replayed across a cohort and evaluated. What improved, what regressed, before you ship.

A [replay](/kitaru/core-concepts/replay) is one counterfactual. An **experiment** is that counterfactual at population scale: take a [cohort](/kitaru/core-concepts/cohorts) of real runs, apply one change to all of them, evaluate every re-run with the same [evaluators](/kitaru/core-concepts/evaluators), and read what improved and what regressed.

The split of responsibilities is deliberate:

* The **experiment** holds the *change*: an override (model, prompt, params), a tool policy, and the evaluator list. It is reusable.
* An **experiment run** supplies the *population and the code*: one cohort version and one agent version. Run the same experiment against next week's cohort version, or the same cohort against your PR's agent version.

```python
import asyncio
import os
import uuid

from kitaru.client import KitaruAPIClient
from kitaru.api_models.v1.experiment import ExperimentCreateRequest
from kitaru.api_models.v1.experiment_run import ExperimentRunCreateRequest
from kitaru.api_models.v1.replay_config import (
    EvaluatorConfig,
    HistoryConfig,
    ReplayOverride,
    ToolPolicy,
)


async def main() -> None:
    client = KitaruAPIClient()
    agent_id = uuid.UUID(os.environ["KITARU_AGENT_ID"])

    experiment = await client.experiments.create(
        ExperimentCreateRequest(
            agent_id=agent_id,
            name="cheaper-model",
            description="Would gpt-5-nano have held on refund tickets?",
            override=ReplayOverride(model={"openai:gpt-5.4": "openai:gpt-5-nano"}),
            tool_policy=ToolPolicy(
                default=HistoryConfig(scope="cohort_version", on_miss="fail")
            ),
            evaluators=[EvaluatorConfig(evaluator="refund-check")],
        )
    )

    run = await client.experiments.start_run(
        experiment.id,
        ExperimentRunCreateRequest(
            cohort_version_id=COHORT_VERSION_ID,
            agent_version_id=AGENT_VERSION_ID,
            evaluate_baselines=True,
        ),
    )
    print(run.id, run.status, run.progress)


asyncio.run(main())
```

The same two steps from the CLI (the change as JSON on the experiment, the population and code on the run):

```bash
kitaru experiment create cheaper-model \
  --agent support-agent \
  --evaluator refund-check@latest \
  --override '{"model": {"openai:gpt-5.4": "openai:gpt-5-nano"}}' \
  --tool-policy '{"default": {"type": "history", "scope": "cohort_version", "on_miss": "fail"}}'

kitaru experiment run start cheaper-model \
  --cohort-version <cohort-version-id> \
  --agent support-agent@1 \
  --evaluate-baselines --wait
```

Starting a run fans out **one replay per session** in the cohort version. [Workers](/kitaru/core-concepts/workers) in your environment execute them; the run's `progress` counts replays through `pending → evaluating → completed` (plus `failed` / `canceled`), and the run settles when the last replay does. `evaluate_baselines=True` evaluates the original sessions too, so every replay has its baseline numbers to sit next to.

With a `history` tool policy scoped to `cohort_version`, replayed tool calls can be answered from any recording in the cohort (useful when runs share tool traffic), and `on_miss="fail"` keeps anything unrecorded from reaching a live system.

## Reading a run

A run's output is intentionally plain: its replays, each with a result session, and the evaluation rows on both sides. Compare them by reading the evaluations:

```python
from kitaru.api_models.v1.evaluation import EvaluationListParams
from kitaru.api_models.v1.filter import FilterCondition, FilterOp

async for evaluation in client.evaluations.iter(
    EvaluationListParams(
        filter=FilterCondition(field="session_id", op=FilterOp.EQ, value=session_id)
    )
):
    print(evaluation.name, evaluation.score, evaluation.passed)
```

Numbers average, booleans count into pass rates, categorical labels diff as transitions, and free text gets read. Cost and token totals ([tracked per model call](/kitaru/guides/llm-calls)) ride on each result session, so "the cheaper model held on 18 of 20 tickets and cut cost 41%" is two loops over stored rows. The end-to-end workflow, including gating CI on a frozen cohort version, is in [Build a regression suite from production](/kitaru/guides/regression-suite).

A failed replay fails the run: the comparison the experiment exists for cannot be produced for that session, and the numbers never silently shrink their denominator. Watch a run with `kitaru experiment run watch <run>`, inspect its jobs with `kitaru experiment run jobs <run>`, and cancel with `kitaru experiment run cancel <run>`; already finished replays keep their results.


# Workers

The process that runs replays, imports, and evaluators as subprocesses in your environment, so your code and data never leave your systems.

Nothing in Kitaru executes on the server. Replays, imports, and evaluator runs are **tasks**; a **worker** is the process that claims tasks from the server and runs each one as a subprocess in *your* environment, with *your* virtualenv, credentials, and network. The server coordinates, your infrastructure executes, and session payloads are read from the server your team already hosts.

Start one wherever your agent's code can run:

```bash
kitaru worker start --concurrency 4
```

The worker registers itself, polls for pending tasks, heartbeats while work is in flight, and reports results. Stop it with Ctrl-C: the first signal drains in-flight tasks, a second one exits immediately.

## What a worker executes

| Task kind   | What the subprocess is                                                                                                                                                                                 |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `agent`     | Your agent, started from the [agent version's](/kitaru/core-concepts/agents-and-sessions) run spec command; this is how replays, experiment runs, and on-demand session runs re-execute your real code |
| `evaluator` | A registered [evaluator](/kitaru/core-concepts/evaluators) plugin, run against one session                                                                                                             |
| `importer`  | A registered importer parsing an uploaded trace payload into sessions                                                                                                                                  |

Evaluator and importer plugins declare their own dependencies (PEP 723 inline metadata for script plugins, an exact pin for package plugins), and the worker builds each an isolated environment via `uv`. Agent tasks run your command as-is, in the working directory and environment the agent version declares, plus the [secrets](/kitaru/running-in-production/secrets) it references.

The worker hands each subprocess its context through environment variables: `KITARU_API_URL` and a `KITARU_API_TOKEN`, a bearer token scoped to that one task and attempt, with your broader `KITARU_API_KEY` stripped from the child environment. It also provides `KITARU_TASK_ID` to link the recorded session to the task, and `KITARU_REPLAY_ID` when the run is a replay, which is how the adapter knows to apply overrides and answer tool calls from the recording. The worker itself authenticates once with your API key and holds a worker-scoped token it renews on its own; see [Authentication & API keys](/kitaru/running-in-production/authentication).

## Scoping workers

By default a worker claims any pending task. Narrow it when environments differ:

```bash
# only imports and evaluations; no agent code runs here
kitaru worker start --claim importer --claim evaluator

# only tasks for a specific agent version's environment
kitaru worker start --claim agent=<AGENT_VERSION_ID>

# drain one job, then exit; useful in CI
kitaru worker start --job-id <job-id>
```

Every option is also an environment variable with the `KITARU_WORKER_` prefix (`KITARU_WORKER_CONCURRENCY`, `KITARU_WORKER_SCOPE__CLAIMS`, …), so a containerized worker can be configured without flags. Deployment patterns, including long-running workers on Kubernetes and one-shot workers in CI, are in [Workers in production](/kitaru/running-in-production/workers).

Check what's alive:

```bash
kitaru worker list
kitaru worker get <worker-id>
```

`kitaru worker list` shows live workers, add `--include-stale` for the rest. Names are labels shared by any number of workers, so `kitaru worker get` takes an id from that listing. A worker record exposes `last_seen_at`, the time of its last observed heartbeat, and `live`, the server's current liveness calculation. These are observations, not assignment guarantees: a worker can become unavailable after its last heartbeat, and a live worker may not match a task's scope or win its claim. The native MCP server exposes the same list and exact-UUID get operations through the read-only `kitaru_registry_read` tool. It cannot register, update, delete, or control workers.

A worker that stops heartbeating loses its tasks: the server requeues them for the next worker (or fails them at the retry cap), so a crashed pod never strands a replay.


# Under the Hood

The machinery behind the loop: a FastAPI server on Postgres, workers claiming tasks, and where Kitaru sits beside your observability stack.

You can use Kitaru without reading this page. Read it when you want to know what happens between "start a replay" and "read the diff", or when you are deciding where Kitaru sits in your stack.

## Two processes, one contract

Kitaru is a **server** and your **workers**.

The server is a single FastAPI service backed by Postgres. It stores every resource (agents, sessions and their nodes, cohorts, evaluators, experiments, replays, secrets, tags) and exposes them over a plain versioned REST API (`/api/v1/...`). It coordinates work but executes none of it: there is no code execution on the server, ever.

[Workers](/kitaru/core-concepts/workers) run in your environment and pull work from the server. Everything that executes (a replayed agent, an evaluator, an importer parsing a trace export) runs as a subprocess of a worker, next to your credentials, packages, and network. The server never needs access to your model providers or your tools.

Between them sits the **job/task** layer. Commands like "replay this session," "import this export," or "evaluate these sessions" create a job holding one or more tasks; every job carries its kind (`session_run`, `import`, `evaluation`, `replay`), so `kitaru job` listings filter cleanly. Workers claim tasks scoped by *task* kind (`agent`, `evaluator`, `importer`, a different axis than job kinds) or by label, heartbeat while running them, and report results. Crashed workers lose their claim; the server requeues or fails the task, so no replay is ever silently stranded. `kitaru job watch <id>` follows any of it live.

Writes are safe to retry: the client stamps every POST request with an `Idempotency-Key` header, held stable across the transport's own retries, and the server stores the first committed response for that key, scoped to your account. A replay or evaluation request that times out on the wire and gets retried never becomes two replays: the retry gets the original response back, marked with an `Idempotent-Replayed: true` header instead of running again. Reusing a key with a different request body is rejected with 422. A failed request stores nothing, so a retry after an error re-executes normally. Stored keys expire after `KITARU_SERVER_IDEMPOTENCY_KEY_RETENTION_SECONDS` (15 minutes by default) and are cleared by the same sweep loop that requeues tasks.

## How replay works

1. `POST /api/v1/replays` stores the replay (baseline session, agent version, override, tool policy, evaluators) and creates its job with one agent task.
2. A worker claims the task and starts your agent from the agent version's run spec command, with the baseline's inputs (rewritten by the override, if any) and `KITARU_REPLAY_ID` in the environment.
3. Your agent runs for real. The adapter sees `KITARU_REPLAY_ID`, fetches the override and tool policy, applies model swaps at the model-call boundary, and answers tool calls per policy; a `history` policy looks up the recorded result by a hash of the tool name and arguments.
4. The re-run records a fresh session, node by node, `origin: replay`.
5. When the agent task completes, the server appends one evaluator task per configured evaluator; workers evaluate the result session and, with `evaluate_baselines`, the baseline.
6. The job settles, the replay settles, and, inside an experiment run, the run's progress advances. Results are stored rows: the result session, its nodes, its evaluations.

An [experiment run](/kitaru/core-concepts/experiments) is this pipeline fanned out once per session in a cohort version. Nothing about scale changes the mechanics.

## Storage and blobs

Session payloads (inputs, outputs, node payloads) live in Postgres. Uploaded artifacts (trace exports to import, script plugin code) are **blobs**: content-addressed by SHA-256, deduplicated, capped by a server setting. Workers cache blobs locally by hash, so a hundred evaluator runs fetch the evaluator's code once.

Auth is deliberately simple: [API keys](/kitaru/running-in-production/authentication) (`KITKEY_` prefix) or a login token, one trusted team per deployment. Ownership records who created a resource; it does not gate access. Workers and their task subprocesses never hold your key for long; they operate on short-lived tokens scoped to one worker or one task.

## Where Kitaru sits in your stack

Kitaru is a debugger with a memory, sitting **beside** your observability stack, not replacing it. Langfuse, LangSmith, Braintrust, Logfire, and Arize Phoenix remain your system of record for traces; Kitaru holds runnable copies of the runs you care about and the machinery to re-execute and evaluate them. The [import path](/kitaru/import-your-traces/import-your-traces) is that bridge.

On the other side, Kitaru deliberately does **not** run your production agent. Your agent runs wherever it runs today; the adapter records it. Durable execution of agents in production is [ZenML](https://docs.zenml.io)'s job: ZenML runs agents durably; Kitaru replays and improves them.

Everything here is open source (Apache 2.0) and [self-hosted](/kitaru/getting-started/deploy): your server, your Postgres, your workers, your data.


# Complete returns-agent tutorial

Investigate supplied returns-agent traces and test one evidence-led improvement.

This tutorial applies Kitaru's complete method to a small customer-support agent. The agent looks up orders, return policies, and shipments, then chooses whether to refund, replace, or escalate each request.

You begin with ten recorded PydanticAI sessions exported from Langfuse. The walkthrough does not reveal or use the example's test-only expected outcomes. You will survey the population, inspect complete traces, record your own judgments, define one observable behavior, freeze its reviewed evidence, and test one bounded agent change.

The tutorial is intentionally more detailed than the [Quickstart](/kitaru/getting-started/quickstart). It explains what each resource preserves and why each command is part of the evidence chain. Your exact sessions, questions, evaluator, candidate, and result will depend on what you observe.

## What you will build

| Phase                                              | You will create                                                                                                                                                                                            | Why it exists                                                                       |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| [1. Observe](/kitaru/guides/returns-agent/observe) | A verified [agent version](/kitaru/core-concepts/agents-and-sessions), ten imported [sessions](/kitaru/core-concepts/agents-and-sessions), and descriptive [evaluations](/kitaru/core-concepts/evaluators) | Confirm what the template preserved and select a bounded, varied review worklist.   |
| [2. Judge](/kitaru/guides/returns-agent/judge)     | An [investigation](/kitaru/core-concepts/investigations), evidence-linked annotations, and verdicts                                                                                                        | Store what a human concluded without rewriting the trace.                           |
| [3. Define](/kitaru/guides/returns-agent/define)   | One accepted behavior, an immutable [cohort version](/kitaru/core-concepts/cohorts), and an evaluator version                                                                                              | Turn reviewed evidence into a repeatable measurement.                               |
| [4. Replay](/kitaru/guides/returns-agent/replay)   | A candidate agent version, [experiment](/kitaru/core-concepts/experiments), and experiment run                                                                                                             | Run one bounded change against the frozen population under an explicit tool policy. |
| [5. Compare](/kitaru/guides/returns-agent/compare) | Paired baseline and replay evidence                                                                                                                                                                        | Decide whether the result is improved, regressed, a trade-off, or inconclusive.     |

Each page begins with the same five-step map. The first four phase pages end with a **Checkpoint**, and the final page summarizes the complete evidence chain. Because this is evidence-led, placeholders such as `YOUR_SESSION_UUID` are deliberate: substitute IDs produced by your own review rather than copying a predetermined ticket list.

## Prepare the public template

Install `jq`, then open the public [`kitaru-template` README](https://github.com/zenml-io/kitaru-template#prepare-the-template) and complete its setup through the ten-session confirmation. That README is the source of truth for cloning, the frozen environment, workspace selection, agent registration, worker startup, and the checked-in Langfuse import. Keep running the commands below from the template repository root.

The template uses synthetic customers, orders, shipments, and actions. Refund and replacement tools modify only an isolated in-memory store. No model-provider or Langfuse credentials are needed for setup, import, or the deterministic parts of this tutorial.

Before continuing, confirm these conditions from the template README:

* the selected workspace does not already contain tutorial resources named `returns-resolver`, `returns-discovery`, `returns-regression`, `returns-behavior`, or `returns-candidate`;
* `returns-resolver@1` is registered from the template root;
* ten imported sessions have the `returns-baseline` tag; and
* the template worker remains running in the second terminal.

Stop and select another workspace if those resource names already exist. Do not delete an existing workspace merely to make its names available.

Some tutorial commands create jobs. The [worker](/kitaru/core-concepts/workers) claims those jobs and performs the work in your environment, so the Kitaru server does not receive your agent code or model credentials. Keep the template worker running while you Observe, Judge, and Define. In the Replay phase you will restart it with `OPENAI_API_KEY` before any paid model call.

## Prefer a coding agent?

The pages that follow teach the manual path so you can see each object and boundary. If you want an agent to guide the same evidence loop, install the [Kitaru skills](/kitaru/getting-started/setup) and use the template-specific prompt in the [`kitaru-template` README](https://github.com/zenml-io/kitaru-template#continue-with-a-coding-agent).

## Start the investigation

Continue to [1. Observe the recorded behavior](/kitaru/guides/returns-agent/observe).


# 1. Observe the recorded behavior

Verify the prepared returns evidence and select a bounded worklist for human review.

**Observe** → Judge → Define → Replay → Compare

The first task is factual: confirm what the template preserved and inspect the population before deciding what was right or wrong. By the end of this page, you will have descriptive measurements and a bounded, varied worklist for human review.

## Confirm the prepared evidence

The [`kitaru-template` setup](https://github.com/zenml-io/kitaru-template#prepare-the-template) registered the logical agent `returns-resolver` and assigned its first immutable run specification the reference `returns-resolver@1`. That version stores the command, working directory, timeout, and declared tools Kitaru can use for later replay. Registration did not run the agent.

The setup also imported `traces/langfuse-traces.jsonl` under that exact version. One complete recorded run became a [session](/kitaru/core-concepts/agents-and-sessions); model calls, tool calls, tool results, and other events inside it became session nodes. Importing preserved the evidence and its source identity without calling the historical agent.

Do not repeat registration or import here. If either `returns-resolver@1` or the ten `returns-baseline` sessions is missing, return to the template README and resolve that setup failure before continuing.

## Survey before judging

An [**evaluator**](/kitaru/core-concepts/evaluators) is a reusable measurement. An **evaluation** is one stored result from applying a particular evaluator version to one session.

Run low-cost deterministic evaluators across the population:

```bash
uv run kitaru session evaluate \
  --tag returns-baseline \
  --evaluator kitaru/session-diagnostics@latest \
  --evaluator kitaru/tool-health@latest \
  --evaluator kitaru/trajectory-signals@latest \
  --evaluator kitaru/llm-call-signals@latest \
  --evaluator kitaru/cost@latest \
  --evaluator kitaru/timing-profile@latest \
  --wait

uv run kitaru evaluation list --size 100
```

These evaluators read stored nodes and make no model calls. They can reveal missing data, failed tools, unusual trajectories, model-call patterns, cost, and timing. They cannot decide whether a refund, replacement, or escalation was correct.

Print a compact inventory:

```bash
uv run kitaru --output json session list \
  --tag returns-baseline \
  --origin imported \
  --size 20 \
| jq -r '.items[] | [.id, .name, .status, .outputs.action, .cost, .llm_call_count, .tool_call_count] | @tsv'
```

Select a bounded worklist that you can review carefully. Include different final actions and tool paths, at least one operational outlier, and at least one random session. Summary fields help choose where to look; they are not verdicts.

## Inspect complete traces

Set the UUID of one selected session and inspect every node with its payload:

```bash
SESSION_ID="YOUR_SESSION_UUID"

uv run kitaru session nodes \
  "$SESSION_ID" \
  --include-payloads \
  --size 100
```

Repeat this command for each selected session. Read the input, model decisions, tool inputs, tool results, and final output together. A final response may claim that an action happened while the tool result proves otherwise; a tool failure may explain behavior that looks irrational in the summary.

Record the session UUIDs and any node UUIDs that contain useful evidence. A node ID is an address for a recorded event, not a judgment about that event.

For each session, write down:

| Field            | What to note                                                                    |
| ---------------- | ------------------------------------------------------------------------------- |
| Selection reason | Why this trace belongs in a varied review worklist.                             |
| Open question    | One concrete point that requires human judgment.                                |
| Evidence         | Exact nodes or fields that help answer the question without stating the answer. |

Keep each question neutral and specific to its trace. "Was this handled correctly?" is too generic. "Given the policy result and accepted action shown here, was escalation required?" identifies the decision without supplying its verdict.

## Checkpoint

You now have:

* `returns-resolver@1`, the registered baseline agent version;
* ten imported sessions tagged `returns-baseline`;
* deterministic survey evaluations;
* a bounded, varied worklist chosen from observed evidence; and
* complete trace notes with exact session and node UUIDs.

The agent itself has not run and no model call has occurred. Continue to [2. Judge the selected behavior](/kitaru/guides/returns-agent/judge).


# 2. Judge the selected behavior

Create an investigation and store human judgments against exact trace evidence.

Observe → **Judge** → Define → Replay → Compare

The traces prove what happened, but they do not contain the conclusion that a decision was acceptable or problematic. This phase stores human judgments separately from the raw evidence.

## Plan the review before writing

For every selected session, prepare one distinct question and optional highlights:

| Field            | Requirement                                                                         |
| ---------------- | ----------------------------------------------------------------------------------- |
| Session          | The exact session UUID and its position in the review.                              |
| Selection reason | The evidence-based reason for including it.                                         |
| Question         | One concise, session-specific question that requires human judgment.                |
| Highlights       | Exact nodes or fields that help answer the question without revealing a conclusion. |

The question and highlight descriptions appear beside the trace in the frontend, so they must make sense without this tutorial or your terminal history.

## Create a fixed investigation

An [**investigation**](/kitaru/core-concepts/investigations) stores an ordered review worklist and the questions asked about each session. The following shape uses two sessions; repeat the arguments for your complete selected worklist:

```bash
SESSION_A="YOUR_FIRST_SESSION_UUID"
SESSION_B="YOUR_SECOND_SESSION_UUID"
NODE_A="A_RELEVANT_NODE_UUID"
NODE_B="A_RELEVANT_NODE_UUID"
QUESTION_A="WRITE_A_QUESTION_FROM_SESSION_A_EVIDENCE"
QUESTION_B="WRITE_A_DIFFERENT_QUESTION_FROM_SESSION_B_EVIDENCE"
HIGHLIGHTS_A="[{\"selector\":{\"node_id\":\"$NODE_A\"},\"description\":\"DESCRIBE_WHY_THIS_NODE_IS_RELEVANT\"}]"
HIGHLIGHTS_B="[{\"selector\":{\"node_id\":\"$NODE_B\"},\"description\":\"DESCRIBE_WHY_THIS_NODE_IS_RELEVANT\"}]"

uv run kitaru investigation create returns-discovery \
  --agent returns-resolver \
  --description "Open review of diverse imported returns sessions." \
  --session "$SESSION_A" \
  --session-question "$SESSION_A:observation=$QUESTION_A" \
  --session-highlights "$SESSION_A:observation=$HIGHLIGHTS_A" \
  --session "$SESSION_B" \
  --session-question "$SESSION_B:observation=$QUESTION_B" \
  --session-highlights "$SESSION_B:observation=$HIGHLIGHTS_B"
```

The investigation links to existing sessions; it does not copy or modify their traces. Save the returned investigation UUID and inspect its ordered queue:

```bash
INVESTIGATION_ID="YOUR_INVESTIGATION_UUID"

uv run kitaru investigation session list \
  "$INVESTIGATION_ID" \
  --size 20
```

Three IDs now have different jobs:

| ID                         | What it identifies                                                          |
| -------------------------- | --------------------------------------------------------------------------- |
| Session UUID               | The recorded agent run.                                                     |
| Node UUID                  | One event inside that run.                                                  |
| Investigation-session UUID | That session's place, question, and review state inside this investigation. |

This separation lets one session participate in different investigations without mixing their questions or answers.

## Review in the frontend

Open the agent's **Investigations** page in the workspace selected by `kitaru status`. For a local workspace, open <http://localhost:8000>. The frontend presents each fixed question beside its highlighted trace evidence. Answer the question and choose a whole-session verdict:

* `acceptable`
* `problematic`
* `uncertain`

The answer and verdict have different meanings. An **annotation** stores the substance of the answer and can point to exact evidence. The verdict classifies the complete session. `uncertain` is appropriate when the trace does not contain enough evidence for a complete judgment.

## Or store an annotation with the CLI

An annotation selector can target the entire node, a field inside it, or a character range inside a string. Start with the whole evidence node you inspected:

```bash
INVESTIGATION_SESSION_ID="YOUR_INVESTIGATION_SESSION_UUID"
EVIDENCE_NODE_ID="YOUR_EVIDENCE_NODE_UUID"

uv run kitaru annotation create \
  --investigation-session "$INVESTIGATION_SESSION_ID" \
  --question-key observation \
  --selector "{\"node_id\":\"$EVIDENCE_NODE_ID\"}" \
  --value '"Write your own observation here."'
```

When only one field is evidence, add an RFC 6901 JSON pointer such as `"path":"/outputs/message"`. Add a `span` with `start` and `end` offsets only when a specific character range inside that string supports the answer. Omit the selector when the judgment depends on the complete session.

Store the whole-session verdict separately:

```bash
REVIEWED_SESSION_ID="THE_RECORDED_SESSION_UUID_FOR_THIS_REVIEW_ITEM"

uv run kitaru investigation session verdict \
  "$INVESTIGATION_ID" \
  "$REVIEWED_SESSION_ID" \
  problematic
```

Replace `problematic` with the verdict supported by your review. Do not set a verdict merely to complete the workflow.

## Confirm the persisted review

After reviewing the complete worklist, inspect both answer and verdict coverage:

```bash
uv run kitaru investigation get "$INVESTIGATION_ID"

uv run kitaru annotation list \
  --filter "{\"field\":\"investigation_id\",\"op\":\"eq\",\"value\":\"$INVESTIGATION_ID\"}" \
  --size 100
```

Complete the investigation only when you accept the current evidence boundary:

```bash
uv run kitaru investigation update \
  "$INVESTIGATION_ID" \
  --status completed
```

The investigation status describes the review process. It does not claim that an agent problem has been fixed or that the reviewed sample represents all traffic.

## Checkpoint

You now have:

* a fixed `returns-discovery` review worklist;
* one neutral, trace-specific question per selected session;
* persisted annotations linked to relevant evidence;
* explicit whole-session verdicts where the evidence supported them; and
* an accepted boundary around what the review did and did not establish.

No agent or model has run. Continue to [3. Define one behavior to test](/kitaru/guides/returns-agent/define).


# 3. Define one behavior to test

Turn accepted human evidence into one frozen cohort and evaluator version.

Observe → Judge → **Define** → Replay → Compare

A verdict says what a reviewer concluded about one complete session. A repeatable test needs a more precise behavior definition, a frozen population, and a measurement that reads observable trace evidence.

## Accept one observable behavior

Use only the persisted annotations and confirmed verdicts from your investigation. Write one binary definition that answers:

1. Under which observable conditions does the behavior matter?
2. Which recorded agent action passes?
3. Which recorded agent action fails?
4. Which tool or external outcome evidence is required?
5. What result should the evaluator return when evidence is missing?
6. Which reviewed counterexamples limit the definition?

For example, "the agent should handle refunds correctly" is too broad. A usable definition names the required recorded conditions and distinguishes an accepted action from a claim in the final response.

Keep agent behavior separate from a tool or provider failure. If a trace lacks the external evidence required to judge an outcome, record that uncertainty instead of turning absence into a pass.

## Freeze the reviewed population

A [**cohort**](/kitaru/core-concepts/cohorts) is a named population of sessions. A **cohort version** freezes one exact membership list so later experiment runs use the same evidence.

Before creating it, list the exact reviewed target cases that exercise the behavior you want to change and the reviewed counterexamples that could expose overcorrection. Confirm the membership, then create the cohort:

```bash
uv run kitaru cohort create returns-regression \
  --agent returns-resolver \
  --description "Human-reviewed sessions for one accepted returns behavior." \
  --display-version initial-review \
  --session YOUR_REVIEWED_SESSION_UUID \
  --session YOUR_COUNTEREXAMPLE_SESSION_UUID
```

Verify the immutable version and its members:

```bash
uv run kitaru cohort version get returns-regression@1
uv run kitaru session list --cohort returns-regression@1 --size 20

COHORT_REFERENCE="returns-regression@1"
```

The cohort should contain only sessions whose role in this behavior is supported by the review. Testing only problematic sessions can make a blunt change look successful. Counterexamples test whether nearby behavior that was already acceptable remains acceptable.

Create a new cohort version when membership changes. Existing versions remain unchanged. Set `COHORT_REFERENCE` to the exact accepted version before continuing.

## Select or create an evaluator

Inspect the installed [evaluator](/kitaru/core-concepts/evaluators) catalog before writing code:

```bash
uv run kitaru evaluator list
```

Use an installed evaluator when it expresses the accepted behavior. Pin its exact version and parameters, then save the reference for the remaining pages:

```bash
BEHAVIOR_EVALUATOR="NAME@VERSION"
```

If no installed evaluator fits, scaffold a narrow deterministic evaluator:

```bash
uv run kitaru evaluator scaffold \
  returns-behavior \
  --path evaluator.py
```

Replace the scaffold with code that implements the behavior you accepted during review. The following generic example demonstrates the `SessionView` and `EvaluationResult` contracts by checking whether one accepted terminal tool call agrees with the final structured action:

```python
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""Evaluate consistency between an accepted action and the final output."""

from typing import Any

from kitaru.api_models.v1.evaluation import EvaluationResult
from kitaru.api_models.v1.session_node import NodeType
from kitaru.task.evaluator import SessionView

ACTION_BY_TOOL = {
    "issue_refund": "refund",
    "create_replacement": "replacement",
    "escalate_to_human": "escalate",
}


def _get_outputs(value: Any) -> dict[str, Any] | None:
    """Return final outputs from a native or imported session."""
    if isinstance(value, dict) and isinstance(value.get("turns"), list):
        turns = value["turns"]
        value = turns[-1].get("outputs") if turns else None
    return value if isinstance(value, dict) else None


def evaluate(session: SessionView) -> EvaluationResult:
    """Check that one accepted terminal tool matches the final action."""
    accepted_tools = [
        node.tool_name
        for node in session.nodes
        if node.node_type is NodeType.TOOL_CALL
        and node.tool_name in ACTION_BY_TOOL
        and isinstance(node.outputs, dict)
        and node.outputs.get("accepted") is True
    ]
    outputs = _get_outputs(session.session.outputs)

    if not accepted_tools or outputs is None:
        return EvaluationResult(
            name="terminal_action_consistency",
            value="unknown",
            passed=None,
            explanation="The trace does not contain enough recorded action evidence.",
        )

    if len(accepted_tools) != 1:
        return EvaluationResult(
            name="terminal_action_consistency",
            value="fail",
            passed=False,
            explanation=f"The trace contains {len(accepted_tools)} accepted actions.",
        )

    accepted_action = ACTION_BY_TOOL[accepted_tools[0]]
    reported_action = outputs.get("action")
    passed = reported_action == accepted_action
    return EvaluationResult(
        name="terminal_action_consistency",
        value="pass" if passed else "fail",
        passed=passed,
        explanation=(
            f"Accepted action: {accepted_action!r}; "
            f"reported action: {reported_action!r}."
        ),
    )
```

This example uses structured output and recorded tool results. It does not search the customer reply for words such as `refund`, and it does not map ticket IDs to expected answers. Adapt the rule, required evidence, and missing-evidence result to the behavior you confirmed during review.

If you want coding-agent help, ask it to implement only the accepted behavior from the persisted investigation and show you how each branch follows from recorded evidence. Tell it not to read or use the example's test-only expected outcomes. Review the resulting code before registering it.

Do not map ticket or session identifiers to expected answers. Do not search the customer reply for words such as `refund` when tool results provide stronger evidence. A useful evaluator distinguishes, for example, an accepted refund from a claimed refund, multiple accepted terminal actions from one, and missing action evidence from a pass.

Validate and register the implementation:

```bash
uv run kitaru evaluator test \
  evaluator.py \
  --entrypoint evaluate

uv run kitaru evaluator register \
  returns-behavior \
  --script evaluator.py \
  --entrypoint evaluate \
  --description "Evaluate one human-reviewed returns behavior from trace evidence." \
  --display-version initial-review
```

Kitaru assigns the first version the reference `returns-behavior@1`. The version pins the evaluator code and parameters used by later comparisons.

Save that reference:

```bash
BEHAVIOR_EVALUATOR="returns-behavior@1"
```

## Calibrate against human evidence

Apply the evaluator to the frozen baseline cohort:

```bash
uv run kitaru session evaluate \
  --cohort "$COHORT_REFERENCE" \
  --evaluator "$BEHAVIOR_EVALUATOR" \
  --wait

uv run kitaru evaluation list --size 100
```

Compare each evaluation with the investigation's annotations and verdicts. Report agreement, disagreement, and unknown results. A script that loads successfully is not necessarily a valid measurement, and agreement on a small reviewed sample does not make the evaluator production-ready.

When the evaluator disagrees with a human judgment, inspect the trace and the rule. The correct response may be to fix the evaluator, refine the behavior, mark the case uncertain, or create a new cohort version. Register changed evaluator code as a new version, then update `BEHAVIOR_EVALUATOR`. Update `COHORT_REFERENCE` whenever you accept a newer cohort version. Do not change the expected label merely to make the metric pass.

## Checkpoint

You now have:

* one precise behavior accepted from persisted human evidence;
* `COHORT_REFERENCE`, set to the exact accepted cohort version;
* `BEHAVIOR_EVALUATOR`, set to the exact installed or custom evaluator version; and
* baseline evaluations checked against the human review.

No agent or model has run yet. Continue to [4. Replay one bounded change](/kitaru/guides/returns-agent/replay).


# 4. Replay one bounded change

Register one investigation-derived candidate and replay the frozen cohort safely.

Observe → Judge → Define → **Replay** → Compare

This is the first phase that runs the agent and can make paid model calls. You will make one change justified by the investigation, register its run specification, review tool safety, and replay the frozen cohort.

## Make one investigation-derived change

Change `returns_agent/agent.py` only after the review has identified one behavior worth changing. Keep the candidate narrow enough that you can explain how it is expected to affect the evaluator and counterexamples.

The example does not include a prewritten candidate or environment switch. That is intentional: the candidate should follow from the behavior you accepted, not from a hidden fixture answer key. If you want coding-agent help, ask it for the smallest code change that implements only that behavior, require it to explain the expected effect on every reviewed target and counterexample, and review the patch before registering it.

Record the source revision or working-tree state you intend the worker to execute. Then register version 2:

```bash
uv run kitaru agent version register \
  returns-resolver \
  --command "python -m returns_agent.agent" \
  --description "Test one investigation-derived behavior change." \
  --display-version candidate-v1 \
  --working-dir . \
  --timeout-seconds 180 \
  --tool lookup_order \
  --tool get_return_policy \
  --tool check_shipping \
  --tool issue_refund \
  --tool create_replacement \
  --tool escalate_to_human
```

Registration does not run the agent. It creates `returns-resolver@2`, an immutable run specification. The specification does not snapshot a mutable `--working-dir`, so reproducibility also requires the worker to use the intended checkout, commit, or container image.

Save the exact candidate reference:

```bash
CANDIDATE_AGENT="returns-resolver@2"
```

## Give the worker model credentials

The agent uses `openai:gpt-5-nano`. Each replay may make more than one paid OpenAI API request.

In Terminal 2, stop the existing worker with `Ctrl-C`. Export the key in that same shell, then restart the worker:

```bash
printf 'OpenAI API key: '
IFS= read -r -s OPENAI_API_KEY
printf '\n'
export OPENAI_API_KEY
uv run kitaru worker start --name kitaru-template-worker
```

Restarting matters because the worker launches the registered agent as a subprocess. A running process cannot inherit environment variables added to another terminal later.

{% hint style="warning" %}
The commands below create remote Kitaru resources and paid model calls. Confirm the candidate version, cohort membership, evaluator versions, and expected number of replays before starting the run.
{% endhint %}

## Choose the replay tool policy

When agent code runs again, its tools need an explicit relationship to the outside world. The tool policy determines whether a call uses recorded history, a static result, or the live tool.

This synthetic example uses passthrough:

```json
{"default":{"type":"passthrough"},"tools":{}}
```

Passthrough is safe here because every action tool writes only to a fresh in-memory store created for the replay. Do not copy this choice for tools that charge cards, send messages, change production data, or trigger other side effects. For those, prefer recorded history with `on_miss=fail` or a reviewed static result. See [Replay and overrides](/kitaru/guides/replay-and-overrides).

Replay safety comes from the configured policy and the actual tool implementations, not from the word "replay." Review both before starting the run.

## Create the experiment

An [**experiment**](/kitaru/core-concepts/experiments) fixes the replay configuration and evaluator versions for an agent. An **experiment run** supplies the exact candidate agent version and immutable cohort version.

Create an experiment with the accepted behavior evaluator and operational measurements:

```bash
uv run kitaru experiment create \
  returns-candidate \
  --agent returns-resolver \
  --description "Test one accepted behavior change against the reviewed cohort." \
  --tool-policy '{"default":{"type":"passthrough"},"tools":{}}' \
  --evaluator "$BEHAVIOR_EVALUATOR" \
  --evaluator kitaru/tool-health@latest \
  --evaluator kitaru/timing-profile@latest
```

The experiment fixes the agent parent, tool policy, and evaluator versions. Reusing it does not by itself preserve the candidate code or population; each run supplies those exact versions.

Resolve the cohort-version UUID:

```bash
COHORT_VERSION_ID="$(
  uv run kitaru --output json cohort version get "$COHORT_REFERENCE" \
  | jq -r '.item.id'
)"
```

Before continuing, inspect `$COHORT_REFERENCE` again and count its members. The run creates one replay per cohort session.

## Start the bounded run

```bash
uv run kitaru experiment run start \
  returns-candidate \
  --cohort-version "$COHORT_VERSION_ID" \
  --agent "$CANDIDATE_AGENT" \
  --evaluate-baselines \
  --wait \
  --timeout 1800
```

Save the experiment-run UUID printed in the receipt:

```bash
RUN_ID="YOUR_EXPERIMENT_RUN_UUID"
```

The worker launches the candidate command once for each cohort session and stores every new run as a session with `origin: replay`. `--evaluate-baselines` applies the same evaluator versions to both the imported sessions and their replays. Without it, you would have candidate measurements but no like-for-like baseline.

`--timeout-seconds 180` limits each agent subprocess. `--timeout 1800` limits how long the CLI waits for the complete experiment run.

If a replay fails or times out, keep it in the denominator. A completed subset is not the complete experiment result.

## Checkpoint

You now have:

* `CANDIDATE_AGENT`, set to the registered candidate run specification;
* `returns-candidate`, the experiment definition;
* `RUN_ID`, identifying one experiment run over `$COHORT_REFERENCE`; and
* explicit terminal states for every attempted replay, with like-for-like evaluations for completed pairs and preserved failures for incomplete pairs.

The worker has made paid model requests. Continue to [5. Compare the paired evidence](/kitaru/guides/returns-agent/compare).


# 5. Compare the paired evidence

Compare baseline and replay evidence and state the result at the size the cohort supports.

Observe → Judge → Define → Replay → **Compare**

A conclusive improvement or regression claim requires every expected original-and-replay comparison under the same evaluator versions. An incomplete run is still useful evidence for diagnosing an inconclusive result. In this final phase, you will inspect run health, read the available pairs, preserve failures and missing results, and state the narrow conclusion supported by your reviewed cohort.

## Confirm the run completed

List experiment runs and inspect the exact run receipt:

```bash
uv run kitaru experiment run list --size 20
uv run kitaru experiment run get "$RUN_ID"
uv run kitaru experiment run jobs "$RUN_ID" --size 100
```

Confirm that the run attempted every member of `$COHORT_REFERENCE` and that every expected replay has an explicit terminal state. A failed, canceled, or missing replay makes the result incomplete. Do not silently remove it and reduce the denominator.

## Inspect replay sessions and evaluations

```bash
uv run kitaru session list \
  --agent returns-resolver \
  --origin replay \
  --size 20

uv run kitaru evaluation list --size 100
```

For every cohort member, put the imported baseline and replay beside one another. Compare:

* the result from `$BEHAVIOR_EVALUATOR`;
* the accepted terminal tool calls and their results;
* the final structured output;
* tool-health and timing measurements;
* cost and token use; and
* replay, evaluation, or job failures.

Open <http://localhost:8000> to inspect paired traces. The evaluator tells you whether its encoded rule passed. The trace shows how the agent reached the outcome and whether the measurement missed important evidence.

## Classify each transition

Use the human-reviewed role of each session when interpreting the transition:

| Baseline     | Replay             | Interpretation to investigate                                                        |
| ------------ | ------------------ | ------------------------------------------------------------------------------------ |
| Fail         | Pass               | The target case may have improved. Check the trace and operational measurements.     |
| Pass         | Pass               | The reviewed behavior was preserved for this case. Check for other regressions.      |
| Pass         | Fail               | The candidate regressed on this reviewed case.                                       |
| Fail         | Fail               | The candidate did not fix this case, or the evaluator still lacks required evidence. |
| Known result | Unknown or missing | The comparison is inconclusive for this case.                                        |

Do not force every change into pass or fail. A candidate can improve the primary behavior while increasing tool failures, latency, or cost enough to create a real trade-off.

## State the conclusion at the right size

Use one overall evidence conclusion:

| Conclusion       | What the evidence says                                                                            | What to do next                                                                 |
| ---------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| **Improved**     | Target cases improved and reviewed counterexamples remained acceptable.                           | Expand the reviewed population or preserve this cohort as a regression check.   |
| **Regressed**    | A target or counterexample became worse.                                                          | Inspect the paired traces, revise the change, and register a new agent version. |
| **Trade-off**    | One important measure improved while another became worse.                                        | Decide whether the trade-off is acceptable or change the candidate.             |
| **Inconclusive** | A replay failed, required evidence is missing, or the population cannot support the needed claim. | Repair execution or add reviewed evidence before deciding.                      |

Your statement should name the exact cohort and behavior. A defensible form is:

> On `$COHORT_REFERENCE`, candidate `$CANDIDATE_AGENT` \[improved, regressed, traded off, or produced inconclusive evidence for] the reviewed behavior measured by `$BEHAVIOR_EVALUATOR`. This result applies to the frozen reviewed sessions; it does not establish general safety or production readiness.

The ten supplied traces are a small synthetic population. Your selected worklist is also adaptive: you chose it partly because the traces looked interesting. Do not infer production prevalence or general agent quality from this experiment.

## If the result is inconclusive

Inconclusive is not a near-pass. Preserve the reason:

* If a replay failed, inspect its child job and rerun only after correcting the execution problem.
* If tool evidence is missing, change the replay policy or instrumentation rather than guessing the outcome.
* If the evaluator is wrong, register a new evaluator version and apply it consistently to both sides.
* If the cohort lacks a necessary counterexample, create a new cohort version with reviewed membership.

Keep the old versions. Their immutability makes it possible to explain why two experiment runs reached different conclusions.

## Optional: generate fresh traces

The supplied export makes setup repeatable, but its model outputs are not an answer key. To create a new export, create `.env` in the example directory with valid `OPENAI_API_KEY`, `LANGFUSE_PUBLIC_KEY`, and `LANGFUSE_SECRET_KEY`, then run:

```bash
./generate.sh
```

The script makes ten paid agent runs, waits for the Langfuse observations, and replaces `traces/langfuse-traces.jsonl`. Model behavior varies. Import the new file, inspect what actually happened, and build a new review worklist from that evidence.

For your own agent, keep collecting traces where you already collect them and use [Import your traces](/kitaru/import-your-traces/import-your-traces) to select the matching importer. Historical investigation does not require the original code to remain runnable. Replay does require a compatible registered candidate and a worker that can execute it.

## Clean up

Stop the worker in Terminal 2 with `Ctrl-C`, then disconnect the CLI:

```bash
uv run kitaru logout
```

For a CLI-managed local workspace, logout stops its containers but keeps the PostgreSQL data volume.

## What you completed

You followed the full evidence chain:

1. **Observed** a trace population before assigning labels.
2. **Judged** selected sessions and stored human reasoning beside exact evidence.
3. **Defined** one behavior with a frozen reviewed cohort and evaluator version.
4. **Replayed** one candidate under an explicit tool policy.
5. **Compared** complete baseline and replay evidence without hiding failures or uncertainty.

The durable result is not a predetermined passing demo. It is an auditable claim about one reviewed behavior, one frozen population, and one candidate.

## Where to go next

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Use kitaru-investigation</strong></td><td>Apply this method to the agent in your own repository.</td><td><a href="/pages/UPvRjygShZEG0jzCwMXc">/pages/UPvRjygShZEG0jzCwMXc</a></td></tr><tr><td><strong>Build a regression suite</strong></td><td>Grow reviewed evidence into a reusable comparison.</td><td><a href="/pages/ktnQrV1qibhxXwZEZKDn">/pages/ktnQrV1qibhxXwZEZKDn</a></td></tr><tr><td><strong>Replay and overrides</strong></td><td>Control models, tools, history, and replay safety.</td><td><a href="/pages/ieoi9kJkRqHsTAPEUquq">/pages/ieoi9kJkRqHsTAPEUquq</a></td></tr><tr><td><strong>Write an evaluator</strong></td><td>Design and calibrate a domain-specific evaluator.</td><td><a href="/pages/uM42ovPcGiVqzN1FYyko">/pages/uM42ovPcGiVqzN1FYyko</a></td></tr></tbody></table>


# Replay a failure and fork it

Reproduce a recorded session, fork it with an override, and compare the two, so you can debug a failure or test a change before shipping it.

Replay re-executes a recorded [session](/kitaru/core-concepts/agents-and-sessions) to produce a **new session**: the same run, with exactly the changes you specify. One mechanism covers two jobs:

* **Debug a failure.** A production run went wrong. Replay it unchanged and you have the failure on your desk, reproducible without touching production.
* **Test a change.** You want to swap the model, tighten the prompt, or ship the code in your working tree. Fork the run with that one change and read what it did.

This guide assumes you prepared the public [`kitaru-template`](https://github.com/zenml-io/kitaru-template) and completed the setup and Define phases of the [returns-agent tutorial](/kitaru/guides/returns-agent): a registered agent with a run command, a registered [evaluator](/kitaru/core-concepts/evaluators), and a [worker](/kitaru/core-concepts/workers) running in the agent's environment.

## Create the one-off replay from the CLI

Create an unchanged reproduction with an explicit recorded-history policy:

```bash
kitaru replay create <baseline-session-id> \
  --evaluator refund-check@1 \
  --tool-policy '{"default":{"type":"history","scope":"baseline","on_miss":"fail"}}' \
  --evaluate-baselines --output json
```

The JSON result contains the replay ID and job ID. The command queues work and returns; it does not wait for the worker:

```bash
kitaru job watch <job-id>
kitaru replay get <replay-id> --output json
```

Use `kitaru job get <job-id> --tasks` for task errors and `kitaru job cancel <job-id>` to request cancellation. `kitaru replay list --output json` lists both standalone and experiment-created replays.

{% hint style="warning" %}
The SDK's automatic retries (timeouts, dropped connections, 5xx) are safe: they reuse the same idempotency key, so a retried create settles as at most one replay. Manually re-running \`kitaru replay create\` after an ambiguous failure is not, since each invocation mints its own key. Check \`kitaru replay list\` before retrying by hand, or the manual retry may create a duplicate replay and job. Omitting \`--tool-policy\` uses the server default and may execute live tools.
{% endhint %}

## The three-session discipline

Every trustworthy comparison involves three sessions:

1. **Observed:** the original recording (recorded or imported).
2. **Reproduced:** an unchanged replay of it. If this does not hold up, because evaluations disagree or the path is wildly different, stop. Your run depends on something the recording does not answer, such as live tool traffic or nondeterminism you have not pinned, and no fork from it can be trusted.
3. **Forked:** the replay with one thing changed. Because the baseline reproduced, the difference between it and the fork is your change.

## Anatomy of a replay

```python
import asyncio
from kitaru.client import KitaruAPIClient
from kitaru.api_models.v1.replay import ReplayCreateRequest
from kitaru.api_models.v1.replay_config import (
    EvaluatorConfig,
    HistoryConfig,
    ReplayOverride,
    ToolPolicy,
)

RECORDED_TOOLS = ToolPolicy(default=HistoryConfig(scope="baseline", on_miss="fail"))


async def main() -> None:
    client = KitaruAPIClient()
    replay = await client.replays.create(
        ReplayCreateRequest(
            baseline_session_id=SESSION_ID,
            # agent_version_id: defaults to the version the baseline recorded
            override=ReplayOverride(model={"openai:gpt-5.4": "openai:gpt-5-nano"}),
            tool_policy=RECORDED_TOOLS,
            evaluators=[EvaluatorConfig(evaluator="refund-check")],
            evaluate_baselines=True,
        )
    )
    print(replay.id, replay.job_id)


asyncio.run(main())
```

Field by field:

* `baseline_session_id`: the recording to re-run.
* `agent_version_id`: which code runs. Omitted, it is the version the baseline was recorded with, which is the faithful choice. Point it at a newly registered version to replay old traffic **against your working tree**.
* `override`: the fork. Omit it for a pure reproduction.
* `tool_policy`: what tool calls hit. If omitted, the server applies its default, which currently passes calls through to live tools. For a replay that touches nothing real, set a `history` policy as above. Details in [Tool policies](/kitaru/guides/tool-policies).
* `evaluators`: at least one, always. A replay is evaluated on arrival.
* `evaluate_baselines`: evaluate the original session with the same evaluators, so the comparison exists as soon as the replay settles.

Watch it with `kitaru job watch <job-id>`; when the replay reads `completed`, `client.replays.get(replay.id)` carries the `result_session_id`.

## Overrides

One `ReplayOverride`, four knobs; change one at a time:

| Field           | Effect                                                                                                          |
| --------------- | --------------------------------------------------------------------------------------------------------------- |
| `model`         | Swap models at the model-call boundary. A string replaces every model; a `{old: new}` map replaces selectively. |
| `system_prompt` | Replace the system prompt for the re-run.                                                                       |
| `prompt`        | Replace the user prompt to ask a different question of the same recorded world.                                 |
| `model_params`  | Adjust sampling parameters (temperature, etc.) at the adapter level.                                            |

Code changes need no override at all: register the new code as an agent version and pass its `agent_version_id`. Replays run **from the top**: the whole agent re-executes against the recorded world, so the entire decision path downstream of your change is real.

{% hint style="warning" %}
With the default \`passthrough\` policy, a replayed \`refund\_payment\` call \*\*refunds the card again\*\*. Set a \`history\` policy for anything with side effects, or use \`static\` to inject a canned result. If your agent must behave differently under replay, check for the \`KITARU\_REPLAY\_ID\` environment variable; it is set only in replayed runs.
{% endhint %}

## Reading the comparison

Both sides are sessions with evaluations. Read them together:

```python
from kitaru.api_models.v1.evaluation import EvaluationListParams
from kitaru.api_models.v1.filter import FilterCondition, FilterOp


async def evaluations_for(client, session_id):
    return {
        e.name: e
        async for e in client.evaluations.iter(
            EvaluationListParams(
                filter=FilterCondition(
                    field="session_id", op=FilterOp.EQ, value=session_id
                )
            )
        )
    }


baseline_evals = await evaluations_for(client, baseline_session_id)
fork_evals = await evaluations_for(client, result_session_id)
for name, b in baseline_evals.items():
    f = fork_evals.get(name)
    print(name, "baseline:", b.score, "fork:", f.score if f else "-")
```

Session rollups carry the operational deltas (`cost`, `tokens`, `llm_call_count`, `tool_call_count`), so "same pass rate, 40% cheaper, one extra model call" is three field reads. For node-level inspection, `list_nodes(include_payloads=True)` on both sessions shows where the paths diverged.

## When a replay fails

A replay settles `failed` when its pipeline cannot produce the comparison: the agent process exited nonzero, a tool call missed under `on_miss="fail"`, or an evaluator crashed. The job's tasks carry the error and a log tail; `kitaru job get <job-id>` and `kitaru job watch` surface them, and [Troubleshooting](/kitaru/get-help/troubleshooting) walks the diagnosis. The common causes:

* **No run spec:** the agent version must carry a run command; registering with `--command` is what makes a session replayable.
* **Worker environment:** the subprocess needs your agent's dependencies and provider keys; it inherits them from the worker's environment.
* **Unrecorded tool call:** the fork took a path the baseline never took. That is information: widen the history scope, add a `static` case for it, or accept `error_result` and let the agent handle it.

## From one replay to many

The same request against many sessions is a [cohort](/kitaru/core-concepts/cohorts) plus an [experiment](/kitaru/core-concepts/experiments): one replay per session, fanned out and evaluated identically. That is the subject of [Build a regression suite from production](/kitaru/guides/regression-suite).


# Build a regression suite from production

Freeze a cohort of recorded runs, replay it against a change, and use the results in CI.

Replaying a change against one session shows how it affects that case. A regression suite repeats the comparison across a fixed set of recorded or imported sessions. These sessions complement synthetic fixtures: they preserve inputs and behavior seen in real runs, while synthetic cases can cover conditions that have not happened in production.

This guide selects a population, freezes it as a cohort version, defines a change as an experiment, and runs that experiment in CI.

## 1. Select the population

Pick sessions that cover important behavior and known failures. You can filter by agent, status, or time, or start with sessions linked to a specific incident:

```python
import asyncio
from kitaru.client import KitaruAPIClient
from kitaru.api_models.v1.filter import FilterCondition, FilterOp
from kitaru.api_models.v1.session import SessionListParams


async def main() -> None:
    client = KitaruAPIClient()
    refund_runs = [
        s.id
        async for s in client.sessions.iter(
            SessionListParams(
                filter=FilterCondition(
                    field="agent_id", op=FilterOp.EQ, value=AGENT_ID
                ),
                size=100,
            )
        )
    ][:50]
```

A useful starting point is a recent sample of traffic plus sessions linked to past incidents. [Imported sessions](/kitaru/import-your-traces/import-langfuse-traces) work like recorded sessions. If you tagged an import, use that tag to select it (`kitaru session list --tag imported-baseline`). You can select directly recorded sessions with `--agent` or `--filter`.

## 2. Freeze it into a cohort version

`cohort create` accepts a session selection through `--tag`, `--session`, `--sessions-file`, or `--filter`. It stores the matching sessions as version 1. Here, `--agent` names the agent that owns the cohort; it does not select sessions:

```bash
kitaru cohort create refund-regression --agent support-agent \
  --tag imported-baseline --display-version week-32
```

The client can create the cohort and its first version from the selection above:

```python
from kitaru.api_models.v1.cohort import CohortCreateRequest
from kitaru.api_models.v1.cohort_version import CohortVersionCreateRequest

cohort = await client.cohorts.create(
    CohortCreateRequest(name="refund-regression", agent_id=AGENT_ID)
)
version = await client.cohorts.create_version(
    cohort.id,
    CohortVersionCreateRequest(add_session_ids=refund_runs, display_version="week-32"),
)
```

[Cohort versions are immutable](/kitaru/core-concepts/cohorts). Version 1 keeps the same 50 sessions. To add or remove sessions, create a new version so later comparisons show that the population changed.

## 3. Make the change an experiment

The experiment holds everything about the change *except* the population:

```python
import os
import uuid

from kitaru.api_models.v1.experiment import ExperimentCreateRequest
from kitaru.api_models.v1.replay_config import (
    EvaluatorConfig,
    HistoryConfig,
    ReplayOverride,
    ToolPolicy,
)

experiment = await client.experiments.create(
    ExperimentCreateRequest(
        agent_id=uuid.UUID(os.environ["KITARU_AGENT_ID"]),
        name="cheaper-model",
        override=ReplayOverride(model={"openai:gpt-5.4": "openai:gpt-5-nano"}),
        tool_policy=ToolPolicy(
            default=HistoryConfig(scope="cohort_version", on_miss="fail")
        ),
        evaluators=[
            EvaluatorConfig(evaluator="refund-check"),
            EvaluatorConfig(evaluator="tone-judge"),
        ],
    )
)
```

Both evaluators must already be registered. In this example, `tone-judge` represents a second evaluator written for your application. See [Write an evaluator](/kitaru/guides/write-an-evaluator).

To test a code change, omit `override` and register the branch as a new agent version. The experiment run selects that version. The `history` policy with `scope="cohort_version"` can answer tool calls from any recording in the cohort. With `on_miss="fail"`, an unmatched call stops its replay instead of reaching the live tool.

The CLI form takes the override and tool policy as JSON:

```bash
kitaru experiment create cheaper-model \
  --agent support-agent \
  --evaluator refund-check@latest --evaluator tone-judge@latest \
  --override '{"model": {"openai:gpt-5.4": "openai:gpt-5-nano"}}' \
  --tool-policy '{"default": {"type": "history", "scope": "cohort_version", "on_miss": "fail"}}'
```

## 4. Run it and read it

If the candidate version does not exist yet, register it with `kitaru agent version register`. The following example uses `support-agent@2`:

```bash
kitaru experiment run start cheaper-model \
  --cohort-version <cohort-version-id> \
  --agent support-agent@2 \
  --evaluate-baselines \
  --wait --timeout 1800
```

Or from the client:

```python
from kitaru.api_models.v1.experiment_run import ExperimentRunCreateRequest

run = await client.experiments.start_run(
    experiment.id,
    ExperimentRunCreateRequest(
        cohort_version_id=version.id,
        agent_version_id=AGENT_VERSION_ID,  # e.g. your PR's registered version
        evaluate_baselines=True,
    ),
)
```

Workers create one replay task per session, and `run.progress` reports how many have finished. When the run settles, each replay has a result session. If `evaluate_baselines=True`, the baseline and result sessions both have evaluations. The example below compares boolean pass results:

```python
from kitaru.api_models.v1.evaluation import EvaluationListParams
from kitaru.api_models.v1.filter import FilterCondition, FilterOp
from kitaru.api_models.v1.replay import ReplayListParams

replays = [
    r
    async for r in client.replays.iter(
        ReplayListParams(
            filter=FilterCondition(
                field="experiment_run_id", op=FilterOp.EQ, value=run.id
            )
        )
    )
]


async def passed(session_id, name="refund_issued"):
    async for e in client.evaluations.iter(
        EvaluationListParams(
            filter=FilterCondition(field="session_id", op=FilterOp.EQ, value=session_id)
        )
    ):
        if e.name == name:
            return e.passed
    return None


baseline_pass = [await passed(r.baseline_session_id) for r in replays]
fork_pass = [await passed(r.result_session_id) for r in replays]
print(f"baseline: {sum(filter(None, baseline_pass))}/{len(replays)}")
print(f"fork:     {sum(filter(None, fork_pass))}/{len(replays)}")
```

You can also aggregate cost from the result sessions' rollups. Report both the summary and the underlying failures, for example: *"gpt-5-nano passed 47 of 50 refund tickets and reduced recorded cost by 41%. The failed cases were sessions 12, 19, and 44."* You can then [replay and inspect](/kitaru/guides/replay-and-overrides) each failed session.

## 5. Gate on it

To use the experiment in CI, register the pull request's code as an agent version and start a run against the frozen cohort version:

```bash
kitaru agent version register support-agent --command "python support.py"
kitaru experiment run start cheaper-model \
  --cohort-version <cohort-version-id> \
  --agent support-agent@<new-version> \
  --evaluate-baselines --wait --timeout 1800
```

`--wait` blocks until the run settles and exits nonzero if it fails, so the CI job can use the command as a gate. `--output jsonl` streams progress in a machine-readable format. A long-running worker pool can execute the suite, or the CI job can start a worker with `kitaru worker start`.

A practical setup uses a small cohort for pull requests and a larger traffic sample for scheduled runs. When you find a new failure, add its session to a new cohort version. Future runs will then include that case, although the evaluator still needs to detect the behavior for the CI gate to catch it.


# Write an evaluator

Turn your domain expert's criteria into a versioned evaluator: code checks, LLM judges, human calibration, and backfilling your history.

Your domain expert already knows what a good run looks like. An [evaluator](/kitaru/core-concepts/evaluators) is that knowledge as code: a small Python callable that reads one recorded session and writes named, typed verdicts. This guide takes you from criteria to a registered, calibrated evaluator you can trust in a release gate.

## From criteria to code

Start from what the expert says. "A good refund resolution issues exactly one refund, quotes the amount, and does not promise anything we do not do" is three checks:

```bash
kitaru evaluator scaffold refund-quality
```

```python
# refund_quality_evaluator.py
from kitaru.task.evaluator import EvaluationResult, SessionView


def evaluate(session: SessionView, **params) -> list[EvaluationResult]:
    refunds = [
        n
        for n in session.nodes
        if n.node_type == "tool_call" and n.tool_name == "refund_payment"
    ]
    reply = str(session.session.outputs or "")

    return [
        EvaluationResult(
            name="single_refund",
            score=len(refunds) == 1,
            passed=len(refunds) == 1,
            explanation=f"{len(refunds)} refund call(s)",
        ),
        EvaluationResult(
            name="amount_quoted",
            score="$" in reply,
            passed="$" in reply,
        ),
    ]
```

`SessionView` is the whole recording: `session.session` is the [session](/kitaru/core-concepts/agents-and-sessions) with its inputs, outputs, and rollups; `session.nodes` is every model call and tool call with payloads. Return one result or a list; each becomes one stored evaluation. Pick the type by how you will read a thousand of them: **numbers average, booleans count, labels diff as transitions, free text gets read.** Use `passed` for the verdict and `explanation` for the sentence you will want when a gate goes red.

## An LLM judge is an evaluator

For criteria that need judgment, such as tone, helpfulness, or whether the reply answered the question, call a model inside `evaluate`. Declare the dependency inline (PEP 723) and the worker builds the environment:

```python
# /// script
# dependencies = ["openai>=2"]
# ///
from kitaru.task.evaluator import EvaluationResult, SessionView
from openai import OpenAI


def evaluate(session: SessionView, **params) -> EvaluationResult:
    reply = str(session.session.outputs or "")
    verdict = (
        OpenAI()
        .responses.create(
            model=params.get("judge_model", "gpt-5-nano"),
            input=f"Customer-support reply:\n{reply}\n\n"
            "Is this reply professional and non-committal about policy? yes/no, one reason.",
        )
        .output_text
    )
    ok = verdict.strip().lower().startswith("yes")
    return EvaluationResult(name="tone", score=ok, passed=ok, explanation=verdict)
```

The judge runs on your [worker](/kitaru/core-concepts/workers), so its API key is worker environment configuration, the same place your agent's keys live. `params` (here `judge_model`) are set per replay or experiment via `EvaluatorConfig(evaluator="tone-judge", params={...})`, so one evaluator serves cheap-per-PR and thorough-nightly configurations.

## Test offline, then register

```bash
kitaru evaluator test refund_quality_evaluator.py --entrypoint evaluate
kitaru evaluator register refund-quality \
  --script refund_quality_evaluator.py --entrypoint evaluate
```

Evaluators are versioned: re-registering with `kitaru evaluator version register refund-quality --script ...` creates version 2, and every evaluation row records exactly which version wrote it. Tightening a criterion never rewrites history: old rows keep their provenance, and you can evaluate any population again with the new version.

## Calibrate against human judgment

Before an evaluator gates anything, check that it agrees with the human it stands in for. The structured way to collect the human side is an [investigation](/kitaru/core-concepts/investigations), and by design your coding assistant authors it for you: it picks the slice of sessions, poses the criteria as questions, and interviews you against the evidence. The answers land as annotations, one per session per question. Labels can also be written directly as evaluations:

```python
from kitaru.api_models.v1.evaluation import EvaluationResult
from kitaru.api_models.v1.session import SessionEvaluationsRequest

await client.sessions.merge_evaluations(
    session_id,
    SessionEvaluationsRequest(
        evaluations=[
            EvaluationResult(
                name="human_tone", score=True, explanation="Good recovery"
            ),
        ]
    ),
)
```

Run the evaluator over the same slice, then compare the `tone` column against `human_tone` per session. Where they disagree, the explanation field tells you which side is confused. Fix the evaluator (new version) or the criteria, and repeat until the agreement rate earns your trust. The labeled slice is worth keeping as a [cohort](/kitaru/core-concepts/cohorts): it is your calibration set for every future evaluator version.

## Backfill your history

Evaluators run against stored sessions, so day one of a new evaluator can cover months of history, recorded and [imported](/kitaru/import-your-traces/import-langfuse-traces) alike. From the CLI, select by tag or take everything:

```bash
kitaru session evaluate --tag imported-baseline \
  --evaluator refund-quality@latest --wait
```

Or from the client, with explicit IDs:

```python
from kitaru.api_models.v1.evaluation import EvaluationBatchCreateRequest
from kitaru.api_models.v1.replay_config import EvaluatorConfig

job = await client.evaluations.create(
    EvaluationBatchCreateRequest(
        input_session_ids=all_session_ids,  # capped per request; batch as needed
        evaluators=[EvaluatorConfig(evaluator="refund-quality")],
    )
)
```

Each (session, evaluator) pair is its own task; one failure never stops the rest. When the backfill lands, the sessions where `passed=False` are your first triage queue, and the ones worth freezing into the cohort your next [experiment](/kitaru/guides/regression-suite) runs against.


# Deterministic evaluations

Run Kitaru's offline deterministic evaluators over recorded and imported sessions

Kitaru includes ten deterministic evaluator plugins for recorded and imported sessions. They read stored session evidence and return repeatable diagnostics or policy results. They do not run the agent, call a model provider, replay a session, invoke a live tool, or read an external service.

The default Kitaru server installation includes the `kitaru-evaluator` package. At startup, Kitaru registers its three basic evaluators and the ten deterministic evaluators below. A fresh workspace creates version 1 for each evaluator.

You start each evaluation explicitly. Importing, recording, or seeding a session does not start an evaluation Job automatically.

## Start with the descriptive bundles

Use these five bundles first when you are investigating unfamiliar traces:

| Evaluator                      | What it reports                                                                                                                                                |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kitaru/session-diagnostics@1` | Session terminality, node ordering, parent linkage, chronology, payload coverage, counts, duration, resource coverage, and malformed negative resource values. |
| `kitaru/trajectory-signals@1`  | Exact adjacent tool-call repetition, exact retry after a recorded failure, and bounded short tool-name cycles.                                                 |
| `kitaru/tool-health@1`         | Recorded tool failures, null or empty results, error/status inconsistencies, and adjacent failures of the same tool.                                           |
| `kitaru/timing-profile@1`      | Wall-clock duration, node timing coverage, slowest recorded nodes, invalid intervals, and overlapping intervals.                                               |
| `kitaru/llm-call-signals@1`    | Recorded LLM failures, null or empty results, exact adjacent repeated inputs, requested/served model mismatches, and metadata coverage.                        |

These results are descriptive. They leave `passed` unset because a repeated call, a slow span, or a failure marker is not by itself a judgment about agent quality or correctness.

Run the first pass from the CLI:

```bash
kitaru session evaluate "$SESSION_ID" \
  --evaluator kitaru/session-diagnostics@1 \
  --evaluator kitaru/trajectory-signals@1 \
  --evaluator kitaru/tool-health@1 \
  --evaluator kitaru/timing-profile@1 \
  --evaluator kitaru/llm-call-signals@1 \
  --wait
```

Without `--wait`, the command returns the created Job immediately. Inspect it with `kitaru job get JOB_ID --tasks`, or read stored results with `kitaru evaluation list` and `kitaru evaluation get EVALUATION_ID`.

## Add a configured rule when you have a real policy

The other five bundles produce pass or fail verdicts only for rules you supply:

| Evaluator                       | Parameters                                                                                                                                                                                                                                  |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kitaru/output-contract@1`      | `expected` for exact non-null JSON equality; `required_paths` for RFC 6901 JSON Pointer presence; `type_requirements` mapping pointers to `null`, `boolean`, `number`, `integer`, `string`, `array`, or `object`. Supply at least one rule. |
| `kitaru/resource-budget@1`      | One or more inclusive non-negative ceilings: `max_duration_seconds`, `max_cost`, `max_total_tokens`, `max_nodes`, `max_llm_calls`, or `max_tool_calls`. Node and call-count ceilings must be integers.                                      |
| `kitaru/tool-policy@1`          | One or more of `required_tools`, `forbidden_tools`, or `max_calls_per_tool`. Tool names are exact and case-sensitive.                                                                                                                       |
| `kitaru/model-policy@1`         | One or more of `allowed_models`, `allowed_providers`, or `require_requested_model_match`. Recorded names are exact and case-sensitive.                                                                                                      |
| `kitaru/workflow-conformance@1` | Required `expected_tools` plus `mode`: `exact_order`, `in_order`, `contains_all`, or `exact_set`.                                                                                                                                           |

For example, apply recorded resource ceilings and a tool policy:

```bash
kitaru session evaluate "$SESSION_ID" \
  --evaluator kitaru/resource-budget@1 \
  --evaluator-params 'kitaru/resource-budget@1={"max_duration_seconds":120,"max_tool_calls":20,"max_total_tokens":50000}' \
  --evaluator kitaru/tool-policy@1 \
  --evaluator-params 'kitaru/tool-policy@1={"required_tools":["search"],"forbidden_tools":["delete_account"]}' \
  --wait
```

Configured rules use conservative evidence semantics:

* A recorded violation can fail immediately.
* A rule passes only when the session is terminal and all evidence required by that rule is present and consistent.
* Insufficient evidence leaves `passed` unset. This is a HOLD, not a pass.
* Invalid configuration fails that evaluator task. Sibling evaluator tasks continue under the existing evaluation Job behavior.

## Understand result evidence

Every bundle emits `input_sha256` and `config_sha256`. The input hash covers the materialized session and node fields used across the deterministic catalog. The configuration hash covers normalized parameters for that evaluator. Use both values to tell whether two attempts analyzed the same fetched evidence with the same configuration.

Finding results use compact JSON in `value`:

```json
{
  "evidence": [{"indexes": [3, 4], "tool_name": "search"}],
  "total": 1,
  "truncated": false
}
```

Evidence uses node indexes or index windows. Most bundles retain at most 20 findings while keeping the full `total` and a `truncated` flag. `timing-profile` accepts `evidence_limit` from 1 to 100.

If any structured result value would exceed 64,000 UTF-8 bytes, Kitaru retains its SHA-256 hash and original byte count instead of allowing the evaluator task to exceed the worker result limit. Verdicts are computed from the full evidence before this result encoding limit is applied.

The exact-output rule retains SHA-256 hashes of the compared values instead of copying potentially large payloads into the evaluation result. The `passed` field still reflects exact canonical JSON equality.

The short-cycle detector examines tool-name cycles with periods from two through five and requires at least three repetitions. No cycle result means no cycle within those bounds, not that the trajectory contains no other repetition.

## Run through the Python SDK

Use the existing evaluation request and pin the registered version. This example uses a fresh workspace, where the version is 1:

```python
import uuid

from kitaru.api_models.v1.evaluation import EvaluationBatchCreateRequest
from kitaru.api_models.v1.replay_config import EvaluatorConfig
from kitaru.client.api_client import KitaruAPIClient


async def start_diagnostics(session_ids: list[uuid.UUID]) -> str:
    async with KitaruAPIClient() as client:
        job = await client.evaluations.create(
            EvaluationBatchCreateRequest(
                input_session_ids=session_ids,
                evaluators=[
                    EvaluatorConfig(evaluator="kitaru/session-diagnostics", version=1),
                    EvaluatorConfig(evaluator="kitaru/trajectory-signals", version=1),
                    EvaluatorConfig(evaluator="kitaru/tool-health", version=1),
                    EvaluatorConfig(evaluator="kitaru/timing-profile", version=1),
                    EvaluatorConfig(evaluator="kitaru/llm-call-signals", version=1),
                ],
            )
        )
        return str(job.id)
```

## Run through MCP

Start `kitaru-mcp` in `standard` mode. Discover each evaluator parent and exact version with `kitaru_registry_read`, then pass their IDs to `kitaru_workflow_start`:

```json
{
  "request": {
    "operation": "evaluation",
    "session_ids": ["00000000-0000-0000-0000-000000000001"],
    "evaluators": [
      {
        "evaluator_id": "00000000-0000-0000-0000-000000000010",
        "version": 1,
        "params": {}
      }
    ]
  }
}
```

The tool returns the submitted Job immediately. Read the Job with `kitaru_activity_read`, then use `list_children` with `kind: "job_tasks"` to inspect evaluator task results. See [MCP Server](/kitaru/getting-started/setup) for capability modes and request envelopes.

## Respect the batch limit

One request may contain at most 100 distinct session/evaluator pairs. The server calculates this as `number of sessions × number of selected evaluators`.

* The five-bundle descriptive first pass supports up to 20 sessions per request.
* Selecting all ten deterministic bundles supports up to 10 sessions per request.
* A two-bundle policy pass supports up to 50 sessions per request.

For larger sets, split the session IDs into chunks that satisfy the formula and submit one normal evaluation Job per chunk. With the CLI, write each chunk to a separate UTF-8 sessions file and use `--sessions-file`. With the SDK or MCP, submit the same evaluator selection once per chunk. This is caller-side batching; Kitaru does not create one aggregate verdict across the Jobs.

## Current evidence limits

The worker fetches the session and its nodes when an attempt runs. These reads are separate and the underlying records can change, so a retry can observe a later or internally mixed materialization. The hashes expose that difference, but they do not create an immutable snapshot. Repeatability also depends on a compatible Kitaru and Python worker runtime.

The current session view cannot distinguish an absent normalized output from an explicit JSON null. An observed null session output is therefore unavailable to `output-contract`, including when the expected value is null. A null tool result has the same ambiguity and is reported as a diagnostic rather than an integrity verdict.

The evaluator bundles see Kitaru's canonical session and node models. They cannot inspect raw importer events that typed ingestion rejected, unknown raw event kinds, or provider fields that were not retained. They also do not classify rate limits, context exhaustion, timeouts, or malformed external responses unless the canonical record exposes enough direct evidence for the specific result.

`timing-profile` and the call-count results report values for one session. They do not label cohort-relative duration or tool-call outliers. Establishing an outlier requires a frozen comparison cohort, a declared statistic, and calibrated thresholds; a high count alone is not an agent-quality failure.

## Versioning

All built-in evaluators share the `kitaru-evaluator` distribution. When that package version changes, startup registration creates a new immutable version for each evaluator definition. Pin the registered evaluator version when you need a stable contract. Re-executing an older version is deterministic only when the fetched materialized view and worker runtime are also equivalent.

Built-in evaluators are ordinary package-backed workspace plugins. Kitaru registers them at server startup with no owner and reserves their `kitaru/` names.


# Tool policies

Choose whether replayed tool calls use recorded results, static results, live tools, or model-generated responses.

When a [replay](/kitaru/core-concepts/replay) reaches a tool call, the **tool policy** determines how the adapter responds. You can configure individual tools and set a default for all others. Without a suitable policy, a replay can call a live tool and repeat its side effects.

```python
from kitaru.api_models.v1.replay_config import (
    HistoryConfig,
    PassthroughConfig,
    StaticCase,
    StaticConfig,
    ToolPolicy,
)

policy = ToolPolicy(
    default=HistoryConfig(scope="baseline", on_miss="fail"),
    tools={
        "get_current_time": PassthroughConfig(),
        "refund_payment": StaticConfig(
            cases=[
                StaticCase(match={"order_id": "4821"}, result="refund issued: $129.00")
            ],
            on_miss="error_result",
        ),
    },
)
```

A policy belongs to a replay or an [experiment](/kitaru/core-concepts/experiments). The adapter applies it when the re-running agent calls a tool. On the CLI, pass the same structure to `kitaru experiment create --tool-policy` as JSON:

```bash
--tool-policy '{"default": {"type": "history", "scope": "baseline", "on_miss": "fail"},
                "tools": {"get_current_time": {"type": "passthrough"}}}'
```

## The four policies

### `history`: use a recorded result

The adapter looks for a recorded call with the same tool name and arguments. If it finds one, it returns the recorded result without executing the live tool.

`scope` says which recordings answer:

| Scope            | Answers come from                                                                    |
| ---------------- | ------------------------------------------------------------------------------------ |
| `baseline`       | Only the session being replayed. This is the narrowest scope.                        |
| `cohort_version` | Any session in the experiment's cohort. This scope is valid only inside experiments. |
| `agent`          | Any session belonging to the agent.                                                  |

`on_miss` controls what happens when no recorded call matches:

* `fail`: stop the replay without executing the tool. Use this for tools with side effects.
* `error_result`: return a tool error to the agent and continue the replay.
* `passthrough`: execute the live tool. Use this only when repeating the call is safe.

A model or prompt change may cause the agent to call a tool that does not appear in the baseline. With `fail`, that call stops the replay. With `error_result`, the agent receives an error and the evaluator can assess its response.

### `static`: return a configured result

Each `StaticCase` matches arguments with `match_mode="exact"` or `"subset"` and returns the configured `result`. Use it to test a specific condition, such as a refund that has already succeeded, or to stub a tool that was absent from the baseline. `on_miss` controls unmatched arguments as described above.

### `passthrough`: execute the live tool

**This is the default when you set no policy.** The adapter executes the live tool and returns its result. This may be appropriate for safe read-only calls such as clocks or search. It is unsafe for calls that write data or trigger external actions. Prefer a `history` default and configure `passthrough` only for specific tools that are safe to repeat.

### `llm`: generate a result with a model

`LLMConfig(model=..., instructions=...)` asks a model to generate a response to the tool call. This can simulate a tool when no recorded result is available.

{% hint style="warning" %}
The API accepts and stores the \`llm\` policy, but the PydanticAI, Mastra, and Vercel AI SDK adapters do not support it. Those adapters reject the policy before executing the configured tool. Use \`static\` when you need to provide a simulated result. Check the relevant adapter page before relying on \`llm\` elsewhere.
{% endhint %}

History matching is guaranteed only within the same TypeScript adapter. Different frameworks can apply schema defaults, coercion, or serialization differently, which changes the cache key even when a tool call looks equivalent. The TypeScript adapters also fail closed on a found `null` history result because the current API cannot distinguish a successful `null` from a recorded failure.

## How matching works

A recorded `tool_call` node has a cache key derived from the tool name and its canonical JSON arguments. During replay, the adapter computes the same key for the attempted call and asks the server for a match within the policy's scope. Calls with different arguments have different keys and do not match.

A baseline can call the same tool with identical arguments more than once and receive different results. With `baseline` scope, the PydanticAI, LangGraph, and TypeScript (Mastra and Vercel AI SDK) adapters consume those recorded results in their original order: the first replayed call gets the first recorded result, the second gets the second, and so on. A replayed call past the last recorded occurrence is a miss and follows the configured `on_miss` behavior. With `cohort_version` and `agent` scope, the newest matching recording answers every call.

If a tool call's arguments cannot be serialized to canonical JSON, the call has no cache key. A history lookup cannot match it, so replay follows the configured `on_miss` behavior. Keep tool arguments JSON-serializable if you plan to replay them from history.

## Choosing a policy

| Situation                               | Policy                                                                          |
| --------------------------------------- | ------------------------------------------------------------------------------- |
| Reproducing a failure faithfully        | `history(baseline, on_miss="fail")` everywhere                                  |
| Fork that may explore new paths         | `history` default with `on_miss="error_result"`; `fail` on side-effecting tools |
| Injecting a counterfactual              | `static` on the tool in question, `history` for the rest                        |
| Read-only tools that are cheap and safe | `passthrough`, scoped per tool                                                  |
| Regression suite over a cohort          | `history(cohort_version, on_miss="fail")`                                       |


# Track cost and model usage

Every model call recorded: tokens, cost, and call counts on every session, and how to total them across cohorts and experiments.

Every model call in a recorded run lands as an `llm_call` node on the [session](/kitaru/core-concepts/agents-and-sessions): the requested and resolved model, inputs and outputs, token usage (input, output, cached, reasoning), and cost. The session rolls those values up as it goes, so the totals are already there when you read a run:

```python
session = await client.sessions.get(session_id)
print(session.cost)  # Decimal, summed across the run's model calls
print(session.tokens)  # input / output / cached_input / reasoning
print(session.llm_call_count, session.tool_call_count)
```

Imported sessions get the same treatment: when your Langfuse export carries usage and cost, the importer preserves them, so your history is costed the moment it lands.

## Per-call detail

When the total isn't enough, the nodes have the breakdown:

```python
from kitaru.api_models.v1.session_node import SessionNodeListParams

nodes = await client.sessions.list_nodes(
    session_id, SessionNodeListParams(include_payloads=True)
)
for node in nodes.items:
    if node.node_type == "llm_call":
        print(node.requested_model, node.model, node.tokens, node.cost)
```

`requested_model` vs `model` is worth watching: it shows an alias or a replay [override](/kitaru/guides/replay-and-overrides) resolving to the model that served the call.

## Cost as an experiment metric

Cost earns its place in the loop as a *delta*. Every replay's result session carries its own rollups, so "did the cheaper model hold?" is a pass-rate comparison and a cost comparison from the same rows:

```python
from kitaru.api_models.v1.filter import FilterCondition, FilterOp
from kitaru.api_models.v1.replay import ReplayListParams

baseline_cost = fork_cost = 0
async for r in client.replays.iter(
    ReplayListParams(
        filter=FilterCondition(field="experiment_run_id", op=FilterOp.EQ, value=RUN_ID)
    )
):
    baseline = await client.sessions.get(r.baseline_session_id)
    fork = await client.sessions.get(r.result_session_id)
    baseline_cost += baseline.cost or 0
    fork_cost += fork.cost or 0

print(f"cohort cost: ${baseline_cost} -> ${fork_cost}")
```

A negative delta across a [cohort](/kitaru/core-concepts/cohorts) is the cheaper model paying for itself, with the pass rates from your [evaluators](/kitaru/guides/write-an-evaluator) sitting right next to it saying whether the savings were free.

{% hint style="info" %}
Recorded cost is an observability number derived from provider usage data, not an invoice. Treat deltas as reliable and absolute values as estimates.
{% endhint %}

Budget-minded evaluator runs matter too: code evaluators cost nothing to run, while [LLM judges](/kitaru/guides/write-an-evaluator) spend judge tokens per session; size your per-PR cohort accordingly and save the wide sweep for the nightly run.


# Overview

Bring the traces you already collect. Langfuse stays your system of record, Kitaru gets a runnable copy.

You don't have to run a single request through Kitaru to start. If your agent already logs to Langfuse (or any tracing system you can export from), your history is the raw material: import it, and every trace lands as a [session](/kitaru/core-concepts/agents-and-sessions), the same object a live-recorded run produces, ready to replay and evaluate like any other.

This is the honest division of labor: **your observability stack stays your system of record**. Kitaru takes a copy of the runs you care about and makes them runnable: the incident from Tuesday becomes a test case, last month's traffic becomes a regression population.

Imports execute on a [worker](/kitaru/core-concepts/workers) in your environment. The export file is parsed by your worker, not by anything outside your infrastructure.

## 1. Register the agent the traces belong to

Importers for **Langfuse, LangSmith, Braintrust, Logfire, Arize Phoenix, and a native JSONL format** are built in, registered at server startup under the `kitaru/` namespace, so there is no importer code to write for those. Anything else comes in through a [custom importer](/kitaru/import-your-traces/custom-importer).

{% hint style="info" %}
**Traces in OpenTelemetry format?** There is no OTel ingestion endpoint yet. Export the spans and convert them to [Kitaru JSONL](/kitaru/import-your-traces/importing-sessions), or wrap that conversion in a [custom importer](/kitaru/import-your-traces/custom-importer) so your exports import directly; the `kitaru-importer-builder` skill drafts one from a sample export.
{% endhint %}

Register the agent these traces belong to, if you haven't:

```bash
kitaru agent register support-agent --command "python support.py"
```

## 2. Import the export

Export your traces from Langfuse as JSONL (trace, observation, and ingestion-event records are all understood), start a [worker](/kitaru/core-concepts/workers) in another terminal (`kitaru worker start`), then:

```bash
kitaru session import langfuse-export.jsonl \
  --importer kitaru/langfuse@latest \
  --agent support-agent@latest \
  --tag imported-baseline \
  --media-type application/x-ndjson \
  --wait
```

`--tag` labels every session this import creates (repeat it for more than one label), so later steps can select them as a group (`kitaru session evaluate --tag imported-baseline ...`) without copying IDs around. (Tagging happens once the import completes, which is why `--tag` requires `--wait`.)

The final receipt reports what happened: sessions `created`, `skipped`, and `failed`, with samples of the failures. Each imported trace becomes one session (`origin: imported`) with its observations as nodes: model calls with token usage and cost, tool calls with arguments and results. List them:

```bash
kitaru session list --agent support-agent --origin imported
```

The same import is two calls on the [Python client](/kitaru/get-help/configuration) when you'd rather script it: upload the export with `client.blobs.upload(...)`, then create the import with `client.imports.create(ImportCreateRequest( importer="langfuse", agent_id=..., payload_blob_id=...))`.

## Re-runs are safe

Every imported session keeps its source identity (`imported_from` + `external_id`). Importing the same export twice, or a bigger export that overlaps an earlier one, skips what's already there instead of duplicating it. Import incrementally, as often as you like.

{% hint style="warning" %}
An import stores the parsed trace content (prompts, tool arguments, tool results) on your Kitaru server. The server is self-hosted, but check your own access and retention rules before importing exports that contain customer data.
{% endhint %}

## What imported sessions can do

Everything recorded sessions can:

* **Inspect** them: nodes, cost, and token rollups all populate.
* **Evaluate** them with [evaluators](/kitaru/core-concepts/evaluators), including backfilling evaluations over your whole history.
* **Group** them into [cohorts](/kitaru/core-concepts/cohorts) and run [experiments](/kitaru/core-concepts/experiments) against them.
* **Replay** them, with one honest caveat. Replay re-runs *your agent's real code*, which the trace itself doesn't contain. Register the agent version whose code produced the traces (its run command), and replay works exactly as for recorded sessions: recorded tool calls answered from the imported history, everything else per your [tool policy](/kitaru/guides/tool-policies).

Other formats: an importer is about a page of Python, a callable that parses your export bytes into sessions, and the `kitaru-importer-builder` agent skill will draft it for you. See [No importer for your format](/kitaru/import-your-traces/custom-importer).

## Next

Evaluate your imported history with your first evaluator ([Write an evaluator](/kitaru/guides/write-an-evaluator)), then pick the sessions that matter into a cohort and put a change to the test with [Build a regression suite from production](/kitaru/guides/regression-suite).


# Langfuse

Langfuse JSONL imports end to end: the accepted export, dedup semantics, and how imports execute on your worker.

[Import your traces](/kitaru/import-your-traces/import-your-traces) covers the shortest path: one `kitaru session import` against the built-in Langfuse importer. This guide is the full contract: what the importer understands, how re-runs dedup, and how to write an importer for any other format.

## How an import executes

An import is a job with one importer task. You upload the export as a blob; a [worker](/kitaru/core-concepts/workers) claims the task, materializes the importer's code and your payload, and runs the parse **in your environment**; the server never parses your data. Each parsed trace becomes one [session](/kitaru/core-concepts/agents-and-sessions) with `origin: imported`, its observations ingested as nodes in batches.

The CLI wraps the upload and the job in one command:

```bash
kitaru session import langfuse-export.jsonl \
  --importer kitaru/langfuse@latest \
  --agent support-agent@latest \
  --params '{"source_instance": "my-langfuse-project"}' \
  --media-type application/x-ndjson \
  --tag imported-baseline --wait
```

`--tag` labels the created sessions once the import completes (so it requires `--wait`); downstream commands select on it. On the Python client the same import is explicit:

```python
from kitaru.api_models.v1.imports import ImportCreateRequest

job = await client.imports.create(
    ImportCreateRequest(
        importer="langfuse",  # importer name in the registry
        agent_id=AGENT_ID,  # sessions land under this agent
        agent_version_id=None,  # optional: stamp a version on them
        payload_blob_id=blob.id,
        params={"source_instance": "my-langfuse-project"},
    )
)
```

Set `agent_version_id` when you know which code produced the traces; it's what lets a later replay default to the right version. The task's result carries the stats: sessions `created`, `skipped`, `failed`, with up to 20 failure samples (line number, external id, error).

## The built-in importers

Kitaru ships provider importers as default plugins, registered at server startup under the `kitaru/` namespace, so `--importer kitaru/langfuse@latest` always resolves. They run on your worker like any other importer; there is nothing to write. See [Import your traces](/kitaru/import-your-traces/import-your-traces) for the current built-in list.

The Langfuse importer parses **Langfuse JSONL exports**, up to 50 MiB per payload (the importer's own cap, separate from the server's configurable blob limit), and understands three record shapes: `trace`, `observation`, and raw `ingestion_event` lines. Traces map to sessions; observations map to nodes with their parent relationships, timings, model names, token usage, and cost preserved. `params`:

| Param                   | Meaning                                                                                                                                                                                                                                                                                                                                                                                         |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `source_instance`       | The Langfuse project the export came from. Optional when the export itself carries project ids; required when it doesn't; it anchors the sessions' external identity.                                                                                                                                                                                                                           |
| `filename`              | Optional label used as a fallback source name.                                                                                                                                                                                                                                                                                                                                                  |
| `infer_tool_call_links` | Optional boolean, default `true`. The importer matches tool-call ids emitted by a generation with `gen_ai.tool.call.id` on tool observations, nests each unambiguous tool call under the requesting generation, and retains its original Langfuse parent as a secondary parent. Unmatched or ambiguous ids remain unchanged. Set this to `false` to keep only the source observation hierarchy. |

Import in slices as often as you like; dedup makes it safe.

## Dedup: one session per (imported\_from, external\_id)

Every imported session records its source identity: `imported_from` (`langfuse`) and the trace's `external_id`. That pair is unique on the server, so re-importing an overlapping export **skips** what's already stored; the stats report it as `skipped`, not as an error. This is the property that makes "export the last 24 hours every night" a safe cron job rather than a duplication engine.

Node identity works the same way inside a session: nodes upsert by index, so a re-parse states each node's full content and replaces it whole.

## No importer for your format?

The importer contract is deliberately small, about a page of Python, and the shipped Langfuse importer is a reference implementation of it. See [No importer for your format](/kitaru/import-your-traces/custom-importer) to scaffold, test, and register your own.

## After the import

Imported sessions are full Kitaru sessions: evaluate them with [evaluators](/kitaru/guides/write-an-evaluator) (backfilling your history is a single batch call), freeze them into [cohorts](/kitaru/core-concepts/cohorts), and [replay](/kitaru/guides/replay-and-overrides) them. Replay re-runs your code, which no trace export contains, so the agent's code must be registered as an agent version with a run command.


# LangSmith

Turn LangSmith runs into replayable Kitaru sessions: the accepted export shapes, how threads group into sessions, dedup semantics, and what the importer cannot recover.

If your agent already reports to LangSmith, you do not need to re-instrument anything to start using Kitaru. Export the runs, import them, and each one lands as a [session](/kitaru/core-concepts/agents-and-sessions) with `origin: imported`, the same object a live-recorded run produces. LangSmith stays your system of record; Kitaru takes a runnable copy of the runs you want to evaluate and replay.

[Import your traces](/kitaru/import-your-traces/import-your-traces) covers the shortest path. This guide is the LangSmith contract: what the built-in importer accepts, how it decides where one session ends and the next begins, and where it tells you it lost fidelity.

## Export your runs

The importer reads **LangSmith run records**, one JSON object per run, in any of these shapes:

* **JSONL**, one run object per line. This is the shape bulk exports arrive in.
* **A JSON array** of run objects.
* **A run-query envelope**: a JSON object with the runs under a `runs` or `data` key. This is what the LangSmith runs-query API returns, so you can pipe its response straight to a file.
* **A single JSON object**, treated as a one-run export.

Payloads must be UTF-8 and 50 MiB or smaller (the importer's own cap, separate from the server's configurable blob limit). Export in slices as often as you like; [dedup](#dedup-one-session-per-project-and-thread) makes overlapping exports safe.

Each run record is read for the fields LangSmith already writes: `id`, `trace_id`, `parent_run_id`, `is_root`, `run_type`, `name`, `status`, `error`, `start_time` / `end_time`, `inputs`, `outputs`, `tags`, `extra.metadata`, `extra.invocation_params`, `serialized.kwargs`, `total_cost`, and token counts. Export whole traces rather than filtered subsets: a run whose parent is missing from the file still imports, but the session is marked partial.

{% hint style="info" %}
\`inputs\`, \`outputs\`, \`extra\`, and \`metadata\` are commonly JSON-encoded strings in bulk exports. The importer decodes them, so you don't have to pre-process the file.
{% endhint %}

## Import the export

Register the agent these runs belong to, if you have not, then start a [worker](/kitaru/core-concepts/workers) in another terminal (`kitaru worker start`) and import:

```bash
kitaru agent register support-agent --command "python support.py"

kitaru session import langsmith-runs.jsonl \
  --importer kitaru/langsmith@latest \
  --agent support-agent@latest \
  --media-type application/x-ndjson \
  --tag imported-baseline --wait
```

The import is a job with one importer task. The export is uploaded as a blob; a worker claims the task and runs the parse **in your environment**, so the server never parses your run data. `--tag` labels the sessions once the import completes, which is why it requires `--wait`. The receipt reports sessions `created`, `skipped`, and `failed`, with failure samples.

`kitaru/langsmith` is one of the built-in importers, registered at server startup, so `@latest` always resolves and there is no importer code to write.

### Parameters

| Param             | Meaning                                                                                                                                                                                                                                                             |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `source_instance` | The LangSmith project the export came from. Optional when the runs carry `session_id`, `project_id`, `session_name`, or `project_name`; required when they don't. It anchors the sessions' external identity, so keep it stable across imports of the same project. |
| `join_on`         | The path whose value groups traces into one session. Accepts a dotted path (`extra.metadata.thread_id`) or an RFC 6901 JSON Pointer (`/extra/metadata/thread_id`), resolved against each trace's root run. Omit it to use the defaults below.                       |

Pass them with `--params '{"source_instance": "my-project"}'`. `join_on` also has its own flag, `--join-on`, which accepts JSON Pointer syntax only (it must start with `/`) and cannot be combined with a `join_on` inside `--params`:

```bash
kitaru session import langsmith-runs.jsonl \
  --importer kitaru/langsmith@latest \
  --agent support-agent@latest \
  --join-on /extra/metadata/conversation_id \
  --media-type application/x-ndjson --wait
```

## What a LangSmith trace becomes

The mapping is one level deeper than a trace-per-session import, because a LangSmith thread is usually a multi-turn conversation spread over several traces:

| LangSmith                                                                | Kitaru                                                         |
| ------------------------------------------------------------------------ | -------------------------------------------------------------- |
| Project                                                                  | The session's `source_instance`, half of its external identity |
| Thread (`thread_id`, `session_id`, or `conversation_id` in run metadata) | One **session**, holding every trace in the thread             |
| Trace                                                                    | One **turn** inside that session's inputs, in start-time order |
| Run with `run_type` `llm` or `chat_model`                                | An **llm\_call** node                                          |
| Run with `run_type` `tool`                                               | A **tool\_call** node, named after the run                     |
| Any other run type                                                       | A **span** node                                                |
| `parent_run_id`                                                          | The node's parent, rebuilt as a tree per trace                 |

Without an explicit `join_on`, the importer looks for a thread value at `extra.metadata.thread_id`, `extra.metadata.session_id`, `extra.metadata.conversation_id`, and then the same three keys under a top-level `metadata`. If none is present, each trace becomes its own session and the session records the warning "No LangSmith thread metadata found; grouped by trace id".

Per node, the importer preserves timings, status and error, inputs and outputs, the requested and resolved model names, the model provider, model invocation parameters, token usage (input, output, and cached input, read from `extra.token_usage`, `outputs.llm_output.token_usage`, `outputs.usage_metadata`, or top-level `prompt_tokens` / `completion_tokens`), and `total_cost`. It also picks out the user prompt, the assistant's visible answer, the system prompt, and any visible model reasoning, so those render as text rather than as raw payload. LangSmith run type, status, and tags are kept as node attributes, and a bounded set of metadata keys (`thread_id`, `session_id`, `conversation_id`, `user_id`, `assistant_id`, `graph_id`, `langgraph_node`, `langgraph_checkpoint_ns`, `revision_id`, `environment`, and `reference_example_id`) is kept under `langsmith.*`.

At session level you get the thread's trace ids, the join paths used, the union of run tags and user ids, the turn count, and a `source_completeness` of `full` or `partial`. The importer also detects the agent framework (PydanticAI, LangGraph, OpenAI Agents, Google ADK, or the Claude Agent SDK) from run metadata when the evidence points to exactly one. Session status follows the latest trace's root run: failed if that run carries an error or a failure status, completed otherwise.

## Dedup: one session per project and thread

Every imported session records `imported_from: langsmith` plus an `external_id` of `<source_instance>:<thread>`. That pair is unique on the server, so re-importing an overlapping export **skips** what is already stored and reports it as `skipped`, not as an error. Nodes upsert by index within a session, so a re-parse restates each node's full content.

This is what makes "export the last 24 hours every night" safe. It also means the grouping key matters: if you change `source_instance` or `join_on` between imports of the same runs, the same thread lands as a second session rather than deduping against the first.

## Limitations

* **Only what the export contains.** Anything LangSmith did not record (intermediate state, code, environment) is not recoverable from the file.
* **Imported threads are frozen.** Once a thread is imported, later traces in the same thread are skipped by dedup rather than appended. Import a thread after it is finished, or scope `join_on` to something that closes.
* **Partial graphs import with a warning.** A trace with more than one root run, a run whose parent is missing from the export, or model output containing `tool_calls` with no corresponding tool runs all set `source_completeness: partial` and add a line to `normalization_warnings`. The session still imports.
* **A bad trace is isolated, not fatal.** A run with no trace id or run id, a trace with conflicting project identities or conflicting thread values, or a trace missing your chosen `join_on` value is reported as a failure and the rest of the file still imports. A malformed file (invalid JSON, non-UTF-8, empty, or over 50 MiB) fails the task as a whole.
* **Replay needs your code.** Imported sessions replay like recorded ones, but only if the agent version whose code produced the runs is registered with a run command. No trace export contains the code.

{% hint style="warning" %}
Imported payloads contain whatever your runs contain: prompts, customer data, tool results. They are stored on your self-hosted server and parsed on your workers, but access and retention are yours to govern.
{% endhint %}

## Next

Evaluate the history you imported with [Write an evaluator](/kitaru/guides/write-an-evaluator), then freeze the sessions that matter into a cohort and put your next change to the test with [Build a regression suite from production](/kitaru/guides/regression-suite).


# Braintrust

Import Braintrust project logs into Kitaru: accepted export shapes, how spans become nodes, multi-turn grouping, and what the importer marks as lossy.

If your agent already logs to Braintrust, you do not need to instrument anything to start using Kitaru. Export the logs, run one import, and each trace lands as a [session](/kitaru/core-concepts/agents-and-sessions): the same object a live-recorded run produces, ready to evaluate and replay.

**Braintrust stays your system of record.** Kitaru takes a runnable copy of the runs you care about so that last Tuesday's incident becomes a test case and last month's traffic becomes a regression population.

Like every import, this one executes on a [worker](/kitaru/core-concepts/workers) in your environment: the server stores the export blob, your worker parses it. See [Import Langfuse traces](/kitaru/import-your-traces/import-langfuse-traces) for the generic importer contract; this page is the Braintrust specifics.

## 1. Export your Braintrust logs

The importer is permissive about the container because Braintrust logs reach you in more than one shape. It accepts a UTF-8 file that is any of:

* **JSONL**, one Braintrust event object per line.
* **A JSON array** of event objects.
* **A JSON object with an `events` array**, the shape the Braintrust API returns for a log fetch.
* **A single JSON object**, treated as a one-event export.

Payloads are capped at 50 MiB per import (the importer's own limit, separate from the server's configurable blob limit). Import in slices as often as you like; [dedup](#re-runs-skip-what-is-already-there) makes overlapping slices safe.

What matters is the fields on each record, not how you got the file. A full project-log export carries span identity, and that is what you want:

```json
{
  "id": "event-llm",
  "project_id": "project-1",
  "span_id": "llm",
  "root_span_id": "root",
  "span_parents": ["root"],
  "span_attributes": {"name": "weather-model", "type": "llm"},
  "input": {"messages": [{"role": "user", "content": "Weather?"}]},
  "output": {"role": "assistant", "content": "Sunny."},
  "metadata": {"session_id": "conversation-1", "model": "gpt-4o"},
  "metrics": {"start": 1785000000.1, "end": 1785000000.4,
              "prompt_tokens": 5, "completion_tokens": 2,
              "estimated_cost": 0.00125},
  "created": "2026-07-24T10:00:00Z"
}
```

Rows that carry `span_id`, `root_span_id`, or `span_attributes` are treated as a **full export**. Rows without them (a flat export copied out of the Braintrust UI, for example) still import, at lower fidelity; see [Lower-fidelity exports](#lower-fidelity-exports).

## 2. Import it

Register the agent the traces belong to, if you have not, and start a worker:

```bash
kitaru agent register support-agent --command "python support.py"
kitaru worker start
```

Then import:

```bash
kitaru session import braintrust-logs.jsonl \
  --importer kitaru/braintrust@latest \
  --agent support-agent@latest \
  --media-type application/x-ndjson \
  --tag imported-baseline --wait
```

`kitaru/braintrust` is one of the built-in importers registered at server startup, so `@latest` always resolves and there is no importer code to write. Use `--media-type application/json` when you upload a JSON array or an `events` object instead of JSONL.

`--tag` labels every session the import creates, so later commands can select them as a group (`kitaru session evaluate --tag imported-baseline ...`). Tagging happens once the import finishes, which is why it requires `--wait`. The receipt reports sessions `created`, `skipped`, and `failed`, with samples of the failures.

List what landed:

```bash
kitaru session list --agent support-agent --origin imported --imported-from braintrust
```

### Importer params

| Param             | Meaning                                                                                                                                                                                                                                   |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `source_instance` | Project identity fallback. The importer prefers each record's `project_id`; `source_instance` is used when the export carries none.                                                                                                       |
| `filename`        | Optional label. When neither `project_id` nor `source_instance` is available, the filename stem becomes the project identity. A record with none of the three fails with "Braintrust export has no project id; provide source\_instance". |
| `join_on`         | Dotted path or RFC 6901 JSON Pointer selecting the value that groups traces into one session. Defaults to the session id found in metadata. See [Grouping traces into sessions](#grouping-traces-into-sessions).                          |

Pass them with `--params '{"source_instance": "my-braintrust-project"}'`, or use the dedicated `--join-on` flag, which accepts a JSON Pointer only (it must start with `/`) and cannot be combined with `join_on` inside `--params`.

## What a trace becomes

Every Braintrust event in a trace becomes one node, and the `span_parents` links are rebuilt as the node tree, so a tool span nested under a model span stays nested. Node type is mapped conservatively:

| Braintrust record                                                                            | Kitaru node                                                                                |
| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `span_attributes.type == "tool"`, or `metadata["tool.name"]` present                         | `tool_call`, with `tool_name` from `metadata["tool.name"]` (falling back to the span name) |
| `span_attributes.type == "llm"`, and `metadata["openinference.span.kind"]` is empty or `LLM` | `llm_call`                                                                                 |
| Everything else, including OpenInference `CHAIN` wrappers                                    | `span`                                                                                     |

An `llm` span whose OpenInference kind says it is really a chain stays a plain span rather than being mislabeled as a model call.

Per node, the importer preserves:

* **Inputs and outputs** from `input` / `output`, falling back to OpenInference's `metadata["input.value"]` and `metadata["output.value"]`, JSON-decoded when those hold encoded JSON strings.
* **Model identity**: requested model from `gen_ai.request.model` or `model`, resolved model from `gen_ai.response.model` or `model`, provider from `gen_ai.provider.name` or `provider`.
* **Token usage** from `metrics.prompt_tokens`, `metrics.completion_tokens`, and `metrics.prompt_cached_tokens`. A non-integer value there fails that session and is reported as an import failure; other sessions in the file still import.
* **Cost** from `metrics.estimated_cost`.
* **Timings** from `metrics.start` / `metrics.end`, with `created` as a start fallback. Both ISO 8601 strings and Unix timestamps parse.
* **Status**: a record with a non-empty `error` becomes a failed node carrying that error.
* **Metadata**, allowlisted. Only session, conversation, model, and provider keys come across (`session_id`, `sessionId`, `thread_id`, `conversation_id`, `gen_ai.conversation.id`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.provider.name`, `model`, `provider`, `turn_index`). Everything else in `metadata` is dropped rather than copied wholesale into Kitaru.

The importer also normalizes each node for the UI and for evaluators: it locates the user input text, the visible assistant output text, the system prompt on model calls, and any visible reasoning, recording pointers into the payload rather than copying the text. Reasoning and tool-call parts are excluded from what counts as visible output. When the export's metadata names a known framework (PydanticAI, LangGraph, OpenAI Agents, Google ADK, or the Claude Agent SDK), the session records it, provided the evidence points at exactly one.

### Grouping traces into sessions

Braintrust's unit is a trace; a multi-turn conversation is usually several root traces. Kitaru groups them:

* By default, traces are grouped by the first session-like key present in the root record's metadata: `session_id`, `sessionId`, `thread_id`, `conversation_id`, or `gen_ai.conversation.id`. A trace with none of these becomes its own single-turn session.
* With `join_on`, traces are grouped by the scalar at that path in each trace's root record instead, which is how you group by your own correlation key: `--join-on '/metadata/case~1id'`. A trace missing that value, or holding an object or list there, is reported as a failure rather than silently grouped elsewhere.

Grouped traces become **turns**, ordered by start time. The session's `inputs` is a versioned turn list (`{"schema_version": 1, "turns": [{"source_trace_id", "inputs", "outputs"}, ...]}`), and the session's outputs come from the last turn. Session status follows the last turn's root record: a tool that failed and was retried successfully leaves the session completed.

Session metadata records the provenance you'll want when reading the import back: `braintrust.project_ids`, `braintrust.session_id`, `braintrust.trace_ids`, `source_trace_count`, `source_completeness`, `braintrust.join_on` when you set one, and `normalization_warnings`.

## Re-runs skip what is already there

Every imported session records its source identity: `imported_from` (`braintrust`) and an `external_id` of `<project>:<session>`. That pair is unique on the server, so re-importing an overlapping export **skips** what is already stored and reports it as `skipped`, not as an error. Exporting the last 24 hours every night is safe; it will not duplicate earlier sessions.

## Limitations

The importer is explicit about fidelity it cannot recover, and writes what it noticed into `normalization_warnings` on the session:

* `"Braintrust UI export omits span identity and hierarchy"` on a flat export.
* `"One or more spans reference a missing parent"` when a `span_parents` entry is not in the file, usually a partial export. Those nodes are kept as roots.
* `"Model output contains tool activity but no explicit tool spans"` when a model output references `tool_calls` that the export never recorded as their own spans. Kitaru does not invent nodes for them.
* `"One or more LLM spans lack recorded input or output"` when a model call came across without its payload.

Two more things worth knowing before you rely on an import:

* Non-allowlisted `metadata` keys and Braintrust's own evaluations do not come across. Evaluate imported sessions with Kitaru [evaluators](/kitaru/core-concepts/evaluators) instead; backfilling your history is a single batch call.
* Replay re-runs your agent's real code, which no trace export contains. Register the agent version whose code produced these traces, with its run command, and imported sessions replay exactly like recorded ones.

### Lower-fidelity exports

A flat export (rows with `input`, `output`, `metadata`, and `metrics`, but no `span_id` or `span_parents`) still imports. The importer marks it `source_completeness: "flat"`, gives each row a synthetic identity, and relaxes one rule: without span types to read, a row that carries `metadata.model` or token metrics is treated as a model call. There is no hierarchy to rebuild, so the nodes land flat. Prefer a full project-log export whenever you can get one.

{% hint style="warning" %}
An import stores the parsed trace content, including prompts, tool arguments, and tool results, on your Kitaru server. The server is self-hosted, but check your own access and retention rules before importing exports that contain customer data.
{% endhint %}

## Next

Evaluate your imported history with [Write an evaluator](/kitaru/guides/write-an-evaluator), then freeze the sessions that matter into a cohort and put a change to the test with [Build a regression suite from production](/kitaru/guides/regression-suite).


# Logfire

Import Pydantic Logfire records-query exports into Kitaru: accepted export shapes, how OpenTelemetry GenAI spans become nodes, conversation grouping, and what the importer marks as lossy.

If your agent already sends spans to Logfire, you do not need to instrument anything to start using Kitaru. Export the records, run one import, and each conversation lands as a [session](/kitaru/core-concepts/agents-and-sessions): the same object a live-recorded run produces, ready to evaluate and replay.

**Logfire stays your system of record.** Kitaru takes a runnable copy of the runs you care about, so last Tuesday's incident becomes a test case and last month's traffic becomes a regression population.

Like every import, this one executes on a [worker](/kitaru/core-concepts/workers) in your environment: the server stores the export blob, your worker parses it. [Import your traces](/kitaru/import-your-traces/import-your-traces) covers the generic importer contract; this page is the Logfire specifics.

## 1. Export your records

The importer reads rows from Logfire's **records** table, one row per span. Export them as JSON or NDJSON. It accepts a UTF-8 file that is any of:

* **JSONL**, one record row per line.
* **A JSON array** of record rows.
* **A JSON object with a `data` array** of rows.
* **A single JSON object**, treated as a one-row export.
* **The Query API's streaming NDJSON**, where each line is a typed message. `schema`, `explain`, and `end` messages are skipped, rows arrive inside `{"type": "data", "rows": [...]}` (or a single `{"type": "data", "data": {...}}`), and a `{"type": "error"}` message fails the import with the message it carries.

Payloads are capped at 50 MiB per import (the importer's own limit, separate from the server's configurable blob limit). Export in slices as often as you like; [dedup](#re-runs-skip-what-is-already-there) makes overlapping slices safe.

Every row needs `trace_id` and `span_id`; a row without both is reported as a failure and the rest of the file still imports. Beyond those, the importer reads `project_id`, `parent_span_id`, `span_name`, `message`, `kind`, `level`, `start_timestamp`, `end_timestamp`, `otel_status_code` / `status_code`, `otel_status_message`, `is_exception`, `exception_message`, `service_name`, `service_namespace`, `service_version`, `deployment_environment`, `otel_scope_name`, `otel_scope_version`, `tags`, and the `attributes` column:

```json
{
  "project_id": "project-1",
  "trace_id": "trace-1",
  "span_id": "llm",
  "parent_span_id": "root",
  "span_name": "chat claude-haiku-4-5",
  "start_timestamp": "2026-07-22T13:15:00.100000Z",
  "end_timestamp": "2026-07-22T13:15:01Z",
  "service_name": "support-agent",
  "deployment_environment": "production",
  "otel_scope_name": "pydantic-ai",
  "attributes": {
    "gen_ai.operation.name": "chat",
    "gen_ai.conversation.id": "conversation-1",
    "gen_ai.request.model": "claude-haiku-4-5",
    "gen_ai.response.model": "claude-haiku-4-5-20251001",
    "gen_ai.provider.name": "anthropic",
    "gen_ai.usage.input_tokens": 507,
    "gen_ai.usage.output_tokens": 77,
    "operation.cost": 0.000892
  }
}
```

{% hint style="info" %}
\`attributes\` and the payload values inside it are commonly JSON-encoded strings in query output. The importer decodes them, so you don't have to pre-process the file.
{% endhint %}

Export whole traces rather than filtered subsets. A span whose parent is missing from the file still imports, but it lands as a root and the session records a warning.

## 2. Import it

Register the agent the traces belong to, if you have not, and start a worker:

```bash
kitaru agent register support-agent --command "python support.py"
kitaru worker start
```

Then import:

```bash
kitaru session import logfire-records.jsonl \
  --importer kitaru/logfire@latest \
  --agent support-agent@latest \
  --media-type application/x-ndjson \
  --tag imported-baseline --wait
```

`kitaru/logfire` is one of the built-in importers registered at server startup, so `@latest` always resolves and there is no importer code to write. Use `--media-type application/json` when you upload a JSON array or a `data` object instead of JSONL or streaming NDJSON.

`--tag` labels every session the import creates, so later commands can select them as a group (`kitaru session evaluate --tag imported-baseline ...`). Tagging happens once the import finishes, which is why it requires `--wait`. The receipt reports sessions `created`, `skipped`, and `failed`, with samples of the failures.

List what landed:

```bash
kitaru session list --agent support-agent --origin imported --imported-from logfire
```

### Importer params

| Param             | Meaning                                                                                                                                                                                               |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `source_instance` | Project identity, and half of the session's external id. The importer prefers this, then `project_id`, then each row's own `project_id` column.                                                       |
| `project_id`      | Alternative spelling of the same fallback, checked after `source_instance`.                                                                                                                           |
| `join_on`         | Dotted path or RFC 6901 JSON Pointer selecting the value that groups traces into one session. Omit it to use the defaults below. See [Grouping traces into sessions](#grouping-traces-into-sessions). |
| `framework`       | Extra evidence for framework detection, matched alongside the scope and span names found in the export.                                                                                               |

Pass them with `--params '{"source_instance": "my-logfire-project"}'`, or use the dedicated `--join-on` flag, which accepts a JSON Pointer only (it must start with `/`) and cannot be combined with `join_on` inside `--params`:

```bash
kitaru session import logfire-records.jsonl \
  --importer kitaru/logfire@latest \
  --agent support-agent@latest \
  --join-on '/attributes/customer.case~1id' \
  --media-type application/x-ndjson --wait
```

If no project identity is available from any of those three sources, the importer falls back to `source_instance` `logfire` and says so in the session's warnings.

## What a trace becomes

Every Logfire record becomes one node, and `parent_span_id` is rebuilt as the node tree, so a tool span nested under a model span stays nested. Node type is read from OpenTelemetry GenAI semantics:

| Logfire record                                                                                                                                                                                     | Kitaru node                                                                       |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `gen_ai.operation.name` is `execute_tool`, `tool`, or `tool_call`, or a `gen_ai.tool.name` / `tool.name` / `tool_name` attribute is present                                                        | `tool_call`, with `tool_name` from that attribute (falling back to the span name) |
| `gen_ai.operation.name` is `chat`, `completion`, `embeddings`, `generate_content`, or `text_completion`, or the record carries `gen_ai.request.model`, `gen_ai.response.model`, or `gen_ai.system` | `llm_call`                                                                        |
| Everything else                                                                                                                                                                                    | `span`                                                                            |

Per node, the importer preserves:

* **Inputs**, from the first present of `input`, `inputs`, `raw_input`, `pydantic_ai.all_messages`, `gen_ai.input.messages`, `gen_ai.prompt`, `tool.arguments`, `gen_ai.tool.call.arguments`.
* **Outputs**, from the first present of `output`, `outputs`, `final_result`, `gen_ai.output.messages`, `gen_ai.completion`, `tool.result`, `gen_ai.tool.call.result`.
* **Model identity**: requested model from `gen_ai.request.model`, resolved model from `gen_ai.response.model` (falling back to the requested model), provider from `gen_ai.provider.name` or `gen_ai.system`.
* **Token usage**: input from `gen_ai.usage.input_tokens` or `gen_ai.usage.prompt_tokens`, output from `gen_ai.usage.output_tokens` or `gen_ai.usage.completion_tokens`, cached input from `gen_ai.usage.details.cache_read_tokens` or `gen_ai.usage.cached_input_tokens`, and reasoning tokens from `gen_ai.usage.details.reasoning_tokens`.
* **Cost** from `gen_ai.usage.cost`, `gen_ai.cost.total`, or `operation.cost`.
* **Model parameters** from `gen_ai.request.parameters`, `model_parameters`, `model_request_parameters`, or `model_settings`.
* **Timings** from `start_timestamp` and `end_timestamp`, parsed as ISO 8601.
* **Status**: a record is failed when its status code is `error`, when `is_exception` is true, when `level` is the string `error` or `fatal`, or when `level` is a number of 17 or higher. A failed node carries `exception_message`, `otel_status_message`, or `message` as its error.
* **Attributes**: the record's `kind`, `level`, `message`, and the full decoded `attributes` object are kept on the node under `logfire.*`.
* **Metadata**, allowlisted. From the row's columns: `deployment_environment`, `service_name`, `service_namespace`, `service_version`, `otel_scope_name`, `otel_scope_version`, `tags`. From `attributes`: `agent_name`, `deployment.environment.name`, `gen_ai.agent.name`, `gen_ai.conversation.id`, `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.system`, `service.name`, `service.version`, `session.id`, `session_id`, `thread_id`, `user.id`.

The importer also detects the agent framework (PydanticAI, LangGraph, OpenAI Agents, Google ADK, or the Claude Agent SDK) from the scope names, span names, and `gen_ai.agent.name` attributes in the export, provided the evidence points at exactly one.

### Grouping traces into sessions

Logfire's unit is a trace; a multi-turn conversation is usually several traces. Kitaru groups them:

* By default, traces are grouped by the first of these attribute paths that any record in the trace carries: `attributes.session.id`, `attributes.session_id`, `attributes.conversation_id`, `attributes.thread_id`, `attributes.gen_ai.conversation.id`, `attributes.conversation.id`. Values that Logfire scrubbed (`[redacted]`, `[scrubbed]`) count as absent.
* A trace with none of them becomes its own single-turn session, keyed by trace id, and records the warning `"No session attribute found; grouped by trace id"`.
* With `join_on`, traces are grouped by the scalar at that path instead, which is how you group by your own correlation key. A trace whose records disagree at the path fails with `"Trace '<id>' has conflicting values at join path '<path>'"`; a trace missing your configured value fails with `"Trace '<id>' has no value at join path '<path>'"`. Either way the rest of the file still imports.

Grouped traces become **turns**, ordered by start time. Each turn's inputs and outputs come from that trace's root record, read from the same attribute lists as node inputs and outputs. The session's `inputs` is a versioned turn list (`{"schema_version": 1, "turns": [{"source_trace_id", "inputs", "outputs"}, ...]}`), and the session's outputs come from the last turn. Session status follows the last turn's root record: a tool that failed and was retried successfully leaves the session completed.

Session metadata records the provenance you'll want when reading the import back: `logfire.session_id`, `logfire.project_id`, `logfire.trace_ids`, `logfire.join_paths`, `logfire.services`, `logfire.environments`, `source_trace_count`, `source_completeness`, and `normalization_warnings`.

## Re-runs skip what is already there

Every imported session records its source identity: `imported_from` (`logfire`) and an `external_id` of `<source_instance>:<session>`. That pair is unique on the server, so re-importing an overlapping export **skips** what is already stored and reports it as `skipped`, not as an error. Exporting the last 24 hours every night is safe; it will not duplicate earlier sessions.

It also means the grouping key matters: if you change `source_instance` or `join_on` between imports of the same records, the same conversation lands as a second session rather than deduping against the first.

## Limitations

Because a records query returns exactly the rows you asked for, the importer never claims a session is complete: `source_completeness` is always `query-dependent`. What it did notice goes into `normalization_warnings` on the session:

* `"No Logfire project identity supplied; using source_instance 'logfire'"` when neither the params nor the rows carry a project id.
* `"No session attribute found; grouped by trace id"` when a trace has no conversation identity to group on.
* `"Trace '<id>' has <n> root records"` when a trace has no single root span, usually a query that sliced through the middle of a trace.
* `"Span '<id>' references missing parent '<id>'"` when a `parent_span_id` is not in the file. Those nodes are kept as roots.

Some problems fail one session or one row rather than the file, and are reported as import failures: `"Logfire row lacks trace_id or span_id"`, `"Session '<id>' contains conflicting Logfire project ids"`, `"The import contains duplicate span ids"`, and `"The imported span graph contains a parent cycle"`. A malformed file (invalid JSON, non-UTF-8, empty, no data rows, or over 50 MiB) fails the task as a whole.

Two more things worth knowing before you rely on an import:

* Logfire's own evaluations and alerts do not come across, and neither do metrics or logs that are not span records. Evaluate imported sessions with Kitaru [evaluators](/kitaru/core-concepts/evaluators) instead; backfilling your history is a single batch call.
* Replay re-runs your agent's real code, which no trace export contains. Register the agent version whose code produced these records, with its run command, and imported sessions replay exactly like recorded ones.

{% hint style="warning" %}
An import stores the parsed trace content, including prompts, tool arguments, and tool results, on your Kitaru server. The server is self-hosted, but check your own access and retention rules before importing exports that contain customer data.
{% endhint %}

## Next

Evaluate your imported history with [Write an evaluator](/kitaru/guides/write-an-evaluator), then freeze the sessions that matter into a cohort and put a change to the test with [Build a regression suite from production](/kitaru/guides/regression-suite).


# Arize Phoenix

Import Arize Phoenix trace exports into Kitaru: supported UI and CLI files, span mapping, source identity, and known limits.

If your agent already sends traces to Arize Phoenix, export the runs you care about and import the file into Kitaru. Each Phoenix trace becomes one [session](/kitaru/core-concepts/agents-and-sessions), with its model calls, tool calls, agent spans, timings, status, token usage, and cost preserved where the export records them.

Phoenix stays your system of record. Kitaru stores a runnable copy for evaluation, cohort building, and replay. The importer runs on a [worker](/kitaru/core-concepts/workers) in your environment; the server stores the uploaded file, but does not parse it.

## 1. Export traces from Phoenix

### Phoenix UI

Open a project in Phoenix, select **Traces**, select the traces to export, and choose **Download selection**. In the download dialog:

1. Choose **Traces** for the data.
2. Choose **JSONL** for the format.
3. Include span or trace annotations if you want them retained as import metadata.
4. Download the file.

Phoenix's UI trace download is one flat span object per JSONL line. The file is still a trace export: `context.trace_id` groups its lines, while `context.span_id` and `parent_id` reconstruct the graph. Line order is not significant.

### Phoenix CLI

The importer also accepts the JSON written by Phoenix CLI trace retrieval. A CLI trace object contains `traceId` and a `spans` array, with optional trace `annotations` and `notes`. You can import one object, a JSON array of objects, or JSONL with one trace object per line.

The UI and CLI therefore carry the same span objects in different containers. You do not need to reshape either one. See Phoenix's [trace retrieval guide](https://arize.com/docs/phoenix/tracing/how-to-tracing/importing-and-exporting-traces/retrieve-traces-via-cli) for the current CLI commands.

Payloads are capped at 50 MiB per import. Split a larger export into smaller files.

## 2. Import the file

Register the agent the traces belong to, if needed, and run a worker:

```bash
kitaru agent register support-agent --command "python support.py"
kitaru worker start
```

Then import a Phoenix UI download:

```bash
kitaru session import phoenix-traces.jsonl \
  --importer kitaru/phoenix@latest \
  --agent support-agent@latest \
  --media-type application/x-ndjson \
  --tag imported-baseline \
  --wait
```

Use `--media-type application/json` for a CLI JSON object or array. On Kitaru 0.22.2 and later, `kitaru/phoenix` is a built-in importer registered at server startup, so there is no importer code to register. Older servers do not have it in their catalog; upgrade the server before importing.

List the imported sessions:

```bash
kitaru session list \
  --agent support-agent \
  --origin imported \
  --imported-from phoenix
```

## What becomes a session

The safe default is one Phoenix trace per Kitaru session. The Phoenix `trace_id` becomes the session's `external_id`, so importing the same trace again skips it rather than creating a duplicate. Phoenix session or conversation attributes remain on the span, but this first importer version does not join several traces into one multi-turn session.

Every exported span becomes a node. The importer sorts spans by time and reconstructs their parent relationships instead of trusting export order.

| Phoenix `span_kind`                          | Kitaru node |
| -------------------------------------------- | ----------- |
| `LLM`                                        | `llm_call`  |
| `TOOL`                                       | `tool_call` |
| `AGENT`, `CHAIN`, `UNKNOWN`, and other kinds | `span`      |

`AGENT` remains a plain span because a Phoenix agent span does not by itself prove that Kitaru should treat it as a separately replayable subagent.

The importer reads common OpenInference and OpenTelemetry GenAI attributes for:

* inputs and outputs, including model messages, tool arguments and results, and Google ADK request and response payloads;
* requested and resolved model names, model provider, and model parameters;
* input, output, cached-input, and reasoning token counts;
* recorded cost;
* tool name;
* PydanticAI or Google ADK framework identity when provider-specific attributes establish it.

The original Phoenix attributes and events remain on each node under `phoenix.attributes` and `phoenix.events`. CLI trace annotations and notes remain in session metadata.

## Status and partial exports

Phoenix `ERROR` spans become failed nodes. `OK` and `UNSET` spans become completed nodes because both are terminal states in exported traces. Session status follows the root span, so a tool call that failed and was successfully retried does not incorrectly fail the whole session.

A span whose parent is absent from the file remains importable as a root node. The session records `source_completeness: partial` and a `normalization_warnings` entry. Duplicate span ids and parent cycles fail only the affected trace; other valid traces in the same file still import.

## Limits

* The importer reads files. It does not connect to the Phoenix API or manage Phoenix credentials.
* It supports Phoenix's native JSON and JSONL trace shapes, not arbitrary OTLP JSON envelopes. Export JSONL from the Phoenix UI or JSON with the Phoenix CLI.
* It does not accept JSONL produced by serializing `get_spans_dataframe()`. That table uses flattened top-level column names rather than the UI and CLI span objects.
* It does not import Phoenix datasets, experiments, evaluators, or project configuration. Trace and span annotations included in the export are retained as metadata, but do not become Kitaru evaluations.
* Replay still needs the registered agent code that produced the trace. No trace export contains runnable agent code.

{% hint style="warning" %}
A trace export can contain prompts, tool arguments, tool results, annotations, and exception stack traces. Importing stores that content on your Kitaru server. Check your access and retention rules before importing production data.
{% endhint %}

## Next

Evaluate the imported history with [Write an evaluator](/kitaru/guides/write-an-evaluator), then freeze the sessions that matter into a cohort with [Build a regression suite from production](/kitaru/guides/regression-suite).


# Kitaru JSONL

Import provider traces or portable Kitaru session JSONL into a workspace.

Kitaru importers convert exported trace data into session graphs. Provider importers decode source records, join related traces into sessions, order turns, reconstruct node relationships, and project common fields for the UI while preserving source inputs and outputs.

Use a provider importer for Langfuse, LangSmith, Braintrust, Logfire, or Arize Phoenix data. Use the `kitaru-jsonl` importer when your producer already emits the Kitaru session and node contract.

## The portable session contract

Each imported session contains session fields and a list of nodes. A session is the user-visible execution or conversation. A node is one recorded model call, tool call, subagent call, or span.

| Session field            | Type                                    | Meaning                                                                                            |
| ------------------------ | --------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `status`                 | `in_progress`, `completed`, or `failed` | Final source status.                                                                               |
| `name`                   | string or null                          | Display name.                                                                                      |
| `inputs`                 | any JSON value                          | Complete session input. Provider importers use a versioned `turns` object for multi-turn sessions. |
| `outputs`                | any JSON value                          | Final session output.                                                                              |
| `error`                  | string or null                          | Failure message.                                                                                   |
| `started_at`, `ended_at` | ISO 8601 timestamp or null              | Session time range.                                                                                |
| `external_id`            | string                                  | Stable identity in the source system. Kitaru uses it with `imported_from` for deduplication.       |
| `metadata`               | JSON object                             | Source identity, normalization warnings, and user metadata.                                        |
| `imported_from`          | string or null                          | Source importer. Kitaru sets this from the selected importer rather than the JSONL record.         |
| `framework`              | string or null                          | Agent framework when the trace identifies one, such as `pydantic-ai` or `langgraph`.               |
| `nodes`                  | node array                              | Flat indexed nodes.                                                                                |

Each node uses the fields below. Optional fields can be omitted or set to null.

| Node field                                   | Type                                                | Meaning                                                                                                                             |
| -------------------------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `index`                                      | integer                                             | Stable position within the session import. Parents must have lower indexes.                                                         |
| `parent_index`                               | integer or null                                     | Primary parent.                                                                                                                     |
| `secondary_parent_indexes`                   | integer array                                       | Additional parents for graph joins.                                                                                                 |
| `external_id`, `trace_id`                    | string or null                                      | Source node and trace identities.                                                                                                   |
| `node_type`                                  | `llm_call`, `tool_call`, `subagent_call`, or `span` | Work represented by the node.                                                                                                       |
| `name`                                       | string                                              | Display name.                                                                                                                       |
| `status`                                     | `in_progress`, `completed`, or `failed`             | Node status.                                                                                                                        |
| `error`                                      | string or null                                      | Failure message.                                                                                                                    |
| `started_at`, `ended_at`                     | ISO 8601 timestamp or null                          | Node time range.                                                                                                                    |
| `input_text_selector`                        | string or null                                      | RFC 6901 JSON Pointer selecting the primary human-readable text inside `inputs`.                                                    |
| `output_text_selector`                       | string or null                                      | RFC 6901 JSON Pointer selecting the primary human-readable text inside `outputs`.                                                   |
| `system_prompt_selector`                     | string or null                                      | RFC 6901 JSON Pointer selecting the system prompt inside `inputs`.                                                                  |
| `reasoning`                                  | string or null                                      | Visible reasoning text when the source exports it.                                                                                  |
| `inputs`, `outputs`                          | any JSON value                                      | Complete source payloads. Importers preserve message history, tool arguments, multimodal parts, and provider-specific content here. |
| `requested_model`, `model`, `model_provider` | string or null                                      | Requested model, served model, and model provider.                                                                                  |
| `tokens`                                     | object or null                                      | Input, output, cached input, and reasoning token counts when reported.                                                              |
| `cost`                                       | decimal or null                                     | Recorded or estimated call cost.                                                                                                    |
| `model_params`                               | object or null                                      | Model request parameters.                                                                                                           |
| `tool_name`, `subagent_id`                   | string or null                                      | Tool or subagent identity for the matching node type.                                                                               |
| `attributes`                                 | any JSON value                                      | Span attributes retained for diagnostics.                                                                                           |
| `metadata`                                   | JSON object                                         | Bounded source metadata.                                                                                                            |

Text selectors avoid copying potentially large values into separate columns. A selector is present only when the importer can identify one relevant string in the corresponding payload. A client resolves that [RFC 6901 JSON Pointer](https://www.rfc-editor.org/rfc/rfc6901.html) when it loads the node payload and can show the complete `inputs` or `outputs` value for inspection. The selectors remain available in node list responses without loading the payload columns. `system_prompt_selector` resolves against `inputs`. A null selector means the importer could not choose one text value without guessing. The empty string is the JSON Pointer for the complete payload, which is useful when the payload itself is the selected string.

`reasoning` contains visible text only. Redacted, encrypted, or unavailable reasoning remains null, while the provider payload stays in `inputs` or `outputs`. Token usage can also include `reasoning_tokens` when a provider reports the count.

## Create Kitaru JSONL

Write one session object per line. The `kitaru-jsonl` importer validates every field and rejects unknown fields. Invalid lines are reported independently, so valid sessions in the same upload can still import.

The formatted object below represents one JSONL record. Serialize it onto one line in the file.

```json
{
  "status": "completed",
  "name": "Weather request",
  "inputs": {"question": "What is the weather in Delft?"},
  "outputs": {"answer": "Delft is rainy and 18 C."},
  "started_at": "2026-07-22T10:00:00Z",
  "ended_at": "2026-07-22T10:00:01Z",
  "external_id": "weather-session-42",
  "metadata": {"environment": "production"},
  "framework": "pydantic-ai",
  "nodes": [
    {
      "index": 0,
      "parent_index": null,
      "secondary_parent_indexes": [],
      "external_id": "model-call-42",
      "trace_id": "trace-42",
      "node_type": "llm_call",
      "name": "answer weather question",
      "status": "completed",
      "started_at": "2026-07-22T10:00:00Z",
      "ended_at": "2026-07-22T10:00:01Z",
      "input_text_selector": "/1/content",
      "output_text_selector": "/0/content",
      "system_prompt_selector": "/0/content",
      "reasoning": "The weather tool reports rain and a temperature of 18 C.",
      "inputs": [{"role": "system", "content": "Answer in one sentence."}, {"role": "user", "content": "What is the weather in Delft?"}],
      "outputs": [{"role": "assistant", "content": "Delft is rainy and 18 C."}],
      "model": "claude-haiku-4-5-20251001",
      "model_provider": "anthropic",
      "tokens": {"input_tokens": 24, "output_tokens": 11, "cached_input_tokens": 0, "reasoning_tokens": 0},
      "attributes": {},
      "metadata": {}
    }
  ]
}
```

Node indexes do not need to be contiguous. Every `parent_index` and `secondary_parent_indexes` value must be lower than the child index. A node index must be unique within its session.

## Import a file

The session import command uploads the file, resolves an exact importer and agent version, and creates an import job:

```bash
kitaru session import sessions.jsonl \
  --importer kitaru/kitaru-jsonl@latest \
  --agent customer-service@latest \
  --media-type application/x-ndjson \
  --wait
```

Use `--tag` with `--wait` to tag every created session. Use `--join-on` to group provider traces by a source value. Use `--params` for other provider-specific settings.

## Join provider traces into sessions

Providers often record one conversation turn as one trace. Importers group related traces into one Kitaru session, then order the traces by start time with a stable trace-ID tie-breaker.

Default grouping uses the provider's native conversation or session identifier. When that identifier is absent, each trace becomes one session. Use `--join-on` when the export carries the shared session identity in another field.

The option takes an [RFC 6901 JSON Pointer](https://www.rfc-editor.org/rfc/rfc6901.html) that selects one scalar value from each source trace:

```bash
kitaru session import langfuse-observations.jsonl \
  --importer kitaru/langfuse@latest \
  --agent customer-service@latest \
  --join-on '/metadata/customer/case_id' \
  --wait
```

The example reads the scalar at `/metadata/customer/case_id`. Five traces with the value `case-42` become five ordered turns in the same Kitaru session. Traces with a different value form a different session.

Escape source keys according to RFC 6901. Use `~1` for `/` and `~0` for `~`. For example, `/metadata/customer~1case~0id` selects the key `customer/case~id` inside `metadata`.

The pointer root depends on the importer:

| Importer   | Pointer root                                                                | Example                       |
| ---------- | --------------------------------------------------------------------------- | ----------------------------- |
| Braintrust | Each raw trace-root record                                                  | `/metadata/customer~1case_id` |
| Langfuse   | Observation records belonging to one trace; every selected value must agree | `/metadata/customer/case_id`  |
| LangSmith  | Each raw trace-root run                                                     | `/extra/metadata/thread_id`   |

The selected value must be a non-empty string, number, or boolean. A missing, conflicting, object, or array value produces an isolated failure for that trace. Kitaru does not silently place the trace into a fallback session. Imported metadata records explicit grouping provenance under `braintrust.join_on`, `langfuse.join_paths`, or `langsmith.join_paths`.

### SDK and REST

The CLI validates `--join-on` and adds it to the importer parameter object. SDK callers pass the same `join_on` parameter directly:

```python
from kitaru.api_models.v1.imports import ImportCreateRequest

job = await client.imports.create(
    ImportCreateRequest(
        importer="kitaru/langfuse",
        version=1,
        agent_id=agent_id,
        agent_version_id=agent_version_id,
        payload_blob_id=blob_id,
        params={"join_on": "/metadata/customer/case_id"},
    )
)
```

The REST request uses the same structure:

```json
{
  "importer": "kitaru/langfuse",
  "version": 1,
  "agent_id": "00000000-0000-0000-0000-000000000000",
  "agent_version_id": "00000000-0000-0000-0000-000000000001",
  "payload_blob_id": "00000000-0000-0000-0000-000000000002",
  "params": {"join_on": "/metadata/customer/case_id"}
}
```

Send this object to `POST /api/v1/imports`. The server stores `params` on the import task, the worker includes them in `ImportTaskDetails`, and the task process calls the selected importer as `parse(payload, params)`.

Existing integrations can continue to send `params.join_on` as a dotted path. The explicit CLI option accepts JSON Pointer syntax only. Langfuse also retains its older `join_path` plus `join_key` parameters for compatibility, but new integrations should use `join_on`.

## What provider importers normalize

Provider importers apply the same output contract to different source formats:

| Source     | Accepted shape                                        | Default grouping                                    |
| ---------- | ----------------------------------------------------- | --------------------------------------------------- |
| Langfuse   | Trace, observation, and ingestion-event JSON or JSONL | `sessionId`, then `traceId`                         |
| LangSmith  | Run-query and bulk-export JSON or JSONL               | Known thread metadata paths, then `trace_id`        |
| Braintrust | Project-log and UI JSON exports                       | Known session or conversation fields, then trace ID |
| Kitaru     | One portable Kitaru session per JSONL line            | No grouping; each line is one session               |

Normalization includes source identity, parent-child graph reconstruction, deterministic ordering, status and error mapping, model fields, token counts, cost, tool arguments and results, text selectors, visible `reasoning`, and framework detection. Source payloads remain in `inputs` and `outputs`. Session metadata reports normalization warnings and source completeness.

Framework detection only sets `framework` when trace metadata identifies one supported framework without conflict. Unknown or sparse traces keep the field null.

## Inspect failures

The import job result reports created, skipped, and failed counts plus a bounded failure sample. Reimporting the same `(imported_from, external_id)` pair skips the duplicate.

## No importer for your provider

You have two ways in, and neither requires waiting for us to ship an importer.

**Convert to Kitaru JSONL.** Write out [Kitaru JSONL](#create-kitaru-jsonl), one session object per line, exactly the contract above. This is the right choice for a one-off backfill or an export you can transform with a script. Nothing gets installed or registered.

**Write an importer.** Worth it when the conversion is ongoing, or when the source needs real normalization rather than a field rename. The contract is one function:

```python
Parser = Callable[[bytes, dict[str, Any]], Iterator[ImportedSession | ImportFailure]]
```

That is the whole interface. You receive the uploaded bytes and the `--params` object, then yield one `ImportedSession` per session you recognize, or an `ImportFailure` for a record you cannot parse. A yielded failure isolates that record instead of failing the whole import:

```python
from kitaru.api_models.v1.imports import ImportFailure
from kitaru.task.importer import ImportedSession


def parse(
    content: bytes, params: dict[str, Any]
) -> Iterator[ImportedSession | ImportFailure]:
    for line_number, line in enumerate(content.decode("utf-8").splitlines(), start=1):
        try:
            yield ImportedSession.model_validate(transform(json.loads(line)))
        except ValueError as exc:
            yield ImportFailure(line=line_number, external_id=None, error=str(exc))
```

Three things matter most because they are where custom importers usually go wrong:

* **`external_id` is your identity, and it must be stable.** Kitaru deduplicates on `(imported_from, external_id)`, so a re-import is only safe if the id does not move between runs. Derive it from the source's own identifier, never from a row number or a timestamp.
* **Decide session boundaries deliberately.** One `ImportedSession` should be one end-to-end run; see [what a session is](/kitaru/core-concepts/agents-and-sessions). If your source splits a run across records, join them in the parser.
* **Yield failures, don't raise them.** An exception ends the import; an `ImportFailure` costs you one record and keeps the rest.

The shipped importers are the reference: `plugins/packages/jsonl-importer` is the smallest at under 80 lines, and the Langfuse one shows real normalization. The `kitaru-importer-builder` [agent skill](/kitaru/getting-started/setup) exists for this job: it turns a representative export into a locally validated importer, keeps the mapping from source evidence to normalized sessions explicit so you can see what is preserved, approximated, or unavailable, and finishes locally until you approve registration:

```bash
npx skills add zenml-io/kitaru-skills
```


# No importer for your format

No built-in importer for your format? An importer is one Python callable: scaffold it, test it offline, register it, and your export imports like any other.

The built-in importers cover [Langfuse](/kitaru/import-your-traces/import-langfuse-traces), [LangSmith](/kitaru/import-your-traces/import-langsmith-traces), [Braintrust](/kitaru/import-your-traces/import-braintrust-traces), [Logfire](/kitaru/import-your-traces/import-logfire-traces), [Arize Phoenix](/kitaru/import-your-traces/import-phoenix-traces), and the [Kitaru JSONL contract](/kitaru/import-your-traces/importing-sessions). Any other trace store, or a homegrown logging format, comes in through a custom importer. An importer is small by design: one callable that parses your export bytes into sessions, usually about a page of Python.

There are two ways to get one, and the fast way is to not write it yourself: the `kitaru-importer-builder` [agent skill](/kitaru/getting-started/setup) turns a representative export into a locally validated importer. It keeps the mapping from source evidence to normalized sessions explicit, so you can see what is preserved, approximated, or unavailable, and it finishes on your machine until you approve registration.

## The contract

```python
from collections.abc import Iterator
from typing import Any

from kitaru.task.importer import ImportFailure, ParsedNode, ParsedSession


def parse(
    payload: bytes, params: dict[str, Any]
) -> Iterator[ParsedSession | ImportFailure]:
    for line_number, line in enumerate(payload.splitlines(), start=1):
        try:
            record = decode_my_format(line)
        except ValueError as error:
            yield ImportFailure(line=line_number, error=str(error))
            continue
        yield ParsedSession(
            status="completed",
            name=record.title,
            inputs=record.question,
            outputs=record.answer,
            error=None,
            started_at=record.started_at,
            ended_at=record.ended_at,
            external_id=record.trace_id,
            metadata={},
            nodes=[
                ParsedNode(
                    node_type="llm_call",
                    name="model",
                    status="completed",
                    inputs=record.prompt,
                    outputs=record.completion,
                ),
            ],
        )
```

Yield lazily; the import consumes one item at a time, so payload size is bounded by disk, not memory. Yield an `ImportFailure` for a bad record and the import counts it and moves on. Only a crash of the parser itself fails the task, with partial stats preserved. The full field reference for `ParsedSession` and `ParsedNode` is the [portable session contract](/kitaru/import-your-traces/importing-sessions).

Set a stable `external_id` from your source system: together with the importer's provider name it is the dedup key, so re-importing an overlapping export skips what is already stored instead of duplicating it.

## Scaffold, test offline, register

```bash
kitaru importer scaffold my-format          # writes my_format_importer.py
kitaru importer test my_format_importer.py \
  --entrypoint parse --payload sample-export.jsonl
kitaru importer register my-format \
  --script my_format_importer.py --entrypoint parse --provider my-format
```

A script importer may declare dependencies as PEP 723 inline metadata (a `# /// script` block); the worker builds it an isolated environment. An importer that outgrows one file ships as a package instead: `--package "my-importer==1.0.0"` with `--entrypoint "my_importer:parse"`. Importers are versioned like evaluators and agents; imports name the importer and pin to its latest version unless you pass one.

Once registered, your format imports exactly like the built-in ones:

```bash
kitaru session import my-export.jsonl \
  --importer my-format@latest \
  --agent support-agent@latest --wait
```

The shipped importers are the reference implementations: `plugins/packages/jsonl-importer` is the smallest at under 80 lines, and the Langfuse one shows real normalization with turn grouping and warnings.

{% hint style="warning" %}
Imported payloads contain whatever your traces contain: prompts, customer data, tool results. They are stored on your self-hosted server and parsed on your workers, but access and retention are yours to govern.
{% endhint %}

## Next

Evaluate the history you imported with [Write an evaluator](/kitaru/guides/write-an-evaluator), then freeze the sessions that matter into a cohort and put a change to the test with [Build a regression suite from production](/kitaru/guides/regression-suite).




---

[Next Page](/llms-full.txt/1)