> For the complete documentation index, see [llms.txt](https://docs.zenml.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zenml.io/kitaru/core-concepts/experiments.md).

# Experiments

A named change, replayed across a cohort and evaluated. What improved, what regressed, before you ship.

A [replay](/kitaru/core-concepts/replay.md) is one counterfactual. An **experiment** is that counterfactual at population scale: take a [cohort](/kitaru/core-concepts/cohorts.md) of real runs, apply one change to all of them, evaluate every re-run with the same [evaluators](/kitaru/core-concepts/evaluators.md), and read what improved and what regressed.

The split of responsibilities is deliberate:

* The **experiment** holds the *change*: an override (model, prompt, params), a tool policy, and the evaluator list. It is reusable.
* An **experiment run** supplies the *population and the code*: one cohort version and one agent version. Run the same experiment against next week's cohort version, or the same cohort against your PR's agent version.

```python
import asyncio
import os
import uuid

from kitaru.client import KitaruAPIClient
from kitaru.api_models.v1.experiment import ExperimentCreateRequest
from kitaru.api_models.v1.experiment_run import ExperimentRunCreateRequest
from kitaru.api_models.v1.plugin import EvaluatorConfig
from kitaru.api_models.v1.replay_config import (
    HistoryConfig,
    ReplayOverride,
    ToolPolicy,
)


async def main() -> None:
    client = KitaruAPIClient()
    agent_id = uuid.UUID(os.environ["KITARU_AGENT_ID"])

    experiment = await client.experiments.create(
        ExperimentCreateRequest(
            agent_id=agent_id,
            name="cheaper-model",
            description="Would gpt-5-nano have held on refund tickets?",
            override=ReplayOverride(model={"openai:gpt-5.4": "openai:gpt-5-nano"}),
            tool_policy=ToolPolicy(
                default=HistoryConfig(scope="cohort_version", on_miss="fail")
            ),
            evaluators=[EvaluatorConfig(evaluator="refund-check")],
        )
    )

    run = await client.experiments.start_run(
        experiment.id,
        ExperimentRunCreateRequest(
            cohort_version_id=COHORT_VERSION_ID,
            agent_version_id=AGENT_VERSION_ID,
            evaluate_baselines=True,
        ),
    )
    print(run.id, run.status, run.progress)


asyncio.run(main())
```

{% hint style="info" %}
The OpenAI Agents adapter does not support a \`history\` default. Keep its default as \`passthrough\` and add a named \`history\` override for each direct function tool you want to replay. See the \[OpenAI Agents adapter page]\(../adapters/openai-agents.md).
{% endhint %}

The same two steps from the CLI (the change as JSON on the experiment, the population and code on the run):

```bash
kitaru experiment create cheaper-model \
  --agent support-agent \
  --evaluator refund-check@latest \
  --override '{"model": {"openai:gpt-5.4": "openai:gpt-5-nano"}}' \
  --tool-policy '{"default": {"type": "history", "scope": "cohort_version", "on_miss": "fail"}}'

kitaru experiment run start cheaper-model \
  --cohort-version <cohort-version-id> \
  --agent support-agent@1 \
  --evaluate-baselines --wait
```

Starting a run fans out **one replay per session** in the cohort version. [Workers](/kitaru/core-concepts/workers.md) in your environment execute them; the run's `progress` counts replays through `pending → evaluating → completed` (plus `failed` / `canceled`), and the run settles when the last replay does. `evaluate_baselines=True` evaluates the original sessions too, so every replay has its baseline numbers to sit next to.

With a `history` tool policy scoped to `cohort_version`, replayed tool calls can be answered from any recording in the cohort (useful when runs share tool traffic), and `on_miss="fail"` keeps anything unrecorded from reaching a live system.

## Reading a run

A run's output is intentionally plain: its replays, each with a result session, and the evaluation rows on both sides. Compare them by reading the evaluations:

```python
from kitaru.api_models.v1.evaluation import EvaluationListParams
from kitaru.api_models.v1.filter import FilterCondition, FilterOp

async for evaluation in client.evaluations.iter(
    EvaluationListParams(
        filter=FilterCondition(field="session_id", op=FilterOp.EQ, value=session_id)
    )
):
    print(evaluation.name, evaluation.score, evaluation.passed)
```

Numbers average, booleans count into pass rates, categorical labels diff as transitions, and free text gets read. Cost and token totals ([tracked per model call](/kitaru/guides/llm-calls.md)) ride on each result session, so "the cheaper model held on 18 of 20 tickets and cut cost 41%" is two loops over stored rows. The end-to-end workflow, including gating CI on a frozen cohort version, is in [Build a regression suite from production](/kitaru/guides/regression-suite.md).

A failed replay fails the run: the comparison the experiment exists for cannot be produced for that session, and the numbers never silently shrink their denominator. Watch a run with `kitaru experiment run watch <run>`, inspect its jobs with `kitaru experiment run jobs <run>`, and cancel with `kitaru experiment run cancel <run>`; already finished replays keep their results.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.zenml.io/kitaru/core-concepts/experiments.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.