> For the complete documentation index, see [llms.txt](https://docs.zenml.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zenml.io/kitaru/core-concepts/under-the-hood.md).

# Under the Hood

The machinery behind the loop: a FastAPI server on Postgres, workers claiming tasks, and where Kitaru sits beside your observability stack.

You can use Kitaru without reading this page. Read it when you want to know what happens between "start a replay" and "read the diff", or when you are deciding where Kitaru sits in your stack.

## Two processes, one contract

Kitaru is a **server** and your **workers**.

The server is a single FastAPI service backed by Postgres. It stores every resource (agents, sessions and their nodes, cohorts, evaluators, experiments, replays, secrets, tags) and exposes them over a plain versioned REST API (`/api/v1/...`). It coordinates work but executes none of it: there is no code execution on the server, ever.

[Workers](/kitaru/core-concepts/workers.md) run in your environment and pull work from the server. Everything that executes (a replayed agent, an evaluator, an importer parsing a trace export) runs as a subprocess of a worker, next to your credentials, packages, and network. The server never needs access to your model providers or your tools.

Between them sits the **job/task** layer. Commands like "replay this session," "import this export," or "evaluate these sessions" create a job holding one or more tasks; every job carries its kind (`session_run`, `import`, `evaluation`, `replay`, `analysis`), so `kitaru job` listings filter cleanly. Workers claim tasks scoped by *task* kind (`agent`, `evaluator`, `importer`, a different axis than job kinds) or by label, heartbeat while running them, and report results. Crashed workers lose their claim; the server requeues or fails the task, so no replay is ever silently stranded. `kitaru job watch <id>` follows any of it live.

Writes are safe to retry: the client stamps every POST request with an `Idempotency-Key` header, held stable across the transport's own retries, and the server stores the first committed response for that key, scoped to your account. A replay or evaluation request that times out on the wire and gets retried never becomes two replays: the retry gets the original response back, marked with an `Idempotent-Replayed: true` header instead of running again. Reusing a key with a different request body is rejected with 422. A failed request stores nothing, so a retry after an error re-executes normally. Stored keys expire after `KITARU_SERVER_IDEMPOTENCY_KEY_RETENTION_SECONDS` (15 minutes by default) and are cleared by the same sweep loop that requeues tasks.

You can also supply the key yourself. Every SDK method that calls an endpoint supporting idempotency takes an `idempotency_key` argument, which replaces the generated one:

```python
await client.api.replays.create(request, idempotency_key=f"nightly-{date}")
```

The endpoints that honor a key are the ones whose OpenAPI operation declares an `Idempotency-Key` header parameter, and the SDK exposes the argument on exactly those methods. A key is at most 255 printable characters and is scoped to your account rather than to a user, so two members of one account choosing the same key collide. Calling again with the same key while the first call is still running returns 409. Retention applies to your own keys too, so they protect against retries and double submits inside the retention window rather than acting as a permanent uniqueness constraint.

## How replay works

1. `POST /api/v1/replays` stores the replay (baseline session, agent version, override, tool policy, evaluators) and creates its job with one agent task.
2. A worker claims the task and starts your agent from the agent version's run spec command, with the baseline's inputs (rewritten by the override, if any) and `KITARU_REPLAY_ID` in the environment.
3. Your agent runs for real. The adapter sees `KITARU_REPLAY_ID`, fetches the override and tool policy, applies model swaps at the model-call boundary, and answers tool calls per policy; a `history` policy looks up the recorded result by a hash of the tool name and arguments.
4. The re-run records a fresh session, node by node, `origin: replay`.
5. When the agent task completes, the server appends one evaluator task per configured evaluator; workers evaluate the result session and, with `evaluate_baselines`, the baseline.
6. The job settles, the replay settles, and, inside an experiment run, the run's progress advances. Results are stored rows: the result session, its nodes, its evaluations.

An [experiment run](/kitaru/core-concepts/experiments.md) is this pipeline fanned out once per session in a cohort version. Nothing about scale changes the mechanics.

## Storage and blobs

Session payloads (inputs, outputs, node payloads) live in Postgres. Uploaded artifacts (trace exports to import, script plugin code) are **blobs**: content-addressed by SHA-256, deduplicated, capped by a server setting. Workers cache blobs locally by hash, so a hundred evaluator runs fetch the evaluator's code once.

Auth is deliberately simple: [API keys](/kitaru/running-in-production/authentication.md) (`KITKEY_` prefix) or a login token, one trusted team per deployment. Ownership records who created a resource; it does not gate access. Workers and their task subprocesses never hold your key for long; they operate on short-lived tokens scoped to one worker or one task.

## Where Kitaru sits in your stack

Kitaru is a debugger with a memory, sitting **beside** your observability stack, not replacing it. Langfuse, LangSmith, Braintrust, Logfire, and Arize Phoenix remain your system of record for traces; Kitaru holds runnable copies of the runs you care about and the machinery to re-execute and evaluate them. The [import path](/kitaru/import-your-traces/import-your-traces.md) is that bridge.

On the other side, Kitaru deliberately does **not** run your production agent. Your agent runs wherever it runs today; the adapter records it. Durable execution of agents in production is [ZenML](https://docs.zenml.io)'s job: ZenML runs agents durably; Kitaru replays and improves them.

Everything here is open source (Apache 2.0) and [self-hosted](/kitaru/getting-started/deploy.md): your server, your Postgres, your workers, your data.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.zenml.io/kitaru/core-concepts/under-the-hood.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.