Welcome to Kitaru

Replay-based evals for AI agents, in Python and TypeScript. Test your next change against what your agent already did in production.

Your agent has already been tested thousands of times in production. Most of that evidence is sitting in a trace store as something you can read but not run. Kitaru makes it runnable: it records or imports each run as a session, then replays it against your real code, with the recording answering for the world the original run saw. Change the prompt, swap the model, or point replay at the fix in your working tree, and see what improved and what broke before it ships.

Who it's for: Teams with an agent in front of real users, where regression testing today means re-running a few samples and eyeballing the output. Kitaru replaces that with evaluators, cohorts, and experiments over your actual traffic. If you're prototyping and haven't shipped, it will feel like more machinery than you need.

Frameworks: adapters ship for PydanticAI, LangGraph, and the OpenAI Agents SDK in Python, and for Mastra and the Vercel AI SDK in TypeScript. Other frameworks still work: import your traces with the built-in Langfuse, LangSmith, Braintrust, Logfire, Arize Phoenix, MLflow, or JSONL importers; write a custom importer, usually about a page of Python; or build a small adapter, where the recording API is two client calls.

Kitaru has both a Python and a TypeScript SDK, and both talk to the same server. The CLI ships with the Python package.

Kitaru is built to be driven by agents. The MCP server gives Claude Code, Codex, Cursor, and other coding assistants bounded Kitaru operations. The agent skills teach the procedures, and the CLI speaks JSON when a shell command is the right tool. You bring the judgment; your assistant handles the investigation work. Set up your coding agent takes a few minutes.

Kitaru is open source (Apache 2.0) and self-hosted, from the team behind ZenML: ZenML is for ML pipelines, Kitaru is for agents.

The loop

• Record. Wrap your agent or import your traces (both shown below). Either way, runs land as sessions.
• Replay. Re-execute a session against your real code. Tool calls are answered from the recording, so nothing touches real systems. An unchanged replay gives you the faithful baseline. Then fork it with a different model, a new prompt, or your working tree's code.
• Improve. This is where your judgment enters. In an investigation, your coding assistant authors the review, walks you through the evidence, asks the questions Kitaru needs answered, and pins your answers to the exact trace as annotations. Those judgments calibrate the evaluators that evaluate both sides; cohorts freeze the population; experiments replay a cohort against a change and show what improved and what regressed. The cohort that caught a failure becomes the regression gate that keeps it caught.

In daily work, that loop becomes five steps: observe a recorded behavior, judge what should have happened, define the behavior to test, replay the changed agent, and compare the evidence. Recording gives you the raw material; observe, judge, and define turn human judgment into durable criteria; replay and compare close the loop. The Quickstart walks all five. To try it in a controlled environment, ask your assistant for the kitaru-guided-tour skill, which runs the loop on the PydanticAI returns agent example, or follow the complete returns agent tutorial manually.

Do I have to run it in production?

No. There are two ways to get sessions, and they end in the same place:

• Import the history you already have. If your agent logs to Langfuse or anything else you can export from, import it. Nothing in your production path changes: your trace store stays your system of record, and Kitaru gets a runnable copy.
• Record with an adapter. Wrap the agent once, no rewrite, and every run becomes a session wherever the agent runs: production, staging, or your laptop.

Import traces

kitaru session import langfuse-export.jsonl \
  --importer kitaru/langfuse@latest \
  --agent support-agent@latest --wait

Record with an adapter

from pydantic_ai import Agent
from kitaru_pydantic_ai import KitaruAgent

agent = Agent(
    "openai:gpt-5.4", name="support-agent", system_prompt="You resolve support tickets."
)


@agent.tool_plain
def refund_payment(order_id: str) -> str:
    return payments.refund(order_id)  # your real API


support = KitaruAgent(agent, agent_id=AGENT_ID)
support.run_sync("Refund order #4821, the card reader double-charged me.")

Replays, imports, and evaluations run offline on workers in your environment. None of that touches your production traffic. An adapter does run inside your agent's process to record; if you do not want Kitaru near production, the import path never gets close to it.

Built to sit in your stack

• Self-hosted. One FastAPI + Postgres server on your infrastructure. Your traces and credentials don't leave your systems.
• Beside your observability, not instead of it. Langfuse, LangSmith, Braintrust, Logfire, Arize Phoenix, and MLflow remain where you watch production. Kitaru is where you re-run it.
• Choose how you drive it: the kitaru CLI, Python SDK, TypeScript SDK, and your coding agent. Kitaru observes your production agents; your coding assistant is how you talk to Kitaru.

Questions, bugs, feedback? Join the Slack community, report bugs at kitaru.ai/help (it goes straight to GitHub issues), or email [e-mail]. All three reach a human.

Next steps

Installation

SDK, CLI, a local server, and a login.

Quickstart

Understand the five-step method before running commands.

PydanticAI returns agent

Prepare a ready agent and checked-in Langfuse traces.

Complete tutorial

Investigate and replay the example's synthetic returns agent.

Import your traces

Start from the history you already have.

Core Concepts

Sessions, replay, evaluators, cohorts, experiments.

Build a regression suite

Production traffic as your test suite.

Deploy Kitaru

Self-host for your team.