> For the complete documentation index, see [llms.txt](https://docs.zenml.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zenml.io/kitaru/core-concepts/evaluators.md).

# Evaluators & Evaluations

Evaluators turn sessions into evaluations: the rows they write. Numbers average, words count, free text gets read.

Replay tells you what a change *did*; evaluators tell you whether it *helped*. An **evaluator** is a small piece of your code that reads one [session](/kitaru/core-concepts/agents-and-sessions.md), node by node, and writes one or more **evaluations**: named, typed verdicts that Kitaru stores against the session.

Because evaluators run against recorded sessions, they evaluate baselines, replays, and imported traces identically. The same evaluator you run over today's production traffic runs over the fork you're thinking about shipping.

An evaluator that calls a model or another external service can declare its provider and a connection schema, the environment variables its SDK reads, so a [connection](/kitaru/import-your-traces/provider-connections.md) supplies the credentials at run time.

## The evaluator contract

An evaluator is a callable (a single Python file or an installable package) that receives the full session and returns results:

```python
"""refund_check.py: did the agent issue the refund?"""

from kitaru.task.evaluator import EvaluationResult, SessionView


def evaluate(session: SessionView, **params) -> EvaluationResult:
    refund_calls = [
        node
        for node in session.nodes
        if node.node_type == "tool_call" and node.tool_name == "refund_payment"
    ]
    return EvaluationResult(
        name="refund_issued",
        score=bool(refund_calls),
        passed=bool(refund_calls),
        explanation=f"{len(refund_calls)} refund tool call(s) in the session",
    )
```

`SessionView` gives you the session and all its nodes with payloads. Return one `EvaluationResult` or a list; each becomes one stored evaluation row. `params` are per-run knobs you pass when you attach the evaluator to a replay or experiment.

Scaffold, exercise, and register it with the CLI:

```bash
kitaru evaluator scaffold refund-check          # writes refund_check_evaluator.py
kitaru evaluator test refund_check_evaluator.py --entrypoint evaluate
kitaru evaluator register refund-check \
  --script refund_check_evaluator.py --entrypoint evaluate
```

Evaluators are versioned like agents: registering again with `kitaru evaluator version register` creates version 2, and every stored evaluation remembers exactly which evaluator version wrote it. An LLM judge follows the same contract by calling a model inside `evaluate`. The walkthrough is in [Write an evaluator](/kitaru/guides/write-an-evaluator.md).

A suite of evaluators comes **built in**, registered at server startup under the `kitaru/` namespace: three cheap signals (`kitaru/cost`, `kitaru/latency`, `kitaru/tool-call-patterns`) plus ten deterministic checks over the recording itself, from `kitaru/output-contract` and `kitaru/tool-health` to `kitaru/timing-profile` and `kitaru/workflow-conformance`. None of them make model calls; they are the triage layer, available as `kitaru/cost@latest` before you have written anything.

## The evaluation row

One evaluation is one named result for one session. The data type is derived from what you set, never declared:

| You set                     | Stored type   | How to read a batch of them    |
| --------------------------- | ------------- | ------------------------------ |
| `score=0.87`                | `float`       | numbers average                |
| `score=True`                | `bool`        | pass rates count               |
| `value="escalated"`         | `str`         | free text gets read            |
| `score=0.9, value="polite"` | `categorical` | labels count, transitions diff |

`passed` is an independent optional verdict, based on a threshold you decided in the evaluator rather than something derived from `score`. `explanation` says why, which is the part you read when a regression gate goes red.

## Human labels are evaluations too

There is no separate labeling system. A human verdict is an evaluation written directly onto the session:

```python
from kitaru.api_models.v1.evaluation import EvaluationResult
from kitaru.api_models.v1.session import SessionEvaluationsRequest

await client.sessions.create_evaluations(
    session_id,
    SessionEvaluationsRequest(
        evaluations=[
            EvaluationResult(
                name="human_quality",
                score=True,
                explanation="Correct refund, good tone",
            ),
        ]
    ),
)
```

Manual evaluation names are unique per session: sending `human_quality` again fails rather than overwriting the earlier verdict. Rows written by evaluator runs carry the evaluator version that produced them, forever, even after that evaluator is deleted; manual rows carry none, which is how you tell them apart. Comparing your evaluator's column against the human column on the same sessions is how you calibrate the evaluator before you let it gate anything. The human column usually comes out of [the interview](/kitaru/core-concepts/investigations.md): your coding assistant authors the investigation, and your answers land as annotations to calibrate against.

## Running evaluators in batch

Evaluate existing sessions without replaying anything. From the CLI, select by IDs, by tag, by agent, by cohort version, by filter, or everything:

```bash
kitaru session evaluate --tag imported-baseline \
  --evaluator refund-check@latest --evaluator kitaru/cost@latest \
  --wait
```

Exactly one selection is required: explicit session IDs (arguments or `--sessions-file`), `--tag`, `--agent`, `--cohort`, `--filter`, or `--all`. An empty match is an error, not a silent no-op. The client form:

```python
from kitaru.api_models.v1.evaluation import EvaluationBatchCreateRequest
from kitaru.api_models.v1.plugin import EvaluatorConfig

job = await client.evaluations.create(
    EvaluationBatchCreateRequest(
        input_session_ids=session_ids,
        evaluators=[EvaluatorConfig(evaluator="refund-check")],
    )
)
```

Each (session, evaluator) pair runs as its own task on a [worker](/kitaru/core-concepts/workers.md) (in your environment, next to your credentials), and one failed pair never cancels the rest. Read results back with `client.evaluations.list(...)`, filtered by session.

Evaluators are also how [replays](/kitaru/core-concepts/replay.md) and [experiments](/kitaru/core-concepts/experiments.md) get their numbers: both require at least one evaluator, so a re-run is evaluated the moment it lands.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.zenml.io/kitaru/core-concepts/evaluators.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.