Evaluators & Evaluations

Evaluators turn sessions into evaluations: the rows they write. Numbers average, words count, free text gets read.

Replay tells you what a change did; evaluators tell you whether it helped. An evaluator is a small piece of your code that reads one session, node by node, and writes one or more evaluations: named, typed verdicts that Kitaru stores against the session.

Because evaluators run against recorded sessions, they evaluate baselines, replays, and imported traces identically. The same evaluator you run over today's production traffic runs over the fork you're thinking about shipping.

An evaluator that calls a model or another external service can declare its provider and a connection schema, the environment variables its SDK reads, so a connection supplies the credentials at run time.

The evaluator contract

An evaluator is a callable (a single Python file or an installable package) that receives the full session and returns results:

"""refund_check.py: did the agent issue the refund?"""

from kitaru.task.evaluator import EvaluationResult, SessionView


def evaluate(session: SessionView, **params) -> EvaluationResult:
    refund_calls = [
        node
        for node in session.nodes
        if node.node_type == "tool_call" and node.tool_name == "refund_payment"
    ]
    return EvaluationResult(
        name="refund_issued",
        score=bool(refund_calls),
        passed=bool(refund_calls),
        explanation=f"{len(refund_calls)} refund tool call(s) in the session",
    )

SessionView gives you the session and all its nodes with payloads. Return one EvaluationResult or a list; each becomes one stored evaluation row. params are per-run knobs you pass when you attach the evaluator to a replay or experiment.

Scaffold, exercise, and register it with the CLI:

kitaru evaluator scaffold refund-check          # writes refund_check_evaluator.py
kitaru evaluator test refund_check_evaluator.py --entrypoint evaluate
kitaru evaluator register refund-check \
  --script refund_check_evaluator.py --entrypoint evaluate

Evaluators are versioned like agents: registering again with kitaru evaluator version register creates version 2, and every stored evaluation remembers exactly which evaluator version wrote it. An LLM judge follows the same contract by calling a model inside evaluate. The walkthrough is in Write an evaluator.

A suite of evaluators comes built in, registered at server startup under the kitaru/ namespace: three cheap signals (kitaru/cost, kitaru/latency, kitaru/tool-call-patterns) plus ten deterministic checks over the recording itself, from kitaru/output-contract and kitaru/tool-health to kitaru/timing-profile and kitaru/workflow-conformance. None of them make model calls; they are the triage layer, available as kitaru/cost@latest before you have written anything.

The evaluation row

One evaluation is one named result for one session. The data type is derived from what you set, never declared:

You set	Stored type	How to read a batch of them
score=0.87	float	numbers average
score=True	bool	pass rates count
value="escalated"	str	free text gets read
score=0.9, value="polite"	categorical	labels count, transitions diff

passed is an independent optional verdict, based on a threshold you decided in the evaluator rather than something derived from score. explanation says why, which is the part you read when a regression gate goes red.

Human labels are evaluations too

There is no separate labeling system. A human verdict is an evaluation written directly onto the session:

from kitaru.api_models.v1.evaluation import EvaluationResult
from kitaru.api_models.v1.session import SessionEvaluationsRequest

await client.sessions.create_evaluations(
    session_id,
    SessionEvaluationsRequest(
        evaluations=[
            EvaluationResult(
                name="human_quality",
                score=True,
                explanation="Correct refund, good tone",
            ),
        ]
    ),
)

Manual evaluation names are unique per session: sending human_quality again fails rather than overwriting the earlier verdict. Rows written by evaluator runs carry the evaluator version that produced them, forever, even after that evaluator is deleted; manual rows carry none, which is how you tell them apart. Comparing your evaluator's column against the human column on the same sessions is how you calibrate the evaluator before you let it gate anything. The human column usually comes out of the interview: your coding assistant authors the investigation, and your answers land as annotations to calibrate against.

Running evaluators in batch

Evaluate existing sessions without replaying anything. From the CLI, select by IDs, by tag, by agent, by cohort version, by filter, or everything:

kitaru session evaluate --tag imported-baseline \
  --evaluator refund-check@latest --evaluator kitaru/cost@latest \
  --wait

Exactly one selection is required: explicit session IDs (arguments or --sessions-file), --tag, --agent, --cohort, --filter, or --all. An empty match is an error, not a silent no-op. The client form:

from kitaru.api_models.v1.evaluation import EvaluationBatchCreateRequest
from kitaru.api_models.v1.plugin import EvaluatorConfig

job = await client.evaluations.create(
    EvaluationBatchCreateRequest(
        input_session_ids=session_ids,
        evaluators=[EvaluatorConfig(evaluator="refund-check")],
    )
)

Each (session, evaluator) pair runs as its own task on a worker (in your environment, next to your credentials), and one failed pair never cancels the rest. Read results back with client.evaluations.list(...), filtered by session.

Evaluators are also how replays and experiments get their numbers: both require at least one evaluator, so a re-run is evaluated the moment it lands.