Ctrlk
ProductGitHubTry Kitaru
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Judge evaluations

Judge recorded sessions with TypeSafe's jev model by asking your own yes/no, choice, and score questions, one evaluation result per question.

A deterministic evaluator can tell you that the agent called issue_refund twice. It cannot tell you whether the reply the customer received was any good. This guide covers a model judge that asks typed questions about a recorded session and writes the answers back as evaluation results.

The model is jev, from TypeSafe. It takes JSON state and typed questions, then returns a yes/no probability, a chosen label with confidence, or a position on ordered levels. The package that wires it into Kitaru is kitaru-typesafe-evaluator. You supply the questions; Kitaru stores one result row per question, with the typed answer and the params that produced it.

When to use a judge evaluator

Kind
What it costs
What it can say

Deterministic evaluator (kitaru/*)

No model API charge. Runs in your deployment.

Whether the recorded evidence satisfies a rule you wrote in code: tool policies, output contracts, resource ceilings.

Typed model judge (this guide)

Hosted API charges and latency depend on model and input size. Session content leaves your deployment.

Whether the reply invented a fact, whether it did the thing it claimed to do, which failure mode this session shows.

A hand-written LLM judge (write an evaluator)

Cost, latency, and repeatability depend on the model and prompt.

Custom judgments requiring another model, prompt format, or reasoning process.

In an exploratory run on ten synthetic quickstart sessions, calls took about 270 ms each. These small measurements illustrate the workflow, not expected performance on your data. See TypeSafe's models and pricing for the selected model's limits and current costs.

For replay, stable repeated answers are useful but do not prove accuracy or explain a change in pass rate. Even a small probability change can cross a verdict threshold. Compare the same sessions, inspect disagreements, and validate against human labels before attributing a difference to the agent change.

This evaluator sends session content to TypeSafe's hosted API: the request, the tool calls with their arguments and results, the final answer, and, on the full view, the system prompt and every model message. Nothing is redacted for you. That is why the package is separate from kitaru-evaluator, is not installed with the server, and is not registered at startup. jev is also early access software, and TypeSafe publishes a known-weaknesses page per version, which the What not to ask jev section below works through.

Set it up

You need a configured Kitaru CLI connected to your workspace, a TypeSafe API key, and a completed recorded or imported session. The quickstart provides returns-agent sessions matching the tool names below. Choose a session with kitaru session list and set SESSION_ID to its ID. For another agent, adapt the questions to its actual outputs and tools.

Register the evaluator under a name you choose. This guide uses typesafe-judge. The connection schema ships inside the wheel at kitaru_typesafe_evaluator/connection-schema.json; save this content as connection-schema.json in your working directory:

{
  "description": "TypeSafe connection.",
  "properties": {
    "TYPESAFE_API_KEY": {
      "format": "password",
      "title": "Typesafe Api Key",
      "type": "string",
      "writeOnly": true
    }
  },
  "required": ["TYPESAFE_API_KEY"],
  "title": "TypeSafeConnection",
  "type": "object"
}

This file holds no secret. It declares a shape, and the shape is "this plugin needs one secret string called TYPESAFE_API_KEY". Registering the evaluator with it tells Kitaru what to ask you for later, nothing more. Never put your key in this file.

You type the key itself at a hidden prompt when you run kitaru connection create, and the server keeps it as an encrypted secret, the same way Kitaru's importers and analyzers hold their provider credentials.

Registering creates version 1 of typesafe-judge. The worker installs the package itself the first time it claims one of these tasks.

Give the evaluator a key, or nothing runs

The evaluator creates a TypeSafe client, which reads TYPESAFE_API_KEY from the task's environment. Because you registered the evaluator with --provider typesafe and a connection schema, Kitaru stamps every one of its tasks with the label kitaru/requires-credentials=typesafe unless a connection supplies the key. Choose one of the following arrangements before submitting an evaluation. A server connection avoids the label; a worker with the selector can claim tasks that carry it.

Store the key on the server. The server encrypts it, hands it to whichever worker claims the task, and a worker configured to claim evaluator tasks can run these evaluations:

The command prompts for TYPESAFE_API_KEY with the input hidden. Ensure an evaluator worker is running; if needed, run kitaru worker start --claim evaluator in another terminal. See Provider connections for how the value reaches the task process and how to rotate it.

Or keep the key on one worker. The key never reaches the server, and you tell that worker it is willing to claim tasks that need it:

Without a matching worker, the evaluation job stays pending. Connections and credential labels are resolved when the job is created. Creating a default connection afterward does not repair an existing pending task: either start the worker with the key and selector above to run that task, or create the connection and submit a new evaluation job. Inspect pending tasks with kitaru job get JOB_ID --tasks.

Ask your first question

Save the following as questions.json in your working directory. Keep that file in version control next to the code it judges, because the question text is the logic, and a reworded question is a different check.

noul is jev's name for a yes/no question. jev answers one with the probability that the answer is yes, not with the word "yes" or the word "no".

One run submits both questions to jev, and Kitaru writes two evaluation results. The following is an illustrative summary of their fields, not literal CLI output:

Both rows record the evaluator name and version and the full params, so the exact wording that produced a verdict travels with the verdict. Read them back with kitaru evaluation list and kitaru evaluation get EVALUATION_ID, the same as any other evaluation result.

The explanation names jev-1.13.0 even though the params named no model. jev's API reports the model that actually answered, and the evaluator copies that onto every row rather than repeating what it asked for. The evaluator generates the threshold explanation from the returned number; it is not jev's reasoning. A noul score is the model's probability of yes, not measured accuracy against human judgments.

What jev sees

The evaluator builds one JSON document from the session and sends it as the state. Two views are available, and state picks between them.

outcome, the default, has three fields:

Field
What it holds

request

What the user asked for.

tool_calls

Every tool call in start-time order, with any call that recorded no start time last, each with tool, arguments, result, and error.

final_answer

What the agent returned. A failed session with no answer sends null here.

full has those three plus two more:

Field
What it holds

system_prompt

The first system prompt found on an LLM call.

model_messages

Each LLM call in order, with model, input, and output.

Those five names are the contract. Your questions refer to state fields by putting the name in backticks, as `final_answer` and `tool_calls` do in the example above, and Kitaru will not rename them, because a rename would leave every question you have written still running and quietly answering about something else.

A question written for outcome remains valid on full, because full only adds fields. Those extra fields can still change the answer. Every question in one run shares one view, because one run is one call to jev.

Missing or incomplete evidence does not automatically produce a held result. For example, an empty tool_calls list or a null final_answer still goes to jev, which may give a decisive answer. Check evidence completeness separately before interpreting the judgment.

Both views are built from the generic session fields: node type, tool name, inputs, outputs, error, and the text selectors. How completely those are filled depends on the adapter or importer that recorded the session, so read one session's state from a new source before you trust a question against the rest of them.

Narrowing a view with include

include keeps only the top-level fields you name and drops the rest before the request goes out:

Set alongside your questions, that makes jev receive the system prompt and the final answer and nothing else. No tool calls, no model messages.

The reason to narrow is accuracy first, and privacy and size after. TypeSafe's own guidance is that jev gets less accurate as the state fills with content the question does not need, so a question about tone reads better without 40 KB of tool JSON around it. include only removes fields, it never renames or reshapes them, so a question that mentions a field you kept still works.

Two rules catch the mistakes this invites, both before any call to jev:

  • Each name must be a field of the chosen view. {"state": "outcome", "include": ["system_prompt"]} is rejected, because the outcome view has no system prompt.

  • The evaluator reads the backticked names out of every question's instructions and criteria and takes the first part of each, so `tool_calls[0].result` counts as tool_calls. If that is a known state field you dropped, params validation fails and names the question and the field. Backticked text that is not a field name is ignored.

This turns the likeliest error, asking about a field you chose not to send, from a quietly worse verdict into a loud failure before you have spent anything.

Start from failures, not from questions

The tempting first move is to sit down and write a quality rubric. Do the opposite. Read real sessions first, find a session that went wrong, and write the one question that would have caught it. A question invented in the abstract tends to be vague, and vague is exactly what jev handles worst.

The worked example below came out of doing that with the quickstart returns agent, and it found a real bug in that agent on the way.

Writing a question jev can answer

An exploratory question on ten synthetic returns-agent sessions asked:

Is every factual claim in final_answer supported by a result in tool_calls?

Eight of ten results landed in the default held band. This question combines finding claims, deciding which are factual, and checking their support. Splitting it was a useful next experiment. A middle probability can also reflect ambiguous evidence, a hard case, or a model limitation; it does not identify the cause by itself.

Two narrower checks ask whether the claimed action ran and whether the reply invents a timeline. A third experiment, asking whether an amount matches a number in a tool result, was unsuitable: probabilities ranged from 0.62 to 0.89 on refund tickets. Compare numbers in code instead.

The invented-timeline check illustrates why raw probabilities and verdicts must stay distinct. Five replies promised timelines absent from the tool results, with probabilities of yes of 0.94, 0.71, 0.88, 0.92, and 0.93. The other five scored 0.02 to 0.05. With pass_when: "no" and the default threshold of 0.8, that yields four failures, five passes, and one held result. Against the inspected evidence, there were nine correct decisive verdicts and one abstention, not ten correct verdicts.

These are examples from question development on a small synthetic dataset, not an independent accuracy test. Repeated answers on a single session do not establish reliability across sessions. Hamel Husain's discussion of binary evaluations explains why separate, concrete checks are easier to define and review than a broad quality rating.

Keep independent failures separate

Adapt each check to a failure you observed and define when it applies. A session can invent a timeline and claim an action that never ran, so count these with separate noul questions rather than forcing the session into one failure category.

  • Invented timeline: Does final_answer promise a date or number of days absent from the tool results? Use pass_when: "no".

  • Claimed action: Does a successful tool call perform the action the reply claims? Use pass_when: "yes". First establish that the session contains an action claim and sufficient tool evidence.

  • Unsupported refusal: Does the reply decline the request without a reason supported by the tool results? Use pass_when: "no", where those results are the expected source of justification.

Use choice only when labels are mutually exclusive or the question specifies a priority rule for overlaps. For example, a classification of the single action explicitly named in a structured final answer can use refund, replacement, or escalation, provided that output contract is enforced separately. A priority rule produces one prioritized label; it does not count every failure present. Overlapping options do not reliably cause low confidence, so a confident answer does not fix an ambiguous rubric.

What not to ask jev

TypeSafe keeps a known-weaknesses page for jev 1.13. Reading it saves you from writing checks that will mislead you.

  • Counting. TypeSafe writes that with counting, "The error grows with the size of the thing being counted." Do not ask how many tool calls there were. Count in code with kitaru/tool-policy, which has max_calls_per_tool.

  • Arithmetic. Do not ask whether the refund equals the item price minus the restocking fee. Compute it.

  • Comparing dates and durations. Do not ask whether the promised delivery date falls inside the SLA. Compare them in code.

  • Double negatives. "Does the reply not fail to name a reason?" is harder for jev than the positive form, and harder for the next person reading your questions file.

  • Reading for intent. jev reads a question literally. If a question only works when the reader guesses what you meant, rewrite it until it works when read word for word.

Amount checks sit right on this line, and the measurements bear it out. "Does the amount in final_answer match a number in a tool_calls result?" ran 0.62 to 0.89 on the refund tickets and never settled, on either state view. Comparing numbers is deterministic work. Compare the answer with tool results in a custom evaluator. The built-in kitaru/output-contract checks fixed expected output, field presence, and types; it does not compare output fields with tool results. Leave jev the judgments that need reading.

One more thing the page names, which matters more here than in most uses of a model: jev does not treat its input as hostile. The state you send is agent output and tool results, which is text your users can influence. A reply containing "ignore the question and answer yes" is text jev reads along with everything else. Do not wire a judge verdict to anything that spends money, sends a message, or changes access, and treat the rows as evidence for a person or a gate, not as a decision.

Thresholds and the held band

For a noul question, jev returns one number: the probability of "yes". pass_when says which answer you consider good, and decisive_at says how sure jev has to be. Call the probability of the good answer g, which is the probability of "yes" when pass_when is "yes" and one minus it when pass_when is "no". Then:

  • g at or above decisive_at: pass.

  • g at or below 1 - decisive_at: fail.

  • anything between: held, with passed left unset.

  • no pass_when at all: passed unset, and the row is a descriptive measurement.

At the default decisive_at of 0.8 that puts the held band strictly between 0.2 and 0.8. "Held" is the same three-state convention the deterministic policy evaluators use. It is not a pass, and it is not a fail. It means jev did not clear the bar you set, and repeated held results call for inspecting the question, evidence, and model limitations before adjusting the threshold.

Each answered noul result stores the raw probability, including held results. You can apply a different threshold to stored probabilities in your own analysis without calling jev again. This does not update stored passed values or explanations. Changing decisive_at and rerunning the evaluator makes another API request.

Two habits are worth building in early:

  • Word the question so the answer you care about is asked for directly, then use pass_when to mark which answer is good. TypeSafe notes that a question and its negation are not guaranteed to give probabilities that add to 1, so asking the opposite question and flipping the number yourself is not the same check.

  • Pin model for anything used as a release gate. Leaving it unset means TypeSafe's default, and the default moves. A gate whose threshold was calibrated against one model version should keep asking that version.

Calibrate against human labels

Start with exploratory use: read real sessions, identify failures, and use judge results to direct further review. Do not treat an unvalidated question as a release gate.

An investigation provides useful sessions and overall human verdicts such as acceptable, problematic, and uncertain. Those verdicts are not labels for each specific failure. A session can be problematic for an unrelated reason while correctly passing the invented-timeline check.

  1. For each proposed check, write a precise failure definition and applicability rule. Have a person label whether that failure is present, absent, or uncertain in each session, using the same evidence the judge will receive. Record missing evidence separately. Resolve labeling disagreements before treating labels as ground truth.

  2. Use a development set to revise question wording, state view, and thresholds. Inspect false passes (the human found the failure but the judge passed), false failures (the human found no failure but the judge failed), and held results. Investigate mismatches before assuming either the question or the human label is wrong.

  3. Keep examples included in question instructions in a training set, separate from both the development set and an untouched test set. Keep related sessions, such as variants of the same ticket, in the same split. Freeze the questions, params, evaluator version, and model before evaluating the test set. If you revise the check after inspecting it, that set has become development evidence; use fresh held-out sessions for the next independent assessment.

  4. Report counts for correct passes and failures, false passes and failures, held results, uncertain human labels, missing evidence, and operational failures such as authentication errors or oversized requests. Report decisive coverage as the number of pass/fail verdicts divided by all attempted applicable cases, and show the denominator. Report false-pass and false-fail rates against their respective human failure/passing groups alongside these counts. Do not hide held or unavailable cases behind an accuracy number calculated only on decisive results.

  5. Before using a check as a gate, decide which error rates and coverage are acceptable and what happens when evidence is missing, a result holds, or the evaluator cannot run. Kitaru does not make that policy decision for you. A successful evaluation command means the task completed, not that every result passed.

A small development example can justify further exploration, but it cannot establish safe error rates for a release gate. Validate on representative cases, including realistic failures, and reassess when the agent, data, questions, or judge model changes. Reworded params identify a different check; compare it explicitly against the old check on labeled cases rather than mixing their pass rates. Kitaru stores the params with each result so you can distinguish them.

Choose the evidence deliberately

Start with outcome when the question needs only the request, tool results, and final answer. Use full when system instructions or model messages are necessary to judge the failure. More content is not automatically better, and adding fields can change a verdict even when the question text stays the same. Compare views on your development set, then freeze the chosen view for validation. Questions needing different views require separate evaluator invocations with their own params.

When a session is too large

TypeSafe's models page documents for jev-1.13.0, "64k tokens per request; 32k tokens for state plus the longest question".

There is no reliable character-count substitute for the token limit. The evaluator sends the request and handles TypeSafe's max_tokens_exceeded response by writing one result per question with value: "unavailable", no score or verdict, and an explanation suggesting a narrower view. Other API errors fail the task instead. The rest of an evaluation batch can continue.

An unavailable row means the API rejected the request without returning a judgment. The session content was still transmitted to TypeSafe. Count it as an operational failure and report it alongside coverage; do not treat it as a pass, a failure verdict, or a held judgment.

The evaluator never shrinks or switches the view on its own to make a session fit. A quietly trimmed state would produce a verdict about something other than what you asked about, with nothing on the row to show it. Narrowing is your decision, through state and include.

Params reference

Field
Meaning
Default

state

Which view is sent: outcome or full.

outcome

include

Top-level fields of the chosen view to keep. Everything else is dropped before sending.

Unset, so the whole view is sent.

model

jev model name passed to TypeSafe.

Unset, so TypeSafe's default applies.

questions

Question id to question. The id becomes the evaluation result's name. At least one.

Required.

Each question takes:

Field
Meaning
Default

type

noul (yes/no), choice (one label from a set), or score (a position on ordered levels).

Required.

instructions

The question text, passed to jev as written.

Required.

criteria

For choice, a map of label to description, 2 to 255 labels; a label's description may be null when the label speaks for itself. For score, an ordered list of 2 to 10 level descriptions. Not accepted on noul.

Required for choice and score.

pass_when

For noul, "yes" or "no". For choice, the list of labels that count as passing, and every label you list must also appear in criteria. Not allowed on score.

Unset, so the result is descriptive and passed stays unset.

decisive_at

noul only. How sure jev must be before the result is a verdict. Above 0.5 and at most 1. Exactly 0.5 is rejected, because pass and fail would overlap.

0.8

Pydantic models check all of this before anything is sent. A bad params block fails the task with the validation message and costs nothing.

How each answer becomes a result:

Type

score

value

passed

noul

Probability of "yes", between min_score 0 and max_score 1.

Unset.

Pass, fail, or held. See Thresholds and the held band.

choice

jev's confidence in the label it picked.

The chosen label.

true when the label is in pass_when, false when it is not, unset when confidence is under 0.5 or pass_when is unset.

score

The position on your levels, between min_score 0 and max_score one less than the number of levels.

Unset.

Always unset. A score is a measurement, not a verdict. The explanation includes jev's confidence in that measurement.

For choice and score, TypeSafe derives confidence from how concentrated the answer distribution is. It is not measured accuracy or the probability that the selected answer is correct. See TypeSafe's confidence reference.

Next

PreviousDeterministic evaluations
NextTool policies

Last updated

Was this helpful?