> For the complete documentation index, see [llms.txt](https://docs.zenml.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zenml.io/kitaru/guides/returns-agent/compare.md). # 5. Compare the paired evidence Compare baseline and replay evidence and state the result at the size the cohort supports. Observe → Judge → Define → Replay → **Compare** A conclusive improvement or regression claim requires every expected original-and-replay comparison under the same evaluator versions. An incomplete run is still useful evidence for diagnosing an inconclusive result. In this final phase, you will inspect run health, read the available pairs, preserve failures and missing results, and state the narrow conclusion supported by your reviewed cohort. ## Confirm the run completed List experiment runs and inspect the exact run receipt: ```bash uv run kitaru experiment run list --size 20 uv run kitaru experiment run get "$RUN_ID" uv run kitaru experiment run jobs "$RUN_ID" --size 100 ``` Confirm that the run attempted every member of `$COHORT_REFERENCE` and that every expected replay has an explicit terminal state. A failed, canceled, or missing replay makes the result incomplete. Do not silently remove it and reduce the denominator. ## Inspect replay sessions and evaluations ```bash uv run kitaru session list \ --agent returns-resolver \ --origin replay \ --size 20 uv run kitaru evaluation list --size 100 ``` For every cohort member, put the imported baseline and replay beside one another. Compare: * the result from `$BEHAVIOR_EVALUATOR`; * the accepted terminal tool calls and their results; * the final structured output; * tool-health and timing measurements; * cost and token use; and * replay, evaluation, or job failures. Open to inspect paired traces. The evaluator tells you whether its encoded rule passed. The trace shows how the agent reached the outcome and whether the measurement missed important evidence. ## Classify each transition Use the human-reviewed role of each session when interpreting the transition: | Baseline | Replay | Interpretation to investigate | | ------------ | ------------------ | ------------------------------------------------------------------------------------ | | Fail | Pass | The target case may have improved. Check the trace and operational measurements. | | Pass | Pass | The reviewed behavior was preserved for this case. Check for other regressions. | | Pass | Fail | The candidate regressed on this reviewed case. | | Fail | Fail | The candidate did not fix this case, or the evaluator still lacks required evidence. | | Known result | Unknown or missing | The comparison is inconclusive for this case. | Do not force every change into pass or fail. A candidate can improve the primary behavior while increasing tool failures, latency, or cost enough to create a real trade-off. ## State the conclusion at the right size Use one overall evidence conclusion: | Conclusion | What the evidence says | What to do next | | ---------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | **Improved** | Target cases improved and reviewed counterexamples remained acceptable. | Expand the reviewed population or preserve this cohort as a regression check. | | **Regressed** | A target or counterexample became worse. | Inspect the paired traces, revise the change, and register a new agent version. | | **Trade-off** | One important measure improved while another became worse. | Decide whether the trade-off is acceptable or change the candidate. | | **Inconclusive** | A replay failed, required evidence is missing, or the population cannot support the needed claim. | Repair execution or add reviewed evidence before deciding. | Your statement should name the exact cohort and behavior. A defensible form is: > On `$COHORT_REFERENCE`, candidate `$CANDIDATE_AGENT` \[improved, regressed, traded off, or produced inconclusive evidence for] the reviewed behavior measured by `$BEHAVIOR_EVALUATOR`. This result applies to the frozen reviewed sessions; it does not establish general safety or production readiness. The ten supplied traces are a small synthetic population. Your selected worklist is also adaptive: you chose it partly because the traces looked interesting. Do not infer production prevalence or general agent quality from this experiment. ## If the result is inconclusive Inconclusive is not a near-pass. Preserve the reason: * If a replay failed, inspect its child job and rerun only after correcting the execution problem. * If tool evidence is missing, change the replay policy or instrumentation rather than guessing the outcome. * If the evaluator is wrong, register a new evaluator version and apply it consistently to both sides. * If the cohort lacks a necessary counterexample, create a new cohort version with reviewed membership. Keep the old versions. Their immutability makes it possible to explain why two experiment runs reached different conclusions. ## Optional: generate fresh traces The supplied export makes setup repeatable, but its model outputs are not an answer key. To create a new export, create `.env` in the example directory with valid `OPENAI_API_KEY`, `LANGFUSE_PUBLIC_KEY`, and `LANGFUSE_SECRET_KEY`, then run: ```bash ./generate.sh ``` The script makes ten paid agent runs, waits for the Langfuse observations, and replaces `traces/langfuse-traces.jsonl`. Model behavior varies. Import the new file, inspect what actually happened, and build a new review worklist from that evidence. For your own agent, keep collecting traces where you already collect them and use [Import your traces](/kitaru/import-your-traces/import-your-traces.md) to select the matching importer. Historical investigation does not require the original code to remain runnable. Replay does require a compatible registered candidate and a worker that can execute it. ## Clean up Stop the worker in Terminal 2 with `Ctrl-C`, then disconnect the CLI: ```bash uv run kitaru logout ``` For a CLI-managed local workspace, logout stops its containers but keeps the PostgreSQL data volume. ## What you completed You followed the full evidence chain: 1. **Observed** a trace population before assigning labels. 2. **Judged** selected sessions and stored human reasoning beside exact evidence. 3. **Defined** one behavior with a frozen reviewed cohort and evaluator version. 4. **Replayed** one candidate under an explicit tool policy. 5. **Compared** complete baseline and replay evidence without hiding failures or uncertainty. The durable result is not a predetermined passing demo. It is an auditable claim about one reviewed behavior, one frozen population, and one candidate. ## Where to go next
Use kitaru-investigationApply this method to the agent in your own repository.Set up your coding agent
Build a regression suiteGrow reviewed evidence into a reusable comparison.Build a regression suite from production
Replay and overridesControl models, tools, history, and replay safety.Replay a failure and fork it
Write an evaluatorDesign and calibrate a domain-specific evaluator.Write an evaluator
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.zenml.io/kitaru/guides/returns-agent/compare.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.