> For the complete documentation index, see [llms.txt](https://docs.zenml.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zenml.io/kitaru/running-in-production/workers.md). # Workers in production Deploying workers as long-running pools, scoped fleets, and one-shot CI workers, all configured through environment variables. [Workers](/kitaru/core-concepts/workers.md) are where everything executes. In production you run them as ordinary long-lived processes (a systemd unit, a container where the published `zenmldocker/kitaru-worker` image works out of the box, or a Kubernetes Deployment), one per environment your agents' code needs. The rule of thumb: **a worker must be able to run what it claims.** An agent replay needs your agent's virtualenv and provider keys; an evaluator, importer, or analyzer brings its own dependencies and needs Python, `uv`, network access to the server, and any provider credentials its implementation uses. An API import or analyzer whose credentials come from the worker's environment rather than a [connection](/kitaru/import-your-traces/provider-connections.md) is only claimed by a worker whose `kitaru/requires-credentials` selector names that provider. ## Configuration Everything `kitaru worker start` takes as a flag is also an environment variable with the `KITARU_WORKER_` prefix, which is how containerized workers are configured: ```bash export KITARU_API_URL="https://kitaru.internal.example.com" export KITARU_API_KEY="KITKEY_..." # a service API key export KITARU_WORKER_CONCURRENCY=4 export KITARU_WORKER_SCOPE__CLAIMS='[{"kind":"evaluator"},{"kind":"importer"},{"kind":"analyzer"}]' # JSON kitaru worker start ``` | Variable | Default | Meaning | | ------------------------------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `KITARU_WORKER_NAME` | hostname-pid | Label shown in worker listings. Every start registers a new worker, names need not be unique. | | `KITARU_WORKER_CONCURRENCY` | 10 | Tasks run in parallel | | `KITARU_WORKER_SCOPE__CLAIMS` | all | JSON list of claims, such as `{"kind":"agent"}` or `{"kind":"agent","agent_version_id":""}` | | `KITARU_WORKER_SCOPE__SELECTORS` | none | JSON label selectors (e.g. limit to one agent version's environment, or name the providers whose [credentials](/kitaru/import-your-traces/provider-connections.md#worker-credentials) the environment holds) | | `KITARU_WORKER_SCOPE__JOB_ID` | none | Claim one job's tasks, drain, exit | | `KITARU_WORKER_TIMEOUT` | none | Wall-clock lifetime; unset runs until stopped | | `KITARU_WORKER_POLL_INTERVAL` | 2s | Sleep after an empty claim | | `KITARU_WORKER_HEARTBEAT_INTERVAL` | 10s | Liveness reporting cadence | | `KITARU_WORKER_BLOB_CACHE_ROOT` / `PAYLOAD_CACHE_ROOT` | `~/.cache/kitaru/...` | Plugin-code and payload caches, keyed by content hash | The worker retains the API key for registration and worker-token renewal. Each task subprocess gets a narrower per-task token, with the API key stripped from its environment. Details in [Authentication & API keys](/kitaru/running-in-production/authentication.md). ## Fleet patterns **One general worker per agent environment.** The simplest useful fleet: each environment that can run an agent gets a worker with no scope, and utility work (imports, evaluations) rides along. **Split agent execution from plugin execution.** Agent replays need your application environment; evaluations and imports don't. A scoped pair keeps them independent: ```bash # in the agent's environment kitaru worker start --claim agent= # anywhere cheap kitaru worker start --claim evaluator --claim importer --claim analyzer --concurrency 8 ``` The versioned `agent` claim matches the agent version attached to each agent task, so a worker only claims replays its environment can actually run. The `analyzer` claim lets this utility worker run [post-import insights](/kitaru/import-your-traces/post-import-insights.md). Without an analyzer-capable worker, an import's analysis task stays queued after parsing finishes. **One-shot workers in CI.** Pin a worker to the job you just created and it drains the job (appended evaluator tasks included), then exits: ```bash kitaru worker start --job-id "$JOB_ID" --timeout 1800 ``` This is the pattern for [CI regression gates](/kitaru/guides/regression-suite.md): the runner that starts the experiment also executes it, using the PR's own checkout as the agent environment. ## Operational behavior * **Draining**: SIGINT/SIGTERM stops claiming and finishes in-flight tasks; a second signal exits immediately. Per-task timeouts (set server-side and on agent versions) bound the wait. * **Crash safety**: a worker that dies stops heartbeating; the server requeues its tasks to the next worker (up to the retry limit). No replay is lost to a pod eviction. * **Liveness**: `kitaru worker list` shows the live fleet and when each worker was last seen. Add `--include-stale` to see workers past the liveness window. * **Subprocess environments**: evaluator and importer plugins run via `uv` in isolated per-plugin environments, cached by content hash; agent tasks run the agent version's command in the worker's own environment plus the version's [secrets](/kitaru/running-in-production/secrets.md). The default plugins (the five `kitaru/` importers and the built-in evaluator suite) run under the same isolation as plugins you write yourself. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.zenml.io/kitaru/running-in-production/workers.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.