> For the complete documentation index, see [llms.txt](https://docs.zenml.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zenml.io/stacks/stack-components/log-stores.md). # Log Stores Storing and retrieving logs from your ML pipelines. The log store is a stack component responsible for collecting, storing, and retrieving logs generated during pipeline and step execution. It captures everything from standard logging output to print statements and any messages written to stdout/stderr, making it easy to debug and monitor your ML workflows. ### How it works ZenML's log capture system is designed to be comprehensive and non-intrusive. Here's what happens under the hood: 1. **stdout/stderr wrapping**: ZenML wraps the standard output and error streams to capture all printed messages and any output directed to these streams. 2. **Root logger handler**: A custom handler is added to Python's root logger to capture all log messages with proper metadata from loggers that propagate to the root. 3. **Log routing**: All captured messages are routed through a `LoggingContext` to the active log store in your stack. This approach ensures that you don't miss any output from your pipeline steps, including: * Standard Python `logging` messages * `print()` statements * Output from third-party libraries * Messages from subprocesses that write to stdout/stderr ### When to use it The Log Store is automatically used in every ZenML stack. If you don't explicitly configure a log store, ZenML will use an [**Artifact Log Store**](/stacks/stack-components/log-stores/artifact.md) by default, which stores logs in your artifact store. You should consider configuring a dedicated log store when: * You want to use a centralized logging backend like Datadog, Jaeger, Grafana Tempo, Honeycomb, Lightstep or Dash0 for log aggregation and analysis * You need advanced log querying capabilities beyond what file-based storage provides * You're running pipelines at scale and need better log management * You want to integrate with your organization's existing observability infrastructure ### How to use it By default, if no log store is explicitly configured in your stack, ZenML automatically creates an Artifact Log Store that uses your artifact store for log storage. This means logging works out of the box without any additional configuration. To use a different log store, you need to register it and add it to your stack: ```shell # Register a log store (example with Datadog) zenml log-store register \ --flavor=datadog \ --api_key= \ --application_key= # Add it to your stack zenml stack register -a -o --log_store --set ``` Once configured, logs are automatically captured during pipeline execution. ### Viewing Logs You can view step logs in the ZenML dashboard or in your logging provider's UI. To retrieve them programmatically, use the Python SDK or REST API. #### Python SDK Use the log store that captured the logs: ```python from zenml.client import Client from zenml.utils.logging_utils import search_logs_by_source client = Client() run = client.get_pipeline_run("") step_run = run.steps[""] logs = search_logs_by_source(step_run.log_collection or [], "step") if logs is None: raise ValueError("This step has no execution log stream.") log_store = client.active_stack.log_store page = log_store.fetch(logs_model=logs, limit=1000) for entry in page.items: print(f"[{entry.level}] {entry.message}") ``` `fetch()` returns a page with entries in `items`, ordered from oldest to newest. Set `start="oldest"` or `start="newest"` to choose the first page, or omit it to use the backend's default: newest for Datadog, oldest for the artifact log store. For backends with pagination, pass the returned `before` cursor to read older entries or `after` to read newer ones. Only directions supported by the backend's native tokens are available; `None` means there is no continuation in that direction. For example, with Datadog, filter the first request and use its cursor to continue: ```python from zenml.models import LogsEntriesFilter page = log_store.fetch( logs_model=logs, start="newest", filter_=LogsEntriesFilter(level="ERROR", search="ValueError"), ) while True: for entry in page.items: print(entry.message) if page.before is None: break page = log_store.fetch(logs_model=logs, before=page.before) ``` Datadog cursors retain the filters, page size, and fixed time bounds. Search follows the backend's matching rules, which may differ from a literal substring match. The artifact log store returns one batch for client-side filtering and pagination. It does not support these filters or `start="newest"`. Each entry has a UUID `id`. IDs remain stable for Datadog events and structured artifact logs. Deduplicate entries within a stream by `(id, chunk_index)` to preserve chunks of the same message. Shared log types, including `LogEntry`, are available from `zenml.models`. #### REST API `GET /api/v1/logs/{logs_id}/entries` accepts `start`, `limit`, `before`, `after`, `search`, `level`, `since`, and `until` as query parameters. For example: ```http GET /api/v1/logs//entries?start=newest&limit=50 GET /api/v1/logs//entries?before= ``` Invalid query values return `422`; unsupported parameters and cursors rejected by ZenML return `400`. Backends without log retrieval return `501`. Shared log store errors return `429` for throttling, `503` for unavailability, or `502` for other backend errors. When provided, `Retry-After` specifies the delay in seconds before retrying the original request. Direct SDK calls raise the corresponding `LogStoreError` subclass. Runner logs return one batch through the workload manager. Apply filtering and pagination in the client; unsupported runner filters or cursors return `400`. The existing run and step log endpoints continue to return a single list of entries. Use the dedicated entries endpoint for pagination. ### Log Store Flavors ZenML provides several log store flavors out of the box: | Log Store | Flavor | Integration | Notes | | ---------------------------------------------------------------------- | ---------- | ----------- | ----------------------------------------------------------------------------------------------- | | [ArtifactLogStore](/stacks/stack-components/log-stores/artifact.md) | `artifact` | *built-in* | Default log store that writes logs to your artifact store. Zero configuration required. | | [OtelLogStore](/stacks/stack-components/log-stores/otel.md) | `otel` | *built-in* | Generic OpenTelemetry log store for any OTEL-compatible backend. Does not support log fetching. | | [DatadogLogStore](/stacks/stack-components/log-stores/datadog.md) | `datadog` | *built-in* | Exports logs to Datadog's log management platform with full fetch support. | | [Custom Implementation](/stacks/stack-components/log-stores/custom.md) | *custom* | | Extend the log store abstraction and provide your own implementation. | If you would like to see the available flavors of log stores, you can use the command: ```shell zenml log-store flavor list ``` {% hint style="info" %} If you're interested in understanding the base abstraction and how log stores work internally, check out the [Develop a Custom Log Store](/stacks/stack-components/log-stores/custom.md) page for a detailed explanation of the architecture. {% endhint %}
ZenML Scarf
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.zenml.io/stacks/stack-components/log-stores.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.