> For the complete documentation index, see [llms.txt](https://docs.zenml.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zenml.io/stacks/stack-components/log-stores/artifact.md). # Artifact Log Store Storing logs in your artifact store. The Artifact Log Store is the default log store flavor that comes built-in with ZenML. It stores logs directly in your artifact store, providing a zero-configuration logging solution that works out of the box. {% hint style="warning" %} The Artifact Log Store is ZenML's implicit default. You don't need to register it as a flavor or add it to your stack. When no log store is explicitly configured, ZenML automatically uses an Artifact Log Store to handle logs. This means logging works out of the box with zero configuration. {% endhint %} ### When to use it The Artifact Log Store is ideal when: * You want logging to work without any additional configuration * You prefer to keep all your pipeline data (artifacts and logs) in one place * You don't need advanced log querying capabilities * You're getting started with ZenML and want a simple setup ### How it works The Artifact Log Store leverages OpenTelemetry's batching infrastructure while using a custom exporter that writes logs to your artifact store. Here's what happens during pipeline execution: 1. **Log capture**: All stdout, stderr, and Python logging output is captured and routed to the log store. 2. **Batching**: Logs are collected in batches using OpenTelemetry's `BatchLogRecordProcessor` for efficient processing. 3. **Export**: The `ArtifactLogExporter` writes batched logs to your artifact store as JSON-formatted log files. 4. **Finalization**: When a step completes, logs are finalized (merged if necessary) to ensure they're ready for retrieval. #### Handling Different Filesystem Types The Artifact Log Store handles different artifact store backends intelligently: * **Mutable filesystems** (local, S3, Azure): Logs are appended to a single file per step. * **Immutable filesystems** (GCS): Logs are written as timestamped files in a directory, then merged on finalization. This ensures consistent behavior across all supported artifact store types. ### Environment Variables The Artifact Log Store uses OpenTelemetry's batch processing under the hood. You can tune the batching behavior using these environment variables: | Environment Variable | Default | Description | | --------------------------------------- | -------- | ---------------------------------------------- | | `ZENML_LOGS_OTEL_MAX_QUEUE_SIZE` | `100000` | Maximum queue size for batch log processor | | `ZENML_LOGS_OTEL_SCHEDULE_DELAY_MILLIS` | `5000` | Delay between batch exports in milliseconds | | `ZENML_LOGS_OTEL_MAX_EXPORT_BATCH_SIZE` | `5000` | Maximum batch size for exports | | `ZENML_LOGS_OTEL_EXPORT_TIMEOUT_MILLIS` | `15000` | Timeout for each export batch in milliseconds | | `ZENML_LOGS_MAX_ENTRIES_PER_REQUEST` | `50000` | Maximum log entries returned by a single fetch | These defaults are optimized for most use cases. You typically only need to adjust them for high-volume logging scenarios. ### Reading logs The artifact log store returns one batch of the oldest entries, capped by `limit` and `ZENML_LOGS_MAX_ENTRIES_PER_REQUEST` (default: 50,000). Entries beyond the cap are omitted. Both cursors are unset; apply filtering and pagination to the returned batch in your client. `before`, `after`, `start=newest`, and the `search`, `level`, `since`, and `until` filters are unsupported. They raise `ValueError` in the SDK or return HTTP `400` through the REST API. ### Log format Logs are stored as newline-delimited JSON (NDJSON) files. Each log entry contains the following fields: ```json { "message": "Training model with 1000 samples", "level": 20, "timestamp": "2024-01-15T10:30:00.000Z", "name": "my_logger", "filename": "train.py", "lineno": 42, "module": "train", "chunk_index": 0, "total_chunks": 1, "id": "550e8400-e29b-41d4-a716-446655440000" } ``` | Field | Description | | -------------- | --------------------------------------------------------------------- | | `message` | The log message content | | `level` | Numeric log level (DEBUG=10, INFO=20, WARN=30, ERROR=40, CRITICAL=50) | | `timestamp` | When the log was created | | `name` | The name of the logger | | `filename` | The source file that generated the log | | `lineno` | The line number in the source file | | `module` | The module that generated the log | | `chunk_index` | Index of this chunk (0 for non-chunked messages) | | `total_chunks` | Total number of chunks (1 for non-chunked messages) | | `id` | UUID shared by all chunks of a message | For large messages (>5KB), logs are automatically split into multiple chunks with sequential `chunk_index` values and a shared `id` for reassembly. Structured logs preserve entry IDs between reads. Deduplicate chunks within a stream by `(id, chunk_index)`. Older plain-text logs have no stored IDs and receive new ones on each read. ### Storage location Logs are stored in the `logs` directory within your artifact store: ``` / └── logs/ ├── .log # For mutable filesystems └── / # For immutable filesystems (GCS) ├── 1705312200.123.log ├── 1705312205.456.log └── 1705312210.789_merged.log ``` ### Best practices 1. **Use the default**: For most use cases, the automatic artifact log store is sufficient. Don't add complexity unless you need it. 2. **Monitor storage**: Logs can accumulate over time. Consider implementing log retention policies for your artifact store. 3. **Large log volumes**: If you're generating very large log volumes, consider using a dedicated log store like Datadog for better scalability and querying. 4. **Sensitive data**: Be mindful of what you log. Avoid logging sensitive information like credentials or PII. For more information and a full list of configurable attributes, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-log_stores.html#zenml.log_stores.artifact.artifact_log_store).
ZenML Scarf
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.zenml.io/stacks/stack-components/log-stores/artifact.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.