OpenTelemetry Log Store

Exporting logs to any OpenTelemetry-compatible backend.

The OpenTelemetry (OTEL) Log Store is a log store flavor that exports logs to any OpenTelemetry-compatible backend using the OTLP/HTTP protocol with JSON encoding. Built on the OpenTelemetry Python SDK, it provides maximum flexibility for integrating with your existing observability infrastructure.

The OTEL Log Store is a write-only log store. It can export logs to an OTEL-compatible endpoint, but it cannot fetch logs back for display in the ZenML dashboard. If you need log retrieval capabilities, you can extend this log store and implement the fetch() method for your backend. See Develop a Custom Log Store for details on how to do this.

When to use it

The OTEL Log Store is ideal when:

• You have an existing OpenTelemetry-compatible observability platform (e.g., Jaeger, Grafana Tempo, Honeycomb, Lightstep, Dash0)

• You want to consolidate ML pipeline logs with your application logs

• You need to export logs to a custom backend that supports OTLP

• You're building a custom log ingestion pipeline

How it works

The OTEL Log Store implements the OpenTelemetry logging specification:

1. Log capture: All stdout, stderr, and Python logging output is captured during pipeline execution.

2. OTEL conversion: Log records are converted to the OpenTelemetry log format with ZenML-specific attributes.

3. Batching: Logs are batched using OpenTelemetry's BatchLogRecordProcessor for efficient export.

4. Export: Batched logs are sent to your configured endpoint using OTLP/HTTP with JSON encoding and optionally, using data compression.

ZenML-specific attributes

Each log record includes ZenML metadata as OTEL attributes:

Attribute	Description
zenml.log.id	Unique identifier for the log stream
zenml.log.source	Source of the log (step, pipeline, etc.)
zenml.log_store.id	ID of the log store component
zenml.log_store.name	Name of the log store component
zenml.user.id	User ID
zenml.user.name	User name
zenml.project.id	Project ID
zenml.project.name	Project name
zenml.stack.id	Stack ID
zenml.stack.name	Stack name
zenml.pipeline.id	Pipeline ID
zenml.pipeline.name	Pipeline name
zenml.pipeline.run.id	Pipeline run ID
zenml.pipeline.run.name	Pipeline run name
zenml.step.run.name	Step name (for step-level logs)

These attributes enable powerful filtering and querying in your observability platform.

How to use it

You need to have an OpenTelemetry-compatible endpoint ready to receive logs. This could be:

• A self-hosted OTEL Collector

• A managed observability platform (Grafana Cloud, Honeycomb, etc.)

• Any service that accepts OTLP/HTTP with JSON encoding

Register the OTEL log store with your endpoint configuration:

# Register an OTEL log store
zenml log-store register my_otel_logs \
    --flavor=otel \
    --endpoint=https://otel-collector.example.com/v1/logs

# Add it to your stack
zenml stack register my_stack \
    -a my_artifact_store \
    -o default \
    -ls my_otel_logs \
    --set

With authentication headers

Most OTEL backends require authentication. You can pass headers using a ZenML secret:

# Create a secret with your API key
zenml secret create otel_auth \
    --api_key=<YOUR_API_KEY>

# Register the log store with the header
zenml log-store register my_otel_logs \
    --flavor=otel \
    --endpoint=https://otel-collector.example.com/v1/logs \
    --headers='{"Authorization": "Bearer {{otel_auth.api_key}}"}'

With TLS certificates

For endpoints requiring client certificates:

zenml log-store register my_otel_logs \
    --flavor=otel \
    --endpoint=https://secure-collector.example.com/v1/logs \
    --certificate_file=/path/to/ca.crt \
    --client_certificate_file=/path/to/client.crt \
    --client_key_file=/path/to/client.key

Configuration options

Parameter	Default	Description
endpoint	required	OTLP/HTTP endpoint URL for log ingestion
headers	None	Optional headers for authentication
certificate_file	None	Path to CA certificate file for TLS verification
client_certificate_file	None	Path to client certificate file for mTLS
client_key_file	None	Path to client key file for mTLS
compression	"none"	Compression type: "none", "gzip", or "deflate"
service_name	"zenml"	Service name in OTEL resource attributes
service_version	ZenML version	Service version in OTEL resource attributes
max_queue_size	100000	Maximum queue size for batch processor
schedule_delay_millis	5000	Delay between batch exports (milliseconds)
max_export_batch_size	5000	Maximum batch size for exports
export_timeout_millis	15000	Timeout for each export batch (milliseconds)

Retry behavior

The OTEL Log Store includes built-in retry logic for transient failures:

• Retried status codes: 408, 429, 500, 502, 503, 504

• Connection retries: 5 attempts with exponential backoff

• Read retries: 5 attempts

• Backoff factor: 0.5 seconds

This ensures reliable log delivery even in unstable network conditions.

Limitations

1. No log fetching: The OTEL Log Store cannot retrieve logs for display in the ZenML dashboard. You must use your observability platform's native interface to view logs.

2. Dashboard integration: Since logs cannot be fetched, the ZenML dashboard will show "Logs not available" for steps using this log store.

3. Endpoint compatibility: Your endpoint must support OTLP/HTTP with JSON encoding. Protobuf-only endpoints are not supported.

Best practices

1. Use compression: Enable gzip compression for high-volume logging to reduce network bandwidth.

2. Tune batch settings: Adjust max_queue_size and max_export_batch_size based on your log volume:

   • High volume: Increase both values

   • Low latency needs: Decrease schedule_delay_millis

3. Monitor the endpoint: Ensure your OTEL collector or backend can handle the log volume from your pipelines.

4. Use secrets for credentials: Always store API keys and tokens in ZenML secrets, not in plain text.

For more information and a full list of configurable attributes, check out the SDK Docs.