Ayush Basak

The OpenTelemetry Collector Is a Loss Budget

Sep 3

By · Updated Sep 15

opentelemetryobservabilitycloud infrastructurereliabilityplatform engineering

During an incident, the OpenTelemetry Collector becomes a buffer between increasing evidence and a slow or unreachable backend. Its configuration decides what survives.

The invariant is:

Telemetry buffering must be bounded, observable, and unable to consume the resources required to serve the product.

When an exporter slows, its sending queue grows. Queued batches retain memory. Eventually the Collector must drop, reject, persist, block, or crash. An unbounded queue is not reliability; it is an out-of-memory failure with a longer fuse.

Size the outage window

Before sizing a production queue, inspect a deliberately small local pipeline. The demo below has no persistent storage or durable delivery guarantee. Its debug exporter makes processing visible; it does not model the backend outage behavior discussed in the rest of this article.

Architecture / inspect the boundariesThe Collector is a bounded processing pipeline
This local demonstration receives OTLP traces, applies memory protection, batches, and writes to the debug exporter. It has no persistent queue and is not a production durability configuration.Application SDKProducerOTLP receiver127.0.0.1:4317Memory limiterAdmission protectionBatch processorThroughput / latencyDebug exporterLocal demonstration
This local demonstration receives OTLP traces, applies memory protection, batches, and writes to the debug exporter. It has no persistent queue and is not a production durability configuration.
Pan, zoom, and select a node. Layout changes stay in your browser.
Read the components and connections
Application SDK
The SDK has its own bounded queue and export timeout. Collector availability does not eliminate application-side loss.
OTLP receiver
Loopback binding is intentionally local-only. Remote ingestion needs explicit networking, TLS, and authentication.
Memory limiter
Refusal under pressure protects the process; upstream retry and queue limits determine whether refused data survives.
Batch processor
Batching changes export efficiency, not durability. In-memory data can disappear on process failure.
Debug exporter
The debug exporter is for inspecting telemetry, not durable production storage. Replace it with an explicitly bounded production export path.
  • Application SDK → OTLP receiver: export
  • OTLP receiver → Memory limiter: admit
  • Memory limiter → Batch processor: process
  • Batch processor → Debug exporter: flush

The example binds OTLP to loopback. Do not expose it remotely without defining authentication, TLS, and network policy. Validate the downloaded file against the Collector distribution and version you actually deploy using otelcol validate --config=collector-learning.yaml; a valid configuration still needs failure tests. See the Collector configuration reference.

Translate queue configuration into time:

text1 line
buffer window ~= usable queue bytes / peak ingest bytes per second

“5,000 items” means little until batch sizes and signal cardinality are measured. Persistent storage can survive a restart, but disk is still finite and replay traffic can overload a recovering backend.

yaml12 lines
exporters:
  otlp/backend:
    endpoint: telemetry.example.com:4317
    sending_queue:
      enabled: true
      storage: file_storage
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 10m

These are example boundaries, not universal values. Retry duration, queue capacity, disk, exporter concurrency, and backend recovery rate must agree.

Memory limiting is load shedding

Put the memory limiter before processors that allocate more state. Leave headroom below the container limit for receivers, exporters, runtime overhead, and allocation spikes.

yaml11 lines
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 300
  batch:

service:
  pipelines:
    traces:
      processors: [memory_limiter, batch]

Refusal must propagate safely. Application SDKs need bounded queues and backoff; otherwise the Collector protects itself by moving memory pressure into the product process.

Horizontal scale can corrupt the picture

Receivers and exporters are often stateless. Tail sampling is not: it holds spans until a trace decision can be made. If spans from one trace reach different replicas, each sees incomplete evidence. Span-to-metrics aggregation has similar grouping requirements.

OpenTelemetry recommends a routing layer with a load-balancing exporter so related spans reach the same processing Collector:

text2 lines
agents -> routing collectors -> stateful collectors -> backend
                    consistent hash by trace or service

More replicas without affinity can increase throughput while decreasing truth.

Make loss policy explicit

Security audit records should not compete blindly with health-check spans. Separate pipelines when retention obligations differ. Preserve a small unbiased trace sample alongside targeted error or latency sampling; tail sampling provides outcome awareness but consumes memory while traces remain incomplete.

Monitor accepted, refused, retried, queued, and dropped records; queue utilization and time to full; exporter latency and throttling; Collector memory and restarts; trace completeness; and configuration version. Keep a minimal independent signal because loss metrics sent through the broken pipeline may disappear too.

Test backend disconnection, slow drain, full disk, Collector restart, cardinality spikes, and stateful scale-out. Record which evidence survives and whether application latency changes.

The CTO decision

Define tolerable loss per signal and incident class. Size buffers for a bounded recovery window, protect application resources first, preserve affinity for stateful processors, and rehearse failure.

Observability is reliable when its own failure leaves enough evidence to explain the product—not when dashboards happen to be green.

References

>