The OpenTelemetry Collector Is a Loss Budget

Sep 3

By

opentelemetryobservabilitycloud infrastructurereliabilityplatform engineering

During an incident, the OpenTelemetry Collector becomes a buffer between increasing evidence and a slow or unreachable backend. Its configuration decides what survives.

The invariant is:

Telemetry buffering must be bounded, observable, and unable to consume the resources required to serve the product.

When an exporter slows, its sending queue grows. Queued batches retain memory. Eventually the Collector must drop, reject, persist, block, or crash. An unbounded queue is not reliability; it is an out-of-memory failure with a longer fuse.

Size the outage window

Translate queue configuration into time:

buffer window ~= usable queue bytes / peak ingest bytes per second

“5,000 items” means little until batch sizes and signal cardinality are measured. Persistent storage can survive a restart, but disk is still finite and replay traffic can overload a recovering backend.

exporters:
  otlp/backend:
    endpoint: telemetry.example.com:4317
    sending_queue:
      enabled: true
      storage: file_storage
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 10m

These are example boundaries, not universal values. Retry duration, queue capacity, disk, exporter concurrency, and backend recovery rate must agree.

Memory limiting is load shedding

Put the memory limiter before processors that allocate more state. Leave headroom below the container limit for receivers, exporters, runtime overhead, and allocation spikes.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 300
  batch:

service:
  pipelines:
    traces:
      processors: [memory_limiter, batch]

Refusal must propagate safely. Application SDKs need bounded queues and backoff; otherwise the Collector protects itself by moving memory pressure into the product process.

Horizontal scale can corrupt the picture

Receivers and exporters are often stateless. Tail sampling is not: it holds spans until a trace decision can be made. If spans from one trace reach different replicas, each sees incomplete evidence. Span-to-metrics aggregation has similar grouping requirements.

OpenTelemetry recommends a routing layer with a load-balancing exporter so related spans reach the same processing Collector:

agents -> routing collectors -> stateful collectors -> backend
                    consistent hash by trace or service

More replicas without affinity can increase throughput while decreasing truth.

Make loss policy explicit

Security audit records should not compete blindly with health-check spans. Separate pipelines when retention obligations differ. Preserve a small unbiased trace sample alongside targeted error or latency sampling; tail sampling provides outcome awareness but consumes memory while traces remain incomplete.

Monitor accepted, refused, retried, queued, and dropped records; queue utilization and time to full; exporter latency and throttling; Collector memory and restarts; trace completeness; and configuration version. Keep a minimal independent signal because loss metrics sent through the broken pipeline may disappear too.

Test backend disconnection, slow drain, full disk, Collector restart, cardinality spikes, and stateful scale-out. Record which evidence survives and whether application latency changes.

The CTO decision

Define tolerable loss per signal and incident class. Size buffers for a bounded recovery window, protect application resources first, preserve affinity for stateful processors, and rehearse failure.

Observability is reliable when its own failure leaves enough evidence to explain the product—not when dashboards happen to be green.

References

>