Tail Sampling Is a Memory Budget Disguised as an Observability Feature

Head sampling decides whether to keep a trace when it begins. It is cheap and cannot know the outcome. Tail sampling waits for spans to arrive, then can retain errors, slow traces, or rare attributes. That power requires buffering an unknown amount of unfinished distributed work.

The invariant is:

Every span for one trace reaches the same decision maker, and the decision is made only after a bounded wait with bounded memory.

The collector does not know when a trace is complete #

Distributed tracing has no universal “final span” message. Services report asynchronously; retries and queues can create late spans. A tail sampler groups spans by trace ID and waits for a configured decision delay.

t=0ms    root span begins
t=80ms   database span arrives
t=200ms  HTTP response completes
t=900ms  async child arrives
t=10s    sampling decision fires

If the delay is too short, the decision sees an incomplete trace. If it is too long, memory and export latency grow. Tail sampling is therefore an approximation over a time window, not omniscient post-processing.

Trace affinity is mandatory #

If spans from one trace are load-balanced across several independent tail samplers, each sees a fragment and may make a different decision. Route by trace ID to a stable collector shard before tail sampling.

agents/gateways → load-balancing exporter(hash trace_id)
                → tail-sampler shard
                → backend

Changing shard membership can move in-flight trace IDs. Plan rolling updates and failures knowing that buffered traces are ephemeral unless an external durable layer exists. Most collector deployments trade perfect preservation for bounded cost.

Memory follows arrival rate and delay #

A first planning estimate is:

buffer bytes ≈ spans/second × decision delay × average span bytes × overhead

Bursts, large attributes, and long traces dominate the tail. Size from measured high percentiles, not average payloads. The collector also needs memory for receivers, processors, queues, and exporters; giving the tail sampler the entire container limit turns pressure into process death.

Configure expected trace capacity, a memory limiter, exporter queues, and refusal/drop metrics together. A sampler that preserves every error until its own OOM kills all buffered errors has not improved observability.

Policies compose in surprising ways #

Useful policies include:

• always keep error status;
• keep latency above a service-specific threshold;
• keep security-sensitive routes;
• probabilistically sample the healthy remainder;
• rate-limit high-volume categories.

Policy order and combination semantics matter. An inverted or broad string/attribute rule can sample nearly everything. A global “over 1 second” rule over-samples naturally long batch jobs and under-samples a 400 ms endpoint whose SLO is 100 ms.

Attach service criticality and route class as governed resource/span attributes, then test policies against recorded distributions before rollout.

Sampling cannot repair bad instrumentation #

An error trace without causal context remains unhelpful. High-cardinality attributes increase memory and backend cost. Secrets copied into spans become a larger exposure when tail policy deliberately preserves exceptional requests.

The schema must define stable service identity, operation name, status, tenant classification where permitted, and links across asynchronous boundaries. Redaction belongs before buffering and export.

Failure policy #

Keep a small independent head-sampled stream if losing every trace during a tail-sampler failure is unacceptable. Redundant evidence is often cheaper than pretending one pipeline is lossless.

What to monitor #

Track traces in memory, spans per trace, decision latency, sampled and dropped traces by policy, late spans, collector RSS versus limit, refused spans, exporter queue occupancy, shard balance, and backend ingestion rate. Alert on sample-rate changes even when collector health appears green.

The CTO decision #

Use tail sampling when outcome-aware selection materially improves diagnosis and its buffering cost is understood. Keep head sampling when low latency, simple scaling, or predictable loss is more valuable. Many platforms need both: a small unbiased baseline plus targeted tail retention.

Sampling is a data-loss policy. Make the loss observable.

References #

• OpenTelemetry: Sampling
• OpenTelemetry Collector: Tail Sampling Processor
• OpenTelemetry sample configurations
• Related: Observability is a data contract