Observability Is a Data Contract, Not a Dashboard Purchase

Aug 20

By

observabilityopentelemetrydistributed systemsplatform engineering

Teams often buy an observability backend and discover that they still cannot answer the incident question. The vendor stored everything it received; the applications never agreed on what the data meant.

Observability begins before collection. It is a contract that maps production behavior into evidence: operation names, resource identity, error meaning, latency boundaries, and business outcomes. OpenTelemetry is useful because it standardizes the transport and much of this vocabulary. It cannot choose the questions your company must answer.

Start with decisions

Before adding an SDK, write the operational questions:

  • Is checkout meeting its user-visible availability objective?
  • Which dependency consumed the latency budget?
  • Did a request fail before or after committing the order?
  • Is one tenant creating abnormal load?
  • Can a deploy be correlated with the regression?

Each question should map to a signal, a bounded set of attributes, and an action. If a metric has no decision attached, challenge its cost.

user objective
  -> service-level indicator
  -> telemetry event and attributes
  -> alert or investigation
  -> operator action

This chain prevents dashboard-driven instrumentation: collecting whatever a library happens to expose and hoping it becomes useful later.

Semantic conventions are an internal API

OpenTelemetry semantic conventions define common names and meanings for traces, metrics, logs, profiles, and resources. They allow service.name, HTTP attributes, database operations, and messaging spans to mean the same thing across languages.

Treat that vocabulary like an internal API:

  • adopt standard attributes before inventing custom ones;
  • assign an owner for custom conventions;
  • version changes to names or meaning;
  • test required resource attributes in CI;
  • document which attributes are safe to collect;
  • pin or deliberately migrate unstable conventions.

The OpenTelemetry specification publishes stability levels for convention groups. “We use OTel” does not guarantee that every emitted convention is stable. Platform teams must know what they have opted into.

Resource, operation, and occurrence

Good telemetry separates three concepts:

  1. Resource: what produced the signal—service, version, environment, region.
  2. Operation: what kind of work occurred—HTTP route, queue consume, database call.
  3. Occurrence: the particular execution—trace ID, result, duration, selected context.

Do not encode occurrences into metric dimensions. A user_id, request ID, raw URL, SQL text, or stack trace can create unbounded cardinality and turn a useful metric into an expensive index.

Prefer bounded dimensions:

http.request.duration{
  service.name="checkout",
  http.request.method="POST",
  http.route="/orders/:id/confirm",
  http.response.status_code="503"
}

The route template is bounded; /orders/98f.../confirm is not. High-detail occurrences belong in traces or logs, governed by sampling and retention.

Correlation is the leverage

OpenTelemetry’s log model supports trace and span identifiers, while W3C Trace Context standardizes propagation through traceparent and tracestate. With consistent propagation, an alert can lead to a trace, the trace to a dependency span, and the span to correlated logs.

This breaks when teams:

  • generate a new trace at every queue boundary;
  • omit context from retry and dead-letter paths;
  • put trace IDs in message bodies without defining precedence;
  • trust inbound baggage and propagate sensitive or unbounded values;
  • sample the parent but retain unrelated child logs with no join key.

Test propagation as part of an end-to-end contract. Send a synthetic request across HTTP, queue, worker, and database boundaries; assert that the causal chain remains queryable.

Instrument business commit points

Auto-instrumentation sees frameworks. It does not know when your invariant becomes true. Add narrow manual spans or events around domain transitions:

order.confirm
  validation
  payment.authorize
  order.commit       <-- durable business point
  event.enqueue

Marking the whole request as failed after order.commit may be operationally correct, but the retry policy must know that the order already exists. Observability should expose this boundary rather than flatten the request into one red span.

Use span status carefully. The OTel trace specification does not require every expected domain rejection to be an infrastructure error. A declined payment or failed validation may be a successful execution of the software contract. Capture the domain outcome separately and alert on what harms the user objective.

Put a budget on telemetry

Telemetry consumes CPU, network, collector memory, backend ingestion, index storage, and operator attention. Define budgets:

  • maximum attributes per span or log;
  • approved high-cardinality fields and their signal type;
  • head and tail sampling policies;
  • retention by data class;
  • redaction at source and collector;
  • ingestion cost per service or product;
  • collector queue and drop thresholds.

Sampling is a policy decision. Head sampling is cheap but cannot know the eventual result. Tail sampling can retain errors and slow traces but requires buffering and a reliable collector tier. Keep aggregate metrics unsampled for SLO calculation; use sampled traces to explain them.

Design the collector as production infrastructure

The collector decouples applications from backends and centralizes batching, enrichment, redaction, sampling, and export. That makes it a failure domain.

Decide what happens when the backend or collector is unavailable. Application requests should generally not block on telemetry export. SDK queues must be bounded. Collectors need memory limits, persistent or acceptable-loss queues, retry budgets, and self-observability. Alert on dropped spans and exporter failures; a green application with a blind monitoring pipeline is not green.

Common mistakes

Using logs as prose. Structured events with stable fields can be queried and correlated; sentence parsing becomes accidental API design.

Naming spans after raw URLs or SQL. It creates cardinality, leaks data, and fragments one operation across thousands of names.

Collecting sensitive context “for debugging.” Telemetry is broadly accessible and replicated. Minimize and redact before export.

Alerting on infrastructure symptoms without a user objective. High CPU may be harmless; exhausted latency budget is not.

Migrating vendors before fixing semantics. Portable bad data remains bad data.

CTO review checklist

  1. Which user-visible decisions does each primary signal support?
  2. Are resource and operation names consistent across languages?
  3. What are the cardinality, privacy, sampling, and retention budgets?
  4. Does context survive asynchronous boundaries and retries?
  5. Are business commit points observable?
  6. Can the telemetry pipeline fail without taking down the product—and can we detect that failure?

The durable investment is not a dashboard layout. It is a shared operational language that survives service rewrites and backend changes while continuing to answer the questions that matter.

References

>