The Dashboard Was Green. The Customer Was Stuck.

The API was up. The database was reachable. The queue had consumers. Every component-level check was green. The customer still could not complete the operation.

This failure appears in asynchronous products: a request is accepted, work crosses several boundaries, and one transition stops making progress without crashing a process. Infrastructure sees healthy machines. The user sees a promise that never resolves.

What the dashboard could not answer #

Our telemetry described request latency, database connections, consumer health, and error counts. It did not describe the lifecycle of the thing the customer cared about. We could not immediately answer where one operation was, how long it had occupied that state, or whether retrying would repeat an effect.

That is not a missing chart. It is a missing domain model.

The immediate resolution #

The first job is to stop creating ambiguity. Disable an unsafe action if repetition can cause harm, identify affected operation IDs, and establish whether the downstream effect occurred before replaying anything. “Restart the consumer” is not recovery when success may have happened before acknowledgement.

What changed #

We moved the operational view from service health to operation state:

accepted → validated → claimed → effect_requested
         → effect_confirmed → completed
         ↘ terminal_failure
         ↘ reconciliation_required

Each transition needed a timestamp, stable ID, attempt record, and owner. Alerts moved toward oldest age and transition rate instead of queue length alone. Support needed a safe explanation without database archaeology.

I had treated observability as evidence for engineers. The incident showed it is also a product capability. If operations and support cannot tell the truth about a customer action, the system is not observable enough.

The fix was not more logs. It was agreement on the state machine and the ambiguous states we refused to hide.

For every important workflow, I now want completion rate, time in state, and age of the oldest unfinished operation. Then I want a reconciliation path for cases metrics cannot resolve automatically.

A green dashboard should mean the customer’s promise is being kept—not merely that the processes are alive.

All Field Notes · Queues and backpressure · OpenTelemetry as a data contract