The worker was the obvious bottleneck. Jobs waited in the queue, processing looked CPU-bound, and each worker handled one job at a time. We increased concurrency and watched throughput climb.
Then the product became slower.
The workers drained the queue faster by creating more concurrent writes against the database. Lock waits increased, interactive requests competed with background work, and the latency users felt became worse even though the worker dashboard looked better.
The metric that misled us #
We optimized jobs completed per worker. The system needed us to optimize completed customer operations within a latency budget.
Those are not the same objective.
queue -> worker -> database -> external API -> customer-visible completionEvery arrow has capacity. Increasing pressure at one stage does not create capacity at the next one. It converts a visible queue into hidden contention.
The resolution #
We reduced worker concurrency first. That felt like moving backwards, but it restored the foreground workload while we measured the real constraint.
Then we separated traffic classes, bounded database work per worker, and made concurrency respond to downstream saturation rather than queue length alone. The queue was allowed to be a queue again.
The useful control signals became:
- age of the oldest job;
- customer completion latency;
- database lock and connection wait;
- downstream rejection rate;
- work admitted per tenant;
- recovery time after a burst.
What I kept #
Throughput is a property of the whole path, not its busiest component. A local speedup is valuable only when the bottleneck moves somewhere prepared to receive it.
The senior decision was not finding a higher concurrency number. It was defining which workload had priority when capacity became scarce.
All Field Notes · Queues hide overload · Rate limiting is admission control