Microservices are easy to draw and difficult to operate. The interesting work begins when a dependency is slow, a message arrives twice, or a deployment leaves two API versions running at once.
Design around failure #
A remote call is never equivalent to a local function call. Give every request a deadline, retry only operations that are safe to repeat, and use exponential backoff with jitter. A retry without a budget can turn one unhealthy service into a system-wide incident.
Make events idempotent #
Event-driven systems should assume duplicate delivery. Attach a stable event identifier and store processed identifiers beside the business transaction. Consumers can then acknowledge repeats without repeating the effect.
Prefer observable boundaries #
Structured logs, correlation IDs, latency histograms, and queue-depth metrics make service boundaries visible. Observability is not polish added after launch; it is part of the interface.
The goal is not to prevent every failure. It is to keep failures contained, explainable, and recoverable.