An SLO that only appears on a dashboard is a reporting artifact. An SLO becomes an operating mechanism when its error budget changes what the organisation is allowed to do next.
Google SRE defines an error budget as the permitted unreliability implied by an SLO. A 99.9% success target permits 0.1% failure over the measurement window. The useful part is not the arithmetic; it is the agreement between product and engineering about when feature risk is acceptable and when reliability work takes priority.
Measure a user outcome #
Begin with a service-level indicator that represents completed user work. CPU utilization and pod health are diagnostic signals, not availability.
For a checkout API:
good events = valid checkout attempts completed correctly within 2 seconds
valid events = checkout attempts the system accepted responsibility for
SLI = good events / valid eventsDefine exclusions narrowly. If every dependency failure, client retry, or overload response is excluded, the SLO measures the team’s ability to classify errors rather than the customer experience.
Use more than one SLI only when the product promise truly has multiple dimensions. Availability and latency often deserve separate objectives; twenty indicators make ownership ambiguous.
Pick the objective from product consequences #
Do not choose 99.99% because it looks serious. Higher targets reduce the budget for deployments and raise infrastructure and operating cost. Ask:
- How long can the workflow be unavailable before customers take another path?
- Is failure recoverable through retry or reconciliation?
- Does the system move money, control safety-critical equipment, or serve convenience?
- Can support and operations mitigate the failure manually?
- What reliability can dependencies actually sustain?
A dependency with a weaker promise can make a stronger end-to-end SLO fictional unless the architecture adds redundancy or graceful degradation.
Translate the budget into decisions #
Suppose a service receives 10 million valid events in 28 days at a 99.9% SLO:
allowed bad events = 10,000,000 × (1 - 0.999)
= 10,000Track remaining budget and burn rate. A slow burn predicts exhaustion before the window ends; a fast burn detects incidents before the total budget looks large.
A policy should say what happens at thresholds. For example:
| State | Condition | Response |
|---|---|---|
| Healthy | projected consumption < 50% | normal releases |
| At risk | projected consumption 50–100% | reduce risky change; fund named reliability work |
| Exhausted | budget consumed | pause non-essential releases; reliability owner controls exceptions |
| Severe incident | one event consumes > 20% | postmortem and highest-priority corrective action |
The exact numbers should match the organisation. The important property is that they are agreed before an incident.
Do not turn the policy into punishment #
Freezing every change can prolong an incident because fixes, observability improvements, and risk-reducing deployments are changes too. Classify changes:
- reliability restoration and security fixes;
- low-risk, reversible changes;
- ordinary product releases;
- migrations or architectural changes with large blast radius.
When the budget is exhausted, stop the last two categories by default, not the first two. Permit exceptions through a named accountable role with written reasoning and rollback criteria.
Make ownership cross-functional #
Product must participate because the policy reallocates roadmap capacity. Engineering must participate because it owns the failure model. Finance may need to participate when reliability requires material redundancy. Support contributes evidence about customer impact that request metrics miss.
A monthly reliability review should cover:
- SLO performance and remaining budget;
- dominant classes of bad events;
- incidents and near misses;
- reliability work completed versus promised;
- upcoming changes that alter risk;
- objectives that no longer represent product reality.
This is portfolio governance, not merely an SRE meeting.
Common failure modes #
Aspirational SLOs. If no action follows a miss, teams learn the target is optional.
Infrastructure-only indicators. A healthy load balancer can return fast errors while the business workflow is unusable.
One objective for every customer. Free and enterprise tiers may have different promises, but the architecture and commercial agreement must support that distinction.
Monthly averages without burn rate. A severe current outage can hide inside a large remaining monthly budget.
Permanent release freezes. A budget policy should restore balanced delivery, not create an indefinite reliability programme with no exit criteria.
The leadership test #
Ask one question in the roadmap meeting: “If this service exhausts its error budget tomorrow, what work stops and who decides when it resumes?”
If nobody can answer, the SLO is documentation. When the answer is explicit, the organisation has converted reliability from opinion into a decision system.