Autoscaling Cannot Recover an Expired Deadline

The dashboard shows desired replicas rising. Customers still see timeouts. Nothing is necessarily broken in the autoscaler: it cannot manufacture useful capacity instantly, and it cannot make an expired request successful retroactively.

A scaling policy is a delayed feedback loop. It observes yesterday’s load, requests capacity, waits for infrastructure and application readiness, then observes the result.

The service must survive the interval between detecting excess demand and making additional capacity useful.

That interval belongs in the capacity model, not just in deployment documentation.

Write down the delay chain #

load rises
 -> metrics become available
 -> controller evaluates
 -> replicas are requested
 -> Pods are scheduled
 -> images and application initialize
 -> readiness passes
 -> traffic reaches new capacity

Each stage can lengthen under the same incident. A shared registry may be slow. Nodes may need provisioning. A Java process may warm up. A new replica may open a database pool while the database is already overloaded.

The HPA documentation describes periodic control, metric-based desired replicas, handling of missing metrics, and stabilization. HPA changes desired scale; it does not guarantee that the requested Pods can be scheduled or become ready.

Quantify the bridge capacity #

For a simplified workload, let arrival rate exceed current service capacity by 200 requests per second for a 60-second capacity delay. The backlog grows by roughly 12,000 requests if nothing is rejected. These are illustrative values, not benchmark results.

If callers have two-second deadlines, much of that backlog is already worthless before new Pods arrive. Draining it can then consume the new capacity and extend the incident.

A queue therefore needs a deadline policy, not just a maximum length. Reject work that cannot plausibly finish. Cancel downstream execution when appropriate, and isolate longer-running background work from interactive requests.

Choose a metric that represents recoverable demand #

CPU utilization can be useful for CPU-bound work. It can be misleading for a service waiting on a saturated database: the application is busy in a customer sense while CPU remains low.

Queue age or outstanding work can better represent some workers. Request concurrency can better represent occupied execution slots. No metric removes the need to ask whether another replica actually increases total throughput.

When scaling on CPU utilization relative to requests, resource requests are part of the control model. Changing requests can change the utilization signal without changing customer traffic. Coordinate resource tuning with autoscaling policy.

Use behavior controls intentionally #

This illustrative HPA assumes a Deployment named api, working resource metrics, and suitable CPU requests on its containers:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 4
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

The numbers are a test hypothesis. The minimum is warm capacity, the maximum is a ceiling, and the downscale behavior limits churn. None is a substitute for load testing.

Verify version-specific behavior against the cluster you operate. Avoid adopting newly documented feature gates merely because a current documentation page mentions them.

Protect the dependencies #

Twenty replicas with ten database connections each create a different database demand envelope from four replicas. Scale-out can also multiply TLS handshakes, cache misses, subscription rebalances, and background polling.

Make dependency capacity an explicit upper bound. An autoscaler can move the bottleneck faster than an operator can recognize it.

Test the full recovery curve #

Run step-load and burst tests, not only gradual ramps. Record metric age, desired versus ready replicas, time to first useful response from a new Pod, queue age, deadline success, database saturation, and restart rate.

Test with cold images, a slow dependency, and temporarily unavailable metrics. Then remove load and observe downscale. A policy that responds quickly but oscillates under steady traffic is not stable.

The CTO decision #

Buy enough warm capacity to bridge measured startup delay, combine scaling with admission control, and cap expansion by downstream limits. Judge success by useful requests completed within their deadlines—not the speed at which a replica count rises.

Further reading #

• Kubernetes: Pod lifecycle
• Kubernetes probes are failure policy
• Rate limiting is admission control