Changing an A record is not the moment traffic moves. DNS answers are copied through recursive resolvers, operating-system caches, application runtimes, and sometimes connection pools. Every layer can continue using an old address after the authoritative record changes.
The invariant is:
A DNS failover plan must tolerate both old and new destinations serving concurrently for at least the maximum effective cache and connection lifetime.
TTL grants permission to be stale #
An authoritative answer carries a time to live. A recursive resolver may reuse it until that TTL expires without asking the authority again. If a record with a 300-second TTL was cached one second before failover, that resolver can legally return the old value for almost five more minutes.
authoritative change at t=0
resolver A cached old answer at t=-299 → expires at t=1
resolver B cached old answer at t=-1 → expires at t=299There is no single global propagation moment. Different resolvers expire at different times.
Lower TTL before a planned migration, wait at least the prior TTL, then change the record. Lowering the TTL at the same moment as the address does not affect caches already holding the old longer TTL.
The effective lifetime can exceed DNS TTL #
Cloudflare’s documentation notes that local caches may take longer than the configured TTL to reflect changes. Applications may resolve once at startup. JVM and language runtimes can maintain their own DNS policies. HTTP keep-alive, HTTP/2, gRPC, and database pools can keep an existing connection alive without resolving again.
Therefore measure:
effective failover time = max(
recursive cache lifetime,
local/runtime cache lifetime,
live connection lifetime,
health-detection + control-plane update time
)DNS TTL is only one term.
Negative answers are cached too #
RFC 2308 defines caching for NXDOMAIN and NODATA responses using information from the zone’s SOA record. Accidentally querying a name before it exists can seed negative caches; creating the record immediately afterward does not force those resolvers to forget the earlier nonexistence.
RFC 9520 extends required negative caching to resolution failures such as SERVFAIL and DNSSEC validation failure, specifically to prevent retry storms against broken authoritative infrastructure. During an outage, resolvers remembering failure can protect the DNS system while delaying visible recovery for that cache window.
Treat positive TTL, negative TTL, and failure-caching behavior as separate controls.
DNS health is not application health #
A health check may remove an address because one probe path failed while existing connections still work. It may retain an address whose TCP port accepts connections but whose critical dependency is broken. A flapping check can churn answers faster than caches can converge, creating a mixed fleet no operator can reason about.
Use thresholds and a health signal that represents the traffic class being routed. Keep the old destination capable of safe service during the overlap. If writes cannot safely go to both regions, DNS alone is the wrong coordination mechanism; put a strongly controlled routing or data-leadership boundary behind it.
A safe migration timeline #
- Inventory authoritative, negative, runtime, and connection cache lifetimes.
- Lower the TTL early and wait out the previous TTL.
- Bring up the new destination and verify it independently.
- Ensure old and new destinations can coexist safely.
- Change the record and observe traffic at both sites.
- Keep the old destination until the measured tail reaches zero plus margin.
- Raise TTL again after stability.
Never destroy the old endpoint immediately after the DNS control plane reports success.
What to observe #
Query authoritative logs by answer, sample major recursive resolvers, measure traffic reaching old and new endpoints, track connection age, separate resolution errors from connection errors, and test NXDOMAIN recovery. Synthetic checks should query through real recursive paths, not only the authoritative API.
The CTO decision #
Use DNS failover when minutes of mixed routing are acceptable and both destinations remain safe during overlap. For sub-second traffic control, stateful session movement, or single-writer databases, use a routing layer and explicit data failover protocol behind a stable name.
DNS is excellent distributed caching. Its failure behavior follows from that strength.