Small notes for expensive decisions.
Concise engineering takeaways from systems I study and build. Each note names the useful idea, its failure boundary, and one way to test it.
20published notes
20 notes2026
A queue is healthy only when it can recover while traffic continues
A queue is healthy only when it can recover while traffic continues
- Keep
- Backlog size is not the decisive metric. Net drain rate determines whether the system can return to steady state.
- Boundary
- A consumer fleet that matches arrival rate can hold the line but can never remove accumulated lag.
- Test
- Inject one hour of backlog, restore normal traffic, and measure time to return to the lag objective.
Connection pools are promises against a finite database budget
Connection pools are promises against a finite database budget
- Keep
- Pool size multiplied by maximum application replicas is the real configured demand.
- Boundary
- Autoscaling the application can turn a safe local default into database exhaustion.
- Test
- Calculate promised connections at maximum replicas, including workers, migrations, failover, and operator access.
Retries spend the same capacity needed for recovery
Retries spend the same capacity needed for recovery
- Keep
- A retry is additional load, not a free reliability mechanism.
- Boundary
- Correlated failures make independent retry assumptions dangerously optimistic.
- Test
- Model the amplification at the incident failure rate and stop every retry at the caller deadline.
An AI agent memory is governed application state
An AI agent memory is governed application state
- Keep
- Memory needs ownership, schema, retention, provenance, and deletion semantics just like any other durable state.
- Boundary
- Retrieval relevance does not prove that stored context is current, authorized, or safe to act on.
- Test
- Trace one remembered fact to its source, consent, version, expiry, and deletion path.
Error budgets are an investment policy, not an SRE ornament
Error budgets are an investment policy, not an SRE ornament
- Keep
- A budget connects user-visible reliability to the amount of delivery risk a team can responsibly take.
- Boundary
- Availability percentages without an enforcement policy do not change engineering decisions.
- Test
- Write the release action triggered at each burn-rate threshold before the next incident.
Cache invalidation is a consistency protocol
Cache invalidation is a consistency protocol
- Keep
- Caching creates another copy of state, so freshness and invalidation must be explicit contracts.
- Boundary
- A high hit rate can coexist with materially incorrect user outcomes.
- Test
- For every cached value, define authority, maximum staleness, invalidation trigger, and recovery from a missed event.
A schema migration is incomplete until rollback is believable
A schema migration is incomplete until rollback is believable
- Keep
- Expand-and-contract works because old and new code can coexist during the dangerous interval.
- Boundary
- Backward-compatible DDL does not make data transformations automatically reversible.
- Test
- Rehearse rollback after new-format data has already been written.
Exactly-once messaging stops where external side effects begin
Exactly-once messaging stops where external side effects begin
- Keep
- Broker guarantees cannot atomically include an email provider, payment gateway, or arbitrary database.
- Boundary
- A committed offset proves broker progress, not business-effect uniqueness.
- Test
- Crash after the external side effect and before acknowledgement; then replay the event.
Structured AI output is still untrusted input
Structured AI output is still untrusted input
- Keep
- Schema validation proves shape, not truth, permission, or business safety.
- Boundary
- A perfectly valid tool call can still target the wrong tenant or exceed the user’s authority.
- Test
- Run generated actions through the same authorization and invariant checks as hostile API input.
Observability must preserve the decision, not only the event
Observability must preserve the decision, not only the event
- Keep
- Logs become useful when they explain why a system chose an outcome using stable causal context.
- Boundary
- More telemetry can increase cost without reducing time to explanation.
- Test
- Start from one failed request and reconstruct its identity, policy decision, dependencies, and final state.
A read replica needs a freshness contract
A read replica needs a freshness contract
- Keep
- Routing a read to a replica changes the product semantics from current state to potentially stale state.
- Boundary
- Low average lag does not protect a read-after-write workflow during a burst or failover.
- Test
- Define which reads tolerate staleness and how the caller detects or escapes an exceeded lag bound.
Autoscaling cannot recover a request whose deadline already expired
Autoscaling cannot recover a request whose deadline already expired
- Keep
- Scaling is a delayed capacity response; admission control protects work that can still succeed now.
- Boundary
- More replicas can worsen overload when they multiply downstream connections or retries.
- Test
- Compare scale-up time with queueing time and caller deadlines under a realistic burst.
AI evaluations belong in release engineering
AI evaluations belong in release engineering
- Keep
- An evaluation becomes operationally useful when it gates a versioned model, prompt, retrieval corpus, and policy bundle.
- Boundary
- A single aggregate score can hide regressions in rare but expensive slices.
- Test
- Block a release using per-slice thresholds and keep the failing examples as regression fixtures.
A service boundary should reduce coordinated change
A service boundary should reduce coordinated change
- Keep
- Independent deployment is an outcome of data and decision ownership, not a box on a diagram.
- Boundary
- Services that share writable tables or require lockstep releases form one operational unit.
- Test
- Change one capability and count the teams, repositories, schemas, and deployments required.
Backups prove copying; restores prove recoverability
Backups prove copying; restores prove recoverability
- Keep
- Recovery objectives are credible only when a timed restore produces a verified usable system.
- Boundary
- A successful backup job says nothing about dependencies, encryption keys, ordering, or application correctness.
- Test
- Restore into isolation, run integrity checks, and measure the real RPO and RTO.
Build versus buy is an exit-cost decision
Build versus buy is an exit-cost decision
- Keep
- Purchase price is only one term; reversibility, data portability, operational ownership, and switching time shape the real bet.
- Boundary
- A fast integration can quietly place a critical capability behind proprietary state or workflow semantics.
- Test
- Write the exit plan and estimate its cost before signing the entry plan.
A webhook endpoint is a durable inbox
A webhook endpoint is a durable inbox
- Keep
- Acknowledge after durable receipt, then process asynchronously with stable event identity.
- Boundary
- Doing business work before acknowledgement couples provider retry behavior to your slowest dependency.
- Test
- Deliver duplicates, reorder events, pause workers, and crash immediately after durable receipt.
Multi-tenant isolation must exist below the API handler
Multi-tenant isolation must exist below the API handler
- Keep
- Tenant identity should constrain every durable read and write through enforceable lower-level policy.
- Boundary
- A forgotten predicate in one code path can become a cross-tenant data incident.
- Test
- Attempt access with a valid user identity and a mismatched tenant context at every storage boundary.
Control planes should fail quietly
Control planes should fail quietly
- Keep
- Existing data-plane work should continue safely when configuration or orchestration becomes unavailable.
- Boundary
- A control-plane outage becomes a product outage when every request requires fresh coordination.
- Test
- Remove the control plane and observe which established workloads stop, retry, or lose policy.
Architecture decisions need expiry dates
Architecture decisions need expiry dates
- Keep
- A decision made under temporary constraints should not become permanent through institutional amnesia.
- Boundary
- Reviewing everything continuously creates ceremony; never reviewing anything creates accidental architecture.
- Test
- Attach a trigger—scale, cost, regulation, or date—that forces the decision back into review.
Start with a decision
- Service boundaries: failure isolation and independent change
- Database choices: data models, invariants, and operational constraints
- Delivery semantics: retries, duplicates, and the outbox boundary
- Observability: context, signals, and a useful data contract
- API invariants: protect concurrent updates with version checks