# ayushworks — full-text corpus

> Technical writing by Ayush Basak. Each document includes its canonical URL. Cite the canonical HTML page, not this aggregate file.

# The Hardware Did Not Read the API Contract

> Timing, partial truth, and designing software around a physical system that cannot be rolled back.

Canonical URL: https://www.ayushworks.xyz/field-notes/hardware-did-not-read-the-api-contract
Author: Ayush Basak
Last modified: 2026-09-01
Topics: myotrek, connected-products, system-design, field-notes

Web software encourages a dangerous reflex: if the contract is wrong, deploy another version.

Connected products remove that comfort. While working around the companion software behind [Myotrek](https://www.myotrek.com/), the physical experience kept forcing a better question: which system currently knows the truth?

The phone may know the intended workout. The backend may know the saved plan. The device may know what physically happened. The person knows whether feedback felt immediate and believable. Those truths arrive at different times.

## The clean model failed first

```text
client sends command → server accepts → device performs → UI confirms
```

Reality contains disconnections, delayed acknowledgements, app suspension, repeated commands, partial movement, firmware differences, and a person who will not wait for a distributed transaction before moving again.

Treating an API response as completed physical work creates false confidence. Treating every disconnect as failure can cause unsafe repetition.

## The resolution was a richer state

We need to separate intent, acceptance, observation, and completion:

```text
requested ≠ accepted ≠ observed ≠ completed
```

“Saved,” “sent,” and “completed” should not collapse into one optimistic checkmark. Commands need stable identities. Device observations need monotonic sequence information where possible. Reconnection needs reconciliation rather than assuming the latest client screen is authoritative.

## Product management across physics

Hardware makes prioritization less abstract. A visual imperfection can wait; a confusing state around physical action cannot. The cheapest backend shortcut may create the most expensive support problem when users cannot distinguish delay from failure.

Compatibility must span devices that will not all update together. The contract needs capability negotiation and safe defaults, not only a version number.

I now review a connected feature as a timeline, not a screen:

- What does each participant believe at every step?
- Which acknowledgement proves receipt, and which proves effect?
- What can repeat safely?
- What happens when the app disappears mid-operation?
- How does an older device respond to a newer command?
- What does the person see while truth is incomplete?

The physical world does not conform to the abstraction. Good architecture makes that mismatch explicit and gives the user a truthful, calm experience while systems converge.

---

[All Field Notes](/field-notes) · [Offline-first payments](/posts/offline-first-payments) · [API evolution without flag days](/posts/api-evolution-without-flag-days)


---

# Support Needed a Truthful Answer, Not Another Dashboard

> Turning an ambiguous asynchronous operation into a supportable product contract.

Canonical URL: https://www.ayushworks.xyz/field-notes/support-needed-a-truthful-answer
Author: Ayush Basak
Last modified: 2026-09-06
Topics: support, product-engineering, operations, field-notes

A customer asked whether an operation had completed. Engineering could see a request log, a queue message, and an outbound timeout. None of them proved the business outcome.

Support did not need access to three observability tools. They needed one truthful answer the system did not yet know how to produce.

## The wrong first instinct

Our first proposal was a better internal dashboard. It would have made investigation faster, but it still required a person to infer state from technical evidence. Two people could reach different conclusions.

That is an operating procedure built over a missing product model.

## The resolution

We introduced an operation record with states that support and customers could understand:

```text
received
processing
completed
failed_before_effect
outcome_being_confirmed
```

The last state mattered most. A downstream timeout was not automatically a failure; the effect might have committed before the response was lost. We refused to translate uncertainty into a reassuring but incorrect status.

Each operation exposed a stable reference, last confirmed transition, expected next action, and safe retry policy. A reconciliation worker resolved ambiguous outcomes against the authoritative system. Support could trigger approved recovery actions without writing SQL or replaying messages manually.

## The management lesson

Support escalations are often treated as interruptions to engineering. Repeated questions are design input. They reveal where internal state cannot be translated into a user promise.

We added one review question for important workflows:

> If this operation stops between any two steps, what exact sentence can support truthfully tell the customer?

If the answer required reading logs, the workflow was not finished.

## What I kept

Operational tooling is most valuable when it removes interpretation, not when it displays more telemetry. The goal is not to make support behave like database engineers. It is to make the product’s state legible and its recovery actions safe.

---

[All Field Notes](/field-notes) · [A webhook endpoint is a durable inbox](/posts/webhook-endpoints-are-durable-inboxes) · [Incident command is a distributed system](/posts/incident-command-is-a-distributed-system)


---

# The Architecture Is Not the Product

> A field note on choosing engineering boundaries for Saleslyt, Myotrek, healthcare services, and offline payments—and why the real constraint matters more than the impressive diagram.

Canonical URL: https://www.ayushworks.xyz/field-notes/the-architecture-is-not-the-product
Author: Ayush Basak
Last modified: 2026-08-30
Topics: product-engineering, technical-leadership, management, field-notes

Architecture diagrams have a dangerous property: every box looks equally important.

In the product, they are not.

The projects I have worked on occupy very different realities. [Saleslyt](https://saleslyt.com/) turns relationship and sales activity into useful decisions. [Myotrek](https://www.myotrek.com/) connects software to a physical fitness product. A [healthcare microservice platform](https://github.com/ayushbasak101/healthcare-microservices) moves information across service boundaries. An [offline payment system](https://github.com/ayushbasak101/UPI-offline) must reason about a transaction when the network cannot be trusted to answer.

All four can be drawn with APIs, databases, queues, and clients. That similarity is almost useless. The cost of being wrong is different in each one.

## The first architecture was the wrong question

Early engineering conversations naturally start with nouns:

- Which database?
- Monolith or microservices?
- REST, gRPC, or events?
- Which model?
- Which cloud?

Those questions feel productive because they produce decisions. They are often premature.

The better opening question is:

> Which uncertainty must this product remove for the person using it?

For a sales product, storing activity is not the outcome. The system must turn scattered activity into a next decision somebody trusts enough to act on. For connected fitness, a technically successful API response is not enough if the experience around the physical movement feels late, confusing, or detached. For healthcare systems, a clean service boundary does not compensate for incomplete context at a clinical boundary. For offline payments, “request timed out” is not an acceptable model of whether value moved.

The architecture begins where the product cannot afford ambiguity.

## The real constraint chooses the boundary

Consider the same design choice—performing work asynchronously—in four contexts.

| Product boundary | Useful async work | Dangerous async ambiguity |
|---|---|---|
| Saleslyt | enrich an interaction, calculate recommendations | silently losing the task a salesperson expects to see |
| Myotrek | aggregate workout history, prepare analytics | delaying feedback that is part of the live physical experience |
| Healthcare services | fan out secondary notifications | separating a clinical decision from required patient context |
| Offline payments | transport and reconcile after connectivity returns | allowing a replay to become a second settlement |

“Use a queue” is not a design. A design says what may be delayed, what must remain ordered, what can be repeated, what must be reconciled, and what the user sees while certainty is unavailable.

This is the dirty part of engineering: the same mechanism can improve one product and damage another.

## Correctness includes the operator

We often write invariants as if only code participates:

```text
one command → at most one committed effect
```

But production correctness includes the person operating the system. Can support determine what happened? Can an engineer distinguish delayed work from lost work? Can a manager explain the state to a customer without inventing certainty? Can the team reverse a bad release without depending on the path that is failing?

A system that preserves data but gives operators no evidence is technically recoverable and operationally broken.

That changes what “done” means. A background job needs a stable identity, visible state, retry history, and a terminal outcome. An AI recommendation needs its source context, model and prompt version, policy result, and an explanation of what the product will do when confidence is inadequate. An offline command needs an identity that survives transport and a reconciliation view that can answer whether it settled.

The audit trail is not paperwork added after the architecture. It is part of the product's ability to tell the truth.

## The shortcut is not the failure

Small teams take shortcuts. This is not a confession; it is resource allocation.

The damaging shortcut is the one whose risk remains implicit.

A deliberate shortcut has four parts:

1. **The constraint:** why the ideal path is too expensive now.
2. **The protected invariant:** what the shortcut is still forbidden to break.
3. **The detection:** how the team will know when the compromise stops working.
4. **The exit:** what evidence will trigger replacement.

For example, keeping two domains in one deployable service may be correct while the team is small. The protected invariant could be separate ownership of their data and explicit internal interfaces. The detection could be release coupling and incident frequency. The exit condition could be independent scaling or a team boundary—not an arbitrary user count.

Technical debt becomes dangerous when it has no interest rate anyone measures.

## Management is system design with people

Engineering leadership is often described as making the best technical decision. More often, the work is sequencing decisions so the team can still learn.

An irreversible architecture chosen with weak evidence is not conviction. It is a large bet with poor observability.

I prefer decisions that preserve the next decision:

- define the domain model before splitting services;
- measure queue age before adding more workers;
- record model and policy versions before optimizing AI routing;
- establish command identity before adding automatic retries;
- prove restore before buying a more complicated backup product;
- put one clear owner on an operational boundary before adding another tool.

This is also a management choice. Every abstraction has a teaching cost. Every service has an on-call cost. Every new datastore creates another failure language the team must learn under pressure.

The architecture must fit not only the load, but the organization available to operate it.

## What I now ask in a product review

I no longer start with whether the proposed architecture is elegant. I ask:

### Product truth

- What promise is the user making based on this system's answer?
- Which wrong answer is worse than no answer?
- When the system is uncertain, what does the user see?

### Failure truth

- Where can the outcome become ambiguous?
- Which effect can repeat?
- What is the smallest blast radius we can enforce?
- How will an operator reconstruct the event without reading raw database rows?

### Delivery truth

- Which part must exist now to learn anything useful?
- Which “scalable” choice creates operational work before it creates value?
- What assumption would make us reverse this decision?

### Team truth

- Who owns the boundary at 2 a.m.?
- Can the team explain and test its recovery path?
- Are we adding a technology because the constraint requires it or because the diagram rewards it?

These questions make architecture less theatrical. They also make disagreement more useful because the team can argue about the constraint instead of defending a favorite tool.

## The resolution is a tighter feedback loop

There is no single architecture lesson shared by Saleslyt, Myotrek, healthcare services, and offline payments. That is the lesson.

The work is to make the product's unique uncertainty visible, place a correctness boundary around it, and shorten the distance between a real outcome and the team learning from it.

Sometimes the resolution is a stronger transaction. Sometimes it is a queue, a reconciliation worker, or a local snapshot. Sometimes it is removing a service. Sometimes it is changing the user flow so the product stops claiming certainty it does not possess. Sometimes it is a management decision: narrow the scope, assign one owner, and give the team enough time to understand the failure before adding machinery.

Technology, chosen with intent and built—not just advised—means staying close enough to see where the intended system and the lived system diverge.

The diagram is a hypothesis.

The product is what happens to people when that hypothesis meets reality.

---

[Browse all Field Notes](/field-notes) · [See the projects behind these lessons](/projects) · [Read the engineering essays](/posts)


---

# The Best Fix Deleted a Workflow

> We stopped automating an accidental process and removed the reason it existed.

Canonical URL: https://www.ayushworks.xyz/field-notes/the-best-fix-deleted-a-workflow
Author: Ayush Basak
Last modified: 2026-09-06
Topics: product-management, simplicity, automation, field-notes

The request sounded straightforward: automate a manual approval that delayed every release.

We could have built a workflow engine, notifications, escalation timers, audit pages, and an override role. Instead, we asked why this approval existed.

The answer was a production incident from much earlier. One unsafe configuration change had escaped review, so an approval step was added for every deployment. The underlying configuration path had since changed, but the process remained.

## Automation would have preserved the wrong thing

The manual step was slow and inconsistent. Automating it would make the same policy faster and more consistent. It would not make the policy useful.

We separated the original risk from the accumulated ritual:

- What unsafe action had caused the incident?
- Could the system reject it automatically?
- Which changes were actually high risk?
- What evidence did an approver inspect?
- Was approval preventing failure or distributing accountability?

Most releases did not touch the risky configuration. The approver usually checked that tests passed and the deployment was staged—facts the pipeline already knew.

## The resolution

We encoded the invariant at the boundary that could enforce it, added a policy test for dangerous configuration combinations, and required explicit review only when a change crossed that boundary. Ordinary releases no longer entered the approval workflow.

The final system had less code than the proposed automation:

```text
ordinary change -> automated evidence -> staged rollout
risky change    -> automated evidence -> named review -> staged rollout
```

The audit trail improved because the exceptional decision now carried a reason. Previously, hundreds of routine approvals hid the few that mattered.

## What I kept

Before automating a process, ask whether the process is a product requirement, a control, or scar tissue. Automation multiplies whichever one it receives.

Deleting a workflow can look less impressive than building a platform. It is often the higher-leverage engineering decision because it removes code, waiting, operational ownership, and an opportunity for the organization to confuse activity with safety.

---

[All Field Notes](/field-notes) · [The cost of complexity](/posts/the-cost-of-complexity) · [Verify what you deploy](/posts/verify-what-you-deploy)


---

# The Dashboard Was Green. The Customer Was Stuck.

> Why component health is not product health, and how to rebuild observability around a customer operation.

Canonical URL: https://www.ayushworks.xyz/field-notes/the-dashboard-was-green-the-customer-was-stuck
Author: Ayush Basak
Last modified: 2026-09-01
Topics: incidents, observability, product-engineering, field-notes

The API was up. The database was reachable. The queue had consumers. Every component-level check was green. The customer still could not complete the operation.

This failure appears in asynchronous products: a request is accepted, work crosses several boundaries, and one transition stops making progress without crashing a process. Infrastructure sees healthy machines. The user sees a promise that never resolves.

## What the dashboard could not answer

Our telemetry described request latency, database connections, consumer health, and error counts. It did not describe the lifecycle of the thing the customer cared about. We could not immediately answer where one operation was, how long it had occupied that state, or whether retrying would repeat an effect.

That is not a missing chart. It is a missing domain model.

## The immediate resolution

The first job is to stop creating ambiguity. Disable an unsafe action if repetition can cause harm, identify affected operation IDs, and establish whether the downstream effect occurred before replaying anything. “Restart the consumer” is not recovery when success may have happened before acknowledgement.

## What changed

We moved the operational view from service health to operation state:

```text
accepted → validated → claimed → effect_requested
         → effect_confirmed → completed
         ↘ terminal_failure
         ↘ reconciliation_required
```

Each transition needed a timestamp, stable ID, attempt record, and owner. Alerts moved toward oldest age and transition rate instead of queue length alone. Support needed a safe explanation without database archaeology.

I had treated observability as evidence for engineers. The incident showed it is also a product capability. If operations and support cannot tell the truth about a customer action, the system is not observable enough.

The fix was not more logs. It was agreement on the state machine and the ambiguous states we refused to hide.

For every important workflow, I now want completion rate, time in state, and age of the oldest unfinished operation. Then I want a reconciliation path for cases metrics cannot resolve automatically.

A green dashboard should mean the customer's promise is being kept—not merely that the processes are alive.

---

[All Field Notes](/field-notes) · [Queues and backpressure](/posts/queues-backpressure-overload-control) · [OpenTelemetry as a data contract](/posts/opentelemetry-observability-data-contract)


---

# The Demo That Changed the Roadmap

> Discovering that correct software was answering the wrong question—and changing the roadmap before adding more machinery.

Canonical URL: https://www.ayushworks.xyz/field-notes/the-demo-that-changed-the-roadmap
Author: Ayush Basak
Last modified: 2026-09-01
Topics: product-engineering, leadership, saleslyt, field-notes

The feature worked. That was the problem: because it worked, we could have spent weeks making the wrong thing faster.

In an early product conversation around [Saleslyt](https://saleslyt.com/), the system could collect activity, enrich it, and present it cleanly. The architecture discussion moved naturally toward more connectors, automated analysis, and a more capable pipeline. Then the demo reached the question that mattered: what should the user do next?

The product had information. It did not yet remove a decision.

## What we mistook for value

Engineering progress is easy to count: an integration completed, an event consumed, a model response generated, a dashboard populated. Product progress exists when a person's next action becomes clearer, safer, or cheaper.

We had optimized the supply of intelligence before defining the decision contract:

```text
Given this evidence, for this person, at this moment,
what action is justified—and why?
```

Without that contract, adding data increased cognitive load. Every new signal created another fact the user had to interpret.

## The decision

We narrowed the roadmap. Instead of treating “more intelligence” as the outcome, the product flow had to earn three things:

1. identify a specific next action;
2. expose the evidence behind it;
3. make uncertainty visible rather than decorating it as confidence.

That changed the engineering sequence. Provenance became more important than another enrichment source. Feedback on whether a recommendation helped became more important than a richer dashboard. A smaller, explainable action loop became more valuable than a general agent with broad access.

## The management lesson

Stopping technically good work is uncomfortable. The code is real and the team has momentum. A leader must make it safe to say that correct implementation is not evidence of product value.

I now ask for a decision sentence before approving a major product system:

> After this ships, which decision can a user make that they cannot make reliably today?

If the answer is a list of features, the problem is still undefined.

The review moved from component completeness to decision quality: false positives, missing context, stale evidence, and the fallback when the system was unsure. Those are product questions and architecture questions at the same time.

The dirty engineering lesson is that a roadmap sometimes improves by deleting a technically impressive path. The cheapest moment to remove the wrong abstraction is before more of the organization depends on it.

---

[All Field Notes](/field-notes) · [AI evals are release engineering](/posts/ai-evals-are-release-engineering) · [Architecture decisions need expiry dates](/posts/architecture-decisions-need-expiry-dates)


---

# The Feature Flag Had No Owner

> A temporary rollout control became permanent production state because nobody owned its removal.

Canonical URL: https://www.ayushworks.xyz/field-notes/the-feature-flag-had-no-owner
Author: Ayush Basak
Last modified: 2026-09-06
Topics: feature-flags, operations, ownership, field-notes

The flag began as a careful rollout control. Months later it still guarded two implementations, nobody remembered the intended default, and changing it required asking in several channels.

The flag had become production state without the ownership we would demand from a database table or API.

## Why it stayed

Creation was part of delivery. Removal belonged to “later.” The ticket closed when exposure reached one hundred percent, not when the old path disappeared.

That left us paying for:

- two behaviors to test;
- two paths to secure;
- ambiguous incident diagnosis;
- configuration drift between environments;
- a control whose blast radius was no longer understood.

The problem was not that flags are bad. The problem was treating their lifecycle as optional cleanup.

## The resolution

We classified the flag before touching it: release flag, operational kill switch, experiment, or entitlement. Each type needed a different lifetime.

For the stale release flag, we inspected actual evaluation telemetry, confirmed that the old branch had no intended traffic, deleted the dead branch, and removed the flag definition in the same change sequence. We did not simply hide it from the dashboard.

New flags needed:

```text
owner
purpose
created_at
expected_default
expiry_or_review_at
removal_issue
safe behavior when provider is unavailable
```

Operational kill switches could remain, but they needed drills and a conservative local default. Experiments needed exposure and analysis boundaries. Entitlements needed an authorization model rather than an informal boolean.

## What I kept

Temporary infrastructure needs a removal path at creation time. An expiry date without an owner is only a future alert.

The mature question is not “Do we use feature flags?” It is “Which persistent decisions have we created, who owns them, and how does each one end?”

---

[All Field Notes](/field-notes) · [Feature flags are production state](/posts/feature-flags-are-production-state) · [Architecture decisions need expiry dates](/posts/architecture-decisions-need-expiry-dates)


---

# The Incident Needed an Owner, Not Another Channel

> Reducing coordination load, separating command from investigation, and making decisions reversible under pressure.

Canonical URL: https://www.ayushworks.xyz/field-notes/the-incident-needed-an-owner-not-another-channel
Author: Ayush Basak
Last modified: 2026-09-01
Topics: leadership, incident-management, management, field-notes

When an incident becomes confusing, teams often add communication: another call, another channel, another person asking for an update. The result can be less information and fewer decisions.

In one recurring incident shape, several people investigated different components while nobody owned the customer outcome. Each observation was locally useful. Together they formed a noisy, leaderless system.

## The bottleneck

The bottleneck was serialized decision-making. Someone needed to maintain the current hypothesis, choose the next safe action, record what changed, and decide when evidence justified rollback. Investigators needed freedom to test bounded hypotheses without simultaneously explaining the entire incident.

We separated responsibilities:

- **incident lead:** owns outcome, priority, and decisions;
- **investigators:** gather evidence and report concise findings;
- **scribe:** preserves timeline, actions, and open questions;
- **communications owner:** translates facts without inventing certainty.

One person can hold multiple roles in a small team. The responsibilities still need names.

## Reversibility beats cleverness

Under pressure, the best action often reduces harm and preserves the next decision: pause a risky worker, disable one feature path, route traffic away from a cell, or roll back when rollback is safer than diagnosing live.

We used a small decision record:

```text
time | observation | hypothesis | action | owner | expected signal
```

The expected signal matters. Without it, an action becomes activity rather than an experiment.

A leader joining an incident can accidentally reset the room by asking everyone to retell the story. Read the timeline first. Ask what decision is blocked. Remove obstacles. Do not become a second incident commander by seniority.

Psychological safety has operational value. Engineers report contradictory evidence faster when they are not defending earlier hypotheses. A clean correction is more valuable than consistency with a guess made twenty minutes ago.

After recovery, review the management system: when ownership became clear, which signal changed the decision, which action increased ambiguity, and whether support could state customer impact.

Incident response is a distributed system made of people. A clear owner, explicit roles, shared timeline, and reversible actions create the control plane the team needs when the technical control plane is already failing.

---

[All Field Notes](/field-notes) · [Incident command is a distributed system](/posts/incident-command-is-a-distributed-system) · [Control planes must fail quietly](/posts/control-planes-must-fail-quietly)


---

# The Microservice We Chose Not to Build

> Preserving a domain boundary without paying for an independent service too early.

Canonical URL: https://www.ayushworks.xyz/field-notes/the-microservice-we-chose-not-to-build
Author: Ayush Basak
Last modified: 2026-09-01
Topics: architecture, management, microservices, field-notes

The proposed service had a reasonable name, a clear diagram, and enough differences from the rest of the system to justify a boundary. We chose not to deploy it independently.

This was not a rejection of the domain model. It was a decision about when the organization should start paying the operational cost of distribution.

## The attractive argument

The capability had its own workflow and would evolve. A service promised independent releases, scaling, and ownership. Those benefits matter after a team has the pressure that requires them.

Before then, it would add a network failure mode, deployment pipeline, telemetry surface, authorization boundary, data-consistency protocol, and another thing to own during an incident. “Small service” does not mean small operational responsibility.

## What we protected instead

We kept the domain boundary inside the existing deployable unit:

- its own module and vocabulary;
- no direct writes into another domain's tables;
- an explicit internal interface;
- separate tests around its invariants;
- events defined by business meaning, not table changes.

That preserved the option to extract later without pretending extraction would be free.

## The extraction trigger

“When we scale” is not a trigger. We wrote observable reasons:

1. materially different scaling;
2. repeated release coupling;
3. a durable team ownership boundary;
4. measurable value from fault isolation;
5. data or compliance rules demanding independent control.

Until one became true, a separate process would have been architecture in advance of evidence.

Every technology choice consumes learning capacity. A small team operating five understandable components can move faster than the same team operating fifteen fashionable ones. The constraint is rarely whether engineers can create another service. It is whether they can explain and recover every interaction at 2 a.m.

Saying no also needs precision. “Keep the monolith” cannot become permission for tangled ownership. The decision worked only because the logical boundary was real and reviewed.

Modularity and distribution are different decisions. Make the first early; earn the second through evidence. The service we did not build let the product learn without making the organization operate a hypothesis as permanent infrastructure.

---

[All Field Notes](/field-notes) · [Service boundaries](/engineering-notes#service-boundaries) · [Architecture decisions need expiry dates](/posts/architecture-decisions-need-expiry-dates)


---

# The Migration Was Safe Until We Needed Rollback

> A schema change passed forward tests but exposed that rollback was never part of the design.

Canonical URL: https://www.ayushworks.xyz/field-notes/the-migration-was-safe-until-we-needed-rollback
Author: Ayush Basak
Last modified: 2026-09-06
Topics: databases, deployments, schema-migrations, field-notes

The migration was additive. A new column, a backfill, and a release that started reading it. Every forward step passed in staging.

Production exposed a behavior we had not modeled. The obvious response was to roll the application back. The previous version could still run, but it no longer understood the partially migrated state.

We had tested whether the new code worked with the old schema. We had not tested whether the old code worked after the new code wrote data.

## The missing compatibility direction

Safe rollout requires more than backward-compatible DDL. It requires a compatibility window across readers, writers, and stored representations.

```text
old reader + old data
new reader + old data
new writer + mixed readers
old reader + data written by new writer
```

The final case was our gap.

## The immediate choice

We stopped expansion rather than reversing blindly. The new write path was disabled, affected rows were identified, and reads temporarily fell back to the old representation. Recovery became a data decision, not a deployment button.

The revised migration had four explicit phases:

1. expand the schema;
2. deploy code that can read both forms while writing the old form;
3. begin dual writes and verify equivalence;
4. switch reads, then contract only after the rollback window closes.

Each phase had a measurable completion condition and a named reversal path.

## What I kept

Rollback is not “deploy the previous commit.” It is a promise that the previous behavior can interpret the state created since it left production.

That promise expires. Once an irreversible backfill or destructive contraction begins, recovery may require a forward fix instead. The runbook should say when that boundary is crossed.

The migration did not fail because the SQL was unsafe. It failed because compatibility had been modeled in only one direction.

---

[All Field Notes](/field-notes) · [Zero-downtime migrations are compatibility problems](/posts/postgresql-zero-downtime-schema-migrations) · [Evolve APIs without flag days](/posts/api-evolution-without-flag-days)


---

# We Made the Worker Faster and the Product Slower

> A throughput improvement moved the bottleneck downstream and made the customer experience worse.

Canonical URL: https://www.ayushworks.xyz/field-notes/we-made-the-worker-faster-and-the-product-slower
Author: Ayush Basak
Last modified: 2026-09-06
Topics: performance, queues, capacity-planning, field-notes

The worker was the obvious bottleneck. Jobs waited in the queue, processing looked CPU-bound, and each worker handled one job at a time. We increased concurrency and watched throughput climb.

Then the product became slower.

The workers drained the queue faster by creating more concurrent writes against the database. Lock waits increased, interactive requests competed with background work, and the latency users felt became worse even though the worker dashboard looked better.

## The metric that misled us

We optimized jobs completed per worker. The system needed us to optimize completed customer operations within a latency budget.

Those are not the same objective.

```text
queue -> worker -> database -> external API -> customer-visible completion
```

Every arrow has capacity. Increasing pressure at one stage does not create capacity at the next one. It converts a visible queue into hidden contention.

## The resolution

We reduced worker concurrency first. That felt like moving backwards, but it restored the foreground workload while we measured the real constraint.

Then we separated traffic classes, bounded database work per worker, and made concurrency respond to downstream saturation rather than queue length alone. The queue was allowed to be a queue again.

The useful control signals became:

- age of the oldest job;
- customer completion latency;
- database lock and connection wait;
- downstream rejection rate;
- work admitted per tenant;
- recovery time after a burst.

## What I kept

Throughput is a property of the whole path, not its busiest component. A local speedup is valuable only when the bottleneck moves somewhere prepared to receive it.

The senior decision was not finding a higher concurrency number. It was defining which workload had priority when capacity became scarce.

---

[All Field Notes](/field-notes) · [Queues hide overload](/posts/queues-backpressure-overload-control) · [Rate limiting is admission control](/posts/rate-limiting-is-admission-control)


---

# Building AI Products for Accessibility

> Product principles for AI-assisted reading experiences.

Canonical URL: https://www.ayushworks.xyz/posts/accessible-ai-products
Author: Ayush Basak
Last modified: 2026-06-12
Topics: ai-infrastructure, accessibility, product-engineering

An accessibility product should reduce effort without taking control away from the person using it. AI can simplify text, explain vocabulary, and restructure information, but the interface around the model matters just as much as the model itself.

## Preserve meaning

Simplification is not summarization. Important qualifications, numbers, names, and instructions must survive a rewrite. Showing the original beside the transformed version makes comparison easy and keeps the user in control.

## Offer adjustable help

There is no single accessible reading mode. Let people choose sentence length, vocabulary level, spacing, contrast, and whether explanations appear inline or on demand.

## Measure trust, not novelty

Useful evaluation includes factual consistency, reading time, correction rate, and whether users feel confident acting on the result. A feature that looks impressive but quietly changes meaning is a failure.

The best assistive AI feels less like automation taking over and more like a capable tool adapting to its user.


---

# AI Agents Need a Control Plane, Not a Larger Prompt

> A production architecture for AI-agent identity, MCP authorization, bounded tools, policy enforcement, budgets, approvals, and auditable execution.

Canonical URL: https://www.ayushworks.xyz/posts/ai-agent-control-plane-mcp-security
Author: Ayush Basak
Last modified: 2026-08-20
Topics: ai-agents, ai-infrastructure, security, system-design

An AI agent becomes a production system when it can read private context, call tools, mutate state, or spend money. At that point, “the model was instructed not to” is not a security boundary.

The correct architecture assumes the model can misunderstand, be manipulated by retrieved content, or choose the wrong tool. A deterministic control plane must decide what the agent may see, what it may do, how much it may spend, and which actions require a person.

## Split reasoning from authority

The model proposes actions. The control plane authorizes and executes them.

```text
user / event
    -> agent runtime (reason and propose)
    -> policy decision point
    -> approval gate when required
    -> typed tool gateway
    -> target system
    -> immutable audit event
```

Never hand the model a general-purpose credential and hope the prompt scopes its use. Tool credentials belong in the gateway. The runtime receives a capability limited to the current subject, tenant, operation, resource, and expiry.

## Identity must survive the chain

Every action needs at least four identities:

- the human or service that initiated the task;
- the agent definition and deployed version;
- the execution or task instance;
- the credentialed tool or downstream service.

Without these, an audit log saying “agent called CRM” cannot answer who authorized it, which instructions were active, or which tenant was affected.

Propagate an execution ID, but do not confuse correlation with authorization. A trace ID helps find events; it grants no permission.

## MCP standardizes transport, not your policy

The Model Context Protocol defines resources, prompts, and tools, plus authorization behavior for HTTP transports. Its authorization specification uses OAuth mechanisms and requires resource indicators so tokens are bound to their intended server. MCP servers must validate that tokens were issued for them and must not pass an inbound token through to an upstream API.

This prevents an important confused-deputy path: a legitimate token for one resource being replayed against another. It still does not answer whether this particular agent may refund this particular invoice for this tenant now. That is application policy.

Represent the decision explicitly:

```json
{
  "subject": "user:1248",
  "agent": "support-agent@2026-08-20.3",
  "action": "invoice.refund",
  "resource": "invoice:inv_72",
  "tenant": "tenant:acme",
  "constraints": {
    "maxAmountCents": 5000,
    "expiresAt": "2026-08-20T16:05:00Z",
    "requiresApproval": true
  }
}
```

Authorize close to execution, not only when the conversation begins. Permissions, resource state, and risk can change during a long-running task.

## Make tools narrow and typed

A tool named `run_sql(query)` or `http_request(url, body)` exposes an enormous authority surface. Prefer domain tools:

```text
get_customer_summary(customer_id)
draft_refund(invoice_id, amount, reason)
submit_refund(draft_id, approval_token)
```

The gateway validates schema, tenant ownership, state preconditions, amount limits, and idempotency keys. Descriptions are user experience for the model, not enforcement.

Separate read, draft, and commit. The agent can explore and prepare a change without possessing immediate write authority. High-impact commits receive a short-lived approval token bound to the exact action digest; editing the amount or target invalidates approval.

## Treat all retrieved content as data

An email, webpage, PDF, ticket, or tool response can contain instructions aimed at the model. The control plane must not promote retrieved text into trusted policy.

Use structural separation:

- trusted system policy comes from versioned configuration;
- user intent is recorded separately;
- retrieved content is labelled untrusted;
- tool output is validated against a schema;
- sensitive actions are authorized from server-side facts, not model claims;
- egress and accessible resources are allowlisted.

Prompt-injection detection may add a signal. It cannot be the only boundary because classification will have false negatives.

## Bound autonomy with budgets

An agent can be logically correct and economically destructive. Give each execution budgets for:

- model tokens and monetary cost;
- wall-clock duration;
- tool calls and retries;
- rows, files, or accounts touched;
- outbound messages;
- concurrent child tasks;
- irreversible operations.

Enforce budgets outside the model. When a limit is reached, stop at a recoverable checkpoint and return evidence. Do not ask the same model that exceeded a budget whether it should receive more authority.

## Design every write for retries

Agent runtimes retry after timeouts, worker loss, or uncertain responses. A timed-out write may already have succeeded. Every mutating tool should accept a stable idempotency key derived from execution and logical action:

```text
idempotency_key = sha256(execution_id + action_index + normalized_arguments)
```

The tool gateway stores the first result and returns it for identical retries. A different payload with the same key is rejected. For systems without native idempotency, add an adapter or a reconciliation state such as `outcome_unknown`; never translate uncertainty into an automatic second irreversible action.

## Audit events must support reconstruction

Capture enough to reconstruct the decision without indiscriminately storing private prompts:

- initiator, agent version, execution ID, and policy version;
- tool name and normalized argument hash;
- resources and tenant affected;
- authorization decision and reason code;
- approval identity and action digest;
- idempotency key, result class, latency, and cost;
- model and retrieval provenance where permitted.

Use append-only storage with retention and access controls. Redact secrets at ingestion. Auditability is not “log everything”; it is preserve the minimum trustworthy chain of custody.

## Common mistakes

**One OAuth token for the entire agent platform.** A compromise becomes cross-tenant and cross-tool.

**Approval of prose rather than action.** “Proceed” is ambiguous. Bind approval to normalized arguments and resource state.

**Tool schemas without server-side invariants.** Types stop malformed inputs, not unauthorized valid inputs.

**Giving write tools to a planning agent.** Separate proposal from execution and keep least privilege per stage.

**Logging secrets to improve debugging.** Agent traces are a new sensitive data store; minimize them deliberately.

## CTO review checklist

1. Can the model execute anything that the policy layer did not independently authorize?
2. Are tokens audience-bound, short-lived, tenant-scoped, and never passed through?
3. Are irreversible actions separated into draft, approval, and commit?
4. What happens after an ambiguous timeout?
5. Which budgets stop loops and economic abuse?
6. Can an incident reviewer reconstruct who authorized the exact action under which policy?

Agent quality will improve. The need for explicit authority will not. Build the control plane so stronger models become more useful without becoming more dangerous.

## References

- [Model Context Protocol: Authorization](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization)
- [Model Context Protocol: Security and Trust Principles](https://modelcontextprotocol.io/specification/2025-03-26/index)
- [RFC 8707: Resource Indicators for OAuth 2.0](https://www.rfc-editor.org/rfc/rfc8707)
- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
- [Related: Scaling Self-Organizing Multi-Agent Clusters](/posts/scaling-patterns-for-self-organizing-multi-agent-clusters-with-kiro)
- [Related: Building AI Products for Accessibility](/posts/accessible-ai-products)


---

# AI Agent Memory Is Governed State

> A production architecture for agent memory: provenance, consent, retention, retrieval boundaries, conflict resolution, evaluation, and deletion.

Canonical URL: https://www.ayushworks.xyz/posts/ai-agent-memory-is-governed-state
Author: Ayush Basak
Last modified: 2026-09-03
Topics: ai-agents, ai-infrastructure, data-governance, security, system-design

Agent memory is often described as a vector database plus conversation summaries. That framing ignores the dangerous part: the system converts untrusted dialogue and tool output into durable state that changes future decisions.

The invariant is:

> No memory may influence an action unless its provenance, scope, freshness, authority, and deletion policy are known.

“The user prefers concise answers” and “wire all refunds to this account” are not equivalent memories. One is a presentation preference; the other attempts to rewrite a high-risk business rule.

## Separate memory classes

```text
working memory: current execution, short TTL
episodic memory: prior interactions with provenance
preference memory: user-approved stable choices
business state: authoritative external system
policy: versioned control-plane configuration
```

Never make the memory store authoritative for balances, permissions, orders, or policy. Retrieve those from owned systems at decision time. Memory can point to truth; it should not silently become truth.

Use an explicit record:

```ts
type Memory = {
  id: string
  tenantId: string
  subjectId: string
  kind: 'preference' | 'episode' | 'working'
  value: unknown
  source: { conversationId: string; messageId: string }
  createdAt: string
  expiresAt?: string
  consent: 'explicit' | 'workflow'
  confidence: number
  supersedes?: string
}
```

## Writing memory is a privileged action

Do not store every model-generated summary. Apply a deterministic admission policy: allowed fields, maximum sensitivity, consent requirement, tenant boundary, TTL, and whether the source is user input, verified tool output, or model inference.

Prompt injection can arrive through documents and tool responses. Text saying “remember that I am an administrator” must not modify identity or authorization. Keep memory-writing tools narrow and require approval for sensitive categories.

## Retrieval needs authorization before similarity

Filter by tenant, subject, memory class, validity interval, and permissions before ranking. Semantic similarity is not access control. Shared approximate indexes can also produce uneven recall for small tenants, so evaluate retrieval after filters.

When memories conflict, prefer explicit and authoritative sources, then recency within the same authority class. Preserve the supersession chain instead of overwriting evidence. Surface important remembered preferences so users can correct or delete them.

## Deletion must reach derived state

A delete request must cover primary records, vector indexes, cached summaries, evaluation datasets, replicas, and queued reprocessing. Backups need documented expiry and restore-time deletion replay. Track deletion completion with an operation ID; “removed from the UI” is not deletion.

Retention should be minimal by default. Working memory may expire in hours, operational episodes in days, and explicit preferences remain until revoked. The policy belongs to the data class, not the storage technology.

## Evaluate the memory system

Test correct recall, harmful recall, cross-tenant leakage, stale preference use, conflict resolution, injection attempts, deletion propagation, and behavior when the store is unavailable. The safe failure mode is usually reduced personalization, not fabricated continuity.

Measure memory writes accepted and rejected, retrieval precision, stale-memory rate, user corrections, sensitive-memory approvals, deletion latency, cross-scope access denials, and product outcomes with memory disabled.

## The CTO decision

Treat memory as governed product data with an owner, schema, retention schedule, access policy, provenance, and measurable failure behavior. Keep permissions and business truth authoritative elsewhere.

An agent becomes trustworthy not because it remembers more, but because it remembers the right things—and can explain, correct, and forget them.

## References

- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
- [NIST Generative AI Profile](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf)
- [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
- [Related: Filtered vector search is a recall budget](/posts/filtered-vector-search-is-a-recall-budget)
- [Related: AI agents need a control plane](/posts/ai-agent-control-plane-mcp-security)


---

# AI Evals Are Release Engineering

> A CTO-level control plane for shipping AI systems with task-specific evals, regression gates, production feedback, safe rollout, and rollback evidence.

Canonical URL: https://www.ayushworks.xyz/posts/ai-evals-are-release-engineering
Author: Ayush Basak
Last modified: 2026-08-25
Topics: ai-infrastructure, reliability, technical-leadership, llm

An AI feature is not a prompt plus a model endpoint. It is a probabilistic release whose behavior depends on model version, instructions, tools, retrieval, data, sampling, policy, and the surrounding application.

If any of those can change without a regression gate, the organization does not have an AI release process. It has production sampling.

## Define the system under test

Pin a release candidate as a complete configuration:

```yaml
release: support-agent-2026-08-25.3
model: pinned-provider-version
prompt_sha: 4e91c2a
tool_schema_sha: b8031f7
retrieval_index: help-center-2026-08-24
policy_version: support-v7
temperature: 0.1
```

“We tested the model” is incomplete when production uses different tools, documents, or policies. Store this manifest with every evaluation result and production trace. Without provenance, a better or worse score cannot be explained or reproduced.

## Evaluate business behavior

Generic benchmarks help compare broad model capabilities. They do not prove that your refund agent applies your policy, cites the right account record, or refuses to expose another tenant’s data.

Build an evaluation set from the real task distribution:

- normal high-volume requests;
- high-value business paths;
- ambiguous and underspecified inputs;
- known production incidents;
- adversarial security cases;
- long-tail languages and formats;
- tool and dependency failures;
- explicit “must refuse” cases.

Each case needs a reason for existing and an owner. A thousand uncurated examples can provide less signal than one hundred cases mapped to failure modes.

## Use layered graders

No single score is trustworthy enough for release control.

**Deterministic checks** validate properties with exact answers: JSON schema, required citations, allowed tool names, arithmetic, SQL syntax, tenant IDs, or absence of secrets.

```python
def grade_tool_call(output, account_id):
    call = output["tool_calls"][0]
    return (
        call["name"] in {"lookup_order", "create_refund_request"}
        and call["arguments"]["account_id"] == account_id
    )
```

**Reference checks** compare extracted facts, selected actions, or expected outcomes.

**Model graders** help assess open-ended qualities such as relevance or completeness. Calibrate them against human judgments, randomize candidate order when comparing outputs, and retain disagreement samples.

**Human review** remains necessary for policy, brand, and novel high-impact failures.

The release gate should expose a scorecard, not collapse every concern into one average:

| Dimension | Gate |
|---|---:|
| tenant isolation | 100% |
| unsafe action prevention | 100% |
| tool selection | ≥ 98% |
| answer correctness | ≥ 95% |
| p95 latency | ≤ 4 s |
| cost per resolved case | within budget |

A gain in writing style must never hide a regression in tenant isolation.

## Test trajectories, not only final text

Agent systems can reach a plausible answer through an unsafe path. Capture the trajectory:

```text
user input
  → retrieved documents
  → model decision
  → tool request
  → tool response
  → final answer
```

Grade whether retrieval crossed tenant boundaries, whether the chosen tool was allowed, whether arguments were validated, whether retries duplicated a side effect, and whether the final statement is supported by observed tool output.

Tool authorization belongs in deterministic application code. An evaluation verifies that the agent behaves correctly; it must not be the only barrier preventing a model from deleting data.

## Prevent evaluation leakage

Keep at least three datasets:

1. **Development set:** visible to people tuning the system.
2. **Release set:** stable regression suite with controlled access.
3. **Holdout set:** refreshed and hidden from routine optimization.

If every failed release case becomes a prompt example, the suite eventually measures memorization. Rotate samples from production distributions and create transformed variants that preserve the failure mechanism.

Version datasets. Never overwrite a case after changing its expected result; record the policy version that changed the expectation.

## Join offline and online evidence

Offline evals are repeatable and safe, but production traffic changes. Online signals reveal new inputs and real dependency behavior.

Instrument model, retrieval, and tool spans using a consistent trace ID. OpenTelemetry’s generative-AI semantic conventions provide a common vocabulary, though the conventions continue to evolve; isolate vendor-specific attributes behind your telemetry boundary.

Collect:

- task success or escalation;
- corrected answers and user feedback;
- tool errors and rejected arguments;
- groundedness and citation failures;
- token usage, latency, and cost;
- policy refusals by category;
- release configuration and trace provenance.

Do not log sensitive prompts by default. Apply redaction, access control, retention, and sampling before payload capture.

Convert production failures into reviewed eval cases. That closes the loop:

```text
incident → minimized case → regression test
→ candidate fix → canary → production evidence
```

## Roll out like infrastructure

Use shadow traffic where privacy and cost allow, then a bounded canary. Compare the candidate against the current release on the same task distribution. Set automatic rollback triggers for safety violations, tool-error spikes, latency, and unit cost.

Keep the previous complete release manifest deployable. Rolling back only the prompt while leaving a new tool schema or retrieval index active is not a rollback.

For high-impact actions, separate recommendation from execution. Require deterministic policy checks, idempotency keys, audit trails, and human approval above a risk threshold.

## CTO review

1. Is the complete AI configuration versioned and reproducible?
2. Which business failures does each eval set represent?
3. Which dimensions are hard gates rather than averages?
4. Are agent trajectories and tool effects evaluated?
5. How are graders calibrated against human judgment?
6. How do production incidents become permanent regression cases?
7. Can the complete previous release be restored quickly?
8. What action remains impossible regardless of model output?

Evals are not a demo score. They are the test, provenance, rollout, and rollback system for probabilistic software.

## References

- [OpenAI: Evaluating model performance](https://platform.openai.com/docs/guides/evals)
- [OpenTelemetry: Generative AI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
- [Related: Durable AI Agent Workflows](/posts/durable-ai-agent-workflows)
- [Related: AI Agent Control Plane and MCP Security](/posts/ai-agent-control-plane-mcp-security)


---

# AI-Generated Code Creates Review Debt

> Govern AI-assisted delivery by bounding change size, preserving authorship, testing behavior, and measuring review load.

Canonical URL: https://www.ayushworks.xyz/posts/ai-generated-code-creates-review-debt
Author: Ayush Basak
Last modified: 2026-09-05
Topics: engineering-leadership, ai-assisted-development, code-review, cto

AI can reduce the cost of producing code faster than it reduces the cost of understanding, validating, and operating that code. The gap is review debt: behavior enters the system faster than the organization can build justified confidence in it.

Review debt is not measured in generated lines. It appears as oversized changes, shallow approvals, duplicated abstractions, unowned dependencies, weak failure policy, and incidents whose code nobody can explain.

## Optimize for verified change, not output

Commit volume is an activity measure. The delivery unit is a change whose intent, risk, tests, rollout, and owner are clear.

An AI-assisted pull request should answer:

```text
Intent: what user or operational outcome changes?
Invariant: what must remain true?
Evidence: which tests or measurements support the change?
Boundary: what is deliberately not changed?
Rollback: how is exposure stopped or reverted?
Owner: who can explain and operate it?
```

The human author owns every generated line. “The model wrote it” is provenance, not accountability.

## Bound the review surface

Large generated diffs are cheap for the author and expensive for every reviewer. Set constraints on behavioral scope rather than arbitrary line counts:

- one migration phase per change;
- no unrelated refactor in a reliability fix;
- generated dependencies require explicit justification;
- public API changes include compatibility tests;
- security-sensitive changes require named reviewers;
- generated tests must prove observable behavior, not mirror implementation.

Ask the tool to produce a plan, risk list, and smallest testable patch before implementation. This makes assumptions reviewable earlier, when correction is cheap.

## Use risk-tiered evidence

| Change | Minimum evidence |
| --- | --- |
| copy or internal tooling | lint, tests, preview |
| ordinary application behavior | unit and integration tests, staged rollout |
| schema or data migration | compatibility proof, backfill plan, rollback |
| auth, payments, tenant isolation | threat review, negative tests, audit trail |
| infrastructure control plane | failure injection, canary, runbook |

AI assistance does not lower the evidence required. It may help generate it, but generated evidence must itself be checked.

## Measure system effects

Track review wait time, change failure rate, rollback rate, escaped defects, median change size, time to understand incidents, and concentration of ownership. Compare AI-assisted and conventional work by risk class. Avoid turning acceptance rate or suggestions per developer into performance targets; they reward volume rather than outcomes.

The SPACE framework is useful here because developer productivity has multiple dimensions—satisfaction, performance, activity, communication, and efficiency. A single throughput number can improve while system comprehension deteriorates.

## Failure boundaries

- Never send secrets or regulated data to an unapproved model endpoint.
- Pin or record relevant tool and model versions for sensitive changes.
- Reject code whose license or provenance cannot be resolved.
- Do not let an agent merge, deploy, and validate its own high-risk change without an independent gate.
- Keep production permissions narrower than repository write permissions.

## Trade-offs

Stricter gates reduce raw speed and can frustrate experienced teams. Loose gates maximize experimentation but shift cost into review and operations. The practical policy varies by risk: fast lanes for reversible, observable changes; strong evidence for irreversible or high-blast-radius changes.

The leadership task is not to maximize AI usage. It is to preserve understanding as the marginal cost of producing code falls.

## Further reading

- [The SPACE of Developer Productivity](https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity-theres-more-to-it-than-you-think/)
- [Google SRE: Release Engineering](https://sre.google/sre-book/release-engineering/)
- [Verify what you deploy](/posts/verify-what-you-deploy)


---

# AI Inference Routing Is Capacity Control

> A production design for routing AI inference by task, risk, latency, cost, capability, evaluation evidence, and deterministic fallback—not model fashion.

Canonical URL: https://www.ayushworks.xyz/posts/ai-inference-routing-is-capacity-control
Author: Ayush Basak
Last modified: 2026-08-29
Topics: ai-infrastructure, ai-agents, reliability, cloud-economics

Many AI systems begin with one model behind one API call. Production introduces conflicting objectives: low latency, bounded spend, regional availability, privacy, tool support, context size, and quality that varies by task.

A router that merely sends requests to the cheapest available model will eventually convert a capacity incident into a correctness incident.

The governing invariant is:

> A request may be routed only to a model configuration proven acceptable for that task and risk class.

Availability does not mean “some model returned text.” It means the system produced an outcome inside the product’s quality and safety boundary.

## Classify the task before selecting the model

Route on a small, deterministic task contract:

```json
{
  "task": "support_reply_draft",
  "risk": "reviewed",
  "latency_slo_ms": 2500,
  "max_cost_usd": 0.03,
  "required": ["json_schema"],
  "data_region": "in",
  "evaluation_set": "support-reply-v7"
}
```

Do not let the model self-declare its risk or budget. Those are product decisions derived from the user action, data class, and consequence of error.

Build a capability registry containing provider, model version, supported features, context limit, regional policy, observed latency, price, and evaluation status. Configuration changes should be versioned and auditable.

## Use a constrained candidate set

Routing is two steps:

1. **Eligibility:** remove models that fail capability, privacy, risk, or evaluation requirements.
2. **Optimization:** among eligible candidates, choose for latency, cost, and capacity.

```text
candidates
→ policy filter
→ evaluation threshold
→ capacity admission
→ cost/latency choice
→ execution
```

This order matters. Cost must never make an unapproved model eligible.

Evaluation gates should be task-specific. A model that performs well at extraction may be unacceptable for SQL generation. Store the dataset version, scoring method, sample size, and approval time. Re-evaluate when prompts, tools, model versions, or decoding settings change.

## Reserve capacity before execution

Tokens, concurrency, and spend are finite shared resources. Estimate an upper bound, then atomically reserve it before calling the provider.

```text
tenant budget + task budget + provider concurrency + deadline
                           ↓
                     admit or reject
```

Settle actual usage afterward and release the difference. Without reservation, one burst can admit thousands of individually valid requests that collectively exceed budget or provider concurrency.

Apply backpressure at the product boundary. Queue only work whose value survives delay. Interactive requests should degrade or reject within a deadline rather than wait behind an unbounded batch.

## Design semantic fallback

Fallback is safe only when the substitute preserves required capability and quality.

| Primary failure | Possible response |
|---|---|
| provider timeout | try one eligible alternate inside remaining deadline |
| rate limit | honor retry signal or route to reserved alternate capacity |
| schema failure | one constrained repair attempt, then deterministic rejection |
| policy filter failure | do not retry with a weaker policy |
| no evaluated model available | use non-AI workflow, cached result, review queue, or fail |
| ambiguous tool effect | reconcile by operation ID before any retry |

Do not silently fall from a high-capability model to a cheap model for a high-risk action. “No answer” can be safer than a plausible wrong one.

## Observe decisions, not private prompts

Record stable routing evidence:

- task and risk class;
- router policy version;
- selected provider and model version;
- candidate rejection reasons;
- estimated and actual tokens and cost;
- queue, provider, and total latency;
- fallback path;
- schema, policy, and evaluation outcome.

Use OpenTelemetry’s generative-AI semantic conventions where they fit, but avoid capturing sensitive prompts by default. High-cardinality request detail belongs in controlled traces or audit storage, not metric labels.

## Roll out routing changes like code

A router policy can alter quality, cost, and customer behavior instantly. Treat it as a production release:

1. replay against a frozen evaluation set;
2. shadow on real traffic where permitted;
3. canary by task and tenant cohort;
4. compare quality, latency, fallback, and cost;
5. promote gradually with an automatic rollback condition.

Keep the last known-good policy locally available. The data path should not require a healthy configuration control plane for every inference request.

## Conclusion

An inference router is not a model switch. It is an admission controller and policy enforcement point for correctness, capacity, risk, and economics.

Filter by capability and evidence first. Reserve scarce resources. Fall back semantically. Observe routing decisions. Release changes gradually. That is how multi-model infrastructure creates resilience instead of hiding nondeterminism behind another layer.

## References

- [NIST Generative AI Profile](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence)
- [OpenTelemetry Generative AI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
- [Google SRE: Handling Overload](https://sre.google/sre-book/handling-overload/)


---

# Evolve APIs Without Flag Days

> A production migration protocol for compatibility envelopes, expand-contract changes, Protobuf schemas, events, observability, and deletion evidence.

Canonical URL: https://www.ayushworks.xyz/posts/api-evolution-without-flag-days
Author: Ayush Basak
Last modified: 2026-08-25
Topics: api-design, distributed-systems, platform-engineering, system-design

An API change is not deployed when the producer ships. It is deployed when every relevant producer, consumer, stored message, retry, cache, replay, and rollback path can coexist with it.

That coexistence period is the compatibility envelope. CTO-level platform design makes the envelope explicit rather than hoping teams deploy in the correct order.

## Model versions that can meet

During a rolling migration, at least four interactions are possible:

```text
old client → old server
old client → new server
new client → old server
new client → new server
```

Queues and stored payloads extend the matrix: a new consumer may read a message written months ago, while an old consumer may receive a freshly produced event after rollback.

Write the required compatibility matrix before changing a schema. If “new client → old server” is unsupported, state how deployment and rollback prevent that pairing. Compatibility is an operational promise, not a serialization feature.

## Expand, migrate, contract

Safe evolution usually has three phases.

**Expand:** add the new representation while preserving the old contract.

```json
{
  "customer_name": "A. Basak",
  "customer": { "display_name": "A. Basak" }
}
```

**Migrate:** update readers, backfill stored data, observe adoption, and move writers.

**Contract:** remove the legacy field only after evidence shows that nothing depends on it.

The hard part is not adding a field. It is proving deletion is safe. Assign every migration an owner, deadline, adoption metric, and removal condition. Otherwise compatibility layers accumulate into permanent complexity.

## Make readers tolerant but semantics strict

Readers should generally tolerate additive fields. They should not silently reinterpret meaning.

Changing `timeout_seconds` from a request timeout to an end-to-end deadline keeps the JSON type but breaks behavior. Renaming `amount` to `amount_cents` without a dual-read period may be syntactically obvious and operationally unsafe.

For each field, document:

- unit and allowed range;
- absence versus explicit zero or empty;
- defaulting owner;
- whether unknown enum values are accepted;
- security and tenancy scope;
- lifetime and retention.

Schema compatibility cannot protect an undocumented semantic contract.

## Protobuf safety is more than “it parses”

Protocol Buffers supports additive binary evolution: old code can ignore unknown fields and new code can read messages without newly added fields. But its official guidance includes constraints that matter in production:

- never change an existing field number;
- reserve deleted field numbers and names;
- do not reuse tags;
- treat type changes marked “compatible” as rollout-sensitive because values can be lossy;
- be careful with `oneof` changes and unknown enum behavior;
- distinguish binary wire behavior from ProtoJSON behavior.

```proto
message Account {
  reserved 3;
  reserved "legacy_tier";

  string id = 1;
  string display_name = 2;
  optional string billing_region = 4;
}
```

Add compatibility checks to CI, but do not confuse them with migration approval. A wire-safe change can still violate authorization or business semantics.

## Version behavior, not every URL

Global `/v2` endpoints are useful when the resource model genuinely changes. They are expensive when used for every additive field: clients split, documentation duplicates, and old versions never disappear.

Choose the smallest mechanism matching the change:

- additive optional field: evolve in place;
- behavior negotiated by capability: explicit header or field;
- incompatible resource model: new version or resource;
- one consumer’s special need: avoid contaminating the shared contract; consider a dedicated boundary.

HTTP semantics matter. A `PUT` remains idempotent; a retry should not create another resource because a version changed. Status codes, cache validators, and content negotiation are part of the contract described by HTTP, not decoration around the JSON.

## Events make old contracts immortal

An event in a durable log can outlive every currently deployed service. Before changing an event schema, test:

```text
new reader × oldest retained event
old reader × new event
replay × current side-effect policy
rollback × messages emitted during new release
```

Prefer facts with stable meaning. If `OrderConfirmed` later needs a tax jurisdiction, add it without redefining what “confirmed” meant historically. When meaning truly changes, create a new event type and make the transition explicit.

Keep golden payloads from real historical versions. Decode and re-encode them in CI. Generated examples are less likely to contain the odd omissions and enum values that break production replays.

## Observe the migration

Instrument contract usage:

- requests by client identity and version;
- reads and writes of legacy fields;
- unknown enum or field incidents;
- fallback/default path usage;
- decode failures by schema version;
- remaining stored rows needing backfill;
- oldest message schema in retention.

A removal gate might be:

```text
legacy writes = 0 for 14 days
legacy reads = 0 for 14 days
backfill remaining = 0
rollback no longer requires old field
named owners approve consumer inventory
```

Logs sampled at one percent may miss the monthly billing job. Combine telemetry with ownership and code search.

## Design rollback before rollout

A deployment is not safely reversible if the new writer emits data the old reader cannot interpret. Sequence changes so rollback remains possible:

1. Deploy readers that understand old and new forms.
2. Confirm fleet convergence.
3. Enable new writes behind a controlled flag.
4. Observe both paths.
5. Stop old writes.
6. Remove old reads only after the rollback window closes.

For destructive database changes, application rollback and schema rollback are separate decisions. Often the safest rollback is forward-fixing the application while leaving expanded storage intact.

## CTO review

1. Which old and new versions can interact during rollout and rollback?
2. Are syntax and semantics both compatible?
3. How are durable messages and stored payloads tested?
4. What telemetry proves old behavior is unused?
5. Who owns lagging consumers outside the team?
6. When does the compatibility layer expire?
7. Can the old binary read data produced by the new writer?

The mature alternative to a flag day is not permanent backward compatibility. It is controlled coexistence followed by evidence-based deletion.

## References

- [Protocol Buffers: Updating a Message Type](https://protobuf.dev/programming-guides/proto3/#updating)
- [Protocol Buffers Best Practices](https://protobuf.dev/best-practices/dos-donts/)
- [RFC 9110: HTTP Semantics](https://datatracker.ietf.org/doc/html/rfc9110)
- [Related: Transactional Outbox Delivery Guarantees](/posts/transactional-outbox-delivery-guarantees)
- [Related: Zero-Downtime PostgreSQL Schema Migrations](/posts/postgresql-zero-downtime-schema-migrations)


---

# Architecture Decisions Need Expiry Dates

> A CTO-level operating model for architecture decisions: explicit context, measurable consequences, review triggers, ownership, and reversible migration paths.

Canonical URL: https://www.ayushworks.xyz/posts/architecture-decisions-need-expiry-dates
Author: Ayush Basak
Last modified: 2026-08-29
Topics: systems-architecture, technical-leadership, architecture-governance, decision-making

Architecture is not the collection of technologies a company uses. It is the set of decisions that are expensive to reverse: ownership boundaries, data authority, consistency guarantees, failure containment, security policy, and the path by which change reaches production.

Most architecture records fail because they document a conclusion as if it were timeless. “We chose Kafka” survives in a repository long after the volume, team shape, and delivery requirements that justified it have disappeared.

The governing invariant should be:

> An architecture decision is valid only while its assumptions remain true.

That changes an ADR from a historical note into an operational control.

## Record the pressure, not just the choice

A useful decision record answers six questions:

1. What constraint forced a decision now?
2. Which invariant must remain true?
3. Which options were seriously considered?
4. What evidence distinguished them?
5. What new operational burden does the choice create?
6. What observation would cause a review?

“Use PostgreSQL” is not a decision. “Keep order acceptance strongly consistent inside one regional failure domain; accept asynchronous propagation to analytics; review when write saturation exceeds 60% for thirty days” is one.

The second form carries a measurable validity boundary. It also prevents a future team from replacing a boring, correct database merely because a different technology looks more scalable in isolation.

## Give decisions triggers, not calendar theatre

A review date is useful for regulated controls and vendor contracts, but many technical decisions should be reviewed when reality changes.

| Trigger | Example |
|---|---|
| scale | p99 latency misses its objective at 60% tested capacity |
| topology | the product becomes multi-region active-active |
| economics | unit cost exceeds the agreed margin envelope |
| organization | one team boundary becomes three independent release trains |
| risk | stored data changes classification |
| dependency | a vendor removes an exit path or changes its SLA |

These triggers belong in dashboards, quarterly risk review, or platform scorecards. A decision with no observable trigger silently becomes doctrine.

## Name an owner and the cost of ownership

Every material choice creates continuing work. Event streaming creates schema governance, consumer lag operations, replay policy, and retention costs. A service mesh creates certificate, upgrade, policy, and debugging work. An AI model creates evaluation, fallback, data-governance, and spend-control work.

Before approval, write the ownership ledger:

```text
decision: introduce durable workflow engine
owner: platform-runtime
on-call impact: new persistence and worker failure modes
controls: queue-age SLO, replay audit, version compatibility test
exit asset: workflow state export + activity interface
review trigger: <3 teams use it after two quarters
```

If no team accepts the operational burden, the organization has not made a decision. It has created an orphan.

## Separate reversible from irreversible decisions

Teams move slowly when every choice is escalated, and recklessly when none are. Classify decisions by blast radius and reversibility.

- A local library behind an interface is usually team-owned and reversible.
- A canonical customer identity model affects many systems and needs cross-team review.
- A new system of record, encryption boundary, or irreversible data migration needs executive risk ownership.

The approval mechanism should scale with the cost of being wrong. A CTO should define the decision protocol, not personally approve every framework.

## Treat migration as part of the decision

The selected destination is only half the architecture. The transition determines whether the company can reach it safely.

A credible decision includes coexistence and rollback:

```text
old path → dual read/write → measured comparison → cutover → observation → removal
```

Define who can stop the migration, which metrics decide progression, how data divergence is repaired, and how long rollback remains possible. If the plan only describes the final diagram, it has ignored the period of highest risk.

## Keep a decision portfolio

Individual ADRs are useful; the portfolio reveals concentration risk. Review decisions by domain:

- critical vendors and switching cost;
- shared data stores and blast radius;
- synchronous dependencies on customer paths;
- bespoke infrastructure with one maintainer;
- controls that fail open or fail closed;
- decisions whose review triggers have fired.

This is architecture governance without an architecture committee blocking every pull request. Teams retain autonomy inside explicit guardrails, while leadership can see where the system is accumulating irreversible risk.

## Conclusion

Good architecture records preserve the reasoning that code cannot. Great ones also say when that reasoning stops being valid.

Record the constraint, invariant, evidence, operating cost, owner, migration path, and review trigger. Then architecture becomes a living portfolio of accountable decisions—not a museum of diagrams.

## References

- [Architecture Decision Records](https://adr.github.io/)
- [Michael Nygard: Documenting Architecture Decisions](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions)
- [Martin Fowler: Feature Toggles](https://martinfowler.com/articles/feature-toggles.html)


---

# Autoscaling Cannot Recover an Expired Deadline

> Model HPA as a delayed feedback loop: metric lag, startup time, dependency capacity, stabilization, and the admission controls needed before new Pods arrive.

Canonical URL: https://www.ayushworks.xyz/posts/autoscaling-cannot-recover-expired-deadlines
Author: Ayush Basak
Last modified: 2026-09-04
Topics: kubernetes, capacity-planning, reliability, cloud-infrastructure

The dashboard shows desired replicas rising. Customers still see timeouts. Nothing is necessarily broken in the autoscaler: it cannot manufacture useful capacity instantly, and it cannot make an expired request successful retroactively.

A scaling policy is a delayed feedback loop. It observes yesterday's load, requests capacity, waits for infrastructure and application readiness, then observes the result.

> The service must survive the interval between detecting excess demand and making additional capacity useful.

That interval belongs in the capacity model, not just in deployment documentation.

## Write down the delay chain

```text
load rises
 -> metrics become available
 -> controller evaluates
 -> replicas are requested
 -> Pods are scheduled
 -> images and application initialize
 -> readiness passes
 -> traffic reaches new capacity
```

Each stage can lengthen under the same incident. A shared registry may be slow. Nodes may need provisioning. A Java process may warm up. A new replica may open a database pool while the database is already overloaded.

The [HPA documentation](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/) describes periodic control, metric-based desired replicas, handling of missing metrics, and stabilization. HPA changes desired scale; it does not guarantee that the requested Pods can be scheduled or become ready.

## Quantify the bridge capacity

For a simplified workload, let arrival rate exceed current service capacity by 200 requests per second for a 60-second capacity delay. The backlog grows by roughly 12,000 requests if nothing is rejected. These are illustrative values, not benchmark results.

If callers have two-second deadlines, much of that backlog is already worthless before new Pods arrive. Draining it can then consume the new capacity and extend the incident.

A queue therefore needs a deadline policy, not just a maximum length. Reject work that cannot plausibly finish. Cancel downstream execution when appropriate, and isolate longer-running background work from interactive requests.

## Choose a metric that represents recoverable demand

CPU utilization can be useful for CPU-bound work. It can be misleading for a service waiting on a saturated database: the application is busy in a customer sense while CPU remains low.

Queue age or outstanding work can better represent some workers. Request concurrency can better represent occupied execution slots. No metric removes the need to ask whether another replica actually increases total throughput.

When scaling on CPU utilization relative to requests, resource requests are part of the control model. Changing requests can change the utilization signal without changing customer traffic. Coordinate resource tuning with autoscaling policy.

## Use behavior controls intentionally

This illustrative HPA assumes a Deployment named api, working resource metrics, and suitable CPU requests on its containers:

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 4
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
```

The numbers are a test hypothesis. The minimum is warm capacity, the maximum is a ceiling, and the downscale behavior limits churn. None is a substitute for load testing.

Verify version-specific behavior against the cluster you operate. Avoid adopting newly documented feature gates merely because a current documentation page mentions them.

## Protect the dependencies

Twenty replicas with ten database connections each create a different database demand envelope from four replicas. Scale-out can also multiply TLS handshakes, cache misses, subscription rebalances, and background polling.

| Failure | Useful response | Misleading response |
| --- | --- | --- |
| CPU-bound service saturation | add ready compute | only lengthen client timeouts |
| database saturated | cap concurrency and optimize work | add unlimited API replicas |
| Pods pending | address placement or node capacity | raise maxReplicas again |
| metric unavailable | alert and retain safe headroom | assume zero demand |
| request already expired | discard or cancel safely | prioritize stale backlog |

Make dependency capacity an explicit upper bound. An autoscaler can move the bottleneck faster than an operator can recognize it.

## Test the full recovery curve

Run step-load and burst tests, not only gradual ramps. Record metric age, desired versus ready replicas, time to first useful response from a new Pod, queue age, deadline success, database saturation, and restart rate.

Test with cold images, a slow dependency, and temporarily unavailable metrics. Then remove load and observe downscale. A policy that responds quickly but oscillates under steady traffic is not stable.

## The CTO decision

Buy enough warm capacity to bridge measured startup delay, combine scaling with admission control, and cap expansion by downstream limits. Judge success by useful requests completed within their deadlines—not the speed at which a replica count rises.

## Further reading

- [Kubernetes: Pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)
- [Kubernetes probes are failure policy](/posts/kubernetes-probes-are-failure-policy)
- [Rate limiting is admission control](/posts/rate-limiting-is-admission-control)


---

# Backups Do Not Prove Recoverability

> A CTO-level recovery design for RPO, RTO, PostgreSQL PITR, dependency order, restore drills, and evidence that the business can survive data loss.

Canonical URL: https://www.ayushworks.xyz/posts/backups-do-not-prove-recoverability
Author: Ayush Basak
Last modified: 2026-08-25
Topics: database-engineering, reliability, disaster-recovery, technical-leadership

A green “backup completed” metric proves that bytes were written somewhere. It does not prove that the bytes are complete, decryptable, compatible with today’s application, or recoverable inside the business deadline.

The production invariant is stronger:

> After a declared class of failure, we can restore an internally consistent service to an approved point in time, within an observed duration, using people and systems available during the incident.

That is a recovery capability. A backup is one input.

## Start with loss, not tooling

Two numbers frame disaster recovery:

- **Recovery point objective (RPO):** the maximum acceptable data loss measured in time.
- **Recovery time objective (RTO):** the maximum acceptable time until the capability is usable again.

Do not assign one pair to an entire company. Authentication, checkout, analytics, and a documentation site rarely have equal business impact.

| Capability | RPO | RTO | Reason |
|---|---:|---:|---|
| payment ledger | near zero | 30 min | money and reconciliation |
| customer workspace | 5 min | 2 h | core product continuity |
| analytics | 24 h | 24 h | recomputable output |

These are business decisions expressed as engineering constraints. If leadership cannot explain the cost of losing four hours of data, engineering cannot rationally price a four-hour RPO.

## Model the whole recovery graph

Restoring PostgreSQL is not the same as restoring the product. A usable recovery may require:

```text
identity and secrets
        ↓
network and compute
        ↓
database + object storage + event log
        ↓
schema-compatible application
        ↓
workers, search indexes, caches
        ↓
DNS, traffic, reconciliation
```

Every edge is an ordering constraint. If the database is restored but encryption keys were deleted with the primary account, the backup is inert. If the data is restored to 10:05 but Kafka consumers replay effects from 09:40, the service can duplicate external actions.

Maintain a machine-readable recovery manifest with:

- source and target environment;
- backup identifier and checksum;
- application and schema version;
- encryption-key dependency;
- recovery target time;
- replay boundaries for brokers and workers;
- validation queries;
- traffic-switch owner.

The manifest turns tribal knowledge into an executable contract.

## PostgreSQL PITR needs an unbroken history

PostgreSQL point-in-time recovery combines a base backup with archived write-ahead log (WAL). PostgreSQL’s documentation is explicit: recovery depends on the required continuous sequence of WAL files. A valid base backup plus a missing WAL segment cannot satisfy the intended recovery point.

Monitor the recovery chain, not just the latest object:

```sql
SELECT
  now() - last_archived_time AS archive_age,
  archived_count,
  failed_count,
  last_failed_wal
FROM pg_stat_archiver;
```

Alert on archive age relative to RPO. Validate object checksums and retention. Keep recovery credentials separate from the failure domain of the production control plane.

A replica is not a backup. Replication quickly copies valid writes, accidental deletes, and some forms of corruption. It improves availability; it does not create historical recovery points by itself.

## Restore into isolation first

Never make the first restore attempt directly into the production destination. Restore into a quarantined environment where automation cannot emit email, charge cards, call webhooks, or consume live queues.

Validation should test business invariants, not merely database startup:

```sql
-- Ledger must balance per currency.
SELECT currency, sum(debit_cents) - sum(credit_cents) AS imbalance
FROM ledger_entries
GROUP BY currency
HAVING sum(debit_cents) <> sum(credit_cents);

-- No paid order may lack a payment reference.
SELECT count(*)
FROM orders
WHERE status = 'paid' AND payment_reference IS NULL;
```

Also compare row counts and age distributions, run application smoke tests, verify schema compatibility, and sample recent high-value entities. “Postgres accepts connections” is necessary and radically insufficient.

## Measure actual recovery time

An RTO in a document is an aspiration. A timed restore is evidence.

Break the exercise into stages:

```text
detection → decision → environment → data restore
→ validation → reconciliation → traffic → stable operation
```

Record p50 and worst observed duration for each stage. This exposes the real bottleneck. Teams often buy faster storage while approval, credentials, DNS, or validation consumes most of the outage.

Run at least three kinds of drills:

1. **Routine restore:** automated restore into an isolated environment.
2. **Scenario exercise:** region loss, credential compromise, or operator deletion.
3. **Unannounced execution:** a bounded exercise proving the runbook works without its author.

If the same person writes, operates, and validates the procedure, key-person risk remains untested.

## Reconcile the gap

Recovery produces a point in time, not necessarily a final truth. External systems may have accepted effects after the restored point: payments settled, emails sent, devices acted, or partners recorded webhooks.

For every external effect, define:

- the system of record;
- a stable idempotency key;
- how to query the external outcome;
- who resolves ambiguity;
- how evidence is retained.

The last mile of recovery is convergence. Without reconciliation, a technically successful restore can create a financially incorrect product.

## CTO review

Ask for evidence, not reassurance:

1. Which business capability does each RPO and RTO protect?
2. When was the last full restore, and what duration was observed?
3. Can recovery proceed if the primary cloud account is unavailable?
4. Which dependencies must be restored in which order?
5. What prevents restored workers from repeating external effects?
6. Which invariant queries decide that traffic may return?
7. Who has successfully executed the runbook besides its author?

Backups are inventory. Recoverability is a repeatedly demonstrated operating capability.

## References

- [PostgreSQL: Continuous Archiving and Point-in-Time Recovery](https://www.postgresql.org/docs/current/continuous-archiving.html)
- [AWS: Disaster recovery options in the cloud](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html)
- [Google SRE: Managing Incidents](https://sre.google/sre-book/managing-incidents/)
- [Related: PostgreSQL Zero-Downtime Schema Migrations](/posts/postgresql-zero-downtime-schema-migrations)
- [Related: CDC Is a Recovery Contract](/posts/cdc-recovery-contract)


---

# Bloom Filters Turn Memory into an Error Budget

> Size and operate Bloom filters from allowed false positives, expected cardinality, and downstream cost.

Canonical URL: https://www.ayushworks.xyz/posts/bloom-filters-turn-memory-into-error-budget
Author: Ayush Basak
Last modified: 2026-09-05
Topics: algorithms, data-structures, databases, performance

A Bloom filter does not answer “is this item present?” It answers a narrower question:

> The item is definitely absent, or it may be present.

That asymmetry makes it valuable in front of expensive negative lookups: storage engines avoiding disk reads, APIs suppressing checks for unknown IDs, or crawlers skipping URLs already seen. It also makes a Bloom filter dangerous when teams treat “maybe” as truth.

## The product decision comes first

Define what a false positive costs. If a filter says “maybe present” for an absent item, the application performs the expensive fallback lookup. Correctness is preserved; efficiency is lost.

If the application instead interprets “maybe present” as “definitely present,” the data structure is being used outside its contract.

Useful input variables are:

- expected inserted items, `n`;
- bit budget, `m`;
- number of hash positions, `k`;
- acceptable false-positive probability, `p`;
- cost of the fallback lookup;
- growth and rebuild strategy.

For a conventional Bloom filter, common sizing approximations are:

```text
m = -n * ln(p) / (ln(2)^2)
k = (m / n) * ln(2)
```

For ten million items at a one-percent target, the bit array is roughly 11.4 MiB and the optimal hash count is about seven. The exact operational number also includes metadata, allocator overhead, replicas, and growth margin.

## Put it on the correct side of the lookup

```ts
async function getObject(id: string) {
  if (!filter.mightContain(id))
    return null // definitive negative

  return objectStore.get(id) // resolve the possible positive
}
```

The filter is an optimization. The authoritative store remains the source of truth.

This distinction shapes availability. If the filter is unavailable, the service should usually bypass it and query the source—not reject valid requests. If a new filter is warming, false negatives must not leak into decisions; build from a snapshot plus changes, or keep the old filter active until the replacement is complete.

## Capacity is not a suggestion

As inserted cardinality exceeds the planned value, more bits become set and false positives increase. A growing implementation may add sub-filters, but reads then check multiple structures and latency grows. Monitor actual inserted cardinality and sampled false-positive rate rather than assuming the configured target still holds.

Deletion is another boundary. Clearing a bit in a conventional Bloom filter can create false negatives for other keys that share it. Counting Bloom filters support deletion with counters, trading more memory and overflow considerations for that capability.

## Production review

| Question | Why it matters |
| --- | --- |
| What does a false positive trigger? | converts `p` into latency and cost |
| Can false negatives ever occur? | they usually indicate lifecycle or implementation bugs |
| What is the cardinality forecast? | determines memory and rebuild timing |
| Is the source queried on “maybe”? | preserves correctness |
| How is the filter rebuilt? | avoids gaps during deploys |
| Is it per tenant or shared? | controls isolation and noisy-neighbor risk |

## Trade-offs

More memory lowers false-positive probability; more hash functions increase CPU work; layered growth avoids a stop-the-world rebuild but makes lookups more expensive. A cache stores answers and can return values. A Bloom filter stores membership evidence and should only remove unnecessary work.

Treat its false-positive rate as an error budget with a downstream price. Then the data structure becomes an engineering decision instead of an interview trick.

## Further reading

- [Redis Bloom filter sizing and behavior](https://redis.io/docs/latest/develop/data-types/probabilistic/bloom-filter/)
- [Network Applications of Bloom Filters](https://www.eecs.harvard.edu/~michaelm/postscripts/im2005b.pdf)
- [Cache invalidation is a consistency protocol](/posts/cache-invalidation-is-consistency-protocol)


---

# BM25 From First Principles: The Math, the Index, and Production Ranking

> A rigorous guide to BM25 search ranking: derive IDF, term-frequency saturation, document-length normalization, calculate a score by hand, implement it, tune it, and place it inside a production retrieval stack.

Canonical URL: https://www.ayushworks.xyz/posts/bm25-search-ranking-from-first-principles
Author: Ayush Basak
Last modified: 2026-09-06
Topics: search-engineering, bm25, information-retrieval, databases, ranking, system-design

Search begins with a deceptively small question: given a query and millions of documents, which documents deserve to appear first?

BM25 remains one of the strongest answers when relevance depends on exact words. It is fast, explainable, requires no labeled training data, and works naturally with an inverted index. Lucene and Elasticsearch use BM25 as their default text similarity; modern retrieval systems frequently keep it beside embeddings because lexical and semantic retrieval fail in different ways.

The important idea is not that BM25 “counts words.” It decides how much evidence a word match contributes. A match is stronger when the term is rare across the collection, when it appears repeatedly in the document—but with diminishing returns—and when its frequency is meaningful relative to the document’s length.

The production invariant is:

> A relevance score is meaningful only inside the collection, analyzer, field, and query structure that produced it. A BM25 score is not a probability and is not safely comparable across unrelated indexes or queries.

## BM25 in one minute

BM25 ranks a document by combining three signals: how rare each query term is across the corpus, how often that term occurs in the document with diminishing returns, and whether that frequency is surprising for a document of that length. The parameters `k₁` and `b` control frequency saturation and length normalization. In production, BM25 is usually executed over an inverted index and evaluated with judged queries—not tuned from intuition alone.

Use BM25 when exact lexical evidence matters: names, identifiers, error codes, product SKUs, API symbols, and specialist terminology. Combine it with dense retrieval when users express the same intent using different words. Fuse independently ranked result lists with a method such as Reciprocal Rank Fusion instead of treating incomparable raw scores as probabilities.

After reading this guide, you should be able to calculate a BM25 score by hand, implement a small ranker, inspect Lucene or Elasticsearch scoring, tune relevance using judgments, and decide when hybrid retrieval earns its operational cost.

![BM25 combines term rarity, saturating frequency, and document-length normalization](/images/bm25-scoring-map.svg)

## The retrieval problem BM25 solves

Let the corpus contain `N` documents. A query `Q` contains terms `q₁, q₂, …, qₘ`. For each candidate document `D`, the ranker needs a score that increases when the document contains useful query evidence.

A Boolean ranker can determine whether every required term exists, but it cannot express that one matching document is better than another. Raw term frequency helps, but creates a bad incentive: repeating a term 100 times becomes 100 times as valuable as writing it once. Plain TF–IDF recognizes rare terms, yet its frequency and length behavior is often too crude.

BM25 is a bag-of-words lexical ranking function. It does not understand word order, entailment, or semantic equivalence by itself. Its job is narrower: rank documents using corpus statistics and token overlap. That narrowness is a feature. It gives us predictable evidence that an embedding model may miss—identifiers, error codes, names, product SKUs, API symbols, and uncommon technical phrases.

## The scoring equation

A common BM25 form is:

```text
                 f(qᵢ,D) · (k₁ + 1)
score(D,Q) = Σ IDF(qᵢ) · ─────────────────────────────────
 qᵢ ∈ Q          f(qᵢ,D) + k₁(1 − b + b·|D|/avgdl)
```

where:

- `f(qᵢ,D)` is the frequency of query term `qᵢ` in document `D`;
- `|D|` is the analyzed length of the document field;
- `avgdl` is the average analyzed field length in the collection;
- `k₁ ≥ 0` controls term-frequency saturation;
- `0 ≤ b ≤ 1` controls document-length normalization;
- `IDF(qᵢ)` measures how rare the term is across documents.

The formula is a sum because each query term contributes independent evidence. BM25’s probabilistic origins are richer than this operational description; Robertson and Zaragoza’s review connects the practical ranking function to the probabilistic relevance framework and documents the assumptions behind it.

Implementations differ. Lucene computes inverse document frequency as:

```text
                     N − n(qᵢ) + 0.5
IDF(qᵢ) = ln(1 + ───────────────────)
                       n(qᵢ) + 0.5
```

Here `n(qᵢ)` is document frequency: the number of documents containing the term at least once. The added 1 keeps the value positive. Other descriptions of BM25 may show a slightly different IDF, an extra query-frequency term, or a constant factor. When reproducing scores, use the exact formula and statistics of the engine you operate.

## Part one: IDF measures discriminative evidence

Suppose a corpus has one million documents:

- `database` appears in 10,000 documents;
- `the` appears in 900,000 documents.

Using Lucene’s IDF:

```text
IDF(database) = ln(1 + (1,000,000 − 10,000 + 0.5) / (10,000 + 0.5))
              ≈ 4.605

IDF(the)      = ln(1 + (1,000,000 − 900,000 + 0.5) / (900,000 + 0.5))
              ≈ 0.105
```

One match on `database` carries roughly 44 times the IDF weight of one match on `the`. This is why analysis matters before ranking. A stop-word filter might remove `the`; a domain-specific analyzer might preserve `C`, `R`, or `Go`, even though generic tokenization can damage them.

IDF is collection-relative. Add a large body of database documentation and `database` becomes less rare. Delete documents or move a tenant to another shard and statistics can shift. The score did not change because the document changed; it changed because the evidence landscape changed.

## Part two: term frequency saturates

More occurrences should help, but the tenth repetition is not as informative as the second. BM25 encodes diminishing returns through the fraction containing `f(qᵢ,D)`.

Ignore length for a moment by setting `b=0`. With `k₁=1.2`, the term-frequency factor is:

```text
TF saturation = f(k₁ + 1) / (f + k₁)
```

| Term frequency (f) | Saturated contribution before IDF |
| ---: | ---: |
| 1 | 1.000 |
| 2 | 1.375 |
| 4 | 1.692 |
| 8 | 1.913 |
| 32 | 2.120 |
| `∞` | `k₁ + 1 = 2.2` |

The asymptote is `k₁ + 1`. Repetition can strengthen evidence, but cannot grow without bound.

`k1` determines the shape:

- `k₁=0` ignores term frequency; a present term contributes its IDF;
- a lower `k₁` saturates earlier;
- a higher `k₁` lets repeated occurrences matter for longer.

This is a product decision disguised as a numeric parameter. Search over short product titles usually needs repetition less than search over long legal or technical bodies. Do not tune `k1` because a blog post recommends a value. Tune it against judgments representing your users’ information needs.

## Part three: length normalization asks whether frequency is surprising

Four occurrences in a 60-token support article may be concentrated evidence. Four occurrences in a 4,000-token manual may be incidental. BM25 compares the document length with the collection average:

```text
length normalization = 1 − b + b·(|D| / avgdl)
```

At `b=0`, length has no effect. At `b=1`, normalization fully follows the document-to-average length ratio. The common default `b=0.75` applies strong, but not complete, normalization.

Length means analyzed token count for the field—not bytes, source characters, visual height, or `_source` size. Synonyms, stemming, stop-word removal, overlap tokens, and field boundaries therefore change the numbers BM25 sees. Lucene’s default `discountOverlaps=true` excludes tokens with zero position increment from length, which commonly affects synonym expansion.

Length normalization can punish genuinely comprehensive documents. That is especially visible when a field mixes titles, summaries, comments, and bodies. The better fix is often field modeling rather than changing `b`: keep semantically distinct text in separate fields and apply deliberate boosts.

## Calculate two documents by hand

Consider the query `database` and these corpus statistics:

```text
N       = 1,000,000 documents
df      = 10,000 documents
avgdl   = 120 analyzed tokens
k1      = 1.2
b       = 0.75
IDF     ≈ 4.605
```

Document A contains the term four times and has length 180:

```text
K_A      = 1.2 · (1 − 0.75 + 0.75 · 180/120) = 1.65
TFNorm_A = 4 · (1.2 + 1) / (4 + 1.65)
         = 8.8 / 5.65 ≈ 1.558
score_A  ≈ 4.605 · 1.558 = 7.175
```

Document B contains the term twice and has length 60:

```text
K_B      = 1.2 · (1 − 0.75 + 0.75 · 60/120) = 0.75
TFNorm_B = 2 · 2.2 / (2 + 0.75) = 1.600
score_B  ≈ 4.605 · 1.600 = 7.368
```

Document B wins despite containing fewer occurrences. Relative to its length, those two matches are stronger evidence. This example also shows why debugging only raw term counts produces wrong conclusions.

## A small implementation you can inspect

This TypeScript implementation follows the formula above. It is suitable for learning and tests, not for scanning a production corpus; a real engine uses an inverted index to visit only documents containing query terms.

```ts
interface CorpusStats {
  documentCount: number
  averageDocumentLength: number
  documentFrequency: Map<string, number>
}

interface DocumentStats {
  length: number
  termFrequency: Map<string, number>
}

function luceneIdf(documentCount: number, documentFrequency: number) {
  return Math.log(
    1 + (documentCount - documentFrequency + 0.5)
    / (documentFrequency + 0.5),
  )
}

export function bm25(
  queryTerms: string[],
  document: DocumentStats,
  corpus: CorpusStats,
  k1 = 1.2,
  b = 0.75,
) {
  if (k1 < 0 || b < 0 || b > 1)
    throw new RangeError('BM25 requires k1 >= 0 and 0 <= b <= 1')

  const lengthRatio = document.length / corpus.averageDocumentLength
  const normalization = k1 * (1 - b + b * lengthRatio)

  return [...new Set(queryTerms)].reduce((score, term) => {
    const tf = document.termFrequency.get(term) ?? 0
    if (tf === 0)
      return score

    const df = corpus.documentFrequency.get(term) ?? 0
    const idf = luceneIdf(corpus.documentCount, df)
    const saturatedTf = tf * (k1 + 1) / (tf + normalization)
    return score + idf * saturatedTf
  }, 0)
}
```

The `Set` is a modeling choice: this version ignores repeated query terms. Some BM25 variants model query-term frequency; query parsers may also generate repeated clauses or boosts. Again, engine behavior is the contract.

## The inverted index makes BM25 operationally cheap

Computing every query against every document would be `O(N)` scoring work. An inverted index changes the access path:

```text
term: database
postings:
  (doc 17, tf 4)
  (doc 92, tf 2)
  (doc 403, tf 1)
```

For each query term, the engine reads its postings list. The index also stores or derives document frequency, per-document field-length norms, and collection statistics. It merges candidate streams, computes scores, and maintains the top (k) results. Advanced engines use skipping and dynamic pruning to avoid fully evaluating candidates that cannot enter the current top (k).

The architecture matters:

1. The analyzer transforms source text into tokens at index time.
2. The query analyzer transforms user text into compatible terms.
3. The term dictionary locates postings.
4. Postings provide document IDs and frequencies.
5. norms provide compact field-length information.
6. BM25 scores candidates.
7. collectors retain the best results.

Most “BM25 problems” are not formula problems. They are analyzer, field, query construction, or corpus problems.

## Configure and inspect BM25 in Elasticsearch

Elasticsearch and Lucene default to `k₁=1.2` and `b=0.75`. A custom similarity is configured when the index is created and assigned at field mapping time:

```json
PUT articles-v1
{
  "settings": {
    "index": {
      "similarity": {
        "technical_bm25": {
          "type": "BM25",
          "k1": 1.1,
          "b": 0.6,
          "discount_overlaps": true
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": { "type": "text", "similarity": "technical_bm25" },
      "body":  { "type": "text", "similarity": "technical_bm25" }
    }
  }
}
```

Then inspect a surprising result with `_explain`:

```json
GET articles-v1/_explain/article-42
{
  "query": {
    "multi_match": {
      "query": "postgresql connection pool",
      "fields": ["title^3", "body"]
    }
  }
}
```

The explanation tree reveals boosts, IDF, term frequency, field length, and normalization factors. Store these explanations for a small evaluation set during relevance work. They turn “search feels worse” into an inspectable difference.

Changing similarity settings on an existing index requires care because scoring depends on index-time norms and collection statistics. Treat a relevance change like a schema migration: create a versioned index, reindex, replay evaluation queries, compare metrics and critical examples, then move an alias.

## Fields are separate evidence channels

A title match and a body match do not mean the same thing. A practical query often combines fields:

```json
{
  "multi_match": {
    "query": "retry budget",
    "type": "best_fields",
    "fields": ["title^4", "summary^2", "body", "tags^3"],
    "tie_breaker": 0.2
  }
}
```

Field boosts are part of the ranking model. They should reflect product semantics, not compensate blindly for bad results. If tags are editorially controlled, a tag match may be reliable. If users can stuff tags, a large boost creates an abuse channel.

BM25F is a formal multi-field extension. Lucene also exposes combined-field behavior for scoring term statistics across fields. Whichever mechanism you choose, document the unit of evidence: a match in `title`, `body`, `author`, and `tenant_private_notes` must not be treated interchangeably.

## Shards can change the statistics

Distributed search introduces a subtlety. IDF and average field length require collection statistics. If each shard computes them from its local subset, scores can vary with shard composition, especially for small or skewed indexes.

Elasticsearch can use a distributed frequency phase (`dfs_query_then_fetch`) to gather more global statistics before scoring, at additional latency and coordination cost. This is not automatically the right default. Large, evenly distributed corpora usually have similar shard statistics; tenant-per-shard or small collections may not.

The operational test is simple: move the same documents across different shard layouts and compare the top results for critical queries. If rankings move materially, either gather global statistics, improve routing, reduce shard skew, or accept the behavior explicitly.

## Tuning BM25 without fooling yourself

Do not optimize `k1` and `b` against a handful of favorite queries. Build a judgment set:

- sample real information needs, not merely raw query strings;
- include navigational, exact-identifier, broad topical, and long-tail queries;
- label graded relevance where possible;
- split tuning and evaluation queries;
- preserve critical “must win” and “must not appear” cases;
- record analyzer, mapping, corpus snapshot, and engine version.

Measure ranking, not clicks alone. Common offline metrics include:

- **Precision@k:** how many of the first (k) results are relevant;
- **Recall@k:** how much known relevant material appears in the first (k);
- **MRR:** how early the first relevant result appears;
- **nDCG@k:** rewards placing highly relevant documents near the top;
- **MAP:** averages precision at relevant positions across queries.

TREC’s Deep Learning track uses judged collections and metrics such as nDCG to compare lexical and learned ranking systems. Pyserini publishes reproducible BM25 workflows over standard corpora, which is useful for learning what a defensible experiment looks like.

Online evaluation adds behavior but also bias. Position affects clicks. Zero-result queries may disappear from click-based datasets. A ranking that increases clicks may still reduce task completion. Use interleaving or controlled experiments, and pair behavioral metrics with guardrails such as reformulation rate, abandonment, latency, and downstream success.

## Common mistakes

### Treating the score as confidence

A score of 12 is not “twice as relevant” as 6 and does not mean 12% or 12 units of confidence. It is an ordering signal generated by one query against one index state.

### Ignoring the analyzer

BM25 scores tokens, not source strings. Lowercasing, stemming, synonyms, n-grams, decompounding, and stop words determine which evidence exists. Always inspect analyzed tokens for failed queries.

### Mixing fields with radically different length distributions

Concatenating title, body, comments, and metadata creates a length signal with unclear meaning. Separate fields preserve interpretable evidence and tunable boosts.

### Tuning parameters before fixing retrieval

If the relevant document is absent from the candidate set because the analyzer removed an identifier or the wrong field was queried, no `k1` value will recover it.

### Comparing scores across queries

Different terms have different IDF values and candidate populations. Use rank positions or query-normalized features when combining downstream signals.

### Forgetting access control

Filtering unauthorized documents after top-(k) retrieval can return too few results and leak distributional information. Apply tenant and authorization filters inside candidate retrieval.

## BM25 and dense retrieval are complements

Dense embeddings can retrieve semantic matches with little token overlap. BM25 can retrieve precise lexical evidence without model inference. A strong production design frequently uses both:

```text
query
  ├── analyzer ──► BM25 top 200
  └── encoder  ──► vector top 200
                       │
        reciprocal-rank fusion
                       │
             reranker top 50
                       │
                 return top 10
```

Reciprocal rank fusion avoids comparing incompatible raw score scales:

```text
RRF(d) = Σ 1 / (k + rankᵣ(d))
         r ∈ retrievers
```

For retrieval-augmented generation, BM25 is especially useful for exact symbols, policy clauses, versions, and error messages. Dense retrieval helps when the user describes a concept using different words. The reranker can then spend expensive computation on a bounded candidate set.

Hybrid retrieval is not automatically better. It adds index cost, latency, failure modes, fusion parameters, and evaluation work. Start with the failure classes visible in your query set. Add a second retriever when it repairs a measured weakness.

## Production review checklist

Before shipping BM25-backed search, answer these questions:

1. What unit is ranked: document, passage, product, comment, or field?
2. Which analyzer produces index and query terms?
3. Which terms, identifiers, and languages must survive analysis?
4. What are the field-length distributions and sources of skew?
5. Are `k1`, `b`, boosts, and tie-breaking policies versioned?
6. Can an operator explain a result using term and field evidence?
7. Are tenant and authorization constraints applied before ranking?
8. Does the judgment set represent real user tasks and long-tail failures?
9. How do shard statistics affect small or skewed collections?
10. What is the rollback path for an analyzer or relevance change?
11. Which exact-match failures justify lexical retrieval beside vectors?
12. Which latency and cost budget constrains candidate count and reranking?

## The CTO-level conclusion

BM25 is valuable because it turns three defensible intuitions into a cheap ranking function:

- rare terms carry more information;
- repetition helps with diminishing returns;
- frequency must be interpreted relative to document length.

But the formula is only one layer of the system. Relevance is produced by corpus boundaries, analyzers, field modeling, query construction, shard statistics, access filters, candidate generation, and evaluation discipline. A team that changes `k1` without understanding those layers is tuning a symptom.

Start by making lexical evidence observable. Inspect tokens. Calculate a score by hand. Build a small judgment set. Version every relevance change. Then decide whether BM25 alone is sufficient, whether field-aware ranking is needed, or whether semantic retrieval repairs a real failure class.

That is the durable lesson: ranking quality does not come from a fashionable model name. It comes from making evidence, constraints, and evaluation explicit.

## References

- Stephen Robertson and Hugo Zaragoza, [The Probabilistic Relevance Framework: BM25 and Beyond](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf).
- Apache Lucene, [`BM25Similarity` API](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/similarities/BM25Similarity.html).
- Elastic, [Similarity settings and BM25 parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html).
- Pyserini, [reconstructing and scoring BM25 vectors](https://github.com/castorini/pyserini/blob/master/docs/conceptual-framework2.md).
- NIST TREC, [Deep Learning Track overview](https://trec.nist.gov/pubs/trec28/papers/OVERVIEW.DL.pdf).

Related on this site: [filtered vector search is a recall budget](/posts/filtered-vector-search-is-a-recall-budget), [RAG quality starts with retrieval evidence](/posts/rag-quality-starts-with-retrieval-evidence), and [cache invalidation is a consistency protocol](/posts/cache-invalidation-is-consistency-protocol).


---

# Build vs Buy Is an Exit-Cost Decision

> A practical CTO framework for evaluating software vendors through differentiation, operating cost, failure ownership, data portability, and credible exit paths.

Canonical URL: https://www.ayushworks.xyz/posts/build-vs-buy-is-an-exit-cost-decision
Author: Ayush Basak
Last modified: 2026-08-29
Topics: technical-leadership, platform-engineering, cloud-economics, vendor-risk

“Build or buy?” is usually presented as a feature and price comparison. That framing misses the hardest part: whichever path you choose, the company still owns the customer outcome.

A managed database can remove patching work; it does not remove responsibility for recovery objectives, schema design, connection pressure, or a regional dependency. An AI API can remove model serving; it does not remove evaluation, privacy, cost, or fallback policy.

The governing invariant is:

> Outsourcing execution never outsources accountability.

## Start with differentiation

Build when the capability encodes a product advantage or a control you cannot delegate. Buy when the capability is necessary but does not distinguish the product—and a vendor can operate it materially better.

That principle is necessary, but not sufficient. Identity, billing, observability, workflow, search, and AI inference often begin as commodities and later become deeply embedded. The evaluation must include the future cost of changing direction.

## Compare total ownership, not subscription price

Use the same cost model for both options:

```text
annual cost = license or infrastructure
            + implementation
            + integration maintenance
            + on-call and incident cost
            + security and compliance work
            + migration and exit reserve
            + opportunity cost
```

Internal software is not free because engineers are already employed. Vendor software is not cheap because the first invoice is small. Model the cost per useful business unit—per active tenant, million events, successful workflow, or evaluated model response—at current and plausible future scale.

## Make failure ownership explicit

Ask what happens at 02:00 when the dependency is slow, inconsistent, or unavailable.

| Question | Required answer |
|---|---|
| Who detects failure first? | named monitor and owner |
| Can the product degrade? | explicit stale, queue, or fail-closed policy |
| Can support inspect state? | audit and diagnostic interface |
| Is recovery tested? | evidence, not an SLA link |
| What is the vendor escalation path? | severity and response contract |
| Can we reconcile ambiguous outcomes? | stable operation identity |

A vendor SLA is an input to your reliability model, not your product guarantee. If a 99.9% dependency is synchronously required by every request, your service cannot honestly promise more without redundancy or degradation.

## Price the coupling

Coupling is not only API syntax. It includes:

- proprietary data models and workflow semantics;
- identity and authorization embedded in the vendor;
- historical data that cannot be exported completely;
- operational knowledge held only in vendor dashboards;
- features whose behavior cannot be reproduced elsewhere;
- egress, contract, and migration timing constraints.

The most dangerous lock-in is semantic. Replacing an SDK is easy; recreating five years of authorization decisions or workflow history is not.

Keep your durable domain identity outside the vendor. Store the mapping from internal IDs to vendor IDs. Capture the minimum event history needed to reconcile. Wrap only the semantics for which you have a plausible second implementation; do not build an imaginary universal abstraction.

## Design a credible exit

An exit plan is an architectural asset. It should specify:

1. the data export format and tested retrieval time;
2. the maximum acceptable dual-running period;
3. the application boundary where traffic can be switched;
4. reconciliation between old and new systems;
5. contractual notice, deletion, and audit requirements;
6. the condition that would justify paying the migration cost.

Not every dependency needs hot portability. A payroll vendor may justify a planned manual migration. A provider on the payment authorization path may require live routing and idempotent failover. Spend portability effort in proportion to business exposure.

## Use a decision scorecard, not a magic score

Scorecards improve questions but should not hide judgment. Review at least:

- strategic differentiation;
- time to validated customer value;
- five-year unit economics;
- reliability and degradation control;
- security and data classification;
- integration depth;
- operational capability of your team;
- exit cost and time.

Write the assumptions next to every number. A weighted total without assumptions produces false precision.

## Conclusion

Build versus buy is not an identity test for an engineering organization. Strong teams buy aggressively where ownership adds no advantage and build deliberately where control creates value.

The mature question is: which option gives us the best customer outcome, at an acceptable operating cost, with a failure model and exit path we can actually own?

## References

- [Google SRE: Embracing Risk](https://sre.google/sre-book/embracing-risk/)
- [FinOps Foundation: Unit Economics](https://www.finops.org/framework/capabilities/unit-economics/)
- [NIST Cybersecurity Framework](https://www.nist.gov/cyberframework)


---

# Building Microservices That Fail Gracefully

> Practical patterns for resilient service-to-service communication.

Canonical URL: https://www.ayushworks.xyz/posts/building-reliable-microservices
Author: Ayush Basak
Last modified: 2026-07-28
Topics: backend-engineering, distributed-systems, reliability, system-design

Microservices are easy to draw and difficult to operate. The interesting work begins when a dependency is slow, a message arrives twice, or a deployment leaves two API versions running at once.

## Design around failure

A remote call is never equivalent to a local function call. Give every request a deadline, retry only operations that are safe to repeat, and use exponential backoff with jitter. A retry without a budget can turn one unhealthy service into a system-wide incident.

## Make events idempotent

Event-driven systems should assume duplicate delivery. Attach a stable event identifier and store processed identifiers beside the business transaction. Consumers can then acknowledge repeats without repeating the effect.

## Prefer observable boundaries

Structured logs, correlation IDs, latency histograms, and queue-depth metrics make service boundaries visible. Observability is not polish added after launch; it is part of the interface.

The goal is not to prevent every failure. It is to keep failures contained, explainable, and recoverable.


---

# Cache Invalidation Is a Consistency Protocol

> A production guide to cache-aside races, versioned keys, bounded staleness, invalidation delivery, stampede control, negative caching, and evidence of correctness.

Canonical URL: https://www.ayushworks.xyz/posts/cache-invalidation-is-consistency-protocol
Author: Ayush Basak
Last modified: 2026-08-26
Topics: caching, distributed-systems, database-engineering, reliability

A cache is a replica with a weaker update protocol. Once that is acknowledged, cache design stops being “pick a TTL” and becomes a consistency decision.

For every cached value, define the maximum acceptable staleness, the authority, the invalidation path, and what happens when that path fails.

## Name the guarantee

Different data deserves different contracts:

| Data | Acceptable behavior |
|---|---|
| article page | stale for minutes |
| product price | stale briefly, revalidated at checkout |
| authorization | short bounded staleness or fail closed |
| account balance | primary read for financial decision |

A TTL is not a universal consistency model. It is an upper bound only if clocks, refresh behavior, and fallback paths obey it.

State the contract as an invariant:

> A revoked permission is no longer accepted within 30 seconds, and every destructive operation rechecks the authoritative policy version.

That statement can be tested. “We use Redis” cannot.

## Follow the cache-aside race

The common read path is:

```text
GET cache → miss → SELECT database → SET cache
```

The common write path updates the database and deletes the cache entry. Redis documentation describes invalidation after the primary write as the simple cache-aside approach. But concurrency still matters:

```text
reader: cache miss
reader: reads database value v1
writer: commits v2
writer: deletes cache
reader: writes stale v1 into cache
```

The stale value survives until expiry.

Mitigations include:

- short TTL where bounded staleness is acceptable;
- versioned values and compare-before-set;
- delete after write, then a delayed second delete for known race windows;
- change-stream invalidation;
- write-through under one owner;
- bypassing cache for correctness-critical reads.

No technique removes every failure mode. Choose against the actual invariant.

## Put versions in the value

Store source version with cached data:

```json
{
  "entity_version": 481,
  "cached_at": "2026-08-26T08:20:00Z",
  "value": {"plan":"enterprise"}
}
```

An invalidation or refresh carrying version 480 cannot overwrite 481. Versioning also makes stale observations diagnosable.

For immutable or expensive derived objects, put the version in the key:

```text
report:customer-42:revision-481
```

Publish the active revision separately. Old values expire without in-place mutation, reducing races during recomputation.

## Treat invalidations as at-least-once messages

Invalidations can be delayed, duplicated, reordered, or lost during disconnects. Handlers should be idempotent; deleting an already absent key is success.

Redis client-side caching tracks keys read by clients and sends invalidation messages when those keys change. It can efficiently reduce database reads, but applications must evict entries on notification and handle reconnects. After a gap, assume local state may be stale and flush or revalidate it.

For durable correctness requirements, ephemeral pub/sub is not sufficient evidence. Use a durable change log or version check so a disconnected consumer can catch up.

Measure:

- source commit to invalidation latency;
- invalidation consumer lag;
- reconnect flushes;
- cached versus authoritative version gaps;
- stale reads detected at the write boundary.

## Prevent stampedes

When a popular key expires, thousands of requests may miss together and overload the source.

Use request coalescing:

```text
first miss → becomes loader
other misses → wait on same in-flight result
loader writes cache → wake waiters
```

Add TTL jitter so many keys do not expire simultaneously. Consider stale-while-revalidate for non-critical reads: serve a recently expired value while one worker refreshes it.

Bound the loader. A distributed lock with no expiry can make a cache miss permanent; a lock with no fencing can let an expired owner overwrite a newer result. Version checks remain necessary.

## Negative caching needs shorter rules

Caching “not found” protects the database from repeated misses and enumeration. It can also hide a newly created resource.

Use a distinct short TTL, include tenant and authorization scope in the key, and invalidate on creation. Never reuse an unauthenticated negative result for an authenticated subject if visibility differs.

Cache keys are part of the security model:

```text
bad:  profile:user-7
good: profile:tenant-42:user-7:policy-19
```

The correct key encodes every dimension that can change the response.

## Design degradation

A cache outage should not automatically redirect its full peak load to the database. That converts one failure into two.

Apply admission control, per-key coalescing, bounded concurrency, and selective degradation. Serve stale public content where safe. Reject expensive optional views. Preserve capacity for checkout or writes.

Test cache loss at realistic peak traffic. A successful steady-state benchmark says little about miss-storm behavior.

## CTO review

1. What staleness is allowed for each data class?
2. Which source is authoritative at the decision boundary?
3. Can a slow reader repopulate an older value after a write?
4. Are invalidations replayable after disconnect?
5. Does every key include tenancy, authorization, and representation dimensions?
6. How are stampedes coalesced and bounded?
7. What happens to the database when the cache disappears?
8. Which metric proves the cache meets its consistency contract?

A cache earns its latency improvement by accepting a consistency cost. Production design makes that cost explicit, bounded, and observable.

## References

- [Redis: Client-side caching introduction](https://redis.io/docs/latest/develop/clients/client-side-caching/)
- [Redis: Cache-aside](https://redis.io/docs/latest/develop/use-cases/cache-aside/)
- [Redis: Client-side caching reference](https://redis.io/docs/latest/develop/reference/client-side-caching/)
- [Related: Delivery Semantics Shape the Data Model](/engineering-notes#delivery-semantics)
- [Related: Transactional Outbox Delivery Guarantees](/posts/transactional-outbox-delivery-guarantees)


---

# Change Data Capture Is a Recovery Contract

> A CTO-level guide to CDC snapshots, offsets, schema evolution, ordering, replay, lag, and proving downstream systems can recover without corrupting business state.

Canonical URL: https://www.ayushworks.xyz/posts/cdc-recovery-contract
Author: Ayush Basak
Last modified: 2026-08-24
Topics: database-engineering, change-data-capture, distributed-systems, reliability

Change Data Capture is usually introduced as integration plumbing: read database changes, publish events, keep search or analytics up to date.

That framing hides the harder obligation. Once downstream systems depend on the stream, CDC becomes a recovery contract between mutable database state and every derived copy. The architecture must answer not only “can we stream today?” but also:

**Can we rebuild the truth after offsets are lost, schemas change, a slot falls behind, or a consumer has been wrong for three months?**

## Name the source of truth

CDC emits a history of database changes. It does not automatically create a domain event model.

```text
row change: orders.status PENDING -> PAID
domain fact: PaymentCaptured
```

These can coincide, but they have different contracts. A row-change stream reflects storage layout, transactions, and connector behavior. A domain event reflects business meaning and an intentionally versioned API.

Use CDC directly when consumers need a faithful projection of database state and can tolerate storage-shaped contracts. Use an outbox when the producer must choose stable business facts. Many systems combine them: write a domain event to an outbox table, then use CDC as the reliable transport.

Do not let downstream teams infer critical business meaning from undocumented column transitions.

## A snapshot and a stream must meet cleanly

A new consumer needs existing state plus future changes. The difficult boundary is the moment between the snapshot and live log.

An unsafe sequence is:

```text
SELECT all rows
then begin reading changes
```

Writes committed between those operations can disappear. Starting the stream first and then snapshotting can produce duplicates or updates before creates.

Production CDC connectors coordinate a consistent snapshot with a log position. Debezium’s PostgreSQL connector documents snapshot modes and uses PostgreSQL logical decoding to continue from a recorded position after the initial snapshot.

The consumer still needs idempotent application because restarts, retries, and re-snapshots can repeat records.

For a projection, prefer an upsert keyed by stable source identity:

```sql
INSERT INTO customer_projection(customer_id, source_version, payload)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id) DO UPDATE
SET source_version = EXCLUDED.source_version,
    payload = EXCLUDED.payload
WHERE customer_projection.source_version < EXCLUDED.source_version;
```

The version guard prevents an older replay from overwriting newer state.

## Offsets are business continuity state

The connector offset, PostgreSQL replication slot, publication configuration, and downstream consumer offsets together define recoverability. Treat them like production data.

For PostgreSQL logical decoding, a replication slot retains required WAL until the consumer advances. If consumption stops, retained WAL can fill disk. If the slot is dropped or allowed to fall behind available history, continuity is lost and the consumer may require a new snapshot.

Operate at least:

- current and confirmed log position;
- retained WAL bytes per slot;
- connector heartbeat freshness;
- transaction and event lag;
- offset commit failures;
- slot existence after failover;
- time and capacity required for re-snapshot.

“Connector is running” is not evidence that the stream is current or recoverable.

Define Recovery Point Objective (RPO) and Recovery Time Objective (RTO) for every derived system. A recommendation index may accept hours of lag; fraud decisions may not.

## Ordering exists only within a declared scope

Database commits provide an order in the log, but a distributed pipeline can repartition, parallelize, and retry records. A consumer should not assume global business order unless the architecture preserves it end to end.

Choose an ordering key—often aggregate or primary key—and include a monotonic source position or version. Then handle:

- duplicate version: ignore;
- older version: reject as stale;
- next version: apply;
- gap: defer, reconcile, or rebuild the key.

Cross-table transactions need special care. A consumer that observes line items before the order header may temporarily violate an invariant even though the database transaction was atomic. Use transaction metadata, an outbox event, or a projection design that tolerates convergence.

Total ordering across every tenant and entity is expensive and rarely the business requirement. Preserve only the order that protects an invariant.

## Schema evolution is a distributed deployment

A column rename is not one database migration once CDC exists. It is a contract change across connectors, schemas, brokers, consumers, replay archives, and projections.

Use expand-and-contract:

1. add the new field;
2. emit both representations where needed;
3. deploy tolerant consumers;
4. backfill and verify;
5. stop old reads;
6. remove the old field after the replay window.

Avoid reusing a field name with different meaning. Old records remain in logs and backups. A consumer rebuilding from history must interpret a schema version, not guess from the current database definition.

Database schema changes can also alter connector output unexpectedly: defaults, data types, replica identity, primary keys, and table inclusion all matter. Test the actual serialized records in a staging stream.

## Deletes need an explicit semantic

A source-row deletion may produce a delete event and, depending on the pipeline, a tombstone. Consumers must decide whether deletion means:

- remove the projection;
- retain a redacted audit record;
- mark inactive;
- trigger a business workflow;
- or ignore a storage-level cleanup.

For privacy deletion, removing a source row does not automatically delete data from compacted topics, warehouses, object storage, search indexes, caches, and backups. Track deletion as a governed workflow with evidence from every materialized copy.

## Rebuild is a product feature

Every important consumer should have a documented rebuild mode:

```text
freeze or version target
-> establish snapshot/log boundary
-> load snapshot
-> replay changes
-> validate counts and invariants
-> atomically switch readers
-> retain rollback target
```

Do not rebuild directly into the live index if partial state would be visible. Build a new generation and switch an alias after validation.

Validation needs more than row counts:

- sums or balances for financial domains;
- per-tenant counts;
- min/max source positions;
- sampled content hashes;
- missing and duplicate keys;
- business invariants;
- lag to current source position.

Run rebuild drills before an incident. The first full replay will expose hidden assumptions about retention, throughput, schema compatibility, and external rate limits.

## Prevent a consumer bug from becoming historical truth

Derived stores should retain provenance: source table or event type, key, source position, schema version, and projection code version. Without this, you cannot identify which records were produced by a faulty consumer release.

Canary new projection code into a shadow target. Compare outputs and invariants before promotion. When logic changes, decide whether old records need reprocessing or only future records use the new rule.

CDC makes changes easy to distribute. It also makes mistakes easy to distribute.

## CTO review questions

1. Is the stream a storage contract or a domain-event contract?
2. How do snapshot and live changes meet without loss?
3. Which offsets and slots must survive disaster recovery?
4. What is the ordering scope and how are gaps detected?
5. Can old records be interpreted after schema evolution?
6. How does a delete propagate to every copy?
7. How long does a full rebuild take, and when was it last tested?
8. Can we identify data produced by a faulty consumer version?

CDC is valuable because it turns committed changes into reusable infrastructure. It becomes trustworthy only when snapshots, offsets, schemas, deletes, and rebuilds are designed as one recovery system.

## References

- [Debezium PostgreSQL Connector](https://debezium.io/documentation/reference/stable/connectors/postgresql.html)
- [PostgreSQL: Logical Decoding](https://www.postgresql.org/docs/current/logicaldecoding.html)
- [Apache Kafka Connect documentation](https://kafka.apache.org/documentation/#connect)
- [Related: PostgreSQL Logical Replication Failover](/posts/postgresql-logical-replication-failover)
- [Related: The Transactional Outbox Is Not the Delivery Guarantee](/posts/transactional-outbox-delivery-guarantees)


---

# Scale the Blast Radius, Not Just the Fleet

> A CTO-level guide to cell-based architecture: partition keys, routing, migration, observability, and deciding when fault isolation earns its operational cost.

Canonical URL: https://www.ayushworks.xyz/posts/cell-based-architecture-blast-radius
Author: Ayush Basak
Last modified: 2026-08-28
Topics: distributed-systems, system-design, reliability, cloud-infrastructure, technical-leadership

Most scaling plans ask how a system will serve ten times more traffic. A better production question is: **when one part fails, how many customers fail with it?**

A larger shared cluster can improve unit economics while steadily increasing the consequence of a bad deployment, noisy tenant, corrupted cache, or overloaded queue. Replicas protect capacity; they do not automatically reduce correlated failure. Cell-based architecture addresses that distinction by dividing a service into bounded, independent copies called cells.

AWS describes a cell as a fixed-size, self-contained unit containing application logic and storage. Requests are assigned to cells through a thin routing layer. The objective is not infinite scale. It is a maximum credible scope of impact.

## A cell is an operating boundary

A useful cell owns the resources required to serve its assigned tenants:

```text
global entry point
       |
 cell assignment router
   /       |       \
cell-a   cell-b   cell-c
 API      API      API
 queue    queue    queue
 data     data     data
```

If every cell writes to one shared database, waits on one shared queue, or calls one mandatory global service, the diagram has cells but the failure model does not. Shared dependencies belong only where their failure can be tolerated, bypassed, or served from cached state.

The router should answer one narrow question: which cell owns this tenant or resource? Keep business workflows out of it. A sophisticated global router simply creates a new, harder control-plane dependency.

## Choose a partition key that follows ownership

For B2B SaaS, `tenant_id` is often the strongest key because requests, data export, rate limits, and incident communication already align to a customer. Consumer systems might use account, geography, or resource ID.

The key must satisfy four properties:

1. Most transactions stay inside one cell.
2. It is available before expensive request processing begins.
3. Assignment changes are rare and auditable.
4. One oversized key cannot exhaust a cell indefinitely.

Cross-cell synchronous transactions erase isolation. If a workflow genuinely spans cells, model it as an asynchronous process with an explicit coordinator, idempotent steps, and reconciliation. Do not hide distributed coordination inside an ORM transaction abstraction.

## Size cells before they are full

A cell needs a tested ceiling, not an aspirational autoscaling policy. Define limits for tenants, requests, queue depth, database size, connections, and background work. Operate below the first limit that becomes unsafe.

```text
safe cell capacity = min(
  API saturation threshold,
  database connection budget,
  replication and recovery budget,
  queue drain capacity,
  largest-tenant headroom
)
```

Fixed maximum size makes capacity planning repeatable: growth creates another known unit rather than one increasingly unique cluster. It also lets teams rehearse restore, deployment, and failover at the same scale they operate.

## Migration is part of the architecture

Cells become operationally dangerous if tenant movement is improvised during an incident. Build a migration state machine early:

```text
planned -> copying -> dual-read validation -> cutover -> verifying -> complete
```

Writes need one authority throughout the move. Common approaches include a brief write pause, change-data capture with a fenced cutover, or application-level dual writing with reconciliation. The destination must be verified for counts, invariants, and recent writes before routing changes. Retain a redirect marker or tombstone in the old cell so stale clients do not recreate state.

## Deployment must preserve isolation

Do not deploy the same build to every cell simultaneously. A safer sequence is:

1. synthetic and integration validation;
2. one canary cell with representative traffic;
3. observation through at least one meaningful workload cycle;
4. staged expansion with automated stop conditions;
5. explicit completion after SLO and business metrics remain healthy.

A failed canary should leave most customers untouched. That is one of the economic returns on the architecture.

## Observe cells individually and as a fleet

Global averages can hide a completely broken cell. Every alert and dashboard needs cell identity, but customer IDs should not become unbounded metric labels. Track per-cell request success, latency, saturation, queue age, database headroom, and deployment version. Send tenant-level evidence to logs or analytics designed for high cardinality.

Fleet views should answer:

- How many cells are healthy?
- Is one version correlated with failure?
- Which cell is closest to a capacity boundary?
- Can the router and assignment store serve during degradation?
- Is a failure isolated, or does it cross cell boundaries?

## When cells are the wrong answer

Cells add routing, provisioning, migrations, duplicated capacity, fleet deployment, and more complicated analytics. They are not justified because “large systems use them.” Start with simpler bulkheads—per-tenant quotas, isolated queues, database partitions, or workload pools—when they bound the credible failure.

Adopt cells when shared infrastructure creates unacceptable customer impact, a natural partition key exists, the product can tolerate asynchronous cross-cell workflows, and the organisation can automate cell lifecycle operations.

The CTO decision is a trade: operational complexity in exchange for a measurable upper bound on harm. If the team cannot state that bound, the architecture is not finished.

## References

- [AWS Guidance for Cell-Based Architecture](https://docs.aws.amazon.com/solutions/cell-based-architecture-on-aws/)
- [AWS: Cell architecture FAQ](https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/faq.html)
- [AWS: Shuffle Sharding](https://aws.amazon.com/blogs/architecture/shuffle-sharding-massive-and-magical-fault-isolation/)
- [Related: Queues, Backpressure, and Overload Control](/posts/queues-backpressure-overload-control)
- [Related: Control Planes Must Fail Quietly](/posts/control-planes-must-fail-quietly)


---

# Cloud Cost Is an Architecture Metric

> A CTO-level method for converting cloud bills into unit economics, ownership, cost attribution, capacity decisions, and engineering release gates.

Canonical URL: https://www.ayushworks.xyz/posts/cloud-unit-economics-for-architects
Author: Ayush Basak
Last modified: 2026-08-25
Topics: cloud-infrastructure, finops, system-design, technical-leadership

The monthly cloud bill tells finance what was spent. It rarely tells engineering which product behavior caused the spend, which customer received the value, or whether the architecture becomes more efficient as the company grows.

For technical leadership, the useful metric is not total cost. It is **cost per business outcome at an explicit quality level**.

## Choose a unit the business recognizes

Good units connect demand to value:

- cost per paid order;
- cost per active workspace;
- cost per document processed;
- cost per successful agent resolution;
- cost per connected-device hour.

Requests, CPU-hours, and tokens are component drivers, not usually business outcomes. AWS cost guidance recommends a small set of output metrics tied to workload success; Google’s framework similarly emphasizes aligning spend with business value.

Define the unit precisely:

```text
successful resolution =
  issue closed without human escalation for 72 hours
  AND no policy violation
  AND customer feedback is not negative
```

Without a quality condition, the cheapest architecture can win by failing quickly.

## Build a cost tree

Map the business unit to technical drivers:

```text
cost / resolved case
├── model tokens / case × token price
├── retrieval queries / case × query price
├── tool calls / case × service cost
├── storage GB-month / active account
├── compute seconds / case
└── shared platform allocation
```

This makes cost actionable. A model-price negotiation affects one branch; reducing repeated retrieval may affect cost and latency; a cache may reduce inference but increase stale-answer risk.

Track both marginal and allocated cost. Marginal cost answers “what does one more unit consume?” Allocated cost includes shared databases, observability, security, and idle capacity. Product pricing needs the latter; architecture experiments often start with the former.

## Attribute before optimizing

Tagging cloud resources by team is useful but insufficient when many tenants share one database or cluster. Add workload-level attribution using request context:

```text
trace_id
tenant_id
product
operation
release
resource_units
```

Aggregate high-cardinality context outside the metrics system if necessary. Do not put unrestricted tenant IDs into a backend that cannot handle the cardinality or privacy boundary.

Allocate shared cost with a declared rule:

```text
tenant share =
  40% × request share
  + 30% × storage share
  + 30% × compute-time share
```

No allocation is perfectly objective. The goal is a stable model good enough to expose direction and ownership. Change the formula through versioned governance, not whenever one team dislikes its result.

## Separate capacity, usage, and waste

A service can be expensive for three different reasons:

1. **Useful demand:** customers are doing more valuable work.
2. **Required headroom:** capacity protects latency and failure tolerance.
3. **Waste:** idle, duplicate, leaked, or incorrectly sized resources.

Do not label all headroom as waste. A database at 90% steady utilization may have excellent accounting and terrible incident tolerance.

For each component, connect provisioned capacity to a reliability constraint:

```text
required capacity =
  forecast peak × burst factor × failure-domain factor
```

If one zone can fail, remaining zones must carry the approved load. Cost review must preserve that invariant.

In Kubernetes, requests influence scheduling while limits constrain runtime behavior. Incorrect requests can strand allocatable capacity; overly tight memory limits can create OOM kills. Rightsizing is therefore a reliability change and needs canarying, not a spreadsheet-only edit.

## Put cost into release evidence

Benchmark representative workloads before changing architecture. Report:

| Metric | Current | Candidate | Gate |
|---|---:|---:|---:|
| cost / successful task | $0.042 | $0.031 | ≤ $0.035 |
| p95 latency | 1.8 s | 2.0 s | ≤ 2.2 s |
| success rate | 98.7% | 98.8% | ≥ 98.5% |
| peak capacity margin | 2.1× | 1.7× | ≥ 1.5× |

A lower bill with worse tail latency, more retries, or less failure headroom is not necessarily an optimization.

Add cost regression budgets to CI or performance environments for changes with material drivers: query amplification, payload size, model tokens, storage writes, cross-region transfer, and telemetry volume.

```python
estimated_cost = (
    input_tokens * input_rate
    + output_tokens * output_rate
    + tool_calls * tool_call_rate
)
assert estimated_cost / successful_cases <= COST_BUDGET
```

The estimate will not equal the invoice. It creates an early engineering signal.

## Design for cost failure modes

Cost can fail suddenly: a retry storm multiplies calls, a missing partition filter scans a warehouse, abusive traffic triggers inference, or a log loop produces terabytes.

Use layered controls:

- per-tenant quotas and rate limits;
- query scan limits and timeouts;
- retry budgets;
- maximum token and tool-call budgets;
- storage lifecycle and retention policies;
- anomaly alerts on units, not only currency;
- kill switches that degrade non-critical work safely.

A billing alert after several hours is detection, not containment.

## Make ownership visible

Every material cost line needs a technical owner and a product beneficiary. Review the top drivers, their unit trends, and the next scaling discontinuity monthly.

The useful questions are:

- Is unit cost improving as volume grows?
- Which fixed cost will become marginal, or vice versa?
- At what demand does the database, vendor tier, or architecture step-change?
- Which tenant or feature has negative gross margin?
- What reliability margin is included in the number?

Forecast from demand drivers, not a flat percentage over last month. Ten thousand more active devices may increase ingestion, retention, and support differently from ten thousand more registered accounts.

## CTO review

1. What business outcome is the primary unit?
2. Is quality included in its definition?
3. Can shared cost be attributed with a versioned rule?
4. Which costs are demand, headroom, and waste?
5. What scaling threshold creates the next discontinuity?
6. Do release gates include cost, latency, reliability, and success together?
7. Which guardrail contains a runaway cost event automatically?
8. Who owns each major driver and its optimization backlog?

Cost is the resource consequence of architecture. When engineers see it per useful outcome, financial discipline becomes system design rather than end-of-month cleanup.

## References

- [AWS: Monitor cost and usage](https://docs.aws.amazon.com/wellarchitected/latest/cost-optimization-pillar/monitor-cost-and-usage.html)
- [Google Cloud Well-Architected: Cost optimization](https://docs.cloud.google.com/architecture/framework/cost-optimization)
- [Kubernetes: Resource requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
- [Related: Deadline Budgets and Retry Amplification](/posts/deadline-budgets-retry-amplification)
- [Related: Queues, Backpressure, and Overload Control](/posts/queues-backpressure-overload-control)


---

# Continuous Batching Is a Scheduling Policy

> Operate LLM continuous batching with explicit admission, fairness, prefill limits, cancellation, and tail-latency budgets.

Canonical URL: https://www.ayushworks.xyz/posts/continuous-batching-is-a-scheduling-policy
Author: Ayush Basak
Last modified: 2026-09-06
Topics: ai-infrastructure, llm-serving, scheduling, performance

Static batching waits for a group of requests, pads work to compatible shapes, and completes the group together. Continuous batching revisits the active set at generation steps: finished requests leave and waiting requests can enter.

That improves accelerator utilization. It also turns the inference runtime into a scheduler deciding whose tokens are computed, whose prompt is admitted, and who waits behind a large prefill.

Continuous batching is therefore a product policy expressed through GPU scheduling.

## The two workloads are different

Prefill processes the input prompt and is highly parallel, but a long prompt can consume substantial compute and allocate a large KV cache. Decode generates one or a small number of tokens per active sequence repeatedly and is often constrained by memory movement.

```text
request -> tokenize -> admission -> prefill -> repeated decode -> finish
                              |             |
                              +-- KV cache -+
```

If the scheduler admits only by request count, one 100,000-token prompt can be treated like one 500-token prompt even though their resource demands differ dramatically.

Use token-aware admission. Bound total scheduled tokens, active sequences, per-request context, and reserved KV-cache headroom. Keep a queue deadline so requests that can no longer meet their SLO are rejected rather than computed uselessly.

## Define fairness explicitly

First-in-first-out is simple but can let large prefills block interactive requests. Always preferring short jobs reduces mean latency while starving long-context work. Tenant-blind scheduling lets one customer occupy the batch.

A practical policy separates traffic classes:

| Class | Objective | Guardrail |
| --- | --- | --- |
| interactive | low time to first token | small prompt cap, reserved capacity |
| standard generation | balanced latency and throughput | weighted fair share |
| batch/offline | tokens per second | preemptible, no interactive reservation |
| administrative | predictable control access | isolated small pool |

Weighted queues should charge by estimated tokens or GPU time, not request count. Enforce tenant concurrency and token-rate budgets before work enters the runtime.

## Cancellation is capacity recovery

Clients disconnect, abandon streams, or reach deadlines. Propagate cancellation into the engine, remove waiting requests immediately, and reclaim finished or cancelled sequence state safely. A proxy noticing disconnect while the model continues generating is a capacity leak.

Cancellation must not corrupt a shared batch. The scheduler removes one sequence at a defined step boundary while preserving the state of others.

## PagedAttention changes memory management, not limits

The vLLM paper introduces PagedAttention to manage KV-cache memory in blocks, reducing fragmentation and enabling sharing. Better allocation increases feasible batch size; it does not make KV cache unlimited. Admission still needs a budget for model weights, runtime workspace, cache blocks, and safety headroom.

When pressure rises, decide deliberately whether to queue, reject, preempt, swap, or route elsewhere. Unplanned preemption storms can improve nominal admission while destroying tail latency.

## Measure the scheduler, not only the GPU

Track:

- queue time and time to first token by class and tenant;
- inter-token latency and end-to-end latency;
- prompt and generated tokens;
- running, waiting, and preempted requests;
- KV-cache utilization and prefix-cache hit rate;
- cancellation-to-reclamation delay;
- deadline misses and rejection reasons;
- tokens per second per accelerator.

GPU utilization at 100% is not success if interactive requests wait behind offline jobs. Throughput and latency must be evaluated together under the production prompt-length distribution.

## Failure policy

Reject requests exceeding hard context or output limits before allocation. Shed low-priority work when the queue cannot meet its deadline. Route only when the destination has compatible model, adapter, cache, and policy. Preserve request identity across routing so retries do not duplicate metered work or tool side effects.

## Trade-offs

Larger batches improve throughput until memory pressure, queueing, or decode interference harms tails. Chunked prefill can interleave long prompts with decode but adds scheduler complexity. Reserved capacity protects interactive latency while leaving hardware idle during quiet periods. Preemption raises utilization but spends recomputation.

The correct configuration is not the largest batch that fits. It is the scheduling policy that meets differentiated service promises at an acceptable cost.

## Further reading

- [vLLM and PagedAttention paper](https://arxiv.org/abs/2309.06180)
- [vLLM engine arguments](https://docs.vllm.ai/en/latest/configuration/engine_args.html)
- [LLM serving is KV-cache capacity planning](/posts/llm-serving-is-kv-cache-capacity-planning)


---

# Control Planes Must Fail Quietly

> A CTO-level architecture for separating control and data planes, bounding reconciliation, surviving stale configuration, and preventing management failures from becoming customer outages.

Canonical URL: https://www.ayushworks.xyz/posts/control-planes-must-fail-quietly
Author: Ayush Basak
Last modified: 2026-08-26
Topics: distributed-systems, control-planes, reliability, platform-engineering

A control plane decides what the system should become. A data plane serves the request in front of the customer. When those responsibilities share the same synchronous failure path, an administrative outage becomes a product outage.

The governing invariant is simple:

> Existing, authorized traffic should continue safely for a bounded period when the control plane is unavailable.

That is not always possible—credential revocation and risk controls may require fail-closed behavior—but it should be a conscious exception.

## Separate desired state from serving state

Kubernetes provides a useful model. Its API server exposes desired state; controllers repeatedly compare desired and observed state and act to reduce the difference. Worker nodes run the workloads. The architecture is not valuable because every product needs Kubernetes. It is valuable because reconciliation tolerates delay and retry.

```text
operator → control API → desired state store
                         ↓
                    reconciler
                         ↓
                  serving snapshot → data plane
```

The data plane should consume a validated, versioned snapshot—not query a mutable management database during every request.

For a feature-routing system, the snapshot might be:

```json
{
  "version": 1842,
  "generated_at": "2026-08-26T03:20:00Z",
  "routes": [{"tenant":"acme","upstream":"cluster-b"}],
  "valid_until": "2026-08-26T04:20:00Z"
}
```

The version allows monotonic application. The validity window bounds staleness. The data plane can keep serving version 1842 while the control API is temporarily unavailable.

## Make reconciliation idempotent

A controller will retry after timeouts, crashes, and leadership changes. Therefore the operation must converge when executed repeatedly.

Bad:

```text
on event: increment desired replica count
```

Better:

```text
observe replicas = 3
desired replicas = 5
create replicas until observed = 5
```

Events can wake a reconciler, but current state should decide the action. Persist an operation identity for external effects and use compare-and-set or resource versions so two reconcilers cannot overwrite each other blindly.

Model the loop explicitly:

```text
read desired → read observed → compute delta
→ apply bounded action → record result → requeue
```

One loop should make limited progress. A controller that attempts to repair ten thousand resources in one transaction creates long locks and large failure domains.

## Define stale-state policy per decision

Not all configuration has the same safety profile.

| State | During control-plane outage |
|---|---|
| route table | serve last valid snapshot |
| price catalogue | serve briefly, then stop checkout |
| revoked credential | fail closed after short TTL |
| UI experiment | use deterministic default |
| rate limit | retain last limit locally |

“Fail open” and “fail closed” are incomplete without a time horizon. Decide the maximum stale age, default behavior, and customer-visible degradation for every control object.

Keep an emergency path small. If disabling a dangerous integration requires the same broken deployment system that introduced it, the control plane cannot control the incident.

## Protect the data plane from control churn

Administrative activity can be bursty: a bulk import, policy rollout, or controller bug may touch every tenant. Bound its effect with:

- rate-limited reconciliation;
- per-tenant work queues;
- jittered retries;
- generation numbers that collapse obsolete updates;
- priority for safety changes;
- circuit breakers around external dependencies.

Never let reconcilers share an unbounded resource pool with customer traffic. Separate connection pools, worker queues, quotas, and ideally compute capacity. Otherwise a repair storm consumes the system it is trying to repair.

## Publish snapshots atomically

Data-plane readers must not observe half a configuration. Build the next snapshot, validate it, then switch a pointer atomically:

```text
write config/version-1843
validate schema + invariants
compare-and-set active: 1842 → 1843
notify readers
```

Readers retain the last known-good version if the new one fails verification. Record rejection reason and control-plane version in telemetry.

Compatibility matters. During rollout, old data planes may read new snapshots. Version the schema, support an overlap window, and test rollback. A new control plane that emits state the old data plane cannot parse removes your rollback path.

## Observe convergence, not only availability

A healthy API server does not prove that desired state reached production. Measure:

- desired-to-observed convergence time;
- age and version of serving snapshots;
- reconciliation attempts and terminal failures;
- queue age by tenant and priority;
- data planes on unsupported versions;
- rejected snapshots;
- control-plane dependency saturation.

Alert on customer-impacting drift: “4% of tenants are more than two versions behind” is more useful than “controller CPU is 80%.”

## CTO review

1. Which customer paths synchronously depend on the control plane?
2. How long may each type of state remain stale?
3. Are reconciler actions idempotent and bounded?
4. Can control churn exhaust data-plane resources?
5. Is serving configuration published atomically and versioned?
6. Can old data planes read state produced during a new rollout?
7. What metric proves desired state converged?
8. Is there an independent emergency-disable path?

A good control plane changes production deliberately. A great one can be broken for an hour without making the customer discover it first.

## References

- [Kubernetes: Cluster Architecture](https://kubernetes.io/docs/concepts/architecture/)
- [Kubernetes: Controllers](https://kubernetes.io/docs/concepts/architecture/controller/)
- [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/)
- [Related: Durable AI Agent Workflows](/posts/durable-ai-agent-workflows)
- [Related: Queues, Backpressure, and Overload Control](/posts/queues-backpressure-overload-control)


---

# Copy-on-Write Moves Cost to the First Mutation

> Use copy-on-write with an explicit model for page faults, write amplification, memory pressure, and snapshot lifetime.

Canonical URL: https://www.ayushworks.xyz/posts/copy-on-write-moves-cost-to-first-mutation
Author: Ayush Basak
Last modified: 2026-09-05
Topics: systems-internals, memory-management, linux, performance

Copy-on-write (CoW) makes copying cheap by postponing the copy. Two owners initially reference the same physical data. The system copies only when one owner mutates it.

That is an excellent default for process creation, filesystem snapshots, and immutable data structures. It is not free. CoW moves cost from creation time to the first write, where latency and memory growth are often less visible.

## The operating-system case

After `fork()`, parent and child have separate virtual address spaces whose page-table entries can initially refer to the same physical pages. Those mappings are protected against direct writes. When either process writes, the CPU raises a page fault; the kernel allocates a new page, copies content, and updates the writer’s mapping.

```text
before write              after child writes

parent VA --+             parent VA ---> page A
            +-> page A
child VA  --+             child VA  ---> page B (copy)
```

Linux page tables translate process virtual addresses into physical addresses. CoW works by changing mappings and permissions, not by teaching application objects to clone themselves.

## Why production latency surprises teams

Consider a large in-memory service that forks a child for a snapshot. Fork can return quickly, but pages modified while the child remains alive must be copied. A write-heavy workload can produce:

- bursts of minor page faults;
- rapid resident-memory growth;
- extra memory bandwidth consumption;
- allocator and page-table work;
- an out-of-memory kill if headroom was planned from steady-state RSS.

The snapshot duration becomes part of the write-amplification budget. A slower child extends the interval during which mutations create private copies.

The same shape appears above the OS. A CoW B-tree copies nodes along an update path. A filesystem snapshot keeps old blocks alive when new versions are written. The unit differs—page, node, block—but the operational question is the same: how much changes while old readers retain the previous version?

## Capacity model

A useful upper-bound model is:

```text
peak memory ~= base working set
            + unique pages dirtied during overlap
            + page tables and process overhead
            + safety margin
```

Do not estimate the second term from average write throughput alone. Measure distinct pages dirtied during the snapshot window. A workload repeatedly changing one hot page behaves differently from one touching the entire heap.

## Production checks

| Risk | Evidence to collect |
| --- | --- |
| first-write latency | minor faults and tail latency |
| memory expansion | RSS and dirty-page growth during overlap |
| slow snapshot | duration distribution and bytes produced |
| workload sensitivity | unique pages dirtied, not just write count |
| failure recovery | behavior when snapshot child exits or is killed |

Run load tests with the real mutation pattern and the longest expected snapshot time. Confirm the service has headroom during deployment, compaction, and backup overlap—not only when each runs alone.

## Trade-offs

Eager copies make creation predictably expensive and reserve capacity early. CoW makes common read-mostly paths efficient but produces workload-dependent first-write costs. Immutable persistent structures provide controlled sharing but add indirection and reclamation work. Log-based snapshots avoid some page copying while introducing replay and retention requirements.

Use CoW when sharing is likely to outlive few mutations. When most of the working set will be rewritten before the old view disappears, the optimization can become deferred full-copy cost with a less convenient latency profile.

## Further reading

- [Linux kernel page-table documentation](https://docs.kernel.org/mm/page_tables.html)
- [Linux `fork(2)` semantics](https://man7.org/linux/man-pages/man2/fork.2.html)
- [PostgreSQL checkpoints are latency events](/posts/postgresql-checkpoints-are-latency-events)


---

# Cursor Pagination Is a Consistency Contract

> Design cursor pagination around stable ordering, snapshot choices, opaque tokens, deletion behavior, authorization scope, and property-based tests.

Canonical URL: https://www.ayushworks.xyz/posts/cursor-pagination-is-a-consistency-contract
Author: Ayush Basak
Last modified: 2026-09-05
Topics: api-design, databases, backend-engineering, pagination, system-design

Cursor pagination is commonly justified as a faster replacement for `OFFSET`. Performance is only half the design. A cursor defines what “continue this listing” means while rows are concurrently inserted, updated, deleted, or hidden by authorization changes.

> A cursor is a serialized continuation contract, not a disguised row number.

Before choosing its encoding, decide which consistency experience the API promises.

## Make ordering total and immutable enough

Pagination needs a deterministic total order. `ORDER BY created_at` is insufficient when multiple records share a timestamp. Add a unique tie-breaker:

```sql
SELECT id, created_at, title
FROM documents
WHERE tenant_id = $1
  AND (created_at, id) < ($2, $3)
ORDER BY created_at DESC, id DESC
LIMIT $4;
```

The cursor carries the last `(created_at, id)` pair. The next query uses the same filter and order.

If the ordering column can change, an item can move across the boundary between requests and appear twice or not at all. Prefer an immutable ordering key for traversal. If the product must sort by mutable popularity or status, document weaker semantics or freeze the ranking version.

## Choose a mutation model

There are three useful contracts.

| Contract | Behavior during traversal | Cost |
| --- | --- | --- |
| Live traversal | New changes may appear according to key order | Simple, not a stable snapshot |
| High-water mark | Excludes items created after page one | Stable upper boundary |
| Snapshot | All pages read one logical dataset version | Strongest, needs retained snapshot/state |

For a high-water mark, the first response includes the maximum eligible ordering key and later queries constrain results beneath it. This prevents newly inserted items from shifting the traversal, but updates and deletions still require defined behavior.

A database snapshot held across human-paced requests is usually impractical. An export job can materialize results or retain a version; an interactive feed often accepts live or high-water semantics.

## Keep tokens opaque but accountable

Google’s API design guidance uses `page_token` and `next_page_token` for list pagination and recommends opaque page tokens. Opaque does not mean unstructured internally. Version the payload so its schema can evolve.

```json
{
  "v": 2,
  "lastCreatedAt": "2026-09-05T04:00:00Z",
  "lastId": "doc_01K...",
  "filterHash": "sha256:...",
  "subjectHash": "sha256:...",
  "expiresAt": "2026-09-05T05:00:00Z"
}
```

Encode and authenticate the token or store a random handle server-side. Plain Base64 prevents casual reading, not tampering. Reject unsupported versions, invalid signatures, expired cursors, and unreasonable decoded values.

## Bind continuation to the original query

A cursor minted for `tenant=A&status=open` must not be reusable with `tenant=B` or `status=closed`. Bind it to normalized filters, sort direction, API version, and the authorization subject or entitlement scope.

Do not trust tenant data carried inside the cursor as authorization. Derive accessible scope from the authenticated request, then verify that it matches the cursor’s binding. This follows the same principle as [treating cache keys as isolation boundaries](/posts/shared-cache-keys-are-data-isolation-boundaries).

Decide how projection changes behave. If `fields=` affects only representation, reuse may be safe. If it affects eligibility or joins, include it in the query fingerprint.

## Define deletion and exhaustion

Keyset pagination does not require the boundary row to still exist; the tuple values in the cursor remain sufficient. That is one advantage over a cursor that stores only an object ID and looks the row up later.

Return an absent next token when traversal is exhausted. An empty page may still legitimately have a next token if post-filtering occurs after fetching, but that design can confuse clients and waste calls. Prefer applying visibility filters inside the query so page size describes deliverable items.

Set a maximum page size and treat client size as a hint. Token continuation must preserve the server’s ordering contract even if the client requests a different size later; either permit only size changes that do not alter eligibility or bind size explicitly.

## Plan index shape with the query

The index should support tenant/filter prefix followed by ordering keys. For the sample query, a likely starting point is:

```sql
CREATE INDEX documents_tenant_created_id_idx
ON documents (tenant_id, created_at DESC, id DESC);
```

Real selectivity and additional filters determine the final index. Inspect the execution plan with production-like distributions. Cursor pagination avoids scanning skipped offsets, but it cannot rescue an index that does not match the predicate.

## Test properties, not examples

Generate datasets with equal timestamps, delete boundary rows, insert before and after the high-water mark, mutate sortable fields, change authorization between pages, tamper with tokens, and switch sort direction.

Assert:

- no item appears twice under the promised mutation model;
- every eligible stable item is eventually returned;
- ordering is total and monotonic;
- a cursor cannot cross tenant or filter scope;
- invalid tokens fail with a documented client error;
- query work remains bounded for late pages.

Pagination bugs rarely live on page one. The API contract becomes visible only when data changes between requests. Design that change explicitly, and the cursor becomes a dependable continuation rather than an encoded accident.

## References

- [Google Cloud API Design Guide: List pagination](https://cloud.google.com/apis/design/design_patterns#list_pagination)
- [Google Cloud API Design Guide: Naming conventions](https://cloud.google.com/apis/design/naming_convention)
- [AIP-158: Pagination](https://google.aip.dev/158)


---

# Data Retention Is a System Design Problem

> A production architecture for retention policy, deletion guarantees, PostgreSQL partitioning, backups, derived data, legal holds, and verifiable erasure.

Canonical URL: https://www.ayushworks.xyz/posts/data-retention-is-a-system-design-problem
Author: Ayush Basak
Last modified: 2026-08-29
Topics: database-engineering, data-governance, postgresql, systems-architecture

Teams often treat retention as a cleanup query added after a product ships. By then the same customer data exists in the primary database, replicas, caches, search indexes, object storage, analytics tables, event logs, model features, exports, and backups.

Deletion is no longer a SQL statement. It is a distributed workflow with an evidence requirement.

The governing invariant is:

> Data must not remain usable beyond its approved purpose and retention window, except under an explicit hold.

## Define policy by data class

“Keep data for 90 days” is incomplete. A retention record needs:

| Field | Meaning |
|---|---|
| data class | authentication log, invoice, message, model input |
| authority | contractual, legal, security, or product requirement |
| start event | creation, account closure, settlement, last activity |
| active retention | time available to the product |
| backup retention | time recoverable from protected copies |
| deletion objective | maximum time from eligibility to verified removal |
| hold behavior | who can place and release a hold |
| owner | team accountable for evidence |

Do not reuse one duration across unrelated classes. Fraud evidence, transient prompts, invoices, and application telemetry have different purposes and risks.

## Put expiry in the data model

Compute deletion eligibility when the governing event occurs and store it explicitly:

```sql
ALTER TABLE audit_events
  ADD COLUMN expires_at timestamptz NOT NULL,
  ADD COLUMN retention_class text NOT NULL,
  ADD COLUMN legal_hold_id uuid;

CREATE INDEX audit_events_expiry_idx
  ON audit_events (expires_at)
  WHERE legal_hold_id IS NULL;
```

An explicit timestamp makes policy inspectable and avoids reconstructing meaning from mutable account state. Changes to policy can then be applied as versioned migrations with an audit trail.

## Delete by partition when volume demands it

Large row-by-row deletes create dead tuples, WAL volume, replica lag, and long cleanup cycles. Time partitioning turns expiry into a metadata operation when retention aligns with the partition key.

```sql
CREATE TABLE request_logs (
  occurred_at timestamptz NOT NULL,
  tenant_id uuid NOT NULL,
  payload jsonb NOT NULL
) PARTITION BY RANGE (occurred_at);
```

Detach an expired partition, verify that no hold applies, then drop it. Choose partition size from deletion cadence and operational cost—not arbitrary calendar aesthetics. Daily partitions may create catalogue overhead; monthly partitions may keep data longer than policy permits.

Partitioning is not the policy engine. Holds, per-tenant exceptions, and records with retention based on a later event may require a separate quarantine or tombstone workflow.

## Track every derived copy

Build a propagation ledger for each data class:

```text
system of record
├── search index       deletion by document ID
├── analytics          partition expiry + subject tombstone
├── object storage     lifecycle rule + inventory verification
├── cache              bounded TTL
└── model features     dataset/version lineage + rebuild
```

Every edge needs an owner, delivery mechanism, retry policy, and reconciliation job. A deletion event alone is insufficient because consumers can miss it. Periodically compare authoritative tombstones with downstream state.

Prefer stable subject identifiers over copying personal attributes into event keys. That makes targeted deletion and evidence collection possible without searching arbitrary payloads.

## Treat backups as delayed deletion

Immutable backups should not be rewritten casually; doing so can damage recoverability. Instead:

- keep backup retention bounded and documented;
- encrypt backups with managed key lifecycle where appropriate;
- restrict restoration access;
- after restore, replay deletion tombstones before returning the environment to service;
- test that procedure during recovery exercises.

The honest guarantee may be “removed from active systems within 24 hours and expires from protected backups within 35 days.” Architecture and policy must say the same thing.

## Make deletion observable

Measure:

- eligible records awaiting deletion;
- age of the oldest overdue record;
- completion lag per downstream system;
- hold count and age;
- reconciliation mismatches;
- backup sets containing expired data;
- failed or poison deletion commands.

Avoid personal identifiers in metric labels. Keep detailed evidence in access-controlled audit records.

A deletion request should reach a terminal state only after required systems acknowledge or reconciliation proves absence. Partial success remains visible and retryable.

## Conclusion

Retention is an end-to-end data lifecycle, not a nightly cron. Model expiry explicitly, align physical storage with deletion units, trace derived copies, bound backup exposure, and collect evidence.

The cheapest byte to govern is the one you never collect. For everything else, deletion must be designed with the same care as creation.

## References

- [PostgreSQL: Table Partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html)
- [PostgreSQL: DROP TABLE](https://www.postgresql.org/docs/current/sql-droptable.html)
- [NIST SP 800-88 Rev. 1: Media Sanitization](https://csrc.nist.gov/pubs/sp/800/88/r1/final)


---

# Database Credential Rotation: How to Replace Secrets Without Breaking Connection Pools

> A rollout protocol for rotating database credentials across cached secrets, connection pools, workers, rollback windows, and emergency revocation.

Canonical URL: https://www.ayushworks.xyz/posts/database-credential-rotation-needs-a-pool-rollout
Author: Ayush Basak
Last modified: 2026-09-11
Topics: security, database-engineering, backend-engineering, operations

A credential can be rotated successfully in a secret store while applications continue using the previous value. The control plane reports success, but a worker with a long-lived cache may fail the next time its pool creates a connection.

The useful unit of rotation is therefore a deployed population of clients, not a string. A complete rollout needs to account for the secret version, credential accepted by the database, process configuration, pool creation, existing sessions, and rollback behavior.

This guide proposes an application rollout protocol. Exact session behavior depends on the database, authentication method, driver, and proxy. Test those boundaries instead of assuming password replacement terminates or refreshes existing connections.

## Map the copies of the credential

Start with an inventory of consumers: request-serving instances, queue workers, scheduled jobs, migration runners, reporting processes, and administrative tools. For each, record how the secret arrives and what event refreshes it.

~~~text
secret version
    |
    +-- process configuration
    |       |
    |       +-- pool factory -> new authenticated sessions
    |
    +-- worker cache
    |
    +-- scheduled job launched from an older image
~~~

Changing the source does not necessarily update any of these copies. An environment variable is a process snapshot. A cache has its own refresh policy. A connection pool may retain sessions while creating replacements with credentials captured during initialization.

AWS documents client-side caching separately from secret retrieval and rotation. Its [Python caching guide](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_cache-python.html) includes refresh behavior and limitations; do not infer that every SDK, wrapper, or application cache uses the same policy.

## Define the invariant in terms of new sessions

For a planned rotation, a useful availability invariant is: every active workload can establish an authorized connection using the intended current credential before the old credential is retired.

A query over an already open session is insufficient evidence. It proves that session can still execute a query. It may say nothing about the next pool expansion, restart, autoscaling event, or failover.

Use a fresh-connection probe through the same driver, network path, TLS settings, and authentication mode as the workload. Keep the probe read-only and narrow. It should not require broader privileges than the application already has.

Record the credential version identifier and probe outcome, never the secret value. Version identifiers also need care if they encode confidential deployment information.

## Choose a rotation strategy deliberately

AWS Secrets Manager describes single-user and alternating-users strategies for supported Lambda-based database rotation. Alternating users maintains two database identities; it adds privilege and identity-management considerations. Single-user rotation updates one identity and requires clients to handle the transition appropriately. These are documented strategies, not a guarantee that a custom application is disruption-free. See the [rotation strategy guide](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotation-strategy.html).

For a provider that supports overlapping credentials or identities, the rollout can make acceptance of the new value precede retirement of the old value. If the backend accepts only one current password, that overlap may not exist. Your client recovery behavior becomes more important, and your availability target may require a different authentication mechanism.

Do not promise zero downtime until the chosen backend and client path have passed the rotation test.

## A proposed state machine

Use observable stages rather than one boolean called rotated:

~~~json
{
  "rotationId": "rotation-example-17",
  "targetVersion": "credential-version-b",
  "phase": "client-adoption",
  "requiredConsumerClasses": [
    "api",
    "queue-workers",
    "scheduled-jobs"
  ],
  "retirePreviousAfter": "all-required-checks-pass"
}
~~~

This is an example control record, not an AWS API payload.

First prepare the new credential with the intended privileges. Next verify new authentication from a canary client. Then roll the pool factory to the new version in a small part of the fleet. Expand only while fresh connections succeed and business error rates remain acceptable.

Drain old pools through driver-supported behavior. Decide what happens to checked-out sessions and long transactions. Closing all connections at once can turn a routine security operation into a reconnect storm.

Finally, retire the prior credential and prove that a fresh authentication attempt with it is rejected. Securely control that negative test and avoid logging credentials in command arguments or diagnostics.

## Bound the reconnect workload

Credential replacement can synchronize thousands of clients. Even with valid authentication, they may overwhelm connection admission, TLS establishment, or an intermediate proxy.

As an illustrative estimate, replacing 1,200 sessions over 60 seconds creates an average of 20 new sessions per second before normal churn. This is a planning calculation, not a benchmark or a safe database limit. Measure the actual bottleneck and account for bursts.

Roll by workload group, cap new-connection concurrency, and add jitter where clients would otherwise reconnect together. Preserve enough working capacity while each group changes. The relevant limit is discussed further in [database connection budgets](/posts/postgresql-connections-are-capacity-budget).

## Failure policy

| Failure | Response |
| --- | --- |
| New credential cannot authenticate | Stop expansion; investigate privilege and configuration mismatch |
| Some clients still use the old version | Keep planned retirement blocked and locate those consumers |
| Pool replacement causes overload | Slow the rollout and reduce connection creation concurrency |
| Old credential is suspected compromised | Follow emergency revocation policy; availability may be sacrificed |
| Rollback image embeds stale configuration | Refresh configuration before admitting it to service |

Emergency rotation has a different objective from planned maintenance. An overlap window that is useful for availability can be unacceptable after compromise. Decide that exception before an incident.

Also separate future authentication from active-session revocation. Disabling a credential may not terminate sessions already established with it. Verify database-specific behavior and define who is authorized to terminate sessions when containment requires it.

## Test the least convenient clients

The most informative tests often involve infrequent consumers. Start a scheduled job from an older deployment template. Trigger scale-out halfway through rotation. Restart a worker whose secret cache has not refreshed. Exercise database failover after client adoption but before retirement.

For each test, record the expected credential version, fresh-connection result, and recovery time. Avoid treating aggregate success rate as proof that every consumer class works: a broken nightly job can disappear inside millions of healthy API queries.

## The engineering decision

Managed rotation reduces secret-management work, but the application still owns adoption and evidence. A reasonable review asks who inventories consumers, who controls retirement, how new authentication is verified, and what recovery is possible after the old value stops working.

The process is complete when the fleet uses the intended version, stale clients are accounted for, and the old access path behaves according to the security policy. A successful secret-store update is one checkpoint in that process.

## References

- [AWS Secrets Manager rotation overview](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html)
- [Single-user and alternating-users rotation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotation-strategy.html)
- [Client-side secret caching](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_cache-python.html)
- [Related: service identity](/posts/service-identity-before-service-mesh)


---

# Your Timeout Budget Is an Architecture Decision

> A CTO-level method for deadline propagation, retry budgets, jitter, and preventing one slow dependency from amplifying into a fleet-wide outage.

Canonical URL: https://www.ayushworks.xyz/posts/deadline-budgets-retry-amplification
Author: Ayush Basak
Last modified: 2026-08-24
Topics: distributed-systems, reliability, backend-engineering, system-design

Most timeout configurations are numbers copied from another service. `30s` feels conservative, `5s` feels responsive, and nobody can explain what either number protects.

That is not configuration. It is an undocumented availability policy.

A deadline decides how long a caller will reserve memory, a connection, a worker, and user patience for an operation. A retry decides how much additional load the system may create when it is already unhealthy. Together they define whether partial slowness remains local or becomes a fleet-wide outage.

The architecture question is therefore not “what timeout should this HTTP client use?” It is: **how will one end-to-end latency budget be spent, propagated, and enforced across every dependency?**

## Start with the user-visible deadline

Assume an API must complete within 800 ms at the 99th percentile. Its path is:

```text
client -> gateway -> orders -> inventory -> database
                         \-> pricing
```

Giving every hop an 800 ms timeout does not preserve an 800 ms experience. It permits sequential work to consume multiples of the budget and leaves no time to return a useful failure.

Instead, carry an absolute deadline or remaining duration:

```text
request budget                         800 ms
gateway admission + routing             40 ms
orders local work                        80 ms
inventory including database            260 ms
pricing                                  180 ms
response serialization + safety margin   90 ms
unallocated contingency                 150 ms
```

These are not universal numbers. The important design is explicit ownership. Every layer knows its maximum spend and stops work that can no longer affect the response.

gRPC recommends setting deadlines because, by default, a client may otherwise wait indefinitely. It also supports deadline propagation, converting the deadline into a timeout for the next hop while accounting for elapsed time. That is the correct mental model even outside gRPC: downstream work inherits a shrinking budget; it does not receive a fresh lease on the user’s patience.

## Cancellation is part of correctness

A caller timing out is not enough. If the downstream service continues querying, rendering, or calling another vendor, the user has left but the cost remains.

Every blocking boundary should answer three questions:

1. Can the operation observe cancellation?
2. Does cancellation release the scarce resource promptly?
3. If the work cannot be cancelled, is it bounded and isolated?

In application code, pass cancellation context rather than inventing a new timeout at each function:

```go
func Quote(ctx context.Context, orderID string) (Quote, error) {
    ctx, cancel := context.WithTimeout(ctx, 180*time.Millisecond)
    defer cancel()
    return pricing.GetQuote(ctx, orderID)
}
```

The local cap protects the caller, while the parent context ensures an earlier end-to-end cancellation wins.

For a database, configure both client cancellation and a server-side statement limit. A dead client socket is not an operational policy. For background work, use leases so abandoned work becomes recoverable rather than immortal.

## Retries spend a second budget

Retries are selfishly rational and globally dangerous. One caller sees a transient failure and tries again. Thousands of callers see the same failure and multiply load at the moment the dependency has the least capacity.

With five layers and three attempts per layer, a single original request can theoretically produce `3^5 = 243` calls at the bottom if every layer retries independently. Real systems have branching and early successes, but the lesson holds: retry placement is an architectural decision.

Choose one retry owner for a call path. Usually it is the layer that:

- knows whether the operation is idempotent;
- has enough remaining deadline;
- understands the user-visible outcome;
- can observe the full attempt history.

A retry policy needs more than `maxAttempts`:

```yaml
retry:
  retryable: [UNAVAILABLE, RESOURCE_EXHAUSTED]
  maxAttempts: 3
  perAttemptTimeout: 120ms
  backoff: exponential
  jitter: full
  budget: 10% of baseline traffic
```

The retry budget is the critical line. It limits retry traffic relative to healthy request volume. When the budget is exhausted, fail fast instead of converting a dependency incident into an overload incident.

## Only retry a safe semantic operation

“POST is not idempotent” is too crude, and “our handler is idempotent” is usually too optimistic. Retry safety belongs to the business effect.

Creating a payment can be retried only if every attempt carries the same idempotency key and the receiver persists the resulting outcome:

```text
Idempotency-Key: checkout_92f1_payment_v1
```

The key must identify the logical operation, not the network attempt. If a timeout leaves the outcome ambiguous, query by that key before issuing a new effect.

Reads are not automatically harmless either. A heavy analytical query retried after a client timeout may double the exact load that caused the timeout. Safety includes capacity, not just data mutation.

## Backoff without jitter synchronizes failure

Exponential backoff spaces attempts, but identical clients still wake together. Jitter randomizes the delay so recovery traffic arrives as a slope rather than a wall. AWS’s Builders’ Library describes this as a core technique for avoiding correlated retry storms.

Use server hints where available (`Retry-After`, explicit overload metadata), but cap them by the remaining deadline. A request with 90 ms left cannot honor a 2-second retry recommendation.

```text
remaining = deadline - now
delay = min(full_jitter(base * 2^attempt), server_hint, remaining - execution_margin)
```

If the remaining time cannot fund both delay and a meaningful attempt, do not retry.

## Measure deadline economics

Average latency will hide the failure mode. Operate the policy with:

- end-to-end deadline-exceeded rate;
- remaining budget at each service entry and exit;
- attempts per logical operation;
- retry success rate by attempt number;
- retry volume as a percentage of baseline traffic;
- cancelled work that continued executing;
- dependency latency distributions by outcome;
- load-shed responses versus accidental timeouts.

A high retry-success count is not automatically good. It may reveal a dependency that is unreliable enough to require constant hidden duplication.

## Common mistakes

**One timeout everywhere.** Different work has different value and cost; the only shared value should be the inherited deadline.

**Retrying at every layer.** This creates multiplicative traffic and destroys causal evidence.

**Timing out without cancelling.** The caller leaves while resource consumption continues.

**Retrying non-idempotent effects.** An ambiguous failure becomes a duplicate business action.

**Using circuit breakers without admission control.** A breaker can reduce calls to one dependency, but it does not ensure the rest of the service can survive the queued demand.

**Treating p99 as a timeout.** A timeout must include the end-to-end objective, downstream cost, false-timeout tolerance, and recovery policy—not merely yesterday’s percentile.

## CTO review questions

1. What user-visible deadline is the system protecting?
2. How is remaining time propagated across protocols and queues?
3. Which layer owns retries, and what prevents amplification?
4. Which business operations have durable idempotency keys?
5. What happens to work after its caller disappears?
6. Can we distinguish healthy retries from retry-driven overload?

Timeouts and retries are not resilience decorations. They are a distributed resource-allocation protocol. Design them with the same care as a database schema, because during an incident they decide who waits, who retries, and whether the system gets a chance to recover.

## References

- [gRPC: Deadlines](https://grpc.io/docs/guides/deadlines/)
- [gRPC: Retry](https://grpc.io/docs/guides/retry/)
- [AWS Builders’ Library: Timeouts, retries, and backoff with jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/)
- [Google SRE: Handling Overload](https://sre.google/sre-book/handling-overload/)
- [Related: Building Microservices That Fail Gracefully](/posts/building-reliable-microservices)


---

# DNS Failover Is a Bounded-Staleness Protocol

> How recursive caches, TTL, negative caching, resolver behavior, health checks, and connection reuse determine the real recovery time of DNS-based failover.

Canonical URL: https://www.ayushworks.xyz/posts/dns-failover-is-bounded-staleness
Author: Ayush Basak
Last modified: 2026-08-31
Topics: dns, networking, reliability, cloud-infrastructure

Changing an A record is not the moment traffic moves. DNS answers are copied through recursive resolvers, operating-system caches, application runtimes, and sometimes connection pools. Every layer can continue using an old address after the authoritative record changes.

The invariant is:

> A DNS failover plan must tolerate both old and new destinations serving concurrently for at least the maximum effective cache and connection lifetime.

## TTL grants permission to be stale

An authoritative answer carries a time to live. A recursive resolver may reuse it until that TTL expires without asking the authority again. If a record with a 300-second TTL was cached one second before failover, that resolver can legally return the old value for almost five more minutes.

```text
authoritative change at t=0
resolver A cached old answer at t=-299 → expires at t=1
resolver B cached old answer at t=-1   → expires at t=299
```

There is no single global propagation moment. Different resolvers expire at different times.

Lower TTL before a planned migration, wait at least the prior TTL, then change the record. Lowering the TTL at the same moment as the address does not affect caches already holding the old longer TTL.

## The effective lifetime can exceed DNS TTL

Cloudflare's documentation notes that local caches may take longer than the configured TTL to reflect changes. Applications may resolve once at startup. JVM and language runtimes can maintain their own DNS policies. HTTP keep-alive, HTTP/2, gRPC, and database pools can keep an existing connection alive without resolving again.

Therefore measure:

```text
effective failover time = max(
  recursive cache lifetime,
  local/runtime cache lifetime,
  live connection lifetime,
  health-detection + control-plane update time
)
```

DNS TTL is only one term.

## Negative answers are cached too

RFC 2308 defines caching for NXDOMAIN and NODATA responses using information from the zone's SOA record. Accidentally querying a name before it exists can seed negative caches; creating the record immediately afterward does not force those resolvers to forget the earlier nonexistence.

RFC 9520 extends required negative caching to resolution failures such as SERVFAIL and DNSSEC validation failure, specifically to prevent retry storms against broken authoritative infrastructure. During an outage, resolvers remembering failure can protect the DNS system while delaying visible recovery for that cache window.

Treat positive TTL, negative TTL, and failure-caching behavior as separate controls.

## DNS health is not application health

A health check may remove an address because one probe path failed while existing connections still work. It may retain an address whose TCP port accepts connections but whose critical dependency is broken. A flapping check can churn answers faster than caches can converge, creating a mixed fleet no operator can reason about.

Use thresholds and a health signal that represents the traffic class being routed. Keep the old destination capable of safe service during the overlap. If writes cannot safely go to both regions, DNS alone is the wrong coordination mechanism; put a strongly controlled routing or data-leadership boundary behind it.

## A safe migration timeline

1. Inventory authoritative, negative, runtime, and connection cache lifetimes.
2. Lower the TTL early and wait out the previous TTL.
3. Bring up the new destination and verify it independently.
4. Ensure old and new destinations can coexist safely.
5. Change the record and observe traffic at both sites.
6. Keep the old destination until the measured tail reaches zero plus margin.
7. Raise TTL again after stability.

Never destroy the old endpoint immediately after the DNS control plane reports success.

## What to observe

Query authoritative logs by answer, sample major recursive resolvers, measure traffic reaching old and new endpoints, track connection age, separate resolution errors from connection errors, and test NXDOMAIN recovery. Synthetic checks should query through real recursive paths, not only the authoritative API.

## The CTO decision

Use DNS failover when minutes of mixed routing are acceptable and both destinations remain safe during overlap. For sub-second traffic control, stateful session movement, or single-writer databases, use a routing layer and explicit data failover protocol behind a stable name.

DNS is excellent distributed caching. Its failure behavior follows from that strength.

## References

- [RFC 2308: Negative Caching of DNS Queries](https://www.rfc-editor.org/rfc/rfc2308)
- [RFC 9520: Negative Caching of DNS Resolution Failures](https://www.rfc-editor.org/rfc/rfc9520)
- [Cloudflare DNS: Time to Live](https://developers.cloudflare.com/dns/manage-dns-records/reference/ttl/)
- [Related: Service identity before service mesh](/posts/service-identity-before-service-mesh)


---

# Durable AI Agents Need Workflow Semantics, Not Chat Loops

> A production architecture for replayable agent state, idempotent tools, human approval, versioning, budgets, and recovery across long-running AI work.

Canonical URL: https://www.ayushworks.xyz/posts/durable-ai-agent-workflows
Author: Ayush Basak
Last modified: 2026-08-24
Topics: ai-agents, ai-infrastructure, durable-execution, system-design

An agent demo is a loop:

```text
prompt -> model -> tool -> model -> answer
```

A production agent is a long-running distributed workflow in which model calls are nondeterministic, tools have side effects, humans may approve work hours later, credentials expire, deployments change code, and any process can crash between “effect happened” and “state recorded.”

The core architecture question is not which framework can call tools. It is: **what durable state lets the system explain, resume, and safely compensate every step?**

## Separate reasoning from execution

Treat the model as a planner that proposes the next action, not as the source of truth for completed actions.

```text
goal
  -> planner proposes action
  -> policy checks authority and budget
  -> workflow records intent
  -> tool executor performs effect
  -> workflow records evidence
  -> planner receives normalized result
```

The workflow owns lifecycle state. The model never decides that an email was sent merely because it asked a tool to send one.

A minimal state model might be:

```ts
type RunState = {
  runId: string
  goal: string
  status: 'running' | 'waiting_approval' | 'completed' | 'failed'
  policyVersion: number
  modelPolicy: { provider: string; model: string; maxTokens: number }
  budget: { tokens: number; toolCalls: number; moneyCents: number }
  steps: StepEvidence[]
}
```

Store prompts and model outputs according to an explicit privacy policy. For many systems, retaining structured decisions, hashes, redacted arguments, and tool evidence is safer than retaining every raw conversation indefinitely.

## Durable execution changes the failure model

Workflow systems such as Temporal persist an event history and reconstruct workflow state after worker failure. The important property is not “the process runs forever.” It is that the orchestration can resume from recorded history rather than improvising from a partially updated row.

Keep workflow logic deterministic. Put network calls, model invocations, clocks, randomness, and side effects in activities:

```text
workflow (replayable decisions)
  activity: call model
  activity: fetch account
  wait: human approval signal
  activity: update CRM
  activity: send notification
```

An activity may execute more than once if a worker completes the external effect and crashes before recording completion. Therefore durable orchestration does not remove idempotency; it tells you exactly where idempotency is required.

## Give every effect a stable identity

Tool calls need an operation ID derived from durable workflow state, not generated anew on every attempt:

```text
operation_id = run_id + step_number + tool_contract_version
```

Pass it to downstream systems as an idempotency key. Persist the request fingerprint and outcome:

```sql
INSERT INTO tool_effects(operation_id, tool, request_hash, status)
VALUES ($1, $2, $3, 'started')
ON CONFLICT (operation_id) DO NOTHING;
```

If an external tool does not support idempotency, define an ambiguity strategy:

- query the remote system for evidence;
- require human reconciliation;
- make the effect append-only and compensatable;
- or prohibit automatic retry.

“Retry the tool” is not a universal recovery policy.

## Authorization belongs at execution time

The model’s proposed arguments are untrusted input. Tool descriptions and retrieved documents are also untrusted because they may contain prompt injection.

Authorize the concrete action immediately before execution:

```text
principal: sales_rep_17
tenant: acme
tool: crm.update_contact
resource: contact/882
fields: [next_follow_up]
reason: approved campaign workflow
policy_version: 42
```

The Model Context Protocol authorization specification uses OAuth-based patterns for protected servers and explicitly defines resource-server behavior. Regardless of protocol, do not hand a general-purpose bearer token to the model. The executor should hold credentials, bind them to a trusted service identity, and expose only policy-filtered capabilities.

Recheck authorization after a long wait. A user who approved an action yesterday may have lost access today; a customer may have revoked the integration; a policy may have changed.

## Human approval is a durable state

Do not implement approval as an HTTP request waiting for a button click. Record a transition:

```text
proposed -> awaiting_approval -> approved -> executing -> evidenced
                        \-> rejected
                        \-> expired
```

The approval request should display the actual effect: recipient, amount, changed fields, data leaving the system, and cost. An approval for “continue” is not informed consent.

Bind approval to a hash of the proposed action. If the planner changes arguments after approval, require a new decision. Set an expiry and record who approved, under which role and policy version.

## Version workflows without corrupting open runs

Agent runs can outlive deployments. A new prompt, tool schema, or branching rule can make replay diverge or reinterpret old state.

Version at least:

- workflow code paths;
- system prompt and model policy;
- tool contracts;
- authorization policy;
- structured output schemas;
- evaluation rubric.

Existing runs should continue on compatible behavior or pass through an explicit migration. “Deploy and hope no run is between steps” is not a release strategy.

Temporal documents workflow replay and versioning constraints because nondeterministic code changes can break reconstruction. The same issue exists in home-grown orchestrators; it is simply less visible until recovery fails.

## Budget loops before they become incidents

Bound every dimension that can grow:

- model tokens and calls;
- wall-clock duration;
- tool calls by class;
- external spend;
- retrieved bytes;
- repeated planning without state change;
- delegated child agents;
- approval wait time.

Detect cycles semantically. Three differently worded plans that call the same tool with the same arguments are one stuck loop.

```text
if same_effect_fingerprint >= 2 and no_new_evidence:
    stop(reason="non-progressing loop")
```

Budgets should be visible to the planner but enforced outside it.

## Observe evidence, not chain-of-thought

You do not need private reasoning traces to operate an agent. You need structured evidence:

- run and step IDs;
- model and prompt policy versions;
- input/output token counts;
- proposed tool and normalized arguments;
- policy decision and reason code;
- approval state;
- tool latency, retries, and outcome;
- external effect identifier;
- final outcome and evaluation result.

OpenTelemetry’s generative-AI semantic conventions provide a developing vocabulary for model and agent telemetry. Treat the conventions as evolving, and keep sensitive content capture opt-in.

## CTO review questions

1. What durable record proves each side effect happened?
2. Which operations can execute more than once, and how are they deduplicated?
3. Is authorization evaluated against the concrete action at execution time?
4. Can a human see and approve the exact effect?
5. Can open runs survive a deployment and policy change?
6. What stops a non-progressing loop?
7. Can we reconstruct an incident without storing unnecessary private reasoning?

The reliable agent is not the one with the cleverest loop. It is the one whose progress is durable, whose authority is bounded, whose effects are evidenced, and whose failures produce a recoverable state instead of a mystery.

## References

- [Temporal: Workflow Execution](https://docs.temporal.io/workflow-execution)
- [Model Context Protocol: Authorization](https://modelcontextprotocol.io/specification/2025-06-18/basic/authorization)
- [OpenTelemetry: Generative AI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
- [Related: AI Agents Need a Control Plane](/posts/ai-agent-control-plane-mcp-security)


---

# Durable Workflows Do Not Remove the Need for Idempotency

> How replay-based workflow engines recover orchestration, where activity ambiguity remains, and how to design deterministic workflows with idempotent effects.

Canonical URL: https://www.ayushworks.xyz/posts/durable-workflows-do-not-remove-idempotency
Author: Ayush Basak
Last modified: 2026-08-30
Topics: distributed-systems, durable-execution, backend-engineering, ai-agents

Durable execution can recover orchestration after a worker crash by replaying an event history. It cannot reach into a payment provider, email server, or model API and make an ambiguous external effect exactly once.

The invariant is:

> Workflow decisions replay deterministically; every external effect is independently safe to repeat or reconcile.

Confusing those two guarantees creates workflows that resume reliably and duplicate money just as reliably.

## Replay restores decisions

In a replay-based engine such as Temporal, the service persists workflow events. A worker re-executes workflow code and matches generated commands against recorded history. Completed activity results are supplied from history rather than invoking the activity again.

This requires workflow code to be deterministic. Wall-clock reads, randomness, network calls, and mutable process state can produce a different command sequence during replay. Use workflow-provided clocks and randomness, and move external I/O into activities.

```text
workflow code: deterministic orchestration and policy
event history: durable decisions and activity outcomes
activities: non-deterministic external effects
```

## The activity boundary is still ambiguous

Consider an activity that charges a card:

1. the provider commits the charge;
2. the worker loses its connection before reporting completion;
3. the engine sees no completion event;
4. the activity is retried.

No orchestration engine can infer the provider's state from a missing acknowledgement. The activity needs a stable idempotency key derived from workflow and business command identity.

```ts
await payments.charge({
  orderId,
  amount,
  idempotencyKey: `charge:${orderId}`,
})
```

If the provider lacks idempotency, record an intent locally, execute with a durable operation ID where possible, and reconcile ambiguous results before another attempt.

## Timeouts describe different failures

One timeout cannot express queue delay, execution time, and the total retry horizon. A production activity policy separates:

| Boundary | Meaning |
|---|---|
| schedule-to-start | worker capacity or task-queue delay |
| start-to-close | one attempt exceeded its execution budget |
| schedule-to-close | the complete activity, including retries, is no longer useful |
| heartbeat | a long activity stopped proving progress |

Retry only failures classified as transient. Authorization, validation, and policy denial should be terminal. Backoff and jitter protect the unhealthy dependency; maximum attempts or total timeout protect the business deadline.

## History is not a data lake

The event history is operational state. Large payloads, token streams, and unbounded loops inflate replay cost and can reach platform history limits. Temporal documents event-count and signal/update limits and provides Continue-As-New to start a fresh history while carrying forward compact state.

For an AI agent, store large transcripts and artifacts in an external durable store. Put references, tool decisions, approvals, and compact outcomes in workflow state. Model calls and tool calls belong in activities because they are non-deterministic and may have external cost or side effects.

## Deployment requires replay compatibility

A workflow can live longer than one application release. Changing its code may cause old histories to produce new commands during replay. Use the platform's versioning mechanisms or pin compatible workers. Test replay against sampled production histories before rollout.

The deployment question is not only “does the new code pass unit tests?” It is “does the new code still explain every active execution's past?”

## Production review

- Is workflow code deterministic?
- Does every activity have a stable business identity?
- Are retryable failures explicitly classified?
- Do timeouts reflect queue, attempt, and total lifetime separately?
- Can long activities heartbeat and resume from checkpoints?
- Is history growth bounded?
- Can the new worker replay old histories?
- Can operators pause, retry, terminate, and audit an execution safely?

## The CTO decision

Use durable workflows when a business process spans failures, time, human input, or several services and deserves a first-class execution record. Do not use them to disguise poorly defined side effects. Durable orchestration reduces state-machine plumbing; idempotency and reconciliation still protect the world outside the engine.

## References

- [Temporal: Events and Event History](https://docs.temporal.io/workflow-execution/event)
- [Temporal: Activities](https://docs.temporal.io/activities)
- [Temporal: Workflow Definitions](https://docs.temporal.io/workflow-definition)
- [Snippet: Idempotent durable activity](/snippets/durable-activity)


---

# Embedding Upgrades Are Data Migrations

> Roll out a new embedding model with versioned vectors, dual indexes, measured retrieval quality, and a reversible cutover.

Canonical URL: https://www.ayushworks.xyz/posts/embedding-upgrades-are-data-migrations
Author: Ayush Basak
Last modified: 2026-09-05
Topics: ai-systems, embeddings, vector-search, data-engineering

Changing an embedding model changes the coordinate system in which similarity is computed. Even when old and new models emit vectors with the same dimension, distances across model versions have no promised meaning.

An embedding upgrade is therefore not a configuration flip. It is a data migration with an online serving path, a backfill, a quality gate, and a rollback plan.

## Version the representation

Store enough identity to reproduce and interpret every vector:

```sql
CREATE TABLE document_embedding (
  document_id     bigint NOT NULL,
  model_id        text NOT NULL,
  model_revision  text NOT NULL,
  chunker_version text NOT NULL,
  content_hash    text NOT NULL,
  embedding       vector NOT NULL,
  created_at      timestamptz NOT NULL,
  PRIMARY KEY (document_id, model_id, model_revision, chunker_version)
);
```

The chunker belongs in the version because changing chunk boundaries changes what each vector represents. A content hash makes backfill idempotent and prevents recomputing unchanged material.

Do not overwrite the old vector in place. The existing index is your rollback path and your experimental control.

## Separate migration phases

### 1. Offline evaluation

Build a query set from real traffic, support failures, and important edge cases. Label relevant results. Compare recall, ranking metrics, latency, and downstream answer quality by cohort—not only as one average.

### 2. Dual write

New or changed documents receive both representation versions. Observe generation failures and queue lag. The old read path remains authoritative.

### 3. Backfill

Scan by stable cursor, claim bounded batches, rate-limit model calls, and record progress durably. Backfill must yield to live indexing traffic.

```text
eligible = source updated AND target version absent
claim batch -> embed -> validate dimension -> persist -> checkpoint
```

Retries should use the same `(document, model, revision, chunker)` key so they converge rather than duplicate work.

### 4. Shadow retrieval

Run the new retriever without serving its result. Log candidate overlap, latency, empty-result rate, and disagreements on a sampled fraction of traffic. Redact or hash sensitive query material.

### 5. Gradual cutover

Route a small cohort to the new index. Hold a rollback switch at the retrieval-router boundary. Increase exposure only when technical and product metrics remain inside agreed thresholds.

### 6. Retire deliberately

After the rollback window, remove old vectors and indexes through a retention job. Confirm that no consumer still requests the old version before reclaiming storage.

## Failure boundaries

| Failure | Safe behavior |
| --- | --- |
| model API unavailable | preserve old reads; retry bounded backfill |
| dimension mismatch | reject write before index insertion |
| partial backfill | route by complete version, not whichever row exists |
| new model quality regression | flip retrieval router back |
| source changes during backfill | compare content hash and requeue |
| index build fails | keep dual-written source rows for rebuild |

## What to measure

Track coverage by version, oldest unembedded update, backfill throughput, embedding cost, index size, query latency, recall@k, reranker outcomes, grounded-answer rate, and user-task success. An upgrade that improves a benchmark while worsening filtered retrieval or multilingual queries is not complete.

## Trade-offs

Dual storage temporarily increases cost. Shadow traffic consumes retrieval capacity. A fast in-place rewrite is cheaper in the short term but removes comparison and rollback. Versioning adds columns and routing logic, while turning an opaque model change into an observable, reversible operation.

Embedding systems are data systems. Give their migrations the same discipline as a database schema change.

## Further reading

- [OpenAI embeddings guide](https://platform.openai.com/docs/guides/embeddings)
- [pgvector indexing and distance functions](https://github.com/pgvector/pgvector)
- [Production AI needs release gates](/posts/production-ai-risk-gates)


---

# An Error Budget Should Change the Roadmap

> How technical leaders turn SLOs and error budgets into an explicit policy for releases, reliability investment, incidents, and product tradeoffs.

Canonical URL: https://www.ayushworks.xyz/posts/error-budgets-engineering-investment-policy
Author: Ayush Basak
Last modified: 2026-08-28
Topics: reliability, technical-leadership, platform-engineering, observability, system-design

An SLO that only appears on a dashboard is a reporting artifact. An SLO becomes an operating mechanism when its error budget changes what the organisation is allowed to do next.

Google SRE defines an error budget as the permitted unreliability implied by an SLO. A 99.9% success target permits 0.1% failure over the measurement window. The useful part is not the arithmetic; it is the agreement between product and engineering about when feature risk is acceptable and when reliability work takes priority.

## Measure a user outcome

Begin with a service-level indicator that represents completed user work. CPU utilization and pod health are diagnostic signals, not availability.

For a checkout API:

```text
good events  = valid checkout attempts completed correctly within 2 seconds
valid events = checkout attempts the system accepted responsibility for
SLI          = good events / valid events
```

Define exclusions narrowly. If every dependency failure, client retry, or overload response is excluded, the SLO measures the team’s ability to classify errors rather than the customer experience.

Use more than one SLI only when the product promise truly has multiple dimensions. Availability and latency often deserve separate objectives; twenty indicators make ownership ambiguous.

## Pick the objective from product consequences

Do not choose 99.99% because it looks serious. Higher targets reduce the budget for deployments and raise infrastructure and operating cost. Ask:

- How long can the workflow be unavailable before customers take another path?
- Is failure recoverable through retry or reconciliation?
- Does the system move money, control safety-critical equipment, or serve convenience?
- Can support and operations mitigate the failure manually?
- What reliability can dependencies actually sustain?

A dependency with a weaker promise can make a stronger end-to-end SLO fictional unless the architecture adds redundancy or graceful degradation.

## Translate the budget into decisions

Suppose a service receives 10 million valid events in 28 days at a 99.9% SLO:

```text
allowed bad events = 10,000,000 × (1 - 0.999)
                   = 10,000
```

Track remaining budget and burn rate. A slow burn predicts exhaustion before the window ends; a fast burn detects incidents before the total budget looks large.

A policy should say what happens at thresholds. For example:

| State | Condition | Response |
|---|---|---|
| Healthy | projected consumption < 50% | normal releases |
| At risk | projected consumption 50–100% | reduce risky change; fund named reliability work |
| Exhausted | budget consumed | pause non-essential releases; reliability owner controls exceptions |
| Severe incident | one event consumes > 20% | postmortem and highest-priority corrective action |

The exact numbers should match the organisation. The important property is that they are agreed before an incident.

## Do not turn the policy into punishment

Freezing every change can prolong an incident because fixes, observability improvements, and risk-reducing deployments are changes too. Classify changes:

- reliability restoration and security fixes;
- low-risk, reversible changes;
- ordinary product releases;
- migrations or architectural changes with large blast radius.

When the budget is exhausted, stop the last two categories by default, not the first two. Permit exceptions through a named accountable role with written reasoning and rollback criteria.

## Make ownership cross-functional

Product must participate because the policy reallocates roadmap capacity. Engineering must participate because it owns the failure model. Finance may need to participate when reliability requires material redundancy. Support contributes evidence about customer impact that request metrics miss.

A monthly reliability review should cover:

1. SLO performance and remaining budget;
2. dominant classes of bad events;
3. incidents and near misses;
4. reliability work completed versus promised;
5. upcoming changes that alter risk;
6. objectives that no longer represent product reality.

This is portfolio governance, not merely an SRE meeting.

## Common failure modes

**Aspirational SLOs.** If no action follows a miss, teams learn the target is optional.

**Infrastructure-only indicators.** A healthy load balancer can return fast errors while the business workflow is unusable.

**One objective for every customer.** Free and enterprise tiers may have different promises, but the architecture and commercial agreement must support that distinction.

**Monthly averages without burn rate.** A severe current outage can hide inside a large remaining monthly budget.

**Permanent release freezes.** A budget policy should restore balanced delivery, not create an indefinite reliability programme with no exit criteria.

## The leadership test

Ask one question in the roadmap meeting: “If this service exhausts its error budget tomorrow, what work stops and who decides when it resumes?”

If nobody can answer, the SLO is documentation. When the answer is explicit, the organisation has converted reliability from opinion into a decision system.

## References

- [Google SRE Workbook: Example Error Budget Policy](https://sre.google/workbook/error-budget-policy/)
- [Google SRE Workbook: Implementing SLOs](https://sre.google/workbook/implementing-slos/)
- [Related: Observability Is a Data Contract](/posts/opentelemetry-observability-data-contract)
- [Related: Verify What You Deploy](/posts/verify-what-you-deploy)


---

# External Consistency Explained: Spanner, TrueTime, and Commit Wait

> What Spanner's TrueTime and commit wait teach architects about real-time ordering, multi-region writes, locality, and honest consistency requirements.

Canonical URL: https://www.ayushworks.xyz/posts/external-consistency-has-a-latency-budget
Author: Ayush Basak
Last modified: 2026-09-11
Topics: distributed-databases, multi-region, consistency, system-design

**External consistency means transactions behave as if they executed serially in an order that respects real time.** If transaction A finishes before B starts, that serial order must place A before B. Overlapping transactions do not have the same real-time ordering constraint.

Google Spanner documents this guarantee through [TrueTime and external consistency](https://docs.cloud.google.com/spanner/docs/true-time-external-consistency). The design question is where a product needs this ordering and what latency it can afford.

External consistency provides that guarantee. It is powerful—and it has a latency, replication, and operational cost.

The invariant is:

> Once a committed result is acknowledged, no later transaction may be ordered before it.

## TrueTime represents uncertainty honestly

Serializable transactions alone can admit a serial order that differs from real time. External consistency adds the real-time requirement, commonly called strict serializability. Linearizability expresses a related real-time condition for individual operations; do not assume that a linearizable register automatically provides multi-object transactions.

Google Spanner's TrueTime API returns an interval in which absolute time is guaranteed to lie, rather than pretending a machine clock is exact. Spanner assigns a commit timestamp and waits until the uncertainty interval has passed before exposing the commit. This “commit wait” ensures the timestamp is in the past when the client receives success, preserving real-time order.

```text
choose commit timestamp s
replicate and commit transaction at s
wait until TT.after(s)
acknowledge success
```

TrueTime is not merely synchronized clocks, and clocks alone do not provide transaction isolation. Spanner also uses concurrency control, replication, and MVCC. TrueTime connects the serialization order to externally observed time.

## The guarantee changes application design

With external consistency, a service can write a policy, return success, and know that a later strong read will not observe an older world that excludes that policy. Cross-region workflows become easier to reason about because the database behaves like one serial machine with real-time ordering.

But not every read needs the newest state. Spanner supports timestamp-bound reads, including stale reads, which can reduce latency or move work closer to replicas. The architectural move is to classify reads by correctness need:

| Read | Required semantics |
|---|---|
| confirm a payment immediately after commit | strong, read-your-write path |
| authorize using a newly revoked credential | strong or tightly bounded staleness |
| render yesterday's analytics | stale snapshot is acceptable |
| build a historical report | exact timestamp or bounded-staleness read |

Paying the strongest consistency cost for every analytics query wastes latency and capacity. Using stale reads for authorization silently weakens the security model.

## Geography remains physics

A synchronous replicated write must communicate with enough replicas to commit. Leader placement and quorum geography shape latency. TrueTime reduces some coordination needs for timestamp assignment and supports efficient strong reads, but it does not remove wide-area round trips or the speed of light.

Before selecting a multi-region database configuration, place the write leader near the dominant writers, model cross-region quorum latency, and quantify recovery objectives. A globally distributed logo on an architecture diagram is not evidence of a globally low-latency write path.

## Model an application-level ordering token

Even with a strongly ordered database, messages and caches can arrive out of order. Carry a monotonic commit version or timestamp across derived systems and reject regressions:

```ts
function applyPolicy(current: Policy, incoming: Policy) {
  if (incoming.commitVersion <= current.commitVersion)
    return current
  return incoming
}
```

The token does not manufacture consistency. It allows downstream components to preserve the database's ordering instead of overwriting new state with delayed old events.

## Common mistakes

1. Treating “serializable” and “external consistency” as synonyms; serializability alone need not respect real-time order.
2. Assuming global replication means local write latency everywhere.
3. Using wall-clock timestamps from ordinary application hosts as ordering authority.
4. Mixing strong database writes with unordered cache invalidation and declaring the whole system strongly consistent.
5. Selecting the guarantee before identifying which business invariant requires it.

## The CTO decision

Specify consistency per decision, not per product. If real-time transaction order eliminates dangerous application reconciliation, external consistency may justify its cost. If a workflow can tolerate bounded staleness, use that freedom deliberately. Measure end-to-end correctness through caches, queues, and read paths—not only the database contract.

The strongest system is not the one with the strongest setting. It is the one whose consistency cost matches the invariant it protects.

## References

- [Spanner: Google's Globally Distributed Database](https://storage.googleapis.com/gweb-research2023-media/pubtools/pdf/44915.pdf)
- [Google Cloud: TrueTime and External Consistency](https://docs.cloud.google.com/spanner/docs/true-time-external-consistency)
- [Google Cloud: Spanner Reads](https://cloud.google.com/spanner/docs/reads)
- [Snippet: Monotonic version gate](/snippets/monotonic-version-gate)


---

# Feature Flags Are Production State

> A CTO-level operating model for typed evaluation, safe defaults, targeting privacy, rollout evidence, flag debt, kill switches, and control-plane failure.

Canonical URL: https://www.ayushworks.xyz/posts/feature-flags-are-production-state
Author: Ayush Basak
Last modified: 2026-08-26
Topics: feature-delivery, reliability, platform-engineering, technical-leadership

Feature flags decouple deployment from release. They also create a second production configuration system capable of changing behavior without a code review, build, or deployment.

Treating flags as temporary booleans hides their real role: they are versioned production state with owners, permissions, failure modes, and retirement obligations.

## Classify the flag before creating it

Different flags require different controls:

| Type | Purpose | Expected lifetime |
|---|---|---:|
| release | progressive rollout | days or weeks |
| experiment | measure variants | bounded by analysis |
| operational | degrade or reroute | long-lived |
| permission | contractual entitlement | product lifetime |
| emergency | disable dangerous path | long-lived, rarely changed |

Do not use a release flag as an entitlement system. Do not use an experiment flag as a security boundary. Classification defines who may change it, its default, required telemetry, and removal date.

Store metadata with the flag:

```yaml
key: checkout.new-tax-engine
type: release
owner: team-commerce
created: 2026-08-26
expires: 2026-09-30
safe_default: false
rollback_signal: tax_error_rate
ticket: COM-1842
```

A flag without an owner and expiry is deferred code complexity.

## Make evaluation typed and deterministic

OpenFeature defines typed evaluation methods with a caller-supplied default. The type and default are part of the application contract.

```ts
const enabled = client.getBooleanValue(
  'checkout.new-tax-engine',
  false,
  { targetingKey: accountId, region },
)
```

Choose defaults by failure analysis:

- new optional UI: default off;
- safety rate limit: last known value or conservative limit;
- dangerous integration: default disabled;
- purchased entitlement: use durable entitlement authority, not an unavailable experiment provider.

Evaluate once per operation and pass the decision through the call path. Re-evaluating at several layers can produce mixed behavior if configuration changes mid-request.

Fractional rollout must be sticky. Hash a stable, non-sensitive targeting key with the flag key so the same subject receives the same variant. Random evaluation per request destroys user experience and makes results uninterpretable.

## Minimize targeting data

Evaluation context may include users, services, regions, or hosts. OpenFeature defines a targeting key and custom fields; it does not require sending an entire customer record.

Send only attributes needed by rules. Avoid email, names, access tokens, and free-form profiles. Document where context is evaluated and retained. Client-side evaluation can expose flag rules and targeting data to the browser; use server-side evaluation for sensitive policy.

Tenancy is non-negotiable. If a rule targets `account_id`, obtain it from authenticated context rather than a request parameter the caller can forge.

## Make rollout a sequence of evidence

A safe rollout is not “10%, then 50%, then 100%.” Each step has entry and rollback conditions.

```text
internal → 1% → 5% → 25% → 50% → 100% → remove flag
```

At each stage compare candidate and control across:

- business success;
- errors and saturation;
- p95 and p99 latency;
- database and dependency load;
- cost per successful outcome;
- segment-specific harm.

Percentage alone can hide concentration. One percent of traffic may contain no enterprise tenant or large account. Include named cohorts for risky boundaries.

Record the evaluated variant in traces and business events. OpenFeature’s observability guidance maps evaluation details such as flag key, variant, and reason into telemetry. Do not attach high-cardinality or sensitive evaluation context indiscriminately.

## Design for provider failure

Flag evaluation may use local snapshots, streaming updates, or synchronous remote calls. A remote network call in every customer request creates a new critical dependency.

Prefer locally evaluated, versioned snapshots for latency-sensitive decisions. Keep the last known-good snapshot, define its validity window, and expose its age.

```text
flag provider unavailable
  → use cached snapshot if valid
  → otherwise use declared safe default
  → emit degraded evaluation telemetry
```

Operational kill switches need a tested path during provider impairment. If the same control plane is down, can on-call still disable the feature? Consider a small independent emergency override with narrow scope and audited access.

## Prevent combinatorial states

Five interacting booleans create up to 32 combinations. Most will never be tested.

Reduce interaction by:

- grouping mutually exclusive behavior into a typed variant;
- forbidding one flag from changing the meaning of another;
- testing supported combinations explicitly;
- keeping evaluation near one architectural boundary;
- retiring release flags quickly.

```text
bad: use_new_api + use_new_schema + use_new_cache
better: checkout_architecture = legacy | shadow | v2
```

Structured variants express the state machine more honestly.

## Remove the flag completely

At 100%, the work is not done. Delete:

- the old branch;
- flag evaluation calls;
- control-plane configuration;
- obsolete tests and metrics;
- fallback schemas and dependencies;
- dashboards that no longer mean anything.

Automate expiry reporting. Block creation without an owner. Track median flag age by type, and place removal in the original delivery plan rather than a future cleanup backlog.

Before deleting, verify there are no evaluations for the old variant across scheduled jobs, mobile versions, or dormant tenants. Telemetry plus a consumer inventory is stronger than code search alone.

## CTO review

1. What class of flag is this, and when does it expire?
2. Is the default safe during provider failure?
3. Is targeting deterministic, authenticated, and privacy-minimized?
4. Which evidence advances or reverses each rollout stage?
5. Can combinations produce untested system states?
6. Does the data plane require a synchronous control-plane call?
7. Is the kill switch available during a control-plane outage?
8. Who removes both branches and the flag?

Flags make release safer only when their own lifecycle is engineered. Otherwise they move deployment risk into an invisible, mutable control plane.

## References

- [OpenFeature: Flag Evaluation API](https://openfeature.dev/specification/sections/flag-evaluation/)
- [OpenFeature: Evaluation Context](https://openfeature.dev/specification/sections/evaluation-context/)
- [OpenFeature: Observability](https://openfeature.dev/specification/appendix-d/)
- [Related: Control Planes Must Fail Quietly](/posts/control-planes-must-fail-quietly)
- [Related: AI Evals Are Release Engineering](/posts/ai-evals-are-release-engineering)


---

# Your Lock Expired. Your Worker Did Not.

> Understand stale-owner writes, monotonically increasing fencing tokens, atomic storage checks, and why leases alone cannot protect an external side effect.

Canonical URL: https://www.ayushworks.xyz/posts/fencing-stale-writers-at-the-storage-boundary
Author: Ayush Basak
Last modified: 2026-09-04
Topics: distributed-systems, concurrency, backend-engineering, reliability

A worker acquires a lease, pauses, and resumes after the lease expires. A second worker has already acquired ownership. The lock service may be behaving perfectly while both processes believe they are allowed to write.

Expiry revokes permission in the coordination service. It does not stop a CPU, recall a network packet, or cancel a request already buffered by a proxy.

> The resource receiving a write must reject obsolete ownership; the worker's belief about its lease is insufficient.

This is the distinction between controlling who should work and controlling whose effects are accepted.

## Walk through the race

```text
worker A acquires epoch 41
A pauses before sending its write
lease expires
worker B acquires epoch 42
B writes with epoch 42
A resumes and writes with epoch 41
```

If storage ignores ownership, A can overwrite B. Extending the lease reduces the frequency of the race but does not prove safety under long pauses or delayed messages.

[Hazelcast's FencedLock documentation](https://docs.hazelcast.com/hazelcast/5.5/data-structures/fencedlock) illustrates the stale-client problem and the use of increasing fencing tokens. The essential enforcement happens at the protected resource, not in the lock acquisition response alone.

## Define the storage contract

A fencing token is an ordered ownership epoch issued by a coordinator with the required consistency guarantees. The resource remembers the greatest accepted epoch and rejects smaller ones.

Do not substitute a random lock-owner UUID. Random identity can support safe lock release, but it does not establish whether one owner is newer than another. Likewise, timestamps from unsynchronized clients are not a reliable epoch authority.

For a row that accepts one result per ownership epoch, an illustrative conditional update is:

```sql
UPDATE report_result
SET payload = $3::jsonb,
    last_fence = $2::bigint
WHERE report_id = $1
  AND last_fence < $2::bigint
RETURNING report_id;
```

The application must treat zero returned rows as rejection, not success. This assumes last_fence is non-null, all writes use this path, and the supplied fence comes from a trusted coordinator. Access control must prevent arbitrary clients from submitting enormous epochs.

The comparison and mutation must be atomic. A separate SELECT followed by an unconditional UPDATE recreates the race.

## Fencing has a precise boundary

The example rejects epoch 41 after storage accepts epoch 42. It does not automatically reject A merely because B acquired a lease somewhere else: storage must learn the newer epoch.

If the requirement is “no old writes after new ownership is acquired,” ownership activation must include advancing the fence at the resource before the new owner starts work, or ownership must be checked atomically with the write. State that stronger contract explicitly.

Strictly increasing checks also mean repeated writes using the same epoch are rejected. That suits one-result-per-epoch operations. Multi-write owners need a separate operation sequence, transaction model, or idempotency ledger. Do not change the comparison to allow equal epochs without defining replay behavior.

## A lock service cannot fence every API

A payment provider or email gateway may not understand your token. Passing an extra header achieves nothing unless the receiver enforces it.

For external effects, use the provider's supported idempotency key where available, record operation identity durably, and reconcile ambiguous outcomes. If the provider offers neither fencing nor deduplication, acknowledge that duplicate effects remain possible and design compensation or human review.

That limitation is architectural. It cannot be repaired by selecting a more fashionable lock algorithm.

## Test the stale process, not just lock expiry

A useful failure test pauses A after ownership acquisition, lets B acquire and activate a newer epoch, commits B's result, then resumes A. Storage must reject A and preserve B's value.

Also test delayed delivery, coordinator failover, duplicate tokens, token reuse after backup restoration, and a write path that accidentally bypasses the guarded update. Monitor rejected epochs, lease renewal failures, ownership transitions, and operation reconciliation.

A restore that resets the epoch allocator while storage retains high epochs can stop all progress. A restore that resets storage's fence while old clients remain alive can re-admit obsolete writes. Recovery procedures must preserve the ordering contract across both systems.

## Trade-offs and ownership

Fencing adds state and coupling to the resource API. For occasional duplicate cache rebuilds, that may be unnecessary. For mutable authoritative data, avoiding this coupling often means accepting a correctness hole.

The [Redis locking guidance](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/) is useful for understanding lease assumptions and safe release. Treat those assumptions as something to audit, not a replacement for downstream enforcement.

## The CTO decision

Classify each lock as efficiency-only or correctness-critical. For correctness-critical work, locate the final write boundary, prove obsolete owners are rejected there, and specify what happens when the receiver cannot enforce that proof.

## Further reading

- [PostgreSQL: Conditional UPDATE](https://www.postgresql.org/docs/18/sql-update.html)
- [Fenced lease implementation notes](/snippets/fenced-lease)
- [Saga compensation is not rollback](/posts/saga-compensation-is-not-rollback)


---

# Filtered Vector Search Is a Recall Budget

> Why HNSW and IVFFlat filters can return too few results, how iterative scans help, and how to measure recall per tenant instead of trusting latency alone.

Canonical URL: https://www.ayushworks.xyz/posts/filtered-vector-search-is-a-recall-budget
Author: Ayush Basak
Last modified: 2026-09-02
Topics: ai-infrastructure, postgresql, pgvector, vector-search, database-engineering

An approximate vector index can make a query fast while making the product wrong. The risk becomes sharp when semantic search is combined with tenant, permission, language, freshness, or product filters.

The invariant is:

> Retrieval quality must be measured after every eligibility filter, because the application cannot use a relevant result it is not allowed to return.

Approximate nearest-neighbor indexes search a bounded candidate set. In pgvector, filtering is commonly applied after that approximate scan. If only 10% of scanned candidates satisfy the filter, an HNSW search that examines 40 candidates yields roughly four eligible candidates on average—not the requested ten.

## Why `LIMIT 10` may return three rows

Consider a shared embedding table:

```sql
SELECT id, content
FROM document_chunk
WHERE tenant_id = $1
  AND deleted_at IS NULL
ORDER BY embedding <=> $2
LIMIT 10;
```

The SQL contract appears to request ten rows. The approximate index has a different internal budget: visit a bounded neighborhood, then discard rows that fail the relational filter. The executor cannot return disallowed candidates, so the result can be short.

This is not merely a tuning bug. It is an interaction between two correct mechanisms:

```text
ANN candidate generation -> relational eligibility filter -> LIMIT
```

The smaller or more selective a tenant is within a shared graph, the more its recall and latency can differ from a large tenant’s. Aggregate benchmarks conceal that unfairness.

## Choose the search shape by selectivity

There is no universal index plan.

For highly selective filters, a B-tree can find a small eligible set first and exact distance can rank it:

```sql
CREATE INDEX CONCURRENTLY document_chunk_tenant_idx
ON document_chunk (tenant_id)
WHERE deleted_at IS NULL;
```

Exact search over 500 eligible vectors may be cheaper and more accurate than traversing a global HNSW graph.

For a few large, stable segments, partial vector indexes can isolate search spaces:

```sql
CREATE INDEX CONCURRENTLY document_chunk_en_hnsw
ON document_chunk USING hnsw (embedding vector_cosine_ops)
WHERE language = 'en' AND deleted_at IS NULL;
```

For many tenants, one index per tenant creates operational debt. Partitioning by a bounded grouping, dedicated tables for the largest tenants, or shared indexes with iterative scans may be better.

## Iterative scans turn recall into bounded work

pgvector supports iterative scans that continue searching when filtering leaves too few results:

```sql
BEGIN;
SET LOCAL hnsw.iterative_scan = strict_order;
SET LOCAL hnsw.ef_search = 80;
SET LOCAL hnsw.max_scan_tuples = 20000;

SELECT id, content, embedding <=> $2 AS distance
FROM document_chunk
WHERE tenant_id = $1 AND deleted_at IS NULL
ORDER BY embedding <=> $2
LIMIT 10;
COMMIT;
```

Strict ordering preserves exact distance order among returned candidates. Relaxed ordering can improve recall-performance trade-offs; a materialized CTE can then re-sort the expanded result.

The controls are budgets, not magic. Increasing `ef_search`, scan tuples, IVFFlat probes, or scan memory spends CPU, memory, and latency to recover candidates. Put ceilings around interactive traffic and separate policies for offline retrieval jobs.

## Benchmark against exact truth

Latency-only benchmarks reward empty results. Build an evaluation set of real query embeddings and eligibility filters. For each query, compute an exact top-k baseline by disabling approximate index scans in a controlled environment, then compare the approximate result.

Useful metrics include:

```text
recall@k = relevant exact top-k IDs found / k
fill@k   = returned eligible rows / requested k
```

Also record p50, p95, and p99 latency; candidates visited; filter selectivity; tenant size; index size; model version; embedding dimension; and corpus freshness.

Report distributions by tenant and filter class. A global recall of 0.95 can coexist with a recall of 0.30 for small tenants—the exact customers isolation is supposed to protect.

## Retrieval correctness extends beyond ANN

Even perfect nearest-neighbor recall does not prove a correct AI answer. Retrieval can fail because:

- the source was chunked across the required context;
- embeddings changed without a complete reindex;
- permission metadata is stale;
- deleted content remains in a secondary index;
- the distance operator does not match how embeddings were normalized;
- a reranker optimizes a different relevance definition;
- the model cites a chunk it did not actually use.

Version embeddings and chunking policy. Store source revision and authorization metadata with each chunk. Make deletion and permission change propagation measurable. Never use semantic similarity as authorization.

## Index lifecycle is product lifecycle

HNSW generally offers a better query speed-recall trade-off but costs more memory and build time than IVFFlat. IVFFlat depends strongly on list count, training data, and probes. Both need evaluation after corpus growth or embedding-model changes.

Build production indexes concurrently where write availability matters. Observe `EXPLAIN (ANALYZE, BUFFERS)`, index size, build progress, vacuum behavior, and replica impact. A new embedding model often requires dual-writing and shadow evaluation before traffic moves—not overwriting vectors in place and hoping ranking remains stable.

## The CTO decision

Define retrieval quality as a service objective: minimum recall and fill rate for each important eligibility class, within a latency and resource budget. Make an exact baseline reproducible. Choose exact search, partial indexes, partitioning, HNSW, or IVFFlat from measured selectivity rather than fashion.

Vector search is not “working” because the query is fast. It is working when eligible evidence is found reliably enough for the product decision built on top of it.

## References

- [pgvector documentation](https://github.com/pgvector/pgvector)
- [Efficient and robust approximate nearest-neighbor search using HNSW](https://arxiv.org/abs/1603.09320)
- [PostgreSQL: Using EXPLAIN](https://www.postgresql.org/docs/current/using-explain.html)
- [Related: Production AI needs release gates](/posts/production-ai-risk-gates)
- [Related: AI evals are release engineering](/posts/ai-evals-are-release-engineering)


---

# gRPC’s HTTP/2 Transport Is Not the Reliability Model

> Separate HTTP/2 multiplexing and flow control from deadlines, retries, idempotency, and overload policy in gRPC systems.

Canonical URL: https://www.ayushworks.xyz/posts/grpc-http2-is-not-reliability-model
Author: Ayush Basak
Last modified: 2026-09-05
Topics: systems-internals, grpc, http2, backend-engineering

gRPC gives services a typed RPC model and commonly carries it over HTTP/2. That transport provides multiplexed streams, framing, and flow control. None of those features decides whether retrying a payment is safe, when a request is no longer useful, or how an overloaded dependency sheds work.

Transport capability and application reliability are different layers.

## What HTTP/2 actually provides

HTTP/2 associates concurrent request/response exchanges with streams on one connection. It applies flow control at both stream and connection level. Receivers advertise credit with `WINDOW_UPDATE`; senders must remain inside both windows.

This creates useful efficiency, but also shared fate. If connection-level flow-control credit is exhausted, all data-bearing streams on that connection are constrained. A connection reset can affect many in-flight RPCs. Multiplexing removes HTTP/1.1 request queuing on separate application exchanges, but TCP loss and shared connection state still matter.

## Four policies the transport cannot choose

### Deadline

A caller should declare when the result stops being valuable. Without a deadline, work can accumulate across hops after the user or upstream request has gone away.

```go
ctx, cancel := context.WithTimeout(parent, 250*time.Millisecond)
defer cancel()
reply, err := client.Lookup(ctx, req)
```

Each service must propagate the remaining budget, reserve time for its own cleanup, and stop downstream work on cancellation.

### Retry

A retry is a new attempt and additional load. It needs an error policy, bounded attempt count, backoff with jitter, and sufficient deadline budget. Connection failure does not prove the server did not commit an operation.

### Idempotency

For writes, carry an operation identity and persist its outcome at the same authority that applies the side effect. Transport stream IDs are connection-scoped mechanics, not business idempotency keys.

### Admission control

HTTP/2 flow control protects buffers between adjacent endpoints. It does not know a request’s database cost, tenant quota, model-token demand, or deadline. Concurrency limits and load shedding still belong at the application boundary.

## Failure policy

| Condition | Default response |
| --- | --- |
| caller cancelled | stop work immediately |
| deadline expired | fail; do not begin another attempt |
| explicit retryable unavailable | retry only within budget |
| validation failure | do not retry |
| ambiguous write outcome | resolve by idempotency key |
| local concurrency full | reject early with useful status |
| connection draining | shift new calls; let bounded calls finish |

## Observe both layers

RPC metrics should include method, final status, attempt count, remaining deadline at entry, handler concurrency, and queue time. Transport metrics should include active streams, connection churn, flow-control stalls, bytes, and reset causes. Correlating them prevents a transport symptom from becoming a fictional application diagnosis.

## Trade-offs

Long-lived multiplexed connections reduce handshake and socket overhead, while increasing correlated impact when one connection degrades. More connections can isolate traffic classes but use more sockets and memory. Aggressive retries improve recovery from brief faults but amplify overload. Short deadlines bound waste but fail requests that might have completed.

The architecture is complete only when the protobuf contract, transport behavior, and failure policy agree. HTTP/2 carries the call. It does not decide whether the call remains correct under uncertainty.

## Further reading

- [RFC 9113: HTTP/2](https://www.rfc-editor.org/rfc/rfc9113.html)
- [gRPC deadlines](https://grpc.io/docs/guides/deadlines/)
- [Your timeout budget is an architecture decision](/posts/deadline-budgets-retry-amplification)


---

# Heartbeats Measure Silence, Not Failure

> Design failure detection around suspicion, consequence, and recovery—not a magic heartbeat timeout.

Canonical URL: https://www.ayushworks.xyz/posts/heartbeats-measure-silence-not-failure
Author: Ayush Basak
Last modified: 2026-09-05
Topics: distributed-systems, failure-detection, reliability, architecture

A node misses three heartbeats, so the cluster declares it dead. That sentence hides the hardest part of failure detection: the observer cannot distinguish a dead process from a delayed process, a saturated network, a paused runtime, or its own broken connection.

The signal is silence. Failure is an interpretation.

This distinction matters because the action after suspicion may be destructive: reassigning a partition, promoting a replica, revoking a lease, or allowing another worker to produce the same side effect.

## Start with the consequence

Do not choose a timeout before defining what it authorizes.

| Observation | Safe interpretation | Possible action |
| --- | --- | --- |
| one missed probe | transient delay | record latency |
| repeated missed probes | suspect | probe indirectly |
| quorum cannot reach member | member unavailable to quorum | stop routing new work |
| lease expired at authority | ownership ended | fence the old owner |
| process responds again | communication recovered | reconcile state |

The important boundary is that a detector may inform an ownership protocol, but it must not *be* the ownership protocol. A late worker can wake after its peers replaced it. Storage must reject that stale worker through an epoch, generation, or fencing token.

## Detection is a latency–mistake trade-off

Let heartbeat observations have inter-arrival times `x`. A fixed threshold says:

```text
suspect when now - last_seen > T
```

A lower `T` shortens failover but increases false suspicions. A higher `T` reduces false positives but lengthens the interval in which traffic still targets an unavailable member. There is no universally correct number because pause distributions, network paths, and remediation costs differ.

An adaptive detector can instead compare current silence with historical observations. The output should be suspicion—not an invented certainty—and operators should be able to inspect the evidence behind it.

## Separate probing from dissemination

All-to-all heartbeats make every member send to every other member. Message load grows poorly as membership grows. SWIM’s useful design move is to separate two jobs:

1. Probe a small, changing set of peers to detect possible failure.
2. Disseminate membership changes through gossip.

SWIM also introduces a `suspect` phase before `failed`, allowing indirect probes or contrary evidence to correct an initial observation. This is more than an optimization: it makes uncertainty explicit in the state machine.

```text
alive -> suspect -> failed
   ^        |
   +--------+  refutation with newer incarnation
```

## Production policy

For each detector, document:

- the observation source and its blind spots;
- expected and worst-case detection time;
- false-positive budget under load and deployment pauses;
- whether suspicion removes traffic or transfers authority;
- how a returning member proves its incarnation;
- which metric explains every transition.

Test the detector during CPU starvation, packet loss, asymmetric reachability, long garbage-collection pauses, and rolling deploys. A test that only kills a process validates the easiest case.

## Trade-offs

Fast detection improves recovery time but makes healthy nodes easier to condemn. Indirect probes and suspicion stages reduce false positives at the cost of more protocol state. Central detectors simplify policy but introduce shared bottlenecks and correlated blind spots. Distributed detectors scale better, while different observers can temporarily disagree.

The CTO-level decision is not “use a five-second heartbeat.” It is deciding how much uncertainty the system can tolerate before taking an action, and ensuring that the action remains safe when the detector is wrong.

## Further reading

- [SWIM: Scalable Weakly-consistent Infection-style Process Group Membership](https://www.cs.cornell.edu/projects/Quicksilver/public_pdfs/SWIM.pdf)
- [Unreliable Failure Detectors for Reliable Distributed Systems](https://research.google/pubs/unreliable-failure-detectors-for-reliable-distributed-systems/)
- [Why stale writers need storage-level fencing](/posts/fencing-stale-writers-at-the-storage-boundary)


---

# Hello World

> My first blog post.

Canonical URL: https://www.ayushworks.xyz/posts/hello-world
Author: Ayush Basak
Last modified: 2026-08-16
Topics: personal, engineering

<span id="hello-world"></span>

Welcome to my blog. This is my first post.

I'm excited to share my thoughts, experiences, and learnings here.

More to come soon!


---

# HTTP Early Data Is a Replay Boundary

> A production guide to QUIC and TLS 0-RTT replay risk, idempotent operations, 425 Too Early, anti-replay limits, and safe rollout policy.

Canonical URL: https://www.ayushworks.xyz/posts/http-early-data-is-a-replay-boundary
Author: Ayush Basak
Last modified: 2026-09-02
Topics: distributed-systems, http, quic, security, backend-engineering

TLS and QUIC can reduce repeat-connection latency by allowing a client to send application data before a new handshake completes. That 0-RTT path changes the threat model: early data can be replayed.

The invariant is:

> No request may enter early data unless executing it more than once is acceptable or the application supplies a durable deduplication boundary.

Encryption does not provide uniqueness. An attacker does not need to decrypt captured early data to replay it. Infrastructure-level anti-replay controls can reduce exposure, but distributed deployments cannot casually promise global single execution.

## Latency optimization crosses an application boundary

In an ordinary full handshake, fresh handshake state helps bind application traffic to the connection. In 0-RTT, a client uses material derived from an earlier session and sends data immediately:

```text
full handshake: ClientHello -> ServerHello -> request
0-RTT resume:   ClientHello + request ----------->
```

The second path saves a round trip. It also means the server may receive the same valid early request more than once. RFC 9001 explicitly identifies replay vulnerability for QUIC 0-RTT, while RFC 8470 defines how HTTP intermediaries and origins communicate early-data risk.

## “POST” is not the whole policy

HTTP method semantics are useful but insufficient. A `GET` that triggers an email or increments a paid counter is unsafe despite its method. A `POST` with a strong operation key and transactional deduplication may be replay-safe.

Classify operations by effect:

| Operation | Early-data default | Reason |
| --- | --- | --- |
| static asset read | allow | no business mutation |
| cacheable catalogue read | allow | repeat has equivalent effect |
| account balance read | usually deny | stale or sensitive context may matter |
| payment creation | deny | duplicate financial effect |
| idempotent upsert with durable key | conditional | depends on deduplication scope |
| login or token exchange | deny | replay changes security state |

Do not infer replay safety from a framework annotation alone. Trace the operation through queues, email, billing, inventory, and third-party calls.

## Use 425 as a protocol boundary

RFC 8470 defines `425 Too Early`. An origin can reject a request received in early data when it is unwilling to risk replay. A compliant client can retry after the handshake completes.

At the edge, preserve whether early data was used and enforce an allowlist. A conceptual policy is:

```text
if request.is_early_data:
  if route not in replay_safe_routes:
    return 425
  if request carries authorization with unsafe semantics:
    return 425
forward request with trusted early-data context
```

Never trust a public client header that merely claims the request was or was not early data. The trusted TLS terminator must set or sanitize this context before forwarding.

## Idempotency needs durable scope

For an operation intentionally allowed to retry, store the operation key and outcome atomically with the mutation:

```sql
BEGIN;

INSERT INTO operation_result (tenant_id, operation_id, state)
VALUES ($1, $2, 'started')
ON CONFLICT (tenant_id, operation_id) DO NOTHING;

-- Proceed only when this transaction inserted the key.
-- Store the final result in the same transaction as the business mutation.

COMMIT;
```

The key must be scoped to the authenticated tenant or principal, validated for entropy and length, retained for at least the replay/retry window, and bound to a fingerprint of the intended operation. Reusing one key with a different payload must fail.

An in-memory cache on one instance is not durable deduplication. A request replayed into another region or after a restart bypasses it.

## Anti-replay has topology costs

A single TLS terminator can track accepted tickets more easily than a global anycast fleet. Sharing replay state across regions adds coordination and latency—the very costs 0-RTT tries to remove. Keeping state local leaves cross-region replay windows.

This is why infrastructure anti-replay and business idempotency should be treated as layers:

- edge policy reduces which routes can receive early data;
- TLS ticket policy limits age and scope;
- application idempotency prevents duplicate business effects;
- audit signals detect replays and policy violations.

None should be used to claim exactly-once delivery.

## Roll out from evidence

Before enabling early data, measure how much traffic actually resumes sessions, the round-trip time saved by region, and whether that saving changes a customer SLO. Then canary only replay-safe routes.

Observe:

- early-data requests accepted and rejected by route;
- `425` responses and successful post-handshake retries;
- duplicate operation-key conflicts;
- ticket age and resumption rate;
- region changes between original and resumed connections;
- mutations that reached an early-data path unexpectedly.

Test capture-and-replay explicitly in a non-production environment. Verify that unsafe routes return `425`, safe reads remain correct, and idempotent writes return the original outcome without repeating side effects.

## Common mistakes

- Assuming encrypted means non-replayable.
- Enabling 0-RTT for an entire hostname.
- Allowing early authorization or session-establishment requests.
- Treating all `GET` routes as side-effect free.
- Deduplicating at the HTTP layer while a downstream consumer repeats the effect.
- Retrying a `425` again as early data.
- Counting median handshake improvement while ignoring low resumption rates.

## The CTO decision

0-RTT is valuable when a round trip materially affects the product and the eligible operation set is narrow and provably replay-safe. Make early-data eligibility an explicit route policy, reject unsafe work with `425`, and keep durable operation identity at every side-effect boundary.

The optimization is not “turn on QUIC.” It is buying latency with a carefully bounded replay surface.

## References

- [RFC 8470: Using Early Data in HTTP](https://www.rfc-editor.org/rfc/rfc8470)
- [RFC 9001: Using TLS to Secure QUIC](https://www.rfc-editor.org/rfc/rfc9001)
- [RFC 9308: Applicability of QUIC](https://www.rfc-editor.org/rfc/rfc9308)
- [Related: The transactional outbox is not the delivery guarantee](/posts/transactional-outbox-delivery-guarantees)
- [Related: Thinking through offline-first payments](/posts/offline-first-payments)


---

# Incident Command Is a Distributed System

> How CTOs can design incident response around explicit authority, shared state, bounded coordination, customer communication, and evidence-driven recovery.

Canonical URL: https://www.ayushworks.xyz/posts/incident-command-is-a-distributed-system
Author: Ayush Basak
Last modified: 2026-08-29
Topics: reliability, incident-response, technical-leadership, observability

During a serious outage, the organization itself becomes a distributed system. People have partial observations, messages arrive out of order, ownership changes, actions race, and the shared model of reality becomes stale.

Adding more engineers often reduces throughput because communication grows faster than useful work. Incident command exists to impose a small, reliable coordination protocol on that chaos.

The governing invariant is:

> At any moment, one person owns incident priority and one shared record represents the latest accepted state.

This does not mean one person diagnoses everything. It means authority and information have explicit convergence points.

## Separate command from investigation

The incident commander owns the process: severity, priorities, workstreams, escalation, and the decision to move between mitigation and recovery. Technical leads own investigation within bounded workstreams. A communications lead owns updates to customers and internal stakeholders.

```text
incident commander
├── mitigation lead
├── database investigation
├── dependency investigation
└── communications lead
```

Without this split, the strongest technical investigator becomes a meeting router and status writer. With it, specialists can maintain focus while command sees the complete risk picture.

## Create one state log

Chat is a transport, not a source of truth. Maintain an incident log containing:

- start time, severity, and customer effect;
- current hypothesis and confidence;
- actions underway, owner, and deadline;
- completed changes and observed result;
- decisions explicitly rejected;
- next update time;
- links to dashboards, traces, deploys, and tickets.

Use timestamps in UTC. Record observations separately from interpretations:

```text
14:07 observation: checkout 5xx rose from 0.2% to 18% in eu-west
14:09 hypothesis: connection exhaustion after release 8f31; medium confidence
14:11 action: halt rollout; owner Priya; result expected by 14:16
```

This structure makes handoff possible and reduces repeated investigation.

## Mitigate before explaining

Incident response is an optimization problem under uncertainty. The first objective is reducing customer harm, not finding the intellectually complete root cause.

A good mitigation is:

- reversible;
- narrow in blast radius;
- observable within minutes;
- executable with current authority;
- independent of the suspected failing control plane where possible.

Rollback, disable, shed, isolate, or serve a known-good snapshot before attempting a complex repair. Every action must have a predicted signal. “Restart it” is not an experiment unless you know what observation will confirm or reject the hypothesis.

## Bound coordination

Use short update cadences based on severity. Each workstream reports only:

```text
observation → current hypothesis → action → expected evidence → next checkpoint
```

Create new workstreams only when they test independent hypotheses. Ten people querying the same database are not ten workstreams; they are contention.

The commander should stop risky or duplicated actions. During an outage, production write access is a scarce capability. Make one owner accountable for each mutation and record it before execution when time permits.

## Communicate customer truth

External updates should state what users experience, which surfaces are affected, what mitigation is underway, and when the next update will arrive. Do not publish an unverified root cause.

Internally, distinguish three states:

- **mitigated:** customer impact has stopped;
- **recovered:** service indicators remain healthy for an agreed window;
- **resolved:** temporary controls are removed or converted into durable follow-up.

Calling an incident resolved immediately after a graph falls creates repeat incidents during cleanup.

## Learn from the coordination system

A useful review asks more than “what bug caused this?”

- Why did detection take this long?
- Which dependency made mitigation difficult?
- Which access or runbook was missing?
- Where did responders hold conflicting state?
- Which action increased risk?
- Could the system have failed statically instead?
- What reduces recurrence, impact, or recovery time most economically?

Track actions to owners and evidence of completion. “Improve monitoring” is not an action. “Page when admitted database sessions exceed 80% for ten minutes; validate in game day” is.

## Conclusion

Incident management is architecture for humans under partial failure. Explicit authority prevents conflicting writes. A shared log creates convergent state. Bounded workstreams reduce coordination load. Reversible mitigation controls blast radius.

Do not wait for the outage to invent this protocol. Practice it until the organization can execute it while its assumptions—and its dashboards—are failing.

## References

- [Google SRE: Incident Management Guide](https://sre.google/resources/practices-and-processes/incident-management-guide/)
- [Google SRE: Managing Incidents](https://sre.google/sre-book/managing-incidents/)
- [PagerDuty Incident Response Documentation](https://response.pagerduty.com/)


---

# JWT Verification Is a Key-Distribution System

> How issuer binding, algorithm allowlists, JWKS rotation, cache policy, unknown key IDs, and outage behavior determine whether JWT verification is secure and available.

Canonical URL: https://www.ayushworks.xyz/posts/jwt-verification-is-key-distribution
Author: Ayush Basak
Last modified: 2026-09-02
Topics: security, backend-engineering, identity, jwt, distributed-systems

Teams often describe JWT validation as a local cryptographic operation. In production it is also a distributed key-distribution protocol: verifiers must discover, cache, rotate, select, and retire public keys without accepting attacker-controlled trust inputs or turning an identity-provider outage into a total platform outage.

The invariant is:

> A token is acceptable only when its issuer, key, algorithm, audience, type, and claims all satisfy one locally configured trust policy.

A valid signature proves only that one key signed some bytes. It does not prove the key belongs to an issuer you trust or that the token was intended for your API.

## Bind configuration before parsing claims

The verifier should begin with a local issuer policy:

```ts
type IssuerPolicy = {
  issuer: string
  jwksUri: string
  algorithms: readonly ['RS256'] | readonly ['ES256']
  audiences: readonly string[]
  requiredType: 'at+jwt'
  maxTokenAgeSeconds: number
}
```

Do not read an arbitrary `jku` or `x5u` header and fetch keys from it. RFC 8725 warns that blindly following attacker-provided URLs can create SSRF. Resolve the JWKS location from trusted issuer configuration or validated authorization-server metadata.

Pin an algorithm allowlist. The token’s `alg` header is an input to verify, not a command to the verifier. Bind each key to its intended algorithm and reject unexpected or ambiguous combinations.

## `kid` is a selector, not authority

The JWT `kid` selects a key from the trusted issuer’s JWK Set. It must never select an issuer, build a file path, become an unparameterized database query, or cause an unbounded network fetch.

Normal rotation looks like:

```text
JWKS publishes old + new key
        -> issuer begins signing with new kid
        -> verifiers already know both keys
        -> old tokens expire
        -> old key is removed
```

Publishing the new verification key before using it avoids a fleet-wide miss at the signing cutover. Retaining the old key until every legitimately issued token expires avoids rejecting valid sessions.

## Cache for availability without freezing trust

Fetching JWKS on every request makes authentication latency depend on the identity provider and amplifies an outage into a request storm. Cache the set according to bounded policy and HTTP caching signals, but retain controls for emergency refresh and revocation.

A safe unknown-`kid` path is single-flight and rate-limited:

```ts
async function resolveKey(issuer: IssuerPolicy, kid: string) {
  const cached = keyCache.get(issuer.issuer, kid)
  if (cached)
    return cached

  await refreshSingleFlight(issuer.issuer, {
    minRefreshIntervalMs: 30_000,
    deadlineMs: 1_000,
  })

  return keyCache.get(issuer.issuer, kid) ?? null
}
```

Without a minimum refresh interval, an attacker can send random key IDs and force outbound requests. Without single-flight, one legitimate rotation can make every instance refresh simultaneously.

Keep a last-known-good set through a short provider outage, but do not extend token expiration or accept keys past an explicit revocation policy merely because refresh failed. Availability and revocation have different priorities for different systems; document the choice.

## Validate the semantic envelope

After cryptographic verification, validate at least:

- exact issuer match;
- intended audience;
- expiration and not-before with a small, bounded clock tolerance;
- expected token type and mutually exclusive validation rules for different token kinds;
- subject requirements for the endpoint;
- scopes or permissions using server-side policy;
- maximum token age where compromise exposure demands it.

Explicit typing reduces token confusion. An ID token, access token, email-verification token, and internal job token should not be interchangeable merely because they share an issuer and signing key.

Do not authorize from mutable human labels such as an email domain alone. Translate validated identity into product permissions through an owned authorization model.

## Rotation and emergency revocation are different

Routine rotation is an overlap protocol. Emergency revocation is a containment event. If a private key may be compromised, leaving its public key available until all tokens expire preserves attacker access.

Your incident plan should answer:

- How quickly can the issuer stop signing with the key?
- How quickly do verifiers refresh?
- Can a key be denylisted before cache expiry?
- What is the maximum remaining token lifetime?
- Which services accept the affected issuer and audience?
- Can high-risk operations require fresh introspection or step-up authentication?

Short-lived access tokens reduce exposure but increase dependence on refresh infrastructure. Longer tokens improve outage tolerance but enlarge the compromise window. That is a product-security trade-off, not a library default.

## Observe without leaking credentials

Record issuer, audience-validation outcome, algorithm, key ID, policy version, cache hit or refresh, rejection reason, and verifier latency. Never log the raw token. Hash identifiers only when the operational need and retention policy justify it.

Alert on:

- sudden unknown-`kid` volume;
- JWKS refresh failures and last-success age;
- algorithm or issuer mismatches;
- token-expiry failures by client version;
- one key remaining active beyond the rotation window;
- abnormal verification latency or refresh concurrency.

Test rotations with old and new tokens across every service. Test issuer downtime, stale caches, random key IDs, duplicate keys, clock skew, wrong audience, wrong token type, and emergency denial.

## The CTO decision

Treat JWT verification as shared security infrastructure with a versioned trust policy, not middleware copied into every repository. Centralize the hard rules while keeping applications responsible for product authorization.

The signature check is the smallest part. The system succeeds when key rotation is boring, outages are bounded, untrusted headers cannot redirect trust, and every accepted token is proven to belong to the exact context in which it is used.

## References

- [RFC 7517: JSON Web Key](https://www.rfc-editor.org/rfc/rfc7517)
- [RFC 8725: JWT Best Current Practices](https://www.rfc-editor.org/rfc/rfc8725)
- [RFC 8414: OAuth 2.0 Authorization Server Metadata](https://www.rfc-editor.org/rfc/rfc8414)
- [Related: Design service identity before buying a service mesh](/posts/service-identity-before-service-mesh)
- [Related: Verify what you deploy](/posts/verify-what-you-deploy)


---

# Kafka Exactly-Once Stops at the Side-Effect Boundary

> Where Kafka transactions provide atomicity, where they do not, and how to design external effects without magical guarantees.

Canonical URL: https://www.ayushworks.xyz/posts/kafka-exactly-once-stops-at-side-effects
Author: Ayush Basak
Last modified: 2026-09-01
Topics: kafka, distributed-systems, event-driven-architecture, reliability

“Exactly once” is useful only after naming exactly which state changes share a transaction.

Kafka can atomically commit consumed offsets, state-store changes, and records produced to Kafka. That is a powerful boundary. It does not automatically include a payment gateway, email provider, warehouse, or arbitrary database.

The invariant is narrower than the slogan:

```text
one committed Kafka input
→ one visible Kafka/state-store result
```

The moment a handler crosses into a system that does not participate in the Kafka transaction, the outcome can become ambiguous.

## The crash window still exists

Consider a consumer that charges a card and then commits its offset:

```text
1. consume order
2. payment provider accepts charge
3. process crashes before offset commit
4. order is consumed again
```

Kafka correctly redelivers because it cannot know whether the external call succeeded. Retrying without a stable idempotency key can charge twice. Committing before the call merely reverses the failure: a crash can lose the charge.

No ordering removes the uncertainty. The boundary needs a protocol.

## Choose the protocol by ownership

If the durable state is in your relational database, use a local transaction to write both domain state and an outbox record. A relay publishes the outbox record later. Consumers remain idempotent because delivery can repeat.

If the result remains inside Kafka, transactions and `read_committed` consumers can keep offsets and output records atomic.

If the effect belongs to an external provider, send a stable operation identity and reconcile:

```ts
await payments.charge({
  idempotencyKey: `order:${order.id}:capture:v1`,
  amount: order.total,
})
```

The provider must actually persist and enforce that key. A UUID generated on every retry is not idempotency.

| Effect | Practical guarantee |
|---|---|
| Kafka input → Kafka output | Kafka transaction |
| Database mutation → event | transactional outbox |
| External API call | provider idempotency plus reconciliation |
| Email or webhook | deduplication where possible; tolerate repeated delivery |
| Human action | expose state and prevent unsafe repetition |

## Consumer policy

Classify failures rather than retrying everything. Validation failures belong in a terminal path. Transient infrastructure failures may retry with a deadline and jitter. Ambiguous writes require a status lookup or reconciliation job—not blind repetition.

Record four identities:

- business operation ID;
- source topic, partition, and offset;
- external provider request ID;
- resulting domain-state version.

Those identifiers make an incident reconstructable. Without them, “exactly once” becomes a dashboard label that cannot answer whether value moved.

## Production review

Ask these before enabling exactly-once processing:

1. Which writes are inside the transaction coordinator?
2. Which side effects remain outside?
3. What happens after success but before acknowledgement?
4. Can the same business operation arrive through another channel?
5. How is a stuck or ambiguous operation reconciled?
6. Are transaction timeouts and consumer rebalances visible?

Kafka transactions have operational costs and configuration requirements. Use them when atomic Kafka state is the requirement, not as a substitute for defining business idempotency.

## Test the acknowledgement gaps

Happy-path integration tests do not exercise the guarantee. Inject a crash after the external provider accepts a request but before the consumer commits. Repeat after the database commits but before an outbox relay publishes. Restart a producer after its transaction times out. In each case, assert the final business state and the evidence available to an operator.

Also test poison records. Retrying a deterministic validation failure can block a partition and hide useful work behind it. A terminal path must preserve the original record, failure reason, code version, and replay authorization without creating an ungoverned “dead-letter queue forever.”

## Conclusion

Exactly-once semantics are not false; they are scoped. Good architecture writes that scope beside the promise.

Keep atomic work inside one coordinator. Where that is impossible, make repetition safe, preserve operation identity, and build reconciliation as a first-class path. The honest guarantee is often not “the message runs once,” but “the business effect converges to one defensible outcome.”

---

Further reading: [Apache Kafka documentation](https://kafka.apache.org/documentation/), [Kafka Streams core concepts](https://kafka.apache.org/20/streams/core-concepts/), [transactional outbox guarantees](/posts/transactional-outbox-delivery-guarantees), and [Kafka rebalance coordination](/posts/kafka-rebalances-are-stop-the-world-coordination).


---

# Adding Kafka Partitions Changes the Ordering Contract

> Scale Kafka partitions without accidentally changing key affinity, consumer concurrency, and the order your product relies on.

Canonical URL: https://www.ayushworks.xyz/posts/kafka-partitions-change-ordering-contract
Author: Ayush Basak
Last modified: 2026-09-05
Topics: distributed-systems, kafka, event-streaming, system-design

“Add partitions” sounds like a capacity change. For a keyed event stream, it can be a semantics change.

Kafka orders records within a partition, not across a topic. Producers commonly map a key to a partition. If the partition count changes, that mapping may change. Events for the same customer can land in the old and new partitions during the transition, and consumers can observe them in an order the business never anticipated.

## Write the real invariant

“Messages are ordered” is not precise enough. Useful invariants look like:

```text
For one account, balance mutations are applied in source sequence.
For one device, configuration version n+1 never applies before n.
For one order, cancelled never transitions back to paid.
```

Now identify the mechanism enforcing each invariant. If the answer is “Kafka ordering,” record the key, partitioner, producer behavior, and consumer-side checks. A topic-wide statement is almost certainly wrong.

## Why partition growth is not transparent

Suppose a default key hash behaves conceptually as:

```text
partition = hash(key) % partition_count
```

Changing the count from 12 to 24 does not merely create empty lanes. Many keys receive a different result. Old records remain in their original partitions while new records go elsewhere.

More partitions also change operational limits:

- maximum useful consumer concurrency rises;
- per-partition traffic can become more skewed;
- rebalances coordinate a larger assignment set;
- broker metadata, files, and replication work increase;
- downstream systems may receive more parallel requests.

Capacity moved; the failure and ordering surfaces moved too.

## Safer migration patterns

### Create a new versioned topic

Publish to `account-events-v2` with the desired partition count. Dual-write or replay through a controlled cutover, validate consumer positions, then retire the old topic. This costs migration work but gives the contract a visible boundary.

### Route through stable virtual shards

Map entity keys to a fixed number of logical shards, then map shards to physical partitions. The extra indirection lets operators move shard ownership deliberately. It is useful when key affinity is valuable enough to justify a routing layer.

### Make consumers reject stale transitions

Carry an aggregate sequence, source version, or monotonic update number. A consumer can then detect gaps and reject regressions even if transport ordering changes.

```sql
UPDATE account_projection
SET balance = :balance, source_version = :version
WHERE account_id = :id
  AND source_version = :version - 1;
```

A zero-row update is not “just retry.” It signals a duplicate, a gap, or concurrent processing that needs reconciliation.

## Decision table

| Requirement | Design response |
| --- | --- |
| order only within one entity | key by stable entity identifier |
| strict sequence after repartitioning | versioned topic or logical shards |
| tolerate duplicate delivery | idempotent consumer effect |
| detect missing events | per-entity source sequence |
| scale consumers only | verify partitions are actually the bottleneck |
| global ordering | use a single sequencing authority and accept its limit |

## Common mistakes

- Increasing partitions in production without replaying an entity-order test.
- Treating a producer acknowledgement as proof that a projection applied the event.
- Choosing high-cardinality keys without measuring hot-key skew.
- Scaling consumers beyond the downstream database’s concurrency budget.
- Assuming an idempotency key repairs an out-of-order state transition.

## Trade-offs

A versioned-topic migration consumes temporary storage and doubles operational paths. Stable virtual shards add routing complexity. Consumer-side versions require domain support and gap recovery. Each is more work than changing one integer, because each preserves an invariant that the integer can break.

Partitions are simultaneously a throughput unit, an ordering scope, and a parallelism boundary. Change them only after deciding which of those contracts the product is allowed to change.

## Further reading

- [Apache Kafka concepts and guarantees](https://kafka.apache.org/documentation/#intro_concepts_and_terms)
- [Kafka producer partitioner configuration](https://kafka.apache.org/documentation/#producerconfigs_partitioner.class)
- [Why a rebalance is a coordination protocol](/posts/kafka-rebalances-are-stop-the-world-coordination)


---

# Kafka Poison Messages: Dead-Letter Queues Without Silent Data Loss

> Design Kafka consumer recovery around offset ownership, durable quarantine, per-key ordering, and safe replay instead of endless retries.

Canonical URL: https://www.ayushworks.xyz/posts/kafka-poison-messages-need-an-ordering-policy
Author: Ayush Basak
Last modified: 2026-09-11
Topics: kafka, event-driven, distributed-systems, reliability

A poison message is a record that repeatedly fails under the current consumer implementation. It may contain malformed bytes, an unsupported schema, an impossible domain transition, or a value that exposes a code defect. Calling every failure “transient” lets one record become an indefinite operating condition.

The difficult decision is what happens to records behind it. A dead-letter queue preserves a failed record somewhere else. It does not establish whether later records are safe to process, whether the consumer may advance its offset, or whether anyone will repair the abandoned business operation.

This article proposes a recovery policy for a conventional Kafka consumer group. The API references are for Kafka 4.1 Java clients; confirm configuration behavior against the version and group protocol you deploy.

## Begin with the business sequence

Suppose one partition contains:

~~~text
offset 120: OrderCreated(order-7)
offset 121: OrderAddressChanged(order-7)  <- consumer fails
offset 122: OrderDispatched(order-7)
offset 123: OrderCreated(order-9)
~~~

Skipping 121 may let a parcel leave for the wrong address. Blocking the partition preserves that sequence but also delays order-9, which may be unrelated.

Neither behavior is universally correct. Identify whether ordering is required across the whole partition, within one business key, or only between specific event types. Kafka's partition organization is a transport choice; the domain determines which reorderings are acceptable.

Write the invariant before the retry configuration: a dispatch must not execute while an earlier required address change for that order is unresolved.

## Separate four failure classes

A temporary dependency outage should trigger controlled backoff and reduced admission. Retrying malformed bytes against the same parser will not repair them. An unknown schema may require a compatible deployment. A business rejection may be an expected terminal result.

Keep these classifications visible:

| Failure | Candidate policy | Evidence required to resume |
| --- | --- | --- |
| Dependency timeout | Pause affected work; retry within a budget | Dependency can complete useful work |
| Unsupported schema | Quarantine or stop, according to ordering needs | Compatible consumer and a replay test |
| Invalid business transition | Record rejection or request repair | Domain owner defines the correct outcome |
| Unknown consumer defect | Contain and investigate | Reproducer, fix, and regression test |

Do not send every timeout to a dead-letter topic. During an outage that converts infrastructure failure into a large backlog of manual repair.

## The offset is an acknowledgment boundary

The [Kafka consumer API](https://kafka.apache.org/41/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html) distinguishes the current position from the committed position used for recovery. Committed offsets identify where consumption should resume, conventionally the next record to process.

If records are dispatched concurrently, completion order may differ from partition order. Suppose 120 finishes, 121 is still running, and 122 finishes. Committing 123 would skip 121 on restart.

Maintain a contiguous completion frontier per partition. Advance it only when every earlier record has reached an accepted durable outcome. If quarantine counts as completion, make that a deliberate policy with a durable acknowledgment, not an exception handler that logs and continues.

The recovery design must also respect partition ownership during rebalances. An old worker must not independently acknowledge work after ownership has moved.

## Quarantine is a write path

A useful quarantine envelope records original identity and enough evidence to replay safely:

~~~json
{
  "source": {
    "topic": "orders",
    "partition": 2,
    "offset": 121
  },
  "eventId": "evt-address-7",
  "consumerVersion": "orders-handler-v3",
  "failureClass": "unsupported_schema",
  "schemaVersion": 8,
  "payloadReference": "restricted-quarantine/evt-address-7",
  "repairStatus": "pending"
}
~~~

This is a proposed application envelope, not Kafka configuration. Restrict payload access and retention independently from operational metadata. A dead-letter topic can otherwise become a second, less governed customer database.

There is a crash window between acknowledging quarantine and committing the source offset. With ordinary separate writes, a restart can produce another quarantine entry. Make quarantine identity deterministic, such as source topic, partition, and offset, and tolerate duplicate delivery.

For Kafka-to-Kafka processing, transactions can couple output publication and consumed offsets when configured correctly. They do not make an external database mutation or email part of the same atomic outcome. Review the [side-effect boundary](/posts/kafka-exactly-once-stops-at-side-effects) before claiming exactly-once recovery.

## Choose the price of preserving order

Three designs are worth comparing.

A partition stop is easy to reason about but has a large delay radius when unrelated keys share the partition. A per-key holding area can release unrelated work, but adds durable sequencing state, storage, recovery logic, and memory limits. Processing everything after quarantine is operationally simple only when the domain explicitly tolerates missing predecessors.

There is no free bypass. If the system needs per-key holds, decide who owns the blocked-key registry and how a new consumer reconstructs it. Never rely only on an in-memory set that disappears during a rebalance.

This is also why adding partitions is not a neutral throughput change. See [partition changes and ordering contracts](/posts/kafka-partitions-change-ordering-contract).

## Keep the consumer alive without pretending progress

Kafka exposes controls for polling and automatic commits in its [consumer configuration](https://kafka.apache.org/41/configuration/consumer-configs/). Long processing can interact with the poll interval and group ownership. Pausing a partition does not remove the need to service the consumer appropriately.

Design retries so the poll loop and worker ownership remain understandable. Do not block one consumer thread indefinitely inside a failing handler. A green process-health check is weak evidence when one business key has been blocked for hours.

Measure the age of the oldest unresolved operation, quarantine arrival rate, repair completion rate, duplicate replay rate, and the count of blocked keys. Consumer lag alone cannot reveal whether the business has recovered.

## Replay is a release

Replaying a repaired record is another production change. Preserve its original event identity, record the repair version, and prevent an operator from accidentally replaying the same effect repeatedly.

Test these crash points: before quarantine durability, after quarantine but before source acknowledgment, during a rebalance, and after the side effect but before completion is recorded. Add the original failing payload as a restricted regression fixture when policy permits.

The final approval question is concrete: if this record is isolated, which later actions remain valid? A dead-letter queue is useful when the team can answer that question and can bring the operation back through a controlled repair path.

## References

- [KafkaConsumer API: offsets, commits, and partition control](https://kafka.apache.org/41/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html)
- [Kafka 4.1 consumer configuration](https://kafka.apache.org/41/configuration/consumer-configs/)
- [Related: rebalances and ownership](/posts/kafka-rebalances-are-stop-the-world-coordination)


---

# A Kafka Rebalance Is a Coordination Protocol, Not a Consumer Restart

> How Kafka consumer ownership, eager and cooperative rebalancing, offset commits, static membership, and generation fencing shape correctness and availability.

Canonical URL: https://www.ayushworks.xyz/posts/kafka-rebalances-are-stop-the-world-coordination
Author: Ayush Basak
Last modified: 2026-08-31
Topics: kafka, distributed-systems, event-streaming, reliability

A Kafka partition can be processed by only one member of a consumer group at a time. When members join, leave, stall, or when partition counts change, the group must transfer ownership. That transfer is a distributed coordination protocol with correctness boundaries—not a harmless restart detail.

The invariant is:

> At any group generation, each partition has at most one valid owner, and a new owner starts only from an offset consistent with the previous owner's completed effects.

## Eager rebalancing revokes everything

In the classic eager protocol, consumers revoke all partitions, the coordinator calculates a new assignment, and consumers resume. Even partitions whose owners do not change stop processing.

```text
steady state → member change → revoke all → assign all → restore state → resume
```

That pause becomes expensive when consumers maintain caches, local state stores, database connections, or large in-flight batches. Frequent deployments, autoscaling, long garbage-collection pauses, and processing that exceeds poll limits can create a rebalance loop where the group spends more time coordinating than consuming.

## Cooperative rebalancing narrows movement

`CooperativeStickyAssignor` preserves as many assignments as possible and transfers only partitions that must move. Ownership changes happen incrementally: a consumer first revokes a subset, then the next round assigns those partitions elsewhere.

This reduces disruption, but rollout must be compatible across the group. Kafka documentation requires all consumers to support the cooperative assignor. A mixed strategy can fall back to the common protocol selected by the group. Treat changing assignors like a protocol migration, not a one-client configuration tweak.

Kafka 4.x also provides the next-generation consumer rebalance protocol (KIP-848), enabled with `group.protocol=consumer`. Assignment moves server-side and heartbeat/session settings are coordinated differently. The migration choice must match broker and client versions; copying old timeout tuning into the new protocol without checking ownership is unsafe.

## Offset commit is not effect commit

The dangerous boundary is between processing a record and committing its next offset.

```text
read record 42
write invoice to database
crash before committing offset 43
new owner starts at 42
write invoice again
```

Rebalancing makes this ambiguity visible, but does not create it. Automatic offset commits can acknowledge work before external effects finish. Committing after the effect produces at-least-once delivery and therefore requires idempotency. Kafka transactions can atomically combine consumed offsets with records produced to Kafka, but cannot atomically include an arbitrary external database.

Use stable event identity at the destination:

```sql
INSERT INTO processed_event(event_id, processed_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING;
```

The business mutation and deduplication record must share one database transaction.

## Polling is a lease on ownership

Consumers must continue polling to prove liveness and receive group events. If processing blocks the poll loop beyond `max.poll.interval.ms`, the coordinator can remove the member and assign its partitions elsewhere. The old process may still be finishing work even though its ownership is gone.

Separate polling from bounded processing, pause partitions under backpressure, and cap in-flight work. Do not solve slow handlers by setting an enormous poll interval; that also increases the time before genuinely dead processing is recovered.

Static membership with `group.instance.id` can reduce reassignment after brief restarts by giving an instance a stable identity. It does not make two processes with the same identity safe. Deployment systems must prevent overlapping replicas that claim one member identity.

## Revoke callbacks are a deadline

On revocation:

1. stop admitting records from the revoked partitions;
2. finish or cancel bounded in-flight work;
3. persist required state and offsets;
4. release partition-scoped resources;
5. return before the rebalance timeout.

If cleanup depends on a slow external system, the group can stall. Prefer restartable state and idempotent effects over heroic shutdown hooks.

## Operational evidence

Track rebalance rate and duration, assigned partitions per member, time since last poll, records in flight, revoke-handler duration, consumer lag, duplicate suppression, offset-commit failures, and generation-related errors. Correlate lag spikes with deployments and autoscaling events.

## The CTO decision

Select the rebalance protocol from the cost of moving ownership. Use cooperative assignment and static identities to reduce unnecessary movement, but preserve correctness with bounded processing and idempotent effects. Scale on sustained lag and processing capacity, not on momentary queue depth that causes members to churn.

Kafka gives the group one owner at a time. Your application still has to ensure that the old owner's ambiguous effects and the new owner's retries converge.

## References

- [Kafka: Consumer Rebalance Protocol](https://kafka.apache.org/42/operations/consumer-rebalance-protocol/)
- [Kafka: Consumer Configurations](https://kafka.apache.org/42/configuration/consumer-configs/)
- [Related: Transactional outbox delivery guarantees](/posts/transactional-outbox-delivery-guarantees)
- [Related: Queues, backpressure, and overload control](/posts/queues-backpressure-overload-control)


---

# Kubernetes Availability Is Placement Plus Disruption Policy

> Why replicas alone do not create availability, and how topology spread, disruption budgets, probes, and capacity must form one failure-domain policy.

Canonical URL: https://www.ayushworks.xyz/posts/kubernetes-availability-is-placement-plus-disruption
Author: Ayush Basak
Last modified: 2026-08-30
Topics: kubernetes, cloud-infrastructure, reliability, system-design

Three replicas on one node are one failure domain wearing three Pod names. Kubernetes availability comes from where replicas can run, what may evict them, whether replacements can fit, and when traffic considers them ready.

The invariant is:

> After any planned single-domain disruption, enough ready capacity remains to serve the declared load.

Replica count is only an input to that statement.

## Placement encodes the failure model

Topology spread constraints let the scheduler distribute matching Pods across node, zone, or another labeled domain. A strict zone constraint might be:

```yaml
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: checkout
```

`maxSkew: 1` limits imbalance. `DoNotSchedule` preserves the placement invariant by refusing an unsafe placement. `ScheduleAnyway` prioritizes progress and treats spread as a preference. Neither is universally correct: strict policy can leave Pods pending when one zone lacks capacity, while soft policy may silently concentrate the service.

The selector must match the workload's own Pod labels. Otherwise the scheduler can create “ghost” placements that do not count themselves in the calculation.

## A PDB governs only voluntary eviction

A PodDisruptionBudget limits how many selected Pods may be unavailable during voluntary disruptions that use the Eviction API, such as a respectful node drain.

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: checkout
```

It does not create replicas, add capacity, repair a bad readiness probe, or protect against every deletion path. Kubernetes documentation explicitly notes that direct deletion of Pods or Deployments can bypass PDBs. Involuntary failures such as hardware loss are not prevented either.

A PDB is therefore a maintenance admission policy—not an availability guarantee.

## Capacity closes the loop

Strict spreading can deadlock a rollout if the cluster has no spare slot in the required zone. Autoscaling may not recognize a topology domain whose node group is scaled to zero unless the autoscaler understands the full domain set.

Before enabling strict policy, test:

1. one zone unavailable;
2. one node draining during a deployment;
3. a replica already unready when maintenance begins;
4. the largest expected surge plus one failed domain;
5. a scale-from-zero node group.

Reserve enough headroom for replacement Pods. A design requiring all healthy nodes to stay at 90% utilization cannot honestly claim single-node fault tolerance.

## Readiness is part of the budget

The PDB counts health using Pod readiness. A probe that turns green before caches, connections, or migrations are ready allows the platform to evict another replica too early. A probe that depends on every downstream service can make the whole fleet unready during a dependency incident.

Readiness should answer one narrow question: can this instance safely accept its class of traffic now?

## Review the complete policy

| Boundary | Question |
|---|---|
| replicas | how many ready instances does peak load require? |
| placement | which node, zone, or rack losses must be tolerated? |
| disruption | how much planned unavailability may proceed? |
| capacity | where can replacements actually schedule? |
| readiness | when is a replacement truly serving? |
| rollout | can old and new versions coexist within the same budget? |

Observe replicas per topology domain, unschedulable Pod reasons, PDB-blocked evictions, rollout duration, readiness latency, and available capacity after a simulated domain loss.

## The CTO decision

Declare the failure domain before choosing replica count. Use strict spread where concentration violates the service objective, soft spread where degraded placement is preferable to no placement, and PDBs to encode the amount of voluntary disruption the application can absorb. Then prove that capacity and readiness make those declarations true.

Availability is not “replicas: 3.” It is a tested relationship between placement, health, disruption, and spare capacity.

## References

- [Kubernetes: Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/)
- [Kubernetes: Disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)
- [Kubernetes: Running in Multiple Zones](https://kubernetes.io/docs/setup/best-practices/multiple-zones/)
- [Snippet: Zone-aware deployment budget](/snippets/topology-spread-budget)


---

# KYAML in Practice: Explicit Kubernetes Manifests Without a Migration

> How KYAML changes manifest formatting, how to generate it, and where the tradeoff is useful.

Canonical URL: https://www.ayushworks.xyz/posts/kubernetes-kyaml-production-workflow
Author: Ayush Basak
Last modified: 2026-08-17
Topics: kubernetes, cloud-infrastructure, platform-engineering

Kubernetes manifests are data structures disguised as indentation. Most of the time that is convenient. During a large review, however, indentation makes it surprisingly hard to answer basic questions: where does this mapping end, which list owns this item, and did a formatter change structure or only whitespace?

KYAML is an alternative presentation of the same data. It makes structure explicit with braces, brackets, quoted strings, and trailing commas. Crucially, it is not a new Kubernetes API or serialization format. Every valid KYAML document is valid YAML, so existing Kubernetes versions and YAML-aware tooling can consume it without a migration.

The feature is best understood as a formatting decision with operational consequences—not a new configuration language.

## What changes

A conventional manifest relies on indentation:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: demo
  labels:
    app: demo
spec:
  containers:
    - name: nginx
      image: nginx:1.20
```

The KYAML representation makes the containers explicit. A simplified example looks like this:

```yaml
{
  apiVersion: "v1",
  kind: "Pod",
  metadata: {
    name: "demo",
    labels: {
      app: "demo",
    },
  },
  spec: {
    containers: [
      {
        name: "nginx",
        image: "nginx:1.20",
      },
    ],
  },
}
```

The values and hierarchy are unchanged. The additional punctuation carries structural information that block-style YAML leaves to whitespace. That can reduce ambiguity when humans review deeply nested manifests or when generated files produce noisy diffs.

## Generate KYAML with kubectl

Kubernetes added KYAML as a native `kubectl` output format in version 1.34. The initial release was alpha and opt-in:

```bash
export KUBECTL_KYAML=true
kubectl get deployment my-app -o kyaml
```

In Kubernetes 1.35 and later, the beta feature is enabled by default, but the output still must be requested:

```bash
kubectl get deployment my-app -o kyaml > my-app.yaml
```

There are currently no plans to make KYAML the universal default. From Kubernetes 1.36, teams that prefer it can configure `get` output through `kuberc`:

```bash
kubectl kuberc set --section defaults --command get --option output=kyaml
```

Version details matter here. A repository that supports several `kubectl` versions should not assume that every contributor has native `-o kyaml` support.

## Format files outside the cluster

KYAML does not require a live API server. The `sigs.k8s.io/yaml` project includes a formatter that reads a file or directory and writes KYAML to standard output:

```bash
go install sigs.k8s.io/yaml/yamlfmt@latest
yamlfmt -o=kyaml deployment.yaml > deployment.kyaml
yamlfmt -o=kyaml -d deployment.yaml
```

The diff mode is useful in CI because it can show formatting drift without silently changing the working tree.

Google's `yamlfmt` also provides a KYAML formatter. A repository can declare the format once:

```yaml
formatter:
  type: kyaml
```

Then preview or apply it consistently:

```bash
yamlfmt -dry ./k8s/
yamlfmt ./k8s/
```

Pick one formatter and pin its version. Two tools that target the same style can still differ at edge cases or evolve at different times; a floating formatter version turns formatting into an uncontrolled build input.

## A production adoption pattern

Treat KYAML adoption like any other repository-wide formatting change:

1. Choose a formatter and pin the exact version.
2. Convert manifests in a dedicated commit with no semantic edits.
3. Compare parsed objects before and after conversion, not only text.
4. Add a CI check so later pull requests cannot mix styles accidentally.
5. Document how developers reproduce the check locally.

For the semantic comparison, parse both versions and compare normalized JSON. This catches a formatter or conversion mistake that a visual review might miss. Continue running your normal server-side or client-side Kubernetes validation as well; formatting does not validate resource schemas, admission policy, or cluster compatibility.

## Tradeoffs

KYAML adds punctuation. Small manifests can feel heavier, and engineers already fluent in block YAML may find it less pleasant to write by hand. It also does not remove Kubernetes' real sources of configuration complexity: schema evolution, defaulting, controllers, admission, and environment-specific overlays.

Its value appears when explicit structure matters more than minimal syntax—large generated manifests, code review, machine-authored configuration, and repositories where inconsistent YAML styles produce recurring churn.

Do not present KYAML as a correctness boundary. A perfectly formatted manifest can still request the wrong image, omit a resource limit, or fail policy. KYAML makes the document's shape easier to inspect; it does not prove that the desired state is correct.

## Conclusion

KYAML is a low-risk experiment because it preserves YAML compatibility. Start with generated output or one manifest directory, measure whether reviews and diffs become clearer, and keep it only if the explicit syntax earns its extra visual weight.

The useful first-principles question is not “is KYAML better than YAML?” It is “does making structure explicit reduce mistakes in this repository?”

## References

- [Kubernetes: How to Pretty-Print Your Kubernetes YAML as KYAML](https://kubernetes.io/blog/2026/08/11/how-to-pretty-print-kubernetes-yaml-as-kyaml/)
- [Kubernetes API concepts](https://kubernetes.io/docs/reference/using-api/api-concepts/)


---

# Kubernetes Probes Are Failure Policy, Not Health Endpoints

> Designing startup, readiness, and liveness probes around distinct recovery actions without turning dependency failures into restart storms.

Canonical URL: https://www.ayushworks.xyz/posts/kubernetes-probes-are-failure-policy
Author: Ayush Basak
Last modified: 2026-09-02
Topics: kubernetes, cloud-infrastructure, reliability, platform-engineering, system-design

Many services expose one `/health` endpoint and wire it into startup, readiness, and liveness probes. That is convenient configuration and weak failure design. The three probes cause different state transitions, so they should answer different questions.

The invariant is:

> A probe should report only the condition for the recovery action Kubernetes will take when that probe fails.

Liveness failure restarts a container. Readiness failure removes a Pod from Service traffic. Startup failure prevents the other probes from running and eventually restarts a process that cannot initialize. Treating them as synonyms converts a dependency incident into self-inflicted churn.

## Begin with actions, not endpoints

| Probe | Question | Failure action |
| --- | --- | --- |
| startup | Has this process completed initialization? | restart after startup budget expires |
| readiness | Can this instance accept its intended traffic now? | remove it from matching Service endpoints |
| liveness | Is this process incapable of recovering without restart? | restart the container |

A database outage may make an API temporarily unable to serve requests, so readiness could fail. It does not usually mean restarting every API process will repair the database. If liveness checks the database, the outage can trigger a fleet-wide restart storm, cold caches, reconnection spikes, and more load on the failing dependency.

## Keep liveness local and conservative

Liveness should detect states where process replacement is the correct treatment: a deadlocked event loop, a permanently stopped worker, or a corrupted internal state the application knows it cannot repair.

It should be cheap, bounded, and independent of shared remote dependencies:

```ts
app.get('/live', (_req, res) => {
  const eventLoopProgressing = heartbeat.ageMs() < 5_000
  const workerProgressing = worker.lastProgressAgeMs() < 30_000

  if (!eventLoopProgressing || !workerProgressing)
    return res.status(503).end()

  return res.status(204).end()
})
```

Even this logic needs workload context. A worker with no jobs is not stuck. Progress signals must distinguish “nothing to do” from “unable to do it.”

## Readiness is an admission decision

Readiness decides whether a particular instance should receive more work. It may include strictly required local and remote conditions, but avoid a naïve fan-out to every dependency on every probe interval.

If each of 500 Pods probes five downstream services every second, the health system creates 2,500 requests per second before customer traffic arrives. A slow dependency then occupies probe sockets and CPU across the fleet.

Maintain bounded, asynchronous dependency state instead:

```ts
let databaseReady = false

setInterval(async () => {
  databaseReady = await checkDatabaseWithDeadline(200)
}, 2_000)

app.get('/ready', (_req, res) => {
  if (draining || !configurationLoaded || !databaseReady)
    return res.status(503).end()
  return res.status(204).end()
})
```

The endpoint remains fast while the background check has its own deadline. Decide how stale the cached readiness signal may be, and expose its age for diagnosis.

Not every dependency belongs in readiness. If recommendations can fall back to popular items, the recommendation service should not remove itself because its ranking store is down. Readiness should represent minimum viable service, not perfect service.

## Startup needs a measured budget

Startup probes protect slow initialization from premature liveness restarts. Their effective budget is approximately:

```text
failureThreshold × periodSeconds
```

Set it from observed cold-start distributions plus operational margin, not from one developer laptop. Include image startup, secrets delivery, migrations or cache hydration, and runtime compilation where applicable.

Do not make every replica run schema migrations before becoming ready. Separate one-time control-plane work from replica startup when possible. Otherwise a slow lock can prevent an entire deployment from starting.

## Thresholds are temporal policy

The default one-second `timeoutSeconds` can be too aggressive for loaded nodes; an extremely long timeout hides failure. `failureThreshold` filters brief noise, while `successThreshold` on readiness prevents a flapping instance from re-entering too quickly.

Model the timeline:

```text
detection time ~= periodSeconds × failureThreshold + probe timeout effects
re-entry time  ~= periodSeconds × successThreshold
```

That timeline should fit the service SLO and the capacity reserve. If readiness removes Pods faster than autoscaling or recovery can replace capacity, one transient slowdown can cascade into overload on the survivors.

## Probe behavior belongs in load tests

Test probes under conditions that matter:

- CPU throttling and event-loop delay;
- exhausted database pools;
- DNS and dependency latency;
- rolling deployments at minimum replica count;
- node drain and graceful termination;
- cold starts after a regional scale-up;
- partial dependency failure with fallback paths active.

Track probe latency and outcomes, Pod readiness transitions, restarts by reason, time to ready, endpoint churn, traffic error rate, and downstream connection attempts. Correlate them on one incident timeline.

## Common mistakes

- Using the same handler for all three probes.
- Calling every dependency from liveness.
- Returning a large diagnostic payload; probes need a status, not a dashboard.
- Hiding overload behind a readiness failure without retaining capacity headroom.
- Using a TCP-open check for a process whose worker loop can be deadlocked.
- Treating readiness removal as instantaneous connection draining.
- Forgetting that a probe itself consumes CPU, sockets, processes, and logs.

## The CTO decision

Health checks are executable recovery policy. For each failure signal, state who is unhealthy, what automatic action follows, how quickly it should happen, and whether that action improves the situation under a shared dependency outage.

Good probes make the smallest safe state transition. They stop traffic before restarting, distinguish startup from steady state, and never multiply a remote incident into fleet-wide self-destruction.

## References

- [Kubernetes: Liveness, readiness, and startup probes](https://kubernetes.io/docs/concepts/workloads/pods/probes/)
- [Kubernetes: Configure probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
- [Kubernetes: Pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)
- [Related: Kubernetes termination is a distributed protocol](/posts/kubernetes-termination-is-a-distributed-protocol)
- [Related: Kubernetes availability is placement plus disruption](/posts/kubernetes-availability-is-placement-plus-disruption)


---

# Kubernetes Requests Are an Economic Model

> A CTO-level method for setting Kubernetes requests, limits, QoS, autoscaling, and capacity policy from workload evidence instead of copied YAML.

Canonical URL: https://www.ayushworks.xyz/posts/kubernetes-resource-policy-capacity-governance
Author: Ayush Basak
Last modified: 2026-08-28
Topics: kubernetes, cloud-infrastructure, reliability, capacity-planning, technical-leadership

Copied resource YAML creates two kinds of waste: idle nodes paid for by inflated requests, and outages caused by limits that were never tested against real workload behaviour.

Kubernetes requests and limits are not tuning details. Requests express the capacity the scheduler must reserve. Limits constrain consumption. Together with replicas and autoscaling, they encode the service’s economic and failure model.

## Requests buy placement

The scheduler uses requests to decide whether a Pod fits on a node. A request is therefore a claim on cluster capacity even when the process is idle.

```yaml
resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    memory: 768Mi
```

If 100 replicas each request 1 CPU but normally use 100m, the organisation reserves roughly ten times observed steady-state CPU before headroom. Lowering requests blindly can improve packing until synchronized traffic arrives and every Pod competes for CPU at once.

Set requests from representative distributions, not averages. CPU can often begin near a sustained high percentile plus startup and burst evidence. Memory should include working-set peaks, runtime behaviour, and a safety margin because memory exhaustion is not gracefully throttled.

## CPU and memory fail differently

CPU is compressible. When demand exceeds available CPU or a configured limit, work slows and latency rises. A restrictive CPU limit can produce throttling even when a node has spare capacity, turning a protection mechanism into a latency incident.

Memory is incompressible. Exceeding the memory limit can terminate the container. Without a limit, a leak can pressure the node and trigger eviction elsewhere.

That leads to a useful default for many services: specify CPU requests based on scheduling needs, be cautious with CPU limits, and set measured memory requests and limits with an explicit OOM recovery model. This is not universal—multi-tenant or untrusted compute may require hard CPU ceilings—but the decision should be intentional.

## QoS is incident ordering

Kubernetes assigns Pods to `Guaranteed`, `Burstable`, or `BestEffort` classes based on requests and limits. Under node pressure, eviction preference considers those classes. QoS is therefore a statement about which workloads the platform sacrifices first.

- `BestEffort`: no CPU or memory requests/limits; suitable only for genuinely disposable work.
- `Burstable`: some resources specified or requests differ from limits; common for elastic services.
- `Guaranteed`: CPU and memory requests equal limits for every container; strongest reservation and least flexible packing.

Do not make everything Guaranteed to feel safe. It can reserve large amounts of idle capacity and reduce the room the scheduler has to respond. Match class to business criticality and workload shape.

## Autoscaling needs compatible signals

Horizontal Pod Autoscaler CPU utilisation is measured relative to requested CPU. Change the request and the same workload produces a different utilisation percentage. A team can “fix” autoscaling by changing requests while silently changing cluster reservation.

For queue consumers, queue age or drain time is often a better scaling signal than CPU. For APIs, combine concurrency, latency, and saturation evidence. Scaling on a lagging signal after capacity is exhausted creates replica storms that compete for the same database connections.

Model the whole path:

```text
new replicas
  -> startup CPU and image pulls
  -> readiness delay
  -> connection pool growth
  -> downstream request growth
  -> actual useful capacity
```

Autoscaling cannot manufacture downstream capacity.

## Establish a resource policy

A platform policy should require:

1. requests for every production container, including sidecars;
2. measured memory limits and an OOM alert;
3. startup, readiness, and liveness probes with different purposes;
4. Pod disruption budgets for quorum and availability needs;
5. a maximum replica count derived from dependency capacity;
6. periodic right-sizing using production percentiles;
7. documented exceptions for CPU limits and Guaranteed QoS.

Use admission policy to reject missing requests, not to impose one universal value. Defaults are useful for development and dangerous as permanent production assumptions.

## Review cost and reliability together

For each service, report:

- requested versus used CPU and memory;
- throttling and OOM events;
- pending time and scheduling failures;
- replicas versus useful throughput;
- node headroom during peak and failure;
- downstream saturation during scale-out;
- cost per successful business operation.

High reservation with low use suggests packing waste. High usage near requests with latency growth suggests capacity risk. OOMs suggest either an incorrect limit or an application memory problem; raising the limit without a heap profile is not diagnosis.

## The executive question

Ask why a service requests the capacity it does. “The template said so” means the cluster has no defensible capacity model.

A production resource policy should let engineering explain how much demand a replica serves, how the fleet behaves during a node loss, what dependency becomes limiting first, and how much safety margin the company is buying. At that point, YAML becomes an operating contract rather than decoration.

## References

- [Kubernetes: Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
- [Kubernetes: Pod Quality of Service Classes](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/)
- [Kubernetes: Resource Management](https://kubernetes.io/docs/concepts/resource-management/)
- [Related: Cloud Unit Economics for Architects](/posts/cloud-unit-economics-for-architects)
- [Related: Queues, Backpressure, and Overload Control](/posts/queues-backpressure-overload-control)


---

# Kubernetes Termination Is a Distributed Protocol

> Why SIGTERM, EndpointSlices, readiness, connection draining, and grace periods must be designed as one shutdown contract.

Canonical URL: https://www.ayushworks.xyz/posts/kubernetes-termination-is-a-distributed-protocol
Author: Ayush Basak
Last modified: 2026-09-01
Topics: kubernetes, cloud-infrastructure, reliability, system-design

Handling `SIGTERM` is necessary for graceful shutdown. It is not sufficient.

A terminating Kubernetes Pod participates in several concurrent systems: the API server marks it for deletion, EndpointSlices change readiness state, kubelet may execute a `preStop` hook, the runtime signals PID 1, proxies and load balancers converge, clients reuse connections, and the application drains work. These events are coordinated, not instantaneous.

The shutdown invariant is:

> Stop accepting new work before the grace budget expires, finish or safely transfer accepted work, then terminate.

## There is no universal ordering

Kubernetes begins local shutdown while the control plane updates service endpoints. A terminating endpoint remains represented but has `ready: false`; systems aware of the newer conditions can inspect whether it is still `serving`. External load balancers and client-side caches may observe that change later.

That creates an overlap:

```text
endpoint withdrawal ──────────────► converges
application signal ──► draining ──► exits
existing connections ─────────────► may still send requests
```

If the process exits immediately on `SIGTERM`, late traffic fails. If it keeps reporting ready and accepts new work forever, the grace period ends with `SIGKILL`.

## Budget the whole path

The termination grace countdown includes the `preStop` hook. A 30-second grace period with a 20-second sleep leaves roughly 10 seconds for the application to drain. Kubernetes may grant a small one-off extension when a hook is still running, but that is not a capacity plan.

Use an explicit budget:

```text
propagation allowance + maximum request duration
+ worker drain time + safety margin
< terminationGracePeriodSeconds
```

The numbers must come from production behavior. A service with 200 ms requests and a stream processor with ten-minute leases need different protocols.

## Application shutdown sequence

```ts
process.on('SIGTERM', async () => {
  readiness.set(false)
  server.close()                 // stop new connections
  await workers.stopClaiming()   // stop new background work
  await Promise.race([
    Promise.all([server.drained(), workers.drained()]),
    deadline(20_000),
  ])
  await telemetry.flush()
  process.exit(0)
})
```

This sketch still needs bounded operations and cancellation. Shutdown code that waits indefinitely converts a graceful deployment into forced termination.

For queue workers, distinguish “claimed” from “completed.” A lease or visibility timeout must allow another worker to recover abandoned work. For HTTP writes, idempotency protects the caller that received a disconnect after the server committed.

## Failure policy

| Condition | Required behavior |
|---|---|
| New request after drain begins | reject quickly or route elsewhere |
| Existing read | finish within deadline |
| Existing write with ambiguous response | preserve idempotency identity |
| Long background job | checkpoint, release lease, or extend deliberately |
| Telemetry exporter unavailable | bounded flush; do not block exit forever |
| Grace exhausted | accept forced termination and recover safely |

## Verify with disruption

Deployment success is not proof. Run repeated rolling updates under load. Track response-code spikes, terminated requests, queue lease recovery, shutdown duration, and forced kills. Test persistent connections and slow requests. Confirm PID 1 receives signals in the actual container image.

Avoid a blind `preStop: sleep`. A small propagation delay can be pragmatic, but it should accompany application draining and be justified by measured convergence.

## Deployment is the acceptance test

A readiness probe that turns false is useful only if every routing layer responds in time. Verify the complete path: Kubernetes Service, ingress, cloud load balancer, service mesh, and any client-side discovery cache. Track requests that arrive after the application entered draining state. That counter turns a timing assumption into evidence.

Keep shutdown dependencies minimal. If draining requires a database that is already unavailable, use bounded cleanup and rely on leases or idempotency for recovery. Termination must remain safe during the same partial failures that often trigger rescheduling.

## Conclusion

Graceful termination is a protocol across routing, process lifecycle, workload ownership, and time. Kubernetes supplies signals and state transitions; the application supplies the business meaning.

Design shutdown with the same care as startup. A service that cannot leave safely is not highly available—it is merely easy to start.

---

Further reading: [Kubernetes Pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/), [container lifecycle hooks](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/), [termination-flow tutorial](https://kubernetes.io/docs/tutorials/services/pods-and-endpoint-termination-flow/), and [Kubernetes availability budgets](/posts/kubernetes-availability-is-placement-plus-disruption).


---

# LLM Serving Is KV-Cache Capacity Planning

> A practical model for PagedAttention, continuous batching, admission control, and the latency-throughput trade-offs of production LLM inference.

Canonical URL: https://www.ayushworks.xyz/posts/llm-serving-is-kv-cache-capacity-planning
Author: Ayush Basak
Last modified: 2026-08-30
Topics: ai-infrastructure, llm-inference, capacity-planning, gpu

Teams often size an inference fleet by model weights and GPU count. That misses the resource whose usage changes with every request: the key-value cache. Long prompts, long generations, and concurrent sequences compete for KV-cache blocks; when the budget is exhausted, requests queue, preempt, recompute, or fail.

The governing invariant is:

> Admit work only when its estimated KV-cache demand and deadline fit the capacity reserved for its service class.

## Why the cache dominates concurrency

During autoregressive decoding, a transformer reuses keys and values from prior tokens. The cache grows with active sequence length, layer count, head geometry, precision, and concurrent sequences. Model weights are mostly fixed; KV state is workload-shaped.

A rough planning model is:

```text
KV bytes per token ≈ 2 × layers × KV heads × head dimension × bytes per element
request KV bytes ≈ KV bytes per token × (prompt tokens + generated tokens)
```

The estimate is deliberately conservative. Parallelism layout, quantization, allocator block size, and model architecture change the real number. Measure the engine rather than treating the equation as billing truth.

## PagedAttention fixes allocation, not economics

The vLLM paper identifies fragmentation and redundant cache duplication as major barriers to batching. PagedAttention divides KV state into blocks and maps logical sequence blocks to non-contiguous physical blocks, borrowing the core idea of virtual memory. That reduces waste and enables sharing such as copy-on-write for related sequences.

This improves usable capacity. It does not make capacity infinite. An unbounded 128k-token request can still displace many short interactive requests.

## Continuous batching changes the queue

Static batching waits for a group and runs it together, leaving capacity idle as sequences finish at different times. Continuous batching can admit new sequences between decode iterations. Throughput improves, but the scheduler is now a production control plane: its policy determines time to first token, inter-token latency, starvation, and fairness.

Separate at least two classes:

- interactive traffic, optimized for time to first token and bounded generation;
- batch traffic, optimized for throughput and willing to wait.

Do not allow offline summarization to occupy every cache block while customer requests queue.

## Admission must happen before allocation

Validate and tokenize early enough to estimate cost. Then gate on:

```ts
const estimatedTokens = promptTokens + maxOutputTokens
const fitsRequest = estimatedTokens <= policy.maxSequenceTokens
const fitsPool = estimatedTokens <= scheduler.availableTokenSlots()

if (!fitsRequest) throw new RequestTooLargeError()
if (!fitsPool) return queueOrReject(policy.overloadMode)
```

Reserve `maxOutputTokens`; do not assume the model will stop early. Put tenant quotas and deadlines above the engine. When overload occurs, reject before spending expensive prefill compute if the request cannot complete usefully.

## Watch prefill and decode separately

Prefill processes the prompt and is compute-heavy. Decode generates tokens iteratively and is often memory-bandwidth constrained. One aggregate “request latency” hides which stage is saturated.

Track:

- time to first token;
- inter-token latency and generation throughput;
- queued, running, and preempted sequences;
- KV-cache utilization and allocation failures;
- prompt and generated-token distributions;
- latency and rejection by service class;
- useful tokens per accelerator-second.

If preemption rises, reducing `max_num_seqs` or maximum batched tokens may improve tail latency even when headline throughput falls. The correct setting depends on the SLO, not a benchmark maximum.

## Common mistakes

1. Advertising maximum context length as the default entitlement.
2. Routing only by model name while ignoring current cache pressure.
3. Mixing batch and interactive work in one unreserved pool.
4. Retrying rejected generations without preserving the caller deadline.
5. Autoscaling on GPU utilization alone; a saturated cache can coexist with misleading compute metrics.

## The CTO decision

Treat context length and output tokens as scarce capacity, not UI parameters. Establish per-class limits, queue bounds, rejection behavior, and cost attribution before increasing fleet size. PagedAttention makes allocation more efficient; admission control decides whether the product remains predictable.

## References

- [PagedAttention paper](https://arxiv.org/abs/2309.06180)
- [vLLM: PagedAttention design](https://docs.vllm.ai/en/latest/design/paged_attention/)
- [vLLM: Optimization and tuning](https://docs.vllm.ai/en/latest/configuration/optimization/)
- [Snippet: KV-cache admission budget](/snippets/kv-cache-admission)


---

# A Modern PostgreSQL JDBC Driver Built for Virtual Threads and Native Protocols

> Explore the architecture of pg-java, a PostgreSQL-first driver leveraging Java 21 virtual threads, ReentrantLock unpinning, and decoupled JDBC layering.

Canonical URL: https://www.ayushworks.xyz/posts/modern-postgresql-jdbc-driver-virtual-threads
Author: Ayush Basak
Last modified: 2026-08-19
Topics: postgresql, java, jdbc, virtual-threads, concurrency

For over a decade, achieving high concurrency and database driver throughput in Java required adopting asynchronous reactive frameworks like R2DBC or RxJava. While reactive drivers freed thread resources, they introduced operational complexity, difficult execution stack traces, and non-linear control flows. With Java 21 virtual threads, this fundamental tradeoff changes. The new `pg-java` driver—a pre-release PostgreSQL driver written from scratch by Sehrope Sarkini and Claude—demonstrates a modern architectural paradigm by combining blocking-style synchronous code with a PostgreSQL-native wire protocol engine ([PostgreSQL News](https://www.postgresql.org/about/news/new-modern-jdbc-driver-for-postgresql-3354/)).

## PostgreSQL-First Architecture vs. Lowest Common Denominator

Traditional Java Database Connectivity (JDBC) drivers build their execution core directly around `java.sql.*` interfaces. Because standard JDBC is designed to support any relational database, its baseline abstractions represent a lowest-common-denominator feature set. Once those generic JDBC assumptions get baked into the execution core, database-specific optimizations and wire-protocol capabilities become difficult to expose natively.

In contrast, `pg-java` reverses this ordering by treating standard JDBC support as a secondary layer built on top of a PostgreSQL-native core API ([PostgreSQL News](https://www.postgresql.org/about/news/new-modern-jdbc-driver-for-postgresql-3354/)).

```
+-------------------------------------------------------------+
|              Application Layer / Microservices              |
+------------------------------+------------------------------+
                               |
            +------------------+------------------+
            |                                     |
            v                                     v
+-----------------------+             +-----------------------+
|  JDBC Layer (java.sql) |             |   Native API Core     |
| (Connection, ResultSet|             |  (Direct Wire Engine  |
|   DataSource, XA)     |             |   & Streaming)        |
+-----------+-----------+             +-----------+-----------+
            |                                     |
            +------------------+------------------+
                               |
                               v
+-------------------------------------------------------------+
|                 PostgreSQL Wire Protocol                    |
+-------------------------------------------------------------+
```

By decoupling wire protocol handling from standard JDBC, `pg-java` allows direct access to PostgreSQL protocol capabilities without sacrificing standard API compatibility for object-relational mappers (ORMs). This layered design aligns with best practices when [building reliable microservices](/posts/building-reliable-microservices) that require standard integrations alongside high throughput.

## Virtual Threads and Preventing Carrier Thread Pinning

To handle thousands of concurrent connections using straightforward blocking-style I/O, `pg-java` relies on Java 21 virtual threads ([PostgreSQL News](https://www.postgresql.org/about/news/new-modern-jdbc-driver-for-postgresql-3354/)). However, virtual thread architectures require strict avoidance of carrier thread pinning. Pinning occurs when a virtual thread executes a blocking operation inside a `synchronized` block or method, locking the underlying OS carrier thread and preventing other virtual threads from scheduling.

To keep its blocking-style I/O friendly to virtual threads, `pg-java` avoids pinning carrier threads; the project specifically describes using `java.util.concurrent.locks.ReentrantLock` rather than `synchronized` around I/O ([PostgreSQL News](https://www.postgresql.org/about/news/new-modern-jdbc-driver-for-postgresql-3354/), [source repository](https://github.com/pgjdbc/pg-java)). The following is an illustrative pattern, not a class copied from the driver:

```java
public final class VirtualThreadFriendlyChannel {
    private final ReentrantLock ioLock = new ReentrantLock();
    private final SocketChannel socketChannel;

    public VirtualThreadFriendlyChannel(SocketChannel socketChannel) {
        this.socketChannel = socketChannel;
    }

    public void sendCommand(byte[] commandPayload) {
        ioLock.lock();
        try {
            // Blocking write executed on Virtual Thread without carrier pinning
            ByteBuffer buffer = ByteBuffer.wrap(commandPayload);
            while (buffer.hasRemaining()) {
                socketChannel.write(buffer);
            }
        } catch (IOException e) {
            throw new RuntimeException("Socket write failed", e);
        } finally {
            ioLock.unlock();
        }
    }
}
```

Because `ReentrantLock` decouples lock acquisition from JVM object monitors, virtual threads waiting on `ioLock.lock()` or blocking on `socketChannel.write()` unmount cleanly from their carrier thread.

## Native Streaming and JDBC Layering

Streaming is the core query execution primitive in `pg-java`. Standard JDBC drivers often buffer row datasets or require explicit fetch-size configurations to prevent memory exhaustion. The native `pg-java` engine streams protocol messages directly from the socket to processing consumers by default ([PostgreSQL News](https://www.postgresql.org/about/news/new-modern-jdbc-driver-for-postgresql-3354/)).

When standard compatibility is required, the `java.sql.*` layer provides wrappers for standard abstractions, including `Connection`, `PreparedStatement`, `ResultSet`, `DatabaseMetaData`, `DataSource`, and `XA` ([PostgreSQL News](https://www.postgresql.org/about/news/new-modern-jdbc-driver-for-postgresql-3354/)). For further reading on high-concurrency JVM patterns, explore our [engineering notes](/engineering-notes).

## Architectural Comparisons

| Feature / Dimension | Standard JDBC Driver | Reactive Driver (e.g., R2DBC) | Modern `pg-java` Core |
| :--- | :--- | :--- | :--- |
| **Concurrency Model** | Platform thread per connection | Non-blocking Event Loops | Java 21 Virtual Threads |
| **API Paradigm** | Blocking `java.sql.*` | Async Callbacks / Reactive Streams | Simple Blocking + Native Streams |
| **Wire Protocol Core** | Generic standard JDBC core | Custom Reactive core | Native PostgreSQL core |
| **Carrier Pinning Risk** | Depends on implementation | N/A | Driver is designed to avoid pinning during I/O |

## When NOT to Use This Driver

1. **Pre-Java 21 Environments**: `pg-java` depends fundamentally on Java 21 virtual threads and modern concurrency models. Applications running on LTS Java 11 or 17 cannot use this driver.
2. **Production Systems Requiring Strict SLAs**: Because `pg-java` is currently in pre-release status ([PostgreSQL News](https://www.postgresql.org/about/news/new-modern-jdbc-driver-for-postgresql-3354/)), it should not yet be deployed to mission-critical production systems requiring enterprise vendor support.
3. **Multi-Database Agnostic Systems**: If your application explicitly relies on database-agnostic abstractions without PostgreSQL-specific protocol benefits, standard JDBC drivers remain the conventional choice.

## Common Pitfalls

- **Application-Level Pinning**: Using `pg-java` on virtual threads will not prevent pinning if your application code executes driver queries inside custom `synchronized` methods. Use `ReentrantLock` across your application service layer as well.
- **Expecting Immediate 100% JDBC Compliance**: A `java.sql.*` layer already exists, but full JDBC specification compliance remains a long-term project goal ([PostgreSQL News](https://www.postgresql.org/about/news/new-modern-jdbc-driver-for-postgresql-3354/)).

## Conclusion

By prioritizing a PostgreSQL-native core, building standard JDBC abstractions as a secondary layer, and preventing carrier thread pinning with `ReentrantLock`, `pg-java` provides a blueprint for modern database drivers on modern Java runtimes.

## References

- [PostgreSQL News: New Modern JDBC Driver for PostgreSQL](https://www.postgresql.org/about/news/new-modern-jdbc-driver-for-postgresql-3354/)
- [pgjdbc/pg-java source repository](https://github.com/pgjdbc/pg-java)
- [PostgreSQL Documentation: Concurrency Control](https://www.postgresql.org/docs/current/mvcc.html)


---

# Tenant Isolation Is a Data Model, Not a WHERE Clause

> A CTO-level guide to pooled and siloed SaaS tenancy, PostgreSQL row security, identity propagation, noisy-neighbor controls, and proving isolation in production.

Canonical URL: https://www.ayushworks.xyz/posts/multi-tenant-saas-isolation
Author: Ayush Basak
Last modified: 2026-08-24
Topics: saas, postgresql, security, system-design

The most common multi-tenant incident is not a broken cipher. It is a valid query executed with the wrong tenant context.

That is why `WHERE tenant_id = ?` is not an isolation architecture. It is one predicate that every code path, migration, background worker, cache key, search index, export, and support tool must remember forever.

Tenant isolation is the system-wide guarantee that one tenant cannot consume, infer, modify, or starve another tenant’s resources beyond an explicitly approved boundary.

## Choose a tenancy model per resource

“Are we pooled or siloed?” is usually the wrong binary question. A SaaS product can pool stateless compute, silo encryption keys, partition queues, and offer dedicated databases to regulated customers.

Use three broad models:

| Model | Shape | Strength | Cost |
|---|---|---|---|
| Silo | Dedicated resource per tenant | Clear blast radius and customization | Operational multiplication |
| Bridge | Shared service, partitioned resource groups | Tunable isolation | More control-plane complexity |
| Pool | Shared tables and infrastructure | Efficient utilization | Strongest need for policy enforcement |

AWS’s SaaS isolation guidance emphasizes that authentication and isolation are different concerns: a user can be correctly authenticated and still access the wrong tenant’s resource if authorization context is not enforced at every boundary.

Make the choice per resource with explicit drivers:

- regulatory boundary;
- restore and deletion requirements;
- noisy-neighbor tolerance;
- tenant size distribution;
- customization needs;
- unit economics;
- operational maturity.

Siloing every resource maximizes conceptual clarity but can make patching, migrations, and observability unmanageable. Pooling everything minimizes infrastructure count but makes a single context bug catastrophic. Most serious systems use tiers.

## Tenant identity is request state

Tenant identity should be resolved once from authenticated authority, then propagated as typed context. Do not accept a tenant ID from a request body and trust it because it matches a UUID shape.

```text
credential -> principal -> allowed tenant memberships -> selected tenant context
```

The context should include more than an ID:

```json
{
  "tenantId": "t_83f4",
  "principalId": "u_1209",
  "roles": ["billing_reader"],
  "plan": "enterprise",
  "region": "ap-south-1",
  "policyVersion": 17
}
```

Sign or derive it inside a trusted boundary. Propagate the minimum needed to downstream services. Log it on every decision, but never let logging become a source of cross-tenant data leakage.

Background work must carry the same context. “System job” is not a tenant. A job should declare which tenant it acts for, which authority created it, and what scope was granted.

## Put a policy below application code

For pooled PostgreSQL tables, Row-Level Security (RLS) can make the database enforce tenant predicates:

```sql
ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoices FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_invoices ON invoices
USING (tenant_id = current_setting('app.tenant_id')::uuid)
WITH CHECK (tenant_id = current_setting('app.tenant_id')::uuid);
```

Set tenant context within the transaction:

```sql
BEGIN;
SET LOCAL app.tenant_id = '7f35...';
SELECT * FROM invoices WHERE status = 'overdue';
COMMIT;
```

PostgreSQL applies a default-deny policy when RLS is enabled and no applicable policy exists. But there are sharp edges:

- table owners normally bypass RLS unless forced;
- superusers and roles with `BYPASSRLS` bypass policies;
- connection pools can leak session state if context is not transaction-local and reliably reset;
- maintenance and migration roles need deliberate scope;
- security-definer functions can cross policy boundaries;
- referential integrity checks have special behavior and must be reviewed.

RLS is defense in depth, not permission to stop filtering in the application. Application predicates improve clarity and query plans; RLS protects forgotten paths.

## Make identifiers and caches tenant-safe by construction

Prefer globally unique external identifiers, but still include tenant scope in authorization. Knowing an opaque invoice ID must not grant access.

Every derived namespace needs tenant identity:

```text
cache:   tenant/{tenantId}/invoice/{invoiceId}
object:  tenant/{tenantId}/exports/{exportId}.csv
search:  filter tenant_id before ranking and aggregation
queue:   partition or quota by tenant
metric:  avoid raw tenant cardinality in every time series
```

A cache key missing tenant ID can bypass perfect database isolation. An object-store pre-signed URL can outlive revoked membership. A vector search can leak semantic fragments through nearest-neighbor results. Review the entire data path, not only the primary database.

## Isolate capacity as well as rows

Security isolation asks, “Can tenant A see tenant B?” Performance isolation asks, “Can tenant A make tenant B unavailable?”

Controls include:

- per-tenant concurrency and rate budgets;
- weighted fair queues;
- query-cost limits and statement timeouts;
- separate worker pools for bulk exports;
- connection budgets by workload class;
- storage and egress quotas;
- promotion of large tenants to dedicated partitions.

Measure tenant concentration. A pooled design where one customer creates 60% of database load is operationally a single-tenant dependency wearing a multi-tenant label.

Avoid putting unbounded tenant IDs into metric labels. Keep low-cardinality service metrics, then send tenant-attributed events to logs or analytics designed for high-cardinality investigation.

## Design restore, migration, and deletion first

Pooling changes operational promises. Restoring one tenant from a physical database backup may require a logical extraction and reconciliation process. A tenant deletion must cover replicas, search indexes, caches, object storage, analytics, and retained events.

Before choosing pooled storage, demonstrate:

1. tenant-scoped export;
2. tenant-scoped logical restore;
3. verifiable deletion with retention exceptions;
4. online schema migration across all tiers;
5. promotion from pooled to dedicated storage;
6. reconciliation after partial failure.

If enterprise contracts promise tenant restore but the architecture only supports cluster restore, the product has sold a capability the platform cannot perform.

## Test isolation as a negative property

Happy-path tests prove users can access their data. Isolation tests must prove all the ways they cannot access someone else’s.

Create two tenants and systematically attempt:

- direct object ID substitution;
- missing tenant context;
- stale membership tokens;
- background job replay under another tenant;
- cache-key collisions;
- export and search aggregation leakage;
- elevated support-role misuse;
- connection reuse after a previous tenant transaction.

Run these tests against production-like pooling and roles. An application test using a database owner account cannot validate RLS behavior honestly.

## CTO review questions

1. Where is tenant identity established and where can it be overwritten?
2. Which resources are pooled, bridged, or siloed, and why?
3. What policy still protects data when an application predicate is forgotten?
4. Can a connection pool retain another tenant’s session state?
5. How are caches, search, files, events, and analytics scoped?
6. Can one tenant exhaust shared capacity?
7. Can we restore or delete exactly one tenant?

Multi-tenancy is not a table-layout choice. It is an identity, policy, capacity, and operations model. The right design makes tenant context unavoidable and makes violations difficult to express—not merely easy to catch in code review.

## References

- [PostgreSQL: Row Security Policies](https://www.postgresql.org/docs/current/ddl-rowsecurity.html)
- [AWS: SaaS Tenant Isolation Strategies](https://docs.aws.amazon.com/whitepapers/latest/saas-tenant-isolation-strategies/saas-tenant-isolation-strategies.html)
- [PostgreSQL: Client Connection Defaults](https://www.postgresql.org/docs/current/runtime-config-client.html)
- [Related: Zero-Downtime PostgreSQL Migrations](/posts/postgresql-zero-downtime-schema-migrations)


---

# Strongly Consistent Object Storage Still Needs Concurrency Control

> Why S3 strong consistency does not prevent lost updates, and how conditional writes, immutable keys, checksums, versioning, and metadata transactions create safe object workflows.

Canonical URL: https://www.ayushworks.xyz/posts/object-storage-needs-concurrency-control
Author: Ayush Basak
Last modified: 2026-08-31
Topics: cloud-infrastructure, object-storage, concurrency-control, system-design

Amazon S3 provides strong read-after-write consistency for object `PUT` and `DELETE` operations. After a successful write, a later read sees the change. That guarantee removes a large class of stale-read workarounds. It does not prevent two writers from overwriting one another.

The invariant is:

> A mutable object changes only if the writer's precondition still matches the version it read; immutable content is never replaced under the same identity.

## Strong consistency is not compare-and-set

Two workers can read ETag `v1`, independently create updates, and both perform unconditional `PUT`s. Each write is strongly consistent. The later write still destroys the earlier result.

```text
worker A: GET v1 ── compute A ── PUT A
worker B: GET v1 ───── compute B ───── PUT B
final value: B; A is lost
```

Use a conditional request. `If-Match` permits the write only when the current ETag equals the version the writer observed. `If-None-Match: *` creates only when no current object exists.

```ts
await s3.send(new PutObjectCommand({
  Bucket: bucket,
  Key: key,
  Body: payload,
  IfMatch: previousETag,
}))
```

A precondition failure is a concurrency conflict. Reload, recompute the decision, and retry inside a budget. Do not repeat the same stale body automatically.

## ETag is an entity version, not universally a content hash

An ETag changes with the object and is useful for conditional requests. Do not assume it is always the MD5 of payload bytes. Multipart uploads and encryption modes can produce different semantics. Use explicit checksum fields when integrity verification requires a known algorithm.

Separating concerns avoids fragile designs:

- ETag for object version preconditions;
- checksum for transfer/content integrity;
- business version for domain ordering;
- object version ID for recovery when bucket versioning is enabled.

## Immutable keys simplify the system

For artifacts, exports, images, model outputs, and deployment bundles, prefer content-addressed or generation-addressed keys:

```text
reports/tenant-42/2026-08-31/run-8f31/result.parquet
releases/sha256-6c2.../bundle.tar.zst
```

Write the immutable object first, verify it, then update a small pointer or metadata record conditionally. Readers either see the old pointer or the new pointer—never half an upload.

```text
PUT immutable generation
HEAD + verify checksum and metadata
conditional PUT manifest If-Match: prior-manifest-etag
```

This is the object-storage form of publish-then-switch.

## Listing is not a workflow database

Strong listing consistency means a successful write is reflected in subsequent listings. A prefix listing still does not encode job ownership, dependency completion, authorization, retries, or a multi-object transaction.

If a workflow needs “all 20 parts exist and belong to generation 7,” publish a manifest naming those exact objects and checksums. Consumers read the manifest as the commit record. Or keep transactional workflow state in a database and treat objects as immutable payloads.

## Ambiguous writes still exist

The client can lose its connection after S3 accepts a write but before the response arrives. A retry to an immutable key with the same checksum is naturally safe. A retry to a mutable key needs a precondition and reconciliation through `HEAD` or version history.

Do not generate a new object key for every transport retry unless duplicates are intentionally tolerated; otherwise ambiguity becomes orphaned storage and inconsistent metadata.

## Deletion and retention are policy boundaries

Bucket versioning can preserve overwritten and deleted versions, improving recovery while increasing storage and lifecycle complexity. Object Lock can enforce retention, but governance and compliance modes change who can remove data. Lifecycle rules are asynchronous policy execution, not immediate proof of deletion.

Track incomplete multipart uploads, noncurrent versions, delete markers, failed conditional writes, orphan objects not referenced by manifests, checksum failures, and lifecycle backlog.

## The CTO decision

Use object storage for immutable, large payloads and publish their availability through a small transactional or conditional metadata boundary. Use conditional writes for mutable pointers. Specify checksum, version, retention, and ambiguity behavior separately.

Strong consistency tells you which write is visible. Concurrency control decides whether that write was allowed to replace what came before.

## References

- [Amazon S3 consistency model](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html)
- [Amazon S3 conditional writes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html)
- [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
- [Related: Optimistic concurrency snippet](/snippets/optimistic-concurrency)


---

# Thinking Through Offline-First Payments

> Security and distributed-systems lessons from moving value without a live server.

Canonical URL: https://www.ayushworks.xyz/posts/offline-first-payments
Author: Ayush Basak
Last modified: 2026-07-14
Topics: payments, distributed-systems, security, reliability

Offline payment systems force several hard problems into the same room: untrusted transport, replay attacks, concurrent delivery, and eventual settlement.

## Treat the relay as hostile

The device carrying a payment message should not need to be trusted. Sign the payload, bind the signature to the amount and participants, and include a nonce plus an expiry window. Encryption protects privacy; signatures protect integrity and authorship.

## Settlement must be idempotent

The same signed payment may reach the server through multiple relays. Settlement therefore needs an atomic uniqueness check on the transaction identity. The first valid submission wins; later copies receive the same outcome without moving money twice.

## Offline does not mean unlimited

Risk controls still matter. Short validity windows, device-bound keys, and conservative offline limits reduce exposure until the system reconnects.

The larger lesson is useful beyond payments: design the protocol around what an attacker can copy, delay, reorder, and replay.


---

# OpenTelemetry Baggage Is Untrusted Input

> Design trace-context propagation without leaking secrets, trusting forged metadata, or turning observability labels into an authorization channel.

Canonical URL: https://www.ayushworks.xyz/posts/opentelemetry-baggage-is-untrusted-input
Author: Ayush Basak
Last modified: 2026-09-05
Topics: observability, security, opentelemetry, distributed-systems, architecture

OpenTelemetry baggage is attractive because it carries application-defined key-value context across service boundaries. Put a tenant identifier at ingress and every downstream span can become easier to search.

It is also propagated through request headers, can reach services you do not control, and has no built-in integrity guarantee. Those properties make baggage useful telemetry context—and a dangerous place for trust.

> Baggage may describe a request. It must never authorize one.

## Separate claims from decisions

An inbound header such as `baggage=tenant.id=acme,plan=enterprise` is a claim made by the caller. If a downstream service uses `plan=enterprise` to unlock a feature, the client has acquired an authorization interface.

Authentication and policy must produce trusted server-side context. You may then emit a safe derivative into telemetry, but consumers must not reconstruct authority from it.

```text
untrusted request headers
        |
        v
sanitize propagation context -----> tracing only
        |
        v
authenticate + load policy --------> authorization decision
```

Treat trace IDs similarly: they correlate work; they do not prove caller identity, ownership, or causality by themselves.

## Use an allowlist at ingress

Do not automatically accept arbitrary baggage from the public internet. Decide which keys may cross each boundary, validate length and syntax, and drop everything else. For an external request, the safe list may be empty.

At trusted internal ingress, create low-sensitivity identifiers specifically for observability. Prefer an opaque account cohort or irreversible keyed hash to an email address, access token, health identifier, or raw customer ID.

OpenTelemetry explicitly warns that baggage can be propagated to unintended third parties. Automatic instrumentation makes boundary review important: a service that calls a payment provider may forward context unless propagation is restricted.

## Scrub at egress too

Ingress filtering protects against forged data. Egress filtering protects against accidental disclosure.

Maintain an outbound policy by destination class:

| Destination | Trace context | Baggage |
| --- | --- | --- |
| Owned internal service | Allowed, filtered | Small allowlist |
| Vendor with tracing agreement | Explicit decision | Usually none |
| Arbitrary customer URL or webhook | Regenerate or omit | None |

Clear baggage before calling untrusted endpoints. Do not rely exclusively on the collector to repair a secret after it has already crossed the network.

## Control telemetry cardinality

Even non-sensitive baggage can be operationally hazardous. Copying request IDs, URLs, or unrestricted customer values onto every span creates high-cardinality attributes and expensive indexes.

Define a schema for propagated observability context:

- owner for every key;
- allowed source and destinations;
- sensitivity class;
- maximum encoded size;
- whether it becomes a span, log, or metric attribute;
- retention period;
- expected cardinality.

Baggage does not automatically become a span attribute. That explicit conversion is a useful control point. Enrich only the signals that answer a known operational question.

## Defense in depth at the collector

OpenTelemetry’s security guidance recommends data minimization and documents attribute, filter, redaction, and transform processors for managing sensitive telemetry. Use an allowlist or deletion policy near collection as a second boundary.

But collector redaction is not sufficient if application logs already copied the value elsewhere, exporters bypass the collector, or baggage went to an external API. Prevent, restrict, and then redact.

## Test the propagation graph

Build automated boundary tests:

1. send forged baggage from a public client;
2. verify forbidden keys are absent downstream;
3. send approved internal context;
4. verify it appears only on intended telemetry;
5. call a captured third-party endpoint;
6. assert no internal baggage header arrived;
7. inject oversized and malformed values;
8. confirm authorization outcomes never change.

Also inventory auto-instrumentation during upgrades. Propagator defaults and HTTP client coverage can change even when application code does not.

The architectural rule is small: observability context is data moving through the system, so it needs ownership, classification, boundaries, and tests. When baggage is treated as a convenient global variable, it quietly becomes both a leakage path and a shadow control plane.

## References

- [OpenTelemetry: Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/)
- [OpenTelemetry: Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
- [OpenTelemetry: Handling sensitive data](https://opentelemetry.io/docs/security/handling-sensitive-data/)
- [OpenTelemetry specification: Baggage API](https://opentelemetry.io/docs/specs/otel/baggage/api/)


---

# The OpenTelemetry Collector Is a Loss Budget

> How queues, retries, memory limits, persistent storage, and stateful processors determine which production evidence survives an outage.

Canonical URL: https://www.ayushworks.xyz/posts/opentelemetry-collector-is-a-loss-budget
Author: Ayush Basak
Last modified: 2026-09-03
Topics: opentelemetry, observability, cloud-infrastructure, reliability, platform-engineering

During an incident, the OpenTelemetry Collector becomes a buffer between increasing evidence and a slow or unreachable backend. Its configuration decides what survives.

The invariant is:

> Telemetry buffering must be bounded, observable, and unable to consume the resources required to serve the product.

When an exporter slows, its sending queue grows. Queued batches retain memory. Eventually the Collector must drop, reject, persist, block, or crash. An unbounded queue is not reliability; it is an out-of-memory failure with a longer fuse.

## Size the outage window

Translate queue configuration into time:

```text
buffer window ~= usable queue bytes / peak ingest bytes per second
```

“5,000 items” means little until batch sizes and signal cardinality are measured. Persistent storage can survive a restart, but disk is still finite and replay traffic can overload a recovering backend.

```yaml
exporters:
  otlp/backend:
    endpoint: telemetry.example.com:4317
    sending_queue:
      enabled: true
      storage: file_storage
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 10m
```

These are example boundaries, not universal values. Retry duration, queue capacity, disk, exporter concurrency, and backend recovery rate must agree.

## Memory limiting is load shedding

Put the memory limiter before processors that allocate more state. Leave headroom below the container limit for receivers, exporters, runtime overhead, and allocation spikes.

```yaml
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 300
  batch:

service:
  pipelines:
    traces:
      processors: [memory_limiter, batch]
```

Refusal must propagate safely. Application SDKs need bounded queues and backoff; otherwise the Collector protects itself by moving memory pressure into the product process.

## Horizontal scale can corrupt the picture

Receivers and exporters are often stateless. Tail sampling is not: it holds spans until a trace decision can be made. If spans from one trace reach different replicas, each sees incomplete evidence. Span-to-metrics aggregation has similar grouping requirements.

OpenTelemetry recommends a routing layer with a load-balancing exporter so related spans reach the same processing Collector:

```text
agents -> routing collectors -> stateful collectors -> backend
                    consistent hash by trace or service
```

More replicas without affinity can increase throughput while decreasing truth.

## Make loss policy explicit

Security audit records should not compete blindly with health-check spans. Separate pipelines when retention obligations differ. Preserve a small unbiased trace sample alongside targeted error or latency sampling; tail sampling provides outcome awareness but consumes memory while traces remain incomplete.

Monitor accepted, refused, retried, queued, and dropped records; queue utilization and time to full; exporter latency and throttling; Collector memory and restarts; trace completeness; and configuration version. Keep a minimal independent signal because loss metrics sent through the broken pipeline may disappear too.

Test backend disconnection, slow drain, full disk, Collector restart, cardinality spikes, and stateful scale-out. Record which evidence survives and whether application latency changes.

## The CTO decision

Define tolerable loss per signal and incident class. Size buffers for a bounded recovery window, protect application resources first, preserve affinity for stateful processors, and rehearse failure.

Observability is reliable when its own failure leaves enough evidence to explain the product—not when dashboards happen to be green.

## References

- [OpenTelemetry: Collector resiliency](https://opentelemetry.io/docs/collector/resiliency/)
- [OpenTelemetry: Scaling the Collector](https://opentelemetry.io/docs/collector/scaling/)
- [OpenTelemetry: Collector configuration](https://opentelemetry.io/docs/collector/configuration/)
- [Related: Observability is a data contract](/posts/opentelemetry-observability-data-contract)
- [Related: Tail sampling is a memory budget](/posts/tail-sampling-is-a-memory-budget)


---

# Observability Is a Data Contract, Not a Dashboard Purchase

> How engineering leaders can use OpenTelemetry semantics, cardinality budgets, context propagation, and SLO questions to build portable operational evidence.

Canonical URL: https://www.ayushworks.xyz/posts/opentelemetry-observability-data-contract
Author: Ayush Basak
Last modified: 2026-08-20
Topics: observability, opentelemetry, distributed-systems, platform-engineering

Teams often buy an observability backend and discover that they still cannot answer the incident question. The vendor stored everything it received; the applications never agreed on what the data meant.

Observability begins before collection. It is a contract that maps production behavior into evidence: operation names, resource identity, error meaning, latency boundaries, and business outcomes. OpenTelemetry is useful because it standardizes the transport and much of this vocabulary. It cannot choose the questions your company must answer.

## Start with decisions

Before adding an SDK, write the operational questions:

- Is checkout meeting its user-visible availability objective?
- Which dependency consumed the latency budget?
- Did a request fail before or after committing the order?
- Is one tenant creating abnormal load?
- Can a deploy be correlated with the regression?

Each question should map to a signal, a bounded set of attributes, and an action. If a metric has no decision attached, challenge its cost.

```text
user objective
  -> service-level indicator
  -> telemetry event and attributes
  -> alert or investigation
  -> operator action
```

This chain prevents dashboard-driven instrumentation: collecting whatever a library happens to expose and hoping it becomes useful later.

## Semantic conventions are an internal API

OpenTelemetry semantic conventions define common names and meanings for traces, metrics, logs, profiles, and resources. They allow `service.name`, HTTP attributes, database operations, and messaging spans to mean the same thing across languages.

Treat that vocabulary like an internal API:

- adopt standard attributes before inventing custom ones;
- assign an owner for custom conventions;
- version changes to names or meaning;
- test required resource attributes in CI;
- document which attributes are safe to collect;
- pin or deliberately migrate unstable conventions.

The OpenTelemetry specification publishes stability levels for convention groups. “We use OTel” does not guarantee that every emitted convention is stable. Platform teams must know what they have opted into.

## Resource, operation, and occurrence

Good telemetry separates three concepts:

1. **Resource:** what produced the signal—service, version, environment, region.
2. **Operation:** what kind of work occurred—HTTP route, queue consume, database call.
3. **Occurrence:** the particular execution—trace ID, result, duration, selected context.

Do not encode occurrences into metric dimensions. A `user_id`, request ID, raw URL, SQL text, or stack trace can create unbounded cardinality and turn a useful metric into an expensive index.

Prefer bounded dimensions:

```text
http.request.duration{
  service.name="checkout",
  http.request.method="POST",
  http.route="/orders/:id/confirm",
  http.response.status_code="503"
}
```

The route template is bounded; `/orders/98f.../confirm` is not. High-detail occurrences belong in traces or logs, governed by sampling and retention.

## Correlation is the leverage

OpenTelemetry's log model supports trace and span identifiers, while W3C Trace Context standardizes propagation through `traceparent` and `tracestate`. With consistent propagation, an alert can lead to a trace, the trace to a dependency span, and the span to correlated logs.

This breaks when teams:

- generate a new trace at every queue boundary;
- omit context from retry and dead-letter paths;
- put trace IDs in message bodies without defining precedence;
- trust inbound baggage and propagate sensitive or unbounded values;
- sample the parent but retain unrelated child logs with no join key.

Test propagation as part of an end-to-end contract. Send a synthetic request across HTTP, queue, worker, and database boundaries; assert that the causal chain remains queryable.

## Instrument business commit points

Auto-instrumentation sees frameworks. It does not know when your invariant becomes true. Add narrow manual spans or events around domain transitions:

```text
order.confirm
  validation
  payment.authorize
  order.commit       <-- durable business point
  event.enqueue
```

Marking the whole request as failed after `order.commit` may be operationally correct, but the retry policy must know that the order already exists. Observability should expose this boundary rather than flatten the request into one red span.

Use span status carefully. The OTel trace specification does not require every expected domain rejection to be an infrastructure error. A declined payment or failed validation may be a successful execution of the software contract. Capture the domain outcome separately and alert on what harms the user objective.

## Put a budget on telemetry

Telemetry consumes CPU, network, collector memory, backend ingestion, index storage, and operator attention. Define budgets:

- maximum attributes per span or log;
- approved high-cardinality fields and their signal type;
- head and tail sampling policies;
- retention by data class;
- redaction at source and collector;
- ingestion cost per service or product;
- collector queue and drop thresholds.

Sampling is a policy decision. Head sampling is cheap but cannot know the eventual result. Tail sampling can retain errors and slow traces but requires buffering and a reliable collector tier. Keep aggregate metrics unsampled for SLO calculation; use sampled traces to explain them.

## Design the collector as production infrastructure

The collector decouples applications from backends and centralizes batching, enrichment, redaction, sampling, and export. That makes it a failure domain.

Decide what happens when the backend or collector is unavailable. Application requests should generally not block on telemetry export. SDK queues must be bounded. Collectors need memory limits, persistent or acceptable-loss queues, retry budgets, and self-observability. Alert on dropped spans and exporter failures; a green application with a blind monitoring pipeline is not green.

## Common mistakes

**Using logs as prose.** Structured events with stable fields can be queried and correlated; sentence parsing becomes accidental API design.

**Naming spans after raw URLs or SQL.** It creates cardinality, leaks data, and fragments one operation across thousands of names.

**Collecting sensitive context “for debugging.”** Telemetry is broadly accessible and replicated. Minimize and redact before export.

**Alerting on infrastructure symptoms without a user objective.** High CPU may be harmless; exhausted latency budget is not.

**Migrating vendors before fixing semantics.** Portable bad data remains bad data.

## CTO review checklist

1. Which user-visible decisions does each primary signal support?
2. Are resource and operation names consistent across languages?
3. What are the cardinality, privacy, sampling, and retention budgets?
4. Does context survive asynchronous boundaries and retries?
5. Are business commit points observable?
6. Can the telemetry pipeline fail without taking down the product—and can we detect that failure?

The durable investment is not a dashboard layout. It is a shared operational language that survives service rewrites and backend changes while continuing to answer the questions that matter.

## References

- [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/)
- [OpenTelemetry logging specification](https://opentelemetry.io/docs/specs/otel/logs/)
- [OpenTelemetry tracing specification](https://opentelemetry.io/docs/specs/otel/trace/)
- [W3C Trace Context](https://www.w3.org/TR/trace-context/)
- [Related: A Practical Model for Observability](/engineering-notes#observability)
- [Related: Building Microservices That Fail Gracefully](/posts/building-reliable-microservices)


---

# Transpiling Multi-Language DB Functions to PL/pgSQL with plx

> Learn how the plx extension transpiles familiar programming languages like Python, JS, and Go into native PL/pgSQL for zero-overhead execution inside PostgreSQL.

Canonical URL: https://www.ayushworks.xyz/posts/plx-transpiling-postgresql-functions
Author: Ayush Basak
Last modified: 2026-08-19
Topics: postgresql, database-engineering, plpgsql, backend-engineering

Moving domain logic directly into database triggers, stored procedures, and set-returning functions minimizes network round-trips and keeps state mutations consistent. However, PL/pgSQL syntax often introduces a learning curve or cognitive overhead for application developers accustomed to modern programming languages.

The [plx extension](https://www.postgresql.org/about/news/plx-write-postgresql-functions-in-the-language-you-already-know-3358/) bridges this gap. Rather than embedding runtime interpreters like V8 or CPython into PostgreSQL processes, plx acts as a build-time transpiler. When you execute `CREATE FUNCTION`, plx translates the source dialect directly into PL/pgSQL syntax and stores the resulting procedural code in `pg_proc.prosrc`. At query runtime, PostgreSQL executes pure PL/pgSQL without external runtime overhead.

Building data-intensive backends often requires striking a balance between application-layer microservices—such as those described in [building reliable microservices](/posts/building-reliable-microservices)—and database-native execution. Understanding how plx achieves zero-runtime overhead helps software architects evaluate when to push logic into PostgreSQL.

---

## Architecture: Build-Time Transpilation vs Runtime Execution

Traditional procedural extensions (like PL/Python or PL/v8) load heavy language runtimes into every PostgreSQL backend worker process. In contrast, plx shifts the transformation cost entirely to DDL compilation time.

```text
                    DDL Execution (CREATE FUNCTION)
+-------------------+      +-------------------+      +-----------------------+
|  plx Source Code  | ---> |   plx Transpiler  | ---> | PL/pgSQL Intermediate |
| (JS, Python, Go)  |      |  (Dialect Parser) |      |      Representation   |
+-------------------+      +-------------------+      +-----------------------+
                                                                  |
                                                                  v
                                                     +--------------------------+
                                                     | Stored in pg_proc.prosrc |
                                                     +--------------------------+

                               Query Execution
+-------------------+      +-------------------+      +-----------------------+
|   SQL Query Call  | ---> | Native PostgreSQL | ---> | Executed via Standard |
|   SELECT my_fn()  |      | PL/pgSQL Engine   |      | PostgreSQL Engine     |
+-------------------+      +-------------------+      +-----------------------+
```

Because plx transpiles logic straight to native PL/pgSQL statements, every PL/pgSQL capability—including cursor manipulation, exception handling, and transaction controls—is reachable across all supported dialects.

---

## Dialects and Coexistence

plx supports various dialects tailored for popular modern languages as well as legacy database migration pathways:

- `plxjs` and `plxts` (JavaScript / TypeScript)
- `plxpython3` (Python 3)
- `plxgo` (Go)
- `plxruby` and `plxphp` (Ruby / PHP)
- `plxcobol` (ISO/IEC 1989:2023 COBOL)
- `plxplsql` and `plxtsql` (Oracle PL/SQL and SQL Server T-SQL)

By prefixing all language identifiers with `plx`, the extension avoids identifier collisions. A database instance can safely host native PL/PHP or PL/Python alongside `plxphp` or `plxpython3` without conflict.

---

## Practical Examples

### A verified JavaScript-dialect function (`plxjs`)

The project's [official examples](https://github.com/commandprompt/plx) show a grading function whose JavaScript-like body is translated at `CREATE FUNCTION` time:

```sql
CREATE FUNCTION grade(score int) RETURNS text
LANGUAGE plxjs AS $$
  let grade = "F";
  if (score >= 90) { grade = "A"; }
  else if (score >= 80) { grade = "B"; }
  else { grade = "F"; }
  return grade;
$$;
```

The stored body becomes a `DECLARE` plus `IF/ELSIF/ELSE` block and remains visible in `pg_proc.prosrc`. That inspectability matters: teams can review the generated database code, include it in `pg_dump`, and diagnose behavior without a hidden runtime. The repository also documents Ruby, PHP, TypeScript, Python, Go, COBOL, Oracle PL/SQL, and Transact-SQL dialects, but each has deliberate syntax limits; verify examples against the version you install.

When managing core infrastructure code, engineers frequently analyze transpilation efficiency, similar to low-level optimization patterns discussed in [Rust for infrastructure](/posts/rust-for-infrastructure) and detailed in our [engineering notes](/engineering-notes).

---

## Tradeoffs and Architectural Considerations

While plx simplifies database development, backend engineers should evaluate specific tradeoffs:

1. **No External Ecosystem Libraries**: Because code compiles to standard PL/pgSQL, you cannot import arbitrary npm or PyPI packages (such as `lodash` or `numpy`). All logic must map directly to built-in PostgreSQL control structures and SQL functions.
2. **DDL Latency Overhead**: Transpilation occurs during `CREATE FUNCTION`. While runtime query performance is identical to hand-written PL/pgSQL, schema migration scripts creating hundreds of complex functions will incur brief parsing latency.
3. **Debugging Abstraction**: Stack traces and syntax errors produced at runtime originate from the transpiled PL/pgSQL inside `pg_proc.prosrc`, rather than the original source dialect text.

---

## Common Misconceptions

- **Misconception 1: "plx runs a V8 engine or Python runtime inside PostgreSQL."**  
  *Fact:* plx never loads runtime engine binaries into PostgreSQL backends. Execution is handled exclusively by PostgreSQL's native `plpgsql` interpreter ([PostgreSQL plx announcement](https://www.postgresql.org/about/news/plx-write-postgresql-functions-in-the-language-you-already-know-3358/)).

- **Misconception 2: "Installing plx breaks existing PL/Python or PL/Ruby functions."**  
  *Fact:* The plx language family uses distinct prefixes (`plxpython3`, `plxruby`), allowing seamless coexistence with native language extensions.

---

## When NOT to Use plx

Do NOT use plx if:
- You require heavy algorithmic processing relying on external C-extensions (e.g., machine learning models or complex cryptography). Use PL/Python with external libraries or procedural C/Rust extensions instead.
- Your team is already highly proficient in raw PL/pgSQL, as transpilation adds another build-step abstraction layer without performance gains over hand-crafted PL/pgSQL.

---

## Conclusion

The plx extension democratizes database-side logic by allowing engineers to write stored procedures in their native programming languages while guaranteeing zero-overhead, native PL/pgSQL execution. By separating compile-time syntax transformation from runtime execution, plx offers a pragmatic path for modern application teams migrating logic closer to their data.

---

## References

- [PostgreSQL plx Extension Announcement](https://www.postgresql.org/about/news/plx-write-postgresql-functions-in-the-language-you-already-know-3358/)
- [commandprompt/plx source and verified examples](https://github.com/commandprompt/plx)
- [PostgreSQL Documentation: Concurrency Control](https://www.postgresql.org/docs/current/mvcc.html)


---

# A PodDisruptionBudget Is a Maintenance Contract

> Use Kubernetes disruption budgets with topology, rollout policy, capacity headroom, and drain tests instead of treating them as availability insurance.

Canonical URL: https://www.ayushworks.xyz/posts/pod-disruption-budgets-are-maintenance-contracts
Author: Ayush Basak
Last modified: 2026-09-05
Topics: kubernetes, reliability, cloud-infrastructure, operations, system-design

A PodDisruptionBudget (PDB) does not promise that your service stays available. It limits how many selected pods an eviction-aware actor may voluntarily disrupt at once. Hardware failure, resource pressure, direct deletion, and workload rollouts do not become harmless because a PDB exists.

That narrower contract is valuable—provided the team designs the rest of the system around it.

> A PDB protects maintenance coordination, not the application from every cause of unavailability.

## Start from serving capacity

Suppose a service has six replicas. Four are required to sustain peak critical traffic with acceptable tail latency. One replica may already be unavailable during ordinary operation. The maintenance budget is therefore one additional eviction, not “25% because that seems safe.”

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: checkout-api
```

`minAvailable` and `maxUnavailable` are mutually exclusive. Use a number derived from workload capacity, quorum, or recovery cost. Percentages round and may behave unexpectedly at low replica counts, so calculate concrete cases during review.

## Understand which actions participate

Kubernetes documents PDBs as constraints on voluntary disruptions performed through the Eviction API. `kubectl drain` uses that API and retries rejected evictions. Directly deleting a pod or deployment can bypass the budget. Application rollouts are controlled by the workload’s own update strategy rather than by the PDB.

This creates an ownership boundary:

| Mechanism | Governing control |
| --- | --- |
| Node drain or autoscaler eviction | PDB plus Eviction API |
| Deployment rollout | `maxUnavailable` / `maxSurge` |
| Node loss | Replication and topology |
| OOM or disk pressure | Requests, limits, node capacity |
| Direct pod deletion | Access controls and operating procedure |

A platform team cannot infer application safety from a PDB alone. An application team cannot assume every platform action respects one.

## Avoid the impossible budget

`minAvailable: 100%` can block every voluntary eviction. That may be appropriate for a deliberately manual workload, but on an ordinary service it can prevent node maintenance indefinitely. A broken pod can also block a drain when policy requires it to be healthy before eviction.

The `AlwaysAllow` unhealthy-pod eviction policy helps drains make progress when a pod is already unhealthy. It does not repair insufficient replicas or bad placement. Document the trade-off: maintenance progress versus preserving a possibly recoverable unhealthy instance.

## Capacity without topology is fragile

Six replicas placed on two nodes do not provide six independent failure units. Combine the budget with topology spread constraints or anti-affinity, adequate node headroom, and readiness that reflects the ability to serve.

Readiness is part of the disruption calculation. If a replacement pod becomes ready before caches are warm or dependencies are established, the controller may permit the next eviction too soon. Test readiness against actual critical-path behavior. The [probe design guide](/posts/kubernetes-probes-are-failure-policy) covers this failure policy in detail.

## Coordinate with rollouts and autoscaling

The deployment strategy and PDB should express compatible assumptions. If a rollout permits two unavailable replicas while peak capacity tolerates only one, the PDB will not save the rollout. If the HPA scales down near a drain window, a previously safe absolute replica count can disappear.

For critical workloads, define a maintenance floor separate from the demand-driven scaling floor. Check disruption allowance before planned changes, but do not use the current allowance as the only gate: status can trail reality.

## Run the drain game day

In a staging environment that resembles production placement:

1. generate representative traffic;
2. drain one node with a realistic timeout;
3. observe rejected and allowed evictions;
4. confirm replacements become genuinely ready;
5. measure p99 latency and error budget consumption;
6. introduce one unhealthy replica and repeat;
7. simulate a simultaneous involuntary node loss.

The final step matters because real incidents overlap with planned work. A PDB that supports a clean drain but leaves no tolerance for one node failure encodes an optimistic world.

Track blocked drains, `disruptionsAllowed`, unavailable replicas, scheduling latency, and service SLOs together. The operational outcome is the product of all five.

A good PDB makes a precise promise to operators: under stated capacity and health assumptions, this many coordinated evictions may proceed. Everything beyond that promise belongs to topology, workload policy, and application resilience.

## References

- [Kubernetes: Disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)
- [Kubernetes: Specify a disruption budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/)
- [Kubernetes API: PodDisruptionBudget](https://kubernetes.io/docs/reference/kubernetes-api/policy/pod-disruption-budget-v1/)


---

# PostgreSQL Minor Updates: CVE-2026-6464 and Index Maintenance

> An engineering analysis of PostgreSQL 18.6, CVE-2026-6464, skipped 18.5, and post-upgrade REINDEX tasks for GIN, btree_gist, and ltree extensions.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-18-6-security-patching-index-remediation
Author: Ayush Basak
Last modified: 2026-08-19
Topics: postgresql, database-engineering, security, cloud-infrastructure

The PostgreSQL Global Development Group released maintenance updates across all supported versions—18.6, 17.11, 16.15, 15.19, and 14.24—alongside PostgreSQL 19 Beta 3. This release bundle addresses 28 security vulnerabilities and rectifies over 110 operational bugs reported in recent months, as noted in the [PostgreSQL Release Announcement](https://www.postgresql.org/about/news/postgresql-186-1711-1615-1519-1424-and-19-beta-3-released-3365/).

Notably, PostgreSQL skipped version 18.5 entirely during this update cycle due to a late-stage regression discovered during release engineering. Engineering teams maintaining self-hosted or managed PostgreSQL 18 deployments should update directly from 18.4 to 18.6.

Beyond patching vulnerabilities, applying this release demands targeted post-upgrade index maintenance for databases leveraging specific extension features and parallel indexing mechanisms.

## Vulnerability Analysis: CVE-2026-6464

The most high-severity issue addressed in this batch is [CVE-2026-6464](https://www.postgresql.org/about/news/postgresql-186-1711-1615-1519-1424-and-19-beta-3-released-3365/), which carries a CVSS v3.1 score of 8.1. The flaw resides in how the `psql` interactive terminal processes input during an early failure of a `COPY FROM STDIN` command.

When a stream sends data to `psql` executing `COPY table FROM STDIN`, an early failure on the database engine side—such as a schema mismatch, permission failure, or connection reset—causes `psql` to exit the `COPY` protocol state early. However, if the client input stream continues transmitting data lines, `psql` resumes parsing those incoming data lines as native client metaprompts and commands rather than raw data.

```text
+-------------------+                         +--------------------+
|   psql Client     |                         | PostgreSQL Server  |
+-------------------+                         +--------------------+
          |                                             |
          |------- COPY table FROM STDIN -------------->|
          |<------ ERROR: Table Permission Denied ------|
          |   (psql exits COPY data ingestion mode)     |
          |                                             |
          |-- [Data Line: \! malicious_shell_cmd] ------>|
          |   psql evaluates data as client command!    |
```

This behavior allows arbitrary command execution in automated client environments or batch scripts that pipe untrusted network payloads directly into `psql`. Upgrading the client binaries (`psql`) alongside database server instances mitigates this vector.

## Mandatory Post-Upgrade Remediation: GIN, btree_gist, and ltree

While minor PostgreSQL upgrades typically require no index intervention, this release introduces fixes for index corruption scenarios. If your database utilizes [PostgreSQL Indexing](https://www.postgresql.org/docs/current/indexes.html) with parallel Generalized Inverted Index (GIN) builds, `btree_gist`, or `ltree` extensions, specific indexes must be rebuilt post-binary update.

### Affected Index Scenarios

1. **Parallel GIN Builds**: A parallel worker could report an uninitialized row count, leaving `pg_class.reltuples` as `Infinity`, `NaN`, or another bogus value. That can prevent autovacuum and autoanalyze from processing the table. The repair is `ANALYZE`, not a blanket GIN rebuild.
2. **`btree_gist`**: Indexes on `float4`/`float8` values containing `NaN`, and indexes on `bit`/`bit varying`, can require reindexing because of comparison and sort fixes.
3. **`ltree`**: A comparison overflow can affect values with more than roughly 14,653 labels and present as a corrupt B-tree index. Only affected indexes need rebuilding.

To identify and fix affected indexes across your cluster, run the following inspection query:

```sql
-- Official check for tables with GIN indexes and suspicious statistics
SELECT DISTINCT t.oid::regclass, t.reltuples
FROM pg_class t
JOIN pg_index i ON t.oid = i.indrelid
JOIN pg_class ic ON i.indexrelid = ic.oid
WHERE t.relhasindex AND ic.relam = 2742;
```

Run `ANALYZE` on tables whose `reltuples` value is wrong. Use `REINDEX` only for affected `btree_gist` and `ltree` cases:

```sql
ANALYZE schema_name.table_name;
REINDEX INDEX schema_name.affected_extension_index;
```

For large production indexes, evaluate whether `REINDEX CONCURRENTLY` is appropriate and budget extra runtime, I/O, and disk space. Do not turn a narrow release-note remediation into a cluster-wide rebuild.

## PostgreSQL 14 Lifecycle Warning

Infrastructure teams operating PostgreSQL 14 must incorporate major-version upgrade planning into their roadmap. According to official support policies, PostgreSQL 14 will reach End of Life (EOL) on **November 12, 2026**.

After this date, version 14 will no longer receive security patches, bug fixes, or performance backports. Integrating modern deployment patterns—such as those described in [/posts/kubernetes-kyaml-production-workflow](/posts/kubernetes-kyaml-production-workflow)—can simplify cluster lifecycle updates. When designing resilience for microservices, proactive maintenance windows for engine upgrades are critical, as outlined in [/posts/building-reliable-microservices](/posts/building-reliable-microservices).

## Tradeoffs and Architectural Considerations

Upgrading database minor versions and reindexing large datasets requires evaluating specific engineering tradeoffs:

- **I/O Overhead vs. Index Integrity**: Running `REINDEX CONCURRENTLY` across terabyte-scale GIN indexes generates significant I/O pressure and WAL volume. Stagger index rebuilds across low-traffic maintenance windows.
- **Deferred Patching Risks**: Delaying the patch leaves client pipelines vulnerable to shell command execution via CVE-2026-6464 when executing bulk ingest jobs.

## Common Pitfalls

- **Searching for 18.5 packages**: Attempting to deploy `18.5` binaries will fail because the release was pulled before public release. Ensure target automation deployment manifests specify `18.6`.
- **Updating Server Without Client Binaries**: Updating the PostgreSQL server daemon without updating client tooling like `psql` leaves ingestion scripts vulnerable to CVE-2026-6464.
- **Reindexing every GIN index**: The GIN issue concerns table statistics; inspect `reltuples` and run `ANALYZE` where needed. Reserve reindexing for the documented `btree_gist` and extreme-depth `ltree` cases.

## Conclusion

This release closes 28 vulnerabilities and more than 110 reported bugs. Patch server and client packages, inspect GIN table statistics, and rebuild only the affected `btree_gist` or unusually deep `ltree` indexes described by the release notes. Precise remediation is safer than a blanket reindex campaign.

## References

- [PostgreSQL Release Announcement (18.6 / 17.11 / 16.15 / 15.19 / 14.24)](https://www.postgresql.org/about/news/postgresql-186-1711-1615-1519-1424-and-19-beta-3-released-3365/)
- [PostgreSQL Documentation: Index Types](https://www.postgresql.org/docs/current/indexes.html)
- [PostgreSQL Documentation: Concurrency Control](https://www.postgresql.org/docs/current/mvcc.html)
- [PostgreSQL Documentation: High Availability and Replication](https://www.postgresql.org/docs/current/high-availability.html)


---

# PostgreSQL Checkpoints Are Latency Events, Not Maintenance Events

> How checkpoint pacing, WAL pressure, full-page writes, and storage headroom shape PostgreSQL tail latency.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-checkpoints-are-latency-events
Author: Ayush Basak
Last modified: 2026-09-01
Topics: postgresql, database-engineering, performance, reliability

A checkpoint is usually described as a recovery mechanism. Operationally, it is also a latency event.

PostgreSQL must eventually make dirty heap and index pages durable. At a checkpoint it guarantees that pages preceding the checkpoint record have reached their data files. That bounds crash recovery, but it also creates coordinated write pressure across the database, operating system, and storage device.

The useful question is not “how often should checkpoints run?” It is:

> Can the storage system absorb checkpoint work without stealing the latency budget from foreground transactions?

## The hidden feedback loop

PostgreSQL begins a checkpoint because `checkpoint_timeout` elapsed or WAL is approaching `max_wal_size`. The checkpointer spreads writes toward `checkpoint_completion_target`; current PostgreSQL defaults aim to use most of the interval instead of emitting one burst.

That pacing can still break down:

```text
write burst
  → WAL grows quickly
  → requested checkpoint starts early
  → dirty pages compete for I/O
  → commits and reads slow
  → application requests overlap longer
  → concurrency and queueing increase
```

This is why average disk utilization is a weak signal. A device can look comfortable over a minute while millisecond-scale fsync latency destroys the database's p99.

## Full-page writes change the economics

With `full_page_writes` enabled, PostgreSQL logs a full page on its first modification after each checkpoint. This protects against torn pages. It also means overly frequent checkpoints can increase WAL volume, which can trigger more checkpoints. Reducing `checkpoint_timeout` may shorten recovery while increasing steady-state write amplification.

Treat the settings as one policy:

| Control | Benefit | Cost |
|---|---|---|
| Larger `max_wal_size` | fewer requested checkpoints | more disk headroom and potentially longer recovery |
| Longer timeout | fewer full-page-write cycles | potentially more WAL to replay |
| High completion target | smoother writes | less slack if storage falls behind |
| Faster storage | lower flush latency | cost does not repair poor capacity assumptions |

`max_wal_size` is a soft limit, not a disk quota. Archiving failures and replication slots can retain WAL independently. Capacity planning must include those failure states.

## Observe the mechanism

Start with `pg_stat_checkpointer` and correlate it with host telemetry:

```sql
select checkpoints_timed,
       checkpoints_req,
       checkpoint_write_time,
       checkpoint_sync_time,
       buffers_written
from pg_stat_checkpointer;
```

Then compare counter deltas against WAL bytes, device latency, application p95/p99, replication lag, and batch activity. A growing requested-to-timed checkpoint ratio points toward WAL pressure. Long sync time points toward the durability path. Neither is explained by CPU alone.

## Production policy

1. Reserve disk for WAL retention failures, not only normal volume.
2. Alert on checkpoint rate and duration, not merely disk percentage.
3. Rate-limit bulk imports and index builds against an I/O budget.
4. Run crash-recovery tests before trading recovery time for throughput.
5. Change one control at a time and compare counter deltas over representative peaks.

Do not disable durability controls to make a benchmark green. If checkpoints expose a weak storage path, removing the evidence does not remove the constraint.

## A useful capacity exercise

Estimate peak WAL generation in bytes per second, not transactions per second. Then model how much dirty data the device must flush across the checkpoint interval while still serving foreground reads, WAL syncs, autovacuum, and replication. Test the model with the same storage class used in production; local SSD results say little about network-attached volumes.

Include a failure case in which archiving stalls or a replication slot stops advancing. The test should answer two independent questions: how long until latency becomes unacceptable, and how long until retained WAL consumes the reserved disk. Those are different clocks and need different alerts.

## Conclusion

A checkpoint connects recovery objectives to live request latency. The correct configuration is therefore workload-specific: WAL rate, dirty-page rate, device behavior, recovery target, replicas, and archiving all participate.

Design the checkpoint budget before the first write-heavy incident. PostgreSQL will always pay the durability bill; architecture decides whether it is paced, observable, and affordable.

---

Further reading: [PostgreSQL WAL configuration](https://www.postgresql.org/docs/current/wal-configuration.html), [PostgreSQL monitoring statistics](https://www.postgresql.org/docs/current/monitoring-stats.html), [Vacuum is concurrency control](/posts/postgresql-vacuum-is-concurrency-control), and [connections are a capacity budget](/posts/postgresql-connections-are-capacity-budget).


---

# PostgreSQL Connections Are a Capacity Budget

> A production design for connection admission, pool sizing, PgBouncer modes, reserved capacity, queueing, and preventing autoscaling from overwhelming PostgreSQL.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-connections-are-capacity-budget
Author: Ayush Basak
Last modified: 2026-08-26
Topics: postgresql, database-engineering, capacity-planning, reliability

An application pool is not a performance knob. It is a claim on a finite database capacity budget.

If twenty application instances each open fifty connections, the architecture has requested one thousand database sessions—even if PostgreSQL can execute only a fraction of those queries concurrently. Autoscaling then multiplies contention at the exact moment the database is already slow.

## Start from the database budget

PostgreSQL’s `max_connections` is a ceiling, not a target. PostgreSQL allocates resources based on it, and increasing it raises resource requirements. Reserve capacity for operations, replication, migrations, and incident response.

```text
max_connections                     300
- superuser and reserved slots       15
- replication and maintenance        25
- migrations and support             10
= application budget                250
```

Allocate those 250 connections by workload, not by whichever service starts first:

```text
checkout       70
orders         60
workers        50
reporting      20
other services 30
surge reserve  20
```

The allocation forces a product decision: which work may queue or degrade when demand exceeds the safe database concurrency?

## Size from concurrency, not request rate

Little’s Law gives a useful first estimate:

```text
concurrency ≈ throughput × time in system
```

At 400 database operations per second and 25 ms average database time, average active concurrency is about 10. Tail latency, transactions with multiple statements, and bursts require headroom, but they do not justify a pool of 200 by default.

Measure active sessions, not merely checked-out client connections:

```sql
SELECT state, wait_event_type, wait_event, count(*)
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY state, wait_event_type, wait_event
ORDER BY count(*) DESC;
```

Many `idle in transaction` sessions indicate an application correctness problem. Many active sessions waiting on locks indicate contention. More connections amplify both.

## Queue before the database

When all pool slots are busy, callers should wait in a bounded queue with a deadline. This is admission control.

```text
request deadline:       800 ms
pool acquisition limit: 100 ms
query timeout:          500 ms
response budget:        200 ms
```

Reject when the acquisition queue is full or its deadline expires. A fast explicit failure protects useful work already admitted. Creating an emergency connection outside the pool defeats the budget.

Observe pool wait duration and timeout rate. A saturated pool with low database utilization may be undersized; a saturated pool with high database latency is often correctly preventing collapse.

## Understand pooling modes

PgBouncer can multiplex many clients over fewer server connections. Its modes change application semantics:

- **session pooling:** one server connection for the client session;
- **transaction pooling:** server connection returned after each transaction;
- **statement pooling:** returned after each statement, with stronger restrictions.

Transaction pooling improves multiplexing but breaks assumptions tied to a server session: session-level settings, some prepared-statement behavior, temporary tables, advisory locks, and `LISTEN` state need careful review.

Do not deploy transaction pooling because a benchmark looks good. Inventory session features, test migrations and administrative tools, and document which connection endpoint each workload uses.

Pool hierarchy also matters:

```text
application pool → PgBouncer client slots
                 → PgBouncer server pool
                 → PostgreSQL connections
```

If every layer queues without bounded deadlines, latency becomes invisible until requests time out at the edge.

## Make autoscaling database-aware

Suppose each pod has a pool maximum of 20. Scaling from 10 to 40 pods changes potential demand from 200 to 800 connections. CPU-based autoscaling can therefore attack the database during a latency event.

Use a global budget:

```text
per-pod maximum = floor(service allocation / maximum pods)
```

Or place PgBouncer in front of PostgreSQL and treat client concurrency separately from server concurrency. Either way, maximum replicas, job parallelism, and pool settings must be reviewed together.

Background work should have a distinct, smaller pool. A backfill that consumes every connection can make the health endpoint fail and trigger more replicas—the classic positive feedback loop.

## Protect operational access

PostgreSQL supports reserved connection slots. Use them for emergency administration, but do not consider them a substitute for workload isolation. Test that on-call access still works at saturation.

Set per-role and per-database connection limits where they contain a clear failure domain. Use `statement_timeout`, `lock_timeout`, and idle-transaction limits appropriate to the workload. A connection admitted forever is not bounded capacity.

## Operate the budget

Track:

- active, idle, and idle-in-transaction sessions;
- pool utilization and acquisition wait;
- connection creation rate;
- queries and transactions per connection;
- wait events and lock queues;
- transaction duration;
- rejected work by service;
- remaining operational reserve.

Run load tests with the configured maximum replica count. Test database slowdown, not only high request volume. The dangerous case is when query duration expands and every caller holds capacity longer.

## CTO review

1. What is the application connection budget after reserves?
2. Who owns allocations across services and jobs?
3. How were pools sized from observed database concurrency?
4. Where does excess work queue, and what is its deadline?
5. Can autoscaling exceed the global budget?
6. Which session features constrain PgBouncer mode?
7. Can operators connect during saturation?
8. Does the load test include slow queries and lock contention?

The goal is not the largest pool. It is the smallest bounded concurrency that meets latency objectives while leaving the database able to recover.

## References

- [PostgreSQL: Connections and Authentication](https://www.postgresql.org/docs/current/runtime-config-connection.html)
- [PgBouncer Configuration](https://www.pgbouncer.org/config)
- [PostgreSQL: Monitoring Database Activity](https://www.postgresql.org/docs/current/monitoring-stats.html)
- [Related: Deadline Budgets and Retry Amplification](/posts/deadline-budgets-retry-amplification)
- [Related: PostgreSQL Logical Replication Failover](/posts/postgresql-logical-replication-failover)


---

# PostgreSQL Deadlocks: Lock Ordering, Transaction Retries, and Safe Recovery

> Reproduce a PostgreSQL deadlock, establish a lock-ordering contract, and design transaction retries that preserve business correctness.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-deadlocks-are-a-lock-ordering-problem
Author: Ayush Basak
Last modified: 2026-09-11
Topics: postgresql, database-engineering, concurrency, reliability

A PostgreSQL deadlock means transactions have formed a cycle of dependencies: each needs a lock held by another, so waiting alone cannot let everyone finish. The database can break that cycle. It cannot decide whether your application can safely repeat the business operation.

Consider an inventory service that reserves two products in one transaction. One request locks product 10 and then product 20. Another request reaches the same products in the opposite order. Both requests are reasonable individually. Together they expose a missing coordination rule.

The engineering objective is to make that rule explicit, keep transactions short, and recover without repeating an external effect. This guide uses a deliberately small example. The same review applies to transfers, multi-row allocations, and updates crossing aggregates.

## Reproduce the cycle before changing timeouts

Use a disposable database. Create the demonstration table once:

~~~sql
CREATE TABLE lock_order_demo (
  id integer PRIMARY KEY,
  quantity integer NOT NULL
);
INSERT INTO lock_order_demo VALUES (10, 100), (20, 100);
~~~

In connection A, run:

~~~sql
BEGIN;
SELECT id FROM lock_order_demo WHERE id = 10 FOR UPDATE;
~~~

In connection B, run:

~~~sql
BEGIN;
SELECT id FROM lock_order_demo WHERE id = 20 FOR UPDATE;
~~~

Now ask connection A to lock 20. While it waits, ask connection B to lock 10:

~~~sql
-- Connection A: blocks until the cycle is resolved.
SELECT id FROM lock_order_demo WHERE id = 20 FOR UPDATE;

-- Connection B: run separately, while A is blocked.
SELECT id FROM lock_order_demo WHERE id = 10 FOR UPDATE;
~~~

PostgreSQL detects the deadlock and aborts one participant; do not depend on which one. End both demonstration transactions with ROLLBACK after the failure resolves. PostgreSQL describes this behavior and recommends consistent lock acquisition order in its [locking documentation](https://www.postgresql.org/docs/current/explicit-locking.html).

Increasing a statement timeout does not fix the cycle. It only changes how long another class of wait may survive.

## Establish an ordering contract

For this two-row operation, every writer can acquire the lower immutable identifier first and the higher identifier second. Do that before modifying either row:

~~~sql
BEGIN;
SELECT id FROM lock_order_demo WHERE id = 10 FOR UPDATE;
SELECT id FROM lock_order_demo WHERE id = 20 FOR UPDATE;
-- Perform both related changes, then commit.
COMMIT;
~~~

This is a locking demonstration, not a complete reservation implementation. Production code must validate that both rows exist, quantities satisfy the business rule, and the caller owns the operation.

The important word is **every**. Sorting identifiers inside one endpoint is insufficient if a batch job, trigger, administrative script, or another service locks the same resources differently.

For operations spanning tables, document the table order as well as the row order. Review foreign-key interactions and trigger behavior. A local convention reduces a known deadlock pattern; it is not a proof that no other cycle exists.

## Retry the transaction, including the decision

PostgreSQL identifies a detected deadlock with SQLSTATE 40P01. Serialization failure uses 40001. These are different errors, even when the application handles both through a transaction-retry boundary. Use error codes rather than matching localized error strings. See the [error-code reference](https://www.postgresql.org/docs/current/errcodes-appendix.html).

The retry boundary must include reads and application decisions that determined the writes. Suppose an attempt reads available stock, selects a fulfillment location, and then deadlocks. Replaying only the final UPDATE can reuse a decision that no longer fits the current state.

PostgreSQL explicitly calls for retrying the complete transaction, including logic that selects SQL and values, in its [failure-handling guide](https://www.postgresql.org/docs/current/mvcc-serialization-failure-handling.html). An aborted attempt must be rolled back before a fresh attempt begins.

Keep a stable business operation identifier across attempts, but recompute state-dependent choices. These are different forms of state: the intent belongs to the caller; the allocation decision belongs to the current database snapshot.

## Keep external effects outside the replay boundary

Imagine sending an email between two SQL updates. PostgreSQL can undo its own writes after a deadlock. It cannot retract the email. The second attempt may send another.

A safer design records the business change and an outbox event in the same transaction, then performs delivery separately. That still requires a delivery and deduplication contract, as explained in the [transactional outbox guide](/posts/transactional-outbox-delivery-guarantees).

The same concern applies to charging a payment method, calling a partner API, or granting access in another system. A database rollback does not imply those effects failed.

## Failure policy

| Observation | Application response |
| --- | --- |
| 40P01 during a database-only transaction | Roll back; retry the full operation within its budget |
| Caller cancels or deadline expires | Stop creating attempts |
| Business validation fails after rereading | Return that outcome; do not force the original decision |
| Connection disappears during commit | Treat the result as ambiguous; reconcile by operation ID |
| Deadlocks recur on one operation pair | Investigate ordering and transaction scope |

A bounded retry policy is recovery machinery. It should not hide a continuously contested design. Count attempts separately from successful business operations and inspect which operation pairs conflict.

## What to measure and what to change

Track deadlocks by normalized operation name, retry exhaustion, transaction duration, and lock-wait duration. Keep record identifiers and customer data out of metric labels. Retain carefully scoped diagnostic logs when correlation requires them.

Start with transaction scope: remove network calls and user interaction from held-lock intervals. Next, examine lock ordering across all writers. If a small set of hot resources still serializes most traffic, consider whether the domain needs admission control or a different ownership boundary.

That decision has a cost. Serializing an entire tenant may simplify correctness while reducing throughput for unrelated operations. Increasing parallelism can improve unconstrained work while worsening contention on the same inventory rows. Measure useful committed operations, not merely active connections.

## Production review

Before shipping, reproduce reversed acquisition in two sessions, verify both database effects roll back, and verify the business operation can be attempted again. Then interrupt the client near commit and test reconciliation separately: an ambiguous commit is not a deadlock.

Ask whether an old application version or maintenance job can violate the ordering contract. During deployments, both versions may coexist. Tie that review to the [schema compatibility checklist](/posts/postgresql-zero-downtime-schema-migrations).

The defensible outcome is a system whose ordering rule is documented, whose retry boundary includes the decision, and whose external effects remain safe when an attempt disappears.

## References

- [PostgreSQL explicit locking and deadlocks](https://www.postgresql.org/docs/current/explicit-locking.html)
- [PostgreSQL error codes](https://www.postgresql.org/docs/current/errcodes-appendix.html)
- [PostgreSQL transaction failure handling](https://www.postgresql.org/docs/current/mvcc-serialization-failure-handling.html)
- [Related: serializable isolation and retries](/posts/serializable-isolation-is-a-retry-protocol)


---

# PostgreSQL HOT Updates Are a Physical Design Contract

> Increase PostgreSQL HOT updates by aligning indexes, fillfactor, row width, and update patterns with the heap page layout.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-hot-updates-are-a-physical-design-contract
Author: Ayush Basak
Last modified: 2026-09-06
Topics: postgresql, database-engineering, performance, storage-internals

Every PostgreSQL `UPDATE` creates a new row version. The expensive version of that operation also creates entries in every affected index and leaves more work for vacuum. Heap-only tuple updates—HOT updates—can avoid those new index entries, but only when the table’s physical design preserves two conditions.

The update must not change an indexed column, excluding summarizing indexes such as BRIN, and the heap page containing the old tuple must have enough room for the new version.

HOT is therefore not a switch. It is an agreement between schema design, page space, and workload shape.

## Why the page boundary matters

PostgreSQL indexes normally point to a tuple identifier containing a heap block and item offset. During a HOT update, the new version stays on the same heap page. The existing index entry can still lead PostgreSQL to a HOT chain containing the version visible to the current snapshot.

```text
index entry -> heap item A -> tuple v1
                           -> tuple v2
                           -> tuple v3
```

If the new tuple cannot fit on that page, PostgreSQL places it elsewhere and must add new index entries. One extra index on a frequently updated attribute can similarly disqualify the update.

This is why an apparently harmless index on `updated_at`, `last_seen_at`, or a mutable status column can create workload-wide write amplification.

## Measure before changing fillfactor

Start with table statistics:

```sql
SELECT
  relname,
  n_tup_upd,
  n_tup_hot_upd,
  round(100.0 * n_tup_hot_upd / nullif(n_tup_upd, 0), 1) AS hot_pct,
  n_dead_tup
FROM pg_stat_user_tables
WHERE n_tup_upd > 0
ORDER BY n_tup_upd DESC;
```

The ratio is cumulative since the statistics reset. Compare consistent windows and correlate them with WAL volume, index growth, vacuum duration, buffer writes, and transaction latency. A low HOT percentage is not automatically a problem on rarely updated tables.

For a genuinely update-heavy table, lowering `fillfactor` reserves page space during inserts:

```sql
ALTER TABLE account_state SET (fillfactor = 80);
```

The setting affects future page packing. Existing pages do not reorganize themselves, so a controlled rewrite may be required to realize the full effect. Rewriting a large table has locking, WAL, replication, and free-space consequences; treat it as a migration.

## Design indexes around query value

Review indexes containing frequently mutated columns. Do not drop them merely to improve HOT rate. Ask whether each index removes enough query cost to justify its write cost.

| Pattern | Likely consequence |
| --- | --- |
| index immutable identity columns | HOT remains possible for other updates |
| index `updated_at` on a hot table | most updates require index maintenance |
| lower fillfactor | more HOT opportunity, larger base table |
| wider new row version | less chance of fitting on the original page |
| frequent updates to many rows | stronger benefit, more vacuum pressure if missed |

Partial indexes can sometimes narrow the maintained population. Separating volatile counters or presence data into another table can isolate the write pattern, but adds joins and transactional coordination. Schema decomposition should follow a measured bottleneck, not a desire to maximize one ratio.

## Failure and rollout policy

Test with representative row widths and update distributions. Watch replica lag and disk headroom during any rewrite. Preserve enough free space for both the old and rewritten relation. Roll out fillfactor changes to one high-value table first, then compare equivalent traffic windows.

Do not use HOT percentage as an SLO. The product SLO is latency, throughput, recovery time, or cost. HOT is one mechanism influencing those outcomes.

## Common mistakes

- Assuming every non-indexed update is HOT without checking page space.
- Adding indexes through an ORM and ignoring update amplification.
- Changing fillfactor without planning how existing pages change.
- Comparing cumulative statistics across unequal reset windows.
- Optimizing HOT while a different constraint—locks, connections, or checkpoints—dominates latency.

## Trade-offs

Lower fillfactor spends storage and cache density to reserve future update capacity. Fewer indexes reduce write work but may make important reads slower. A table rewrite realizes the new layout sooner while creating operational risk.

The durable principle is simple: logical updates execute through a physical layout. Design both together.

## Further reading

- [PostgreSQL heap-only tuple documentation](https://www.postgresql.org/docs/current/storage-hot.html)
- [PostgreSQL table storage parameters](https://www.postgresql.org/docs/current/sql-createtable.html)
- [Every PostgreSQL index spends a write-amplification budget](/posts/postgresql-indexes-are-a-write-amplification-budget)


---

# Every PostgreSQL Index Spends a Write-Amplification Budget

> How secondary indexes affect updates, HOT eligibility, page splits, vacuum work, and the operational cost of read optimization.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-indexes-are-a-write-amplification-budget
Author: Ayush Basak
Last modified: 2026-09-01
Topics: postgresql, database-engineering, indexes, performance

An index is usually proposed as a read optimization. In PostgreSQL it is also a recurring charge on writes, WAL, cache, vacuum, replication, backup, and deployment.

The design question is not “would this query become faster?” Almost every selective query can become faster with the right structure. The question is:

> Is the saved read work worth the write amplification and operational surface for this workload?

## One update can touch many structures

PostgreSQL uses MVCC: an update creates a new row version. When indexed columns change, new index entries may also be required. More indexes mean more structures to maintain and more bytes competing for memory and storage.

Heap-only tuple (HOT) updates avoid creating new index entries when two conditions hold: the update does not modify an index-referenced column (excluding summarizing indexes such as BRIN), and the same heap page has room for the new tuple version.

This creates a non-obvious consequence: indexing a frequently changing status or timestamp can make every update more expensive even when that index serves little traffic.

Inspect the evidence:

```sql
select relname,
       n_tup_upd,
       n_tup_hot_upd,
       round(100.0 * n_tup_hot_upd / nullif(n_tup_upd, 0), 1) as hot_pct
from pg_stat_user_tables
order by n_tup_upd desc;
```

The ratio is diagnostic, not a target. A low value may be inherent to the update pattern; it may also reveal unnecessary indexes or insufficient page space.

## Covering indexes are not free

`INCLUDE` columns can enable index-only scans, but included values are stored in index tuples. Wide payloads enlarge the index, reduce fanout, increase cache pressure, and can prevent HOT updates when those columns change. “Cover the query” should not mean copying the row into every access path.

Partial indexes are often a better expression of product semantics:

```sql
create index concurrently orders_unsettled_idx
on orders (merchant_id, created_at)
where settled_at is null;
```

If unsettled orders are the operational working set, this index can remain far smaller than indexing the full history. The predicate must match actual query conditions, and deployment still needs monitoring.

## Page splits and fillfactor

B-tree pages eventually split as they fill. PostgreSQL's index `fillfactor` leaves space during builds; a lower value can smooth page splits for update-heavy indexes, at the cost of a larger structure and lower initial density. Table fillfactor can leave heap-page room and improve HOT opportunity.

Do not tune either globally by folklore. A mostly static lookup table and a hot mutable ledger have different economics.

## Index admission policy

Require each proposed index to have:

1. a named query or invariant it serves;
2. production frequency and latency evidence;
3. estimated size and write rate;
4. a deployment and rollback plan;
5. an owner and removal condition.

Review unused-index statistics carefully: counters reset, replicas may serve reads, and rare operational queries can be critical. Removal should be evidence-driven and reversible.

For large tables, `CREATE INDEX CONCURRENTLY` avoids blocking ordinary writes, but it performs more work, takes longer, and can leave an invalid index after failure. Watch progress, replication lag, WAL, disk headroom, and transaction age.

## Review the portfolio, not one query

Two individually reasonable indexes can be redundant together. Compare leading columns, predicates, sort requirements, and operator classes before admitting another structure. Use `EXPLAIN (ANALYZE, BUFFERS)` on representative data, then evaluate the write path under realistic concurrency. Planner estimates from a small development dataset are not production evidence.

Schedule an index review after major product changes. A workflow that disappeared can leave gigabytes of write amplification behind. Removal should use a measured observation window, account for read replicas and rare administrative queries, and retain a rehearsed recreation statement.

## Conclusion

Indexes move cost; they do not erase it. They exchange repeated scanning for persistent write and storage work.

A mature index strategy treats access paths as a portfolio. Keep the structures that protect real latency or correctness, measure their write cost, and retire those whose original product assumption disappeared. The fastest query in isolation is not necessarily the healthiest database.

---

Further reading: [PostgreSQL indexes](https://www.postgresql.org/docs/current/indexes.html), [HOT updates](https://www.postgresql.org/docs/current/storage-hot.html), [`CREATE INDEX`](https://www.postgresql.org/docs/current/sql-createindex.html), [vacuum as concurrency control](/posts/postgresql-vacuum-is-concurrency-control), and [zero-downtime schema migrations](/posts/postgresql-zero-downtime-schema-migrations).


---

# PostgreSQL Logical Replication Failover: The Slot Is Part of Your Recovery Plan

> How to make PostgreSQL logical subscribers survive publisher failover, verify synchronized slots, and design an honest recovery objective.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-logical-replication-failover
Author: Ayush Basak
Last modified: 2026-08-20
Topics: postgresql, database-engineering, distributed-systems, disaster-recovery

Logical replication is often added for a migration, an analytics store, or a regional read model. The happy path looks deceptively clean: publish changes, apply them elsewhere, watch lag approach zero. The recovery path is harder.

If the publisher fails over but its logical replication slot does not, the subscriber's continuity contract disappears with the old primary. A healthy standby is therefore not sufficient. The **replication state that tells the publisher what the subscriber still needs** must also be recoverable.

## Separate the three state machines

Treat logical replication as three related but independent state machines:

```text
publisher database state
        |
        +--> logical slot position and retained WAL
        |
        +--> subscriber apply position and local state
```

Physical replication protects database pages and WAL. A logical slot protects a consumer position. The subscriber maintains its own apply progress. A failover is safe only when the promoted standby has a usable copy of the slot and is far enough ahead for the subscriber to continue.

PostgreSQL's current documentation supports synchronizing failover-enabled logical slots to a physical standby. A subscription created or altered with `failover = true` identifies the logical slot as one that must survive publisher failover. But synchronization is asynchronous: configuration is not proof of readiness.

## The readiness invariant

Before a planned promotion, prove all of the following:

1. the standby has received and replayed the WAL needed for the logical slot;
2. the corresponding slot exists on the standby and reports synchronized readiness;
3. the subscriber is not ahead of the slot state available on the standby;
4. the connection endpoint will resolve to the promoted publisher;
5. the subscriber can resume without recreating the subscription or losing its origin state.

PostgreSQL explicitly warns that the standby must be ahead of the subscriber because slot synchronization happens asynchronously. This converts “HA is configured” into a measurable gate.

## An operating sequence

For a planned failover:

```text
freeze topology changes
  -> inspect publisher/subscriber lag
  -> verify failover slots synchronized on target standby
  -> stop or fence the old primary
  -> promote standby
  -> move publisher endpoint
  -> observe subscriber resume and converge
```

Do not begin with the promotion command. Begin with evidence and fencing. A network partition can leave the old primary writable; logical changes from two primaries cannot be reconciled by wishful thinking.

The exact catalog fields vary by PostgreSQL version, so build checks from that version's documentation rather than pasting a query from an old runbook. At minimum, alert on slot activity, retained WAL, subscriber lag, apply errors, and synchronization state on every eligible failover target.

## WAL retention is a capacity risk

A logical slot prevents removal of WAL still required by its consumer. When a subscriber stalls, storage consumption can grow until it becomes the incident. High availability therefore couples two budgets:

- **continuity budget:** retain enough WAL for subscribers to recover;
- **storage budget:** prevent an abandoned consumer from retaining WAL indefinitely.

Set an explicit maximum acceptable lag in bytes and time. Alert before storage headroom is exhausted. Decide whether an over-budget subscriber is paused and rebuilt, or whether the publisher is allowed to keep retaining WAL. There is no universally correct answer; there must be an owner and a threshold.

## Replication does not copy every database behavior

Logical replication transfers table data changes. It is not a full database clone. PostgreSQL documents important restrictions: schema definitions are not replicated, sequence state is not replicated, and large objects are not replicated. DDL must be coordinated separately, and the subscriber schema must remain compatible with incoming rows.

That creates a deployment order:

```text
expand subscriber schema
 -> expand publisher schema
 -> deploy compatible writers
 -> allow new data shape
 -> validate replication
 -> contract later
```

If the publisher begins emitting a column the subscriber does not understand, availability at the database layer cannot rescue the apply pipeline.

Sequence divergence matters during cutover. If a subscriber becomes the new write primary, initialize its sequences to values that cannot collide with replicated keys. For globally distributed writers, prefer an ID scheme whose uniqueness does not depend on a single local sequence.

## Recovery objectives must name the consumer

“The database has a 30-second RPO” is incomplete. Which state?

- Physical RPO: how much primary database state can be lost?
- Logical-consumer RPO: how far behind can the subscriber be?
- Apply RTO: how long until it resumes after publisher promotion?
- Rebuild RTO: how long to recreate the subscriber if continuity fails?

Analytics may tolerate a rebuild. A payment ledger read model may not. Apply one recovery objective to every logical consumer and the architecture will either waste money or hide risk.

## Failure drills that reveal the truth

Run at least these tests away from customer traffic before trusting the topology:

1. planned promotion while the subscriber is caught up;
2. promotion while the subscriber has controlled lag;
3. subscriber outage long enough to exercise WAL-retention alarms;
4. schema mismatch that stops apply, followed by repair and resume;
5. endpoint failure that proves retry and DNS behavior;
6. old-primary fencing failure to validate split-brain controls.

Record time to detection, time to resume, duplicate or missing rows, retained WAL, and manual steps. A runbook that has never consumed its own telemetry is documentation, not recovery capability.

## CTO review checklist

Ask the team:

- Are logical slots synchronized to every eligible promotion target?
- What automated condition prevents promotion when the slot is not ready?
- How much WAL can a stalled subscriber retain before we intervene?
- Who coordinates subscriber-compatible DDL?
- How are sequences and non-replicated objects handled at cutover?
- Have we measured resume time under real lag?

Logical replication failover is not a checkbox beside physical HA. The slot, subscriber position, endpoint, schema, and fencing mechanism form one recovery system. Operate them as one.

## References

- [PostgreSQL: logical replication failover](https://www.postgresql.org/docs/current/logical-replication-failover.html)
- [PostgreSQL: logical replication restrictions](https://www.postgresql.org/docs/current/logical-replication-restrictions.html)
- [PostgreSQL: logical decoding concepts](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html)
- [Related: Zero-Downtime PostgreSQL Migrations](/posts/postgresql-zero-downtime-schema-migrations)
- [Related: PostgreSQL Minor-Update Remediation](/posts/postgresql-18-6-security-patching-index-remediation)


---

# Why a Prepared PostgreSQL Query Gets Slower After Warm-Up

> Diagnose generic-plan regressions under tenant skew, compare real execution plans, and choose a scoped fix without disabling prepared statements everywhere.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-prepared-plans-and-tenant-skew
Author: Ayush Basak
Last modified: 2026-09-04
Topics: postgresql, database-engineering, performance, multi-tenancy

A query is fast in a SQL console and slow in the application. Adding an index does not help consistently. Restarting the application briefly improves latency. Before blaming connection pooling or storage, investigate whether the application and the console are executing different plans.

Prepared statements introduce a decision about reuse. A custom plan can use the actual parameter values. A generic plan trades that information for avoiding repeated planning. Neither is universally better.

> The performance contract belongs to the distribution of production parameters, not to one successful query execution.

## How tenant skew changes the decision

Imagine a hypothetical orders table where one tenant owns half the rows and most tenants own a few hundred. An index scan may suit a small tenant; reading much of the table may suit the largest. A reusable plan must operate without knowing which tenant the next execution will name.

Under PostgreSQL 18's automatic policy, the first five parameterized executions use custom plans. PostgreSQL then compares a generic plan's estimated cost with the average estimated custom cost. It does not simply switch permanently after five requests, and the comparison is not a benchmark of observed latency. See the [PREPARE documentation](https://www.postgresql.org/docs/18/sql-prepare.html).

The operational consequence is important: the first requests handled by a fresh connection can influence what happens later. A pool also contains multiple sessions, so two apparently identical requests can encounter different preparation histories.

## Reproduce the application path

Start with the exact statement text, parameter types, PostgreSQL version, driver preparation settings, role, and search path. Capture representative small, medium, and large tenants. Do not put sensitive parameter values into unrestricted logs.

In a staging database with representative distributions, compare both modes. This example assumes an existing orders table and intentionally tests a read:

```sql
PREPARE tenant_orders(bigint) AS
SELECT id, created_at, total
FROM orders
WHERE tenant_id = $1
ORDER BY created_at DESC
LIMIT 100;

BEGIN;
SET LOCAL plan_cache_mode = force_custom_plan;
EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_orders(42);
ROLLBACK;

BEGIN;
SET LOCAL plan_cache_mode = force_generic_plan;
EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_orders(42);
ROLLBACK;
```

Repeat with different tenants and cache conditions. A LIMIT can materially change which plan wins; do not replace the real query with a count and assume the comparison still applies.

## Read work, not just elapsed time

Compare estimated and actual rows, loops, rows removed by filtering, buffer hits and reads, sorts, and temporary I/O. PostgreSQL's [EXPLAIN guide](https://www.postgresql.org/docs/18/using-explain.html) explains these measurements. ANALYZE executes the statement; this is not a harmless way to inspect a production mutation.

Planning overhead matters for cheap, high-frequency queries. Execution work dominates an expensive report. Saving a small amount of planning CPU is a poor exchange for scanning millions of unnecessary rows, but forcing every tiny lookup to replan may waste capacity.

Inspect generic and custom execution counters through pg_prepared_statements in the relevant session. An administrator's separate connection cannot show every application's prepared statements through that session-local view.

## Choose the narrowest justified change

| Evidence | Candidate response | Cost to measure |
| --- | --- | --- |
| inaccurate row estimates in both modes | refresh and improve statistics | analyze overhead and estimate stability |
| custom mode consistently wins for one statement | scoped custom planning | additional planning CPU |
| different workloads hidden behind one query | split query shapes | more application paths |
| missing useful access path | evaluate an index | write amplification and storage |
| no meaningful difference | investigate locks, I/O, pooling | avoid an unrelated planner change |

Do not globally force custom plans after examining one endpoint. Also do not disable parameter binding: SQL injection prevention and generic-plan reuse are separate concerns.

For a scoped setting, use SET LOCAL inside a transaction. A session-level setting returned to a connection pool can silently affect unrelated callers. Confirm what the driver and pool support before changing preparation behavior.

## A rollout that can disprove the hypothesis

Canary the change on the affected endpoint. Compare tenant cohorts rather than only fleet-wide p95. Track planning time, execution time, database CPU, buffer work, pool wait, and error rate. Hold query shape and workload mix stable enough to interpret the result.

Include a cold-connection test, because warm-up behavior is the suspected trigger. Include large and small tenants, because an aggregate improvement can hide a severe regression for one cohort. Define rollback as a configuration change with a known owner, not an emergency database restart.

## The architecture decision

Prepared statements are a useful optimization, not a promise of uniform performance. Make parameter skew part of performance testing and preserve evidence about the actual execution path.

The leadership question is not whether to “use prepared statements.” It is whether one reusable plan fits the workload distribution—and how narrowly the system can respond when it does not.

## Further reading

- [PostgreSQL: Prepared-statement inspection](https://www.postgresql.org/docs/18/view-pg-prepared-statements.html)
- [Connections are a capacity budget](/posts/postgresql-connections-are-capacity-budget)
- [Every index spends a write-amplification budget](/posts/postgresql-indexes-are-a-write-amplification-budget)


---

# PostgreSQL Replication Slots Are Retention Leases

> How restart_lsn, catalog_xmin, retained WAL, slot invalidation, and consumer ownership turn replication slots into explicit capacity commitments.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-replication-slots-are-retention-leases
Author: Ayush Basak
Last modified: 2026-09-03
Topics: postgresql, database-engineering, replication, capacity-planning, reliability

A replication slot is often introduced as a convenience: PostgreSQL keeps what a replica or CDC consumer still needs. Operationally, the slot is a lease against finite storage and cleanup capacity.

The invariant is:

> Every slot must have an owner, a recovery objective, and a bounded retention budget.

Physical slots protect WAL required by a standby. Logical slots can protect both WAL and catalog or row visibility horizons needed for decoding. A disconnected consumer stops making progress, but its claim on the primary can continue growing.

## Read the slot as a contract

`pg_replication_slots` exposes several distinct boundaries:

- `restart_lsn`: oldest WAL the consumer may still require;
- `confirmed_flush_lsn`: logical consumer acknowledgement point;
- `xmin` and `catalog_xmin`: row/catalog horizons vacuum must preserve;
- `wal_status`: whether required WAL is reserved, extended, unreserved, or lost;
- `safe_wal_size`: remaining WAL bytes before the slot risks becoming lost;
- `inactive_since` and `invalidation_reason`: lifecycle evidence.

Monitor retained bytes directly:

```sql
SELECT slot_name, slot_type, active, wal_status, safe_wal_size,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal,
       inactive_since, invalidation_reason
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC NULLS LAST;
```

Lag in seconds is insufficient. Ten minutes during a bulk load can retain more WAL than hours of quiet traffic. Capacity is driven by byte rate and recovery time.

## Unlimited retention is delayed failure

With `max_slot_wal_keep_size = -1`, slots may retain unlimited WAL. That protects consumers until `pg_wal` fills the volume and threatens the primary. A finite limit protects the database but allows a lagging slot to become unusable and require re-seeding.

That is the real trade-off:

| Policy | Protects | Sacrifices under long outage |
| --- | --- | --- |
| unlimited retention | consumer continuity | primary disk safety |
| finite slot limit | primary capacity | consumer may need rebuild |
| no slot plus archive | primary working set | recovery depends on archive completeness |

Choose the consumer recovery objective first. If rebuilding a 20 TB downstream store takes three days, a 30-minute retention budget is not honest resilience.

## Lifecycle must be automated carefully

`idle_replication_slot_timeout` can invalidate inactive slots, with enforcement occurring at checkpoint. It is useful for abandoned consumers, not a substitute for ownership. Some slots—such as synchronized standby slots—have special lifecycle behavior.

Maintain a registry outside PostgreSQL:

```text
slot -> owning service -> escalation -> retention bytes/time
     -> rebuild procedure -> criticality -> expected activity
```

Alert before invalidation using burn rate:

```text
time_to_loss ~= safe_wal_size / recent_wal_bytes_per_second
```

Use a conservative high-percentile WAL rate, especially around migrations and backfills. Page rewrites and index work can invalidate yesterday's forecast.

## Dropping a slot is a data decision

Deleting an unknown inactive slot may release disk immediately while permanently removing the consumer's continuation point. Before dropping it, identify the owner, validate whether the consumer has another checkpoint, preserve required evidence, and decide whether a full snapshot is acceptable.

Likewise, restarting the consumer is not enough if `wal_status = lost`. The required history is already gone. Recovery must move to a new base snapshot or another authoritative source.

## The CTO decision

Treat every slot like allocated storage with an accountable customer. Budget retained WAL, test re-seeding, measure time to loss, and make slot invalidation an explicit product trade-off.

Replication slots make retention precise. They do not make it free.

## References

- [PostgreSQL: Replication slots](https://www.postgresql.org/docs/current/warm-standby.html)
- [PostgreSQL: Replication configuration](https://www.postgresql.org/docs/current/runtime-config-replication.html)
- [PostgreSQL: pg_replication_slots](https://www.postgresql.org/docs/current/view-pg-replication-slots.html)
- [Related: Change Data Capture is a recovery contract](/posts/cdc-recovery-contract)
- [Related: PostgreSQL vacuum is concurrency control](/posts/postgresql-vacuum-is-concurrency-control)


---

# Tenant Isolation Should Not Depend on Every WHERE Clause

> A production guide to PostgreSQL row-level security: policy design, connection context, owner bypass, testing, performance, and operational safeguards.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-rls-tenant-isolation
Author: Ayush Basak
Last modified: 2026-08-28
Topics: postgresql, database-engineering, security, multi-tenancy, backend-engineering

In a shared-schema SaaS database, the most expensive missing predicate is usually `tenant_id = ?`.

Application reviews, repository abstractions, and integration tests reduce the risk, but they all rely on every future query preserving the same invariant. PostgreSQL row-level security (RLS) moves that invariant closer to the data: normal access is permitted only when a policy allows the row.

RLS is not a complete tenancy strategy. It is a database-enforced backstop for one of its most important rules.

## Start with the invariant

For an `invoice` table, state the policy before writing SQL:

> A request may read or modify an invoice only when the request tenant equals the row tenant. New rows must belong to the request tenant.

```sql
ALTER TABLE invoice ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoice FORCE ROW LEVEL SECURITY;

CREATE POLICY invoice_tenant_select
ON invoice FOR SELECT
USING (tenant_id = current_setting('app.tenant_id', true)::uuid);

CREATE POLICY invoice_tenant_write
ON invoice FOR ALL
USING (tenant_id = current_setting('app.tenant_id', true)::uuid)
WITH CHECK (tenant_id = current_setting('app.tenant_id', true)::uuid);
```

`USING` filters existing rows. `WITH CHECK` rejects rows whose new values violate the policy. Both matter: protecting reads while permitting a tenant to insert another tenant’s ID is not isolation.

PostgreSQL applies default deny when RLS is enabled and no applicable policy exists. Treat that as a deployment advantage: new tables should fail closed until their access model is explicit.

## Request context must survive pooling

The policy is only as correct as `app.tenant_id`. With transaction pooling, session-level state can leak across requests. Set context inside the transaction and make it local:

```sql
BEGIN;
SELECT set_config('app.tenant_id', $1, true); -- true: transaction-local
-- application queries
COMMIT;
```

Never accept the tenant identifier from an unverified request field. Derive it from authenticated identity and authorised membership. Resetting state is not an adequate substitute for transaction-local context because exceptions and pool behaviour create leak paths.

Background workers need the same contract. A job payload should carry a tenant identity or intentionally use a separate privileged workflow with bounded scope and an audit trail.

## Understand bypass paths

Table owners normally bypass row security. Superusers and roles with `BYPASSRLS` do too. `FORCE ROW LEVEL SECURITY` makes the owner subject to policies in ordinary operation, but privileged roles remain privileged.

Therefore:

- the application must not connect as a superuser;
- the migration owner should be different from the runtime role;
- administrative jobs need narrowly scoped roles;
- connection strings and role grants belong in security review;
- tests must run under the real runtime role, not the table owner.

An RLS test executed as a bypassing role proves nothing.

## Index for the policy

RLS adds policy expressions to query planning. Tenant-scoped access should usually begin with tenant identity in the index:

```sql
CREATE INDEX invoice_tenant_created_idx
ON invoice (tenant_id, created_at DESC);

CREATE UNIQUE INDEX invoice_tenant_number_key
ON invoice (tenant_id, invoice_number);
```

Global uniqueness is often a hidden multi-tenant bug. If invoice numbers need only be unique within a tenant, encode that fact in the constraint.

Inspect representative plans with the runtime role and tenant context. Policies containing subqueries, volatile functions, or joins can create surprising cost and locking behaviour. Keep the hot-path predicate simple.

## Test isolation as a matrix

Do not stop at “tenant A cannot select tenant B.” Test every command and bypass boundary:

| Operation | Expected |
|---|---|
| A selects A row | allowed |
| A selects B row | invisible |
| A updates B row | no affected row |
| A inserts B tenant ID | rejected |
| A changes row tenant to B | rejected |
| request with missing context | denied/error |
| pooled transaction after A serves B | only B visible |
| runtime role attempts to disable RLS | denied |

Add property-style tests that generate tenants and operations. Include prepared statements and the production pooler mode. Isolation failures are data breaches, so this suite deserves the same release authority as payment correctness.

## Operations still need tenant-aware design

RLS does not isolate CPU, connections, storage growth, locks, or noisy queries. It does not solve per-tenant backup and restore. It does not automatically scope caches, object storage, search indexes, events, or analytics.

Log the authenticated tenant separately from any tenant ID supplied in data. Alert on policy violations without logging sensitive row content. Include tenant scope in reconciliation and deletion workflows.

For large or regulated tenants, pooled RLS may eventually give way to schemas, databases, or cells. That is not a failure. Isolation is a spectrum, and the correct boundary follows risk, scale, and recovery requirements.

## The architecture decision

Use RLS when many tenants share tables and the cost of a missed application predicate is unacceptable. Keep application predicates anyway: they communicate intent, improve portability, and can help query plans. The database policy is defense in depth, not permission to make the application tenant-unaware.

The strongest test is simple: can one ordinary application query accidentally cross the tenant boundary? With correctly operated RLS, the database should make the answer no.

## References

- [PostgreSQL: Row Security Policies](https://www.postgresql.org/docs/current/ddl-rowsecurity.html)
- [PostgreSQL: CREATE POLICY](https://www.postgresql.org/docs/current/sql-createpolicy.html)
- [Related: Multi-Tenant SaaS Isolation](/posts/multi-tenant-saas-isolation)
- [Related: PostgreSQL Connections Are a Capacity Budget](/posts/postgresql-connections-are-capacity-budget)


---

# PostgreSQL Synchronous Replication Is a Commit Contract

> How synchronous_commit, quorum standbys, remote_apply, failure domains, and commit latency define the durability you actually sell to customers.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-synchronous-replication-is-a-commit-contract
Author: Ayush Basak
Last modified: 2026-09-02
Topics: postgresql, database-engineering, replication, reliability, system-design

Synchronous replication is often described as a checkbox for “zero data loss.” That description hides the decision that matters: **what evidence must PostgreSQL collect before the application is allowed to call a transaction committed?**

The invariant is:

> A commit acknowledgement is a promise about which failures the transaction can survive, not proof that every replica can already serve it.

PostgreSQL exposes several acknowledgement boundaries. Choosing one changes latency, durability, read-after-write behavior, and availability during a standby failure.

## Follow one WAL record

A transaction becomes durable through a sequence of events:

```text
application
  -> primary inserts commit record into WAL
  -> primary flushes WAL
  -> standby receives WAL
  -> standby writes WAL to its operating system
  -> standby flushes WAL to durable storage
  -> standby replays WAL
  -> standby query can observe the transaction
```

These events are not interchangeable. `synchronous_commit = on` waits for local flush and for the selected synchronous standby to report durable WAL flush. `remote_write` waits for the standby operating system to accept the WAL but not necessarily persist it through an OS crash. `remote_apply` waits until the standby has replayed the transaction, which can support causal reads from that standby at the cost of replay delay in the commit path.

`local` and `off` weaken the remote or local acknowledgement boundary further. They may be sensible for rebuildable telemetry or derived events. They are rarely sensible for a payment ledger merely because an incident made synchronous commits slow.

## Quorum is not geography

PostgreSQL can use priority-based or quorum-based synchronous standbys. A quorum configuration can look like:

```text
synchronous_standby_names = 'ANY 1 (az_a, az_b, az_c)'
```

This allows a commit to proceed after any one listed synchronous candidate acknowledges it. It improves tolerance of one slow or unavailable standby, but the names alone say nothing about failure independence. Three virtual machines on the same storage system are not three durability domains.

The architecture review must map each acknowledgement to power, storage, network, availability-zone, and operator failure domains. If the primary and acknowledged standby share the failure you claim to tolerate, the topology does not satisfy the product promise.

## Availability is part of the contract

If the configuration requires acknowledgements that cannot arrive, commits wait. PostgreSQL has not failed: it is preserving the configured contract. The operational question is whether the business prefers write unavailability or a weaker durability mode during that failure.

Do not let an improvised incident command decide this for the first time. Define a degradation policy:

| Condition | Default action | Business consequence |
| --- | --- | --- |
| one quorum candidate lost | continue with remaining candidates | reduced redundancy |
| all synchronous candidates lost | stop durable writes | availability loss |
| approved emergency downgrade | change named policy explicitly | increased data-loss exposure |
| standby replay slow under `remote_apply` | investigate replay bottleneck | commit latency rises |

An automatic downgrade from synchronous to asynchronous replication can preserve availability while silently violating the recovery objective. If allowed, make it a named mode with an owner, audit event, alert, expiry, and reconciliation procedure.

## Per-transaction policy is powerful and dangerous

`synchronous_commit` can be set per transaction. This lets one cluster carry different durability classes:

```sql
BEGIN;
SET LOCAL synchronous_commit = 'remote_apply';
UPDATE account_balance
SET amount = amount - 500
WHERE account_id = 42;
COMMIT;
```

A ledger mutation may wait for remote durability, while an idempotent analytics event may accept asynchronous loss. The distinction belongs in a reviewed data classification—not scattered ORM settings.

Connection pools also make session settings dangerous. Prefer `SET LOCAL` inside a transaction so a relaxed policy cannot leak to the next borrower of a pooled connection.

## Latency must be budgeted end to end

Synchronous commit latency includes primary WAL work, network transit, standby write or flush, reply transit, and sometimes replay. Tail latency matters more than the average because every qualifying commit depends on a remote critical path.

Measure:

- commit latency by durability class;
- WAL write, flush, and replay locations per standby;
- `write_lag`, `flush_lag`, and `replay_lag` with workload context;
- number and location of eligible synchronous candidates;
- transactions waiting on synchronous replication;
- WAL generation rate versus network and standby disk capacity;
- frequency and duration of degraded durability modes.

Replication lag is not a single number. A standby may have received WAL but not flushed it, or flushed it but not replayed it. Alert on the boundary your contract uses.

## Failover still has ambiguous outcomes

A client can lose its connection after PostgreSQL commits but before the acknowledgement reaches the application. Synchronous replication reduces data-loss exposure; it does not tell the caller whether an interrupted request committed.

Writes still need stable operation identifiers and reconciliation:

```sql
CREATE TABLE transfer (
  operation_id uuid PRIMARY KEY,
  from_account bigint NOT NULL,
  to_account bigint NOT NULL,
  amount numeric(18,2) NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);
```

On retry, look up `operation_id`. Do not create a second transfer because the first response was lost. Durability and idempotency solve different failure boundaries.

## The CTO decision

Write down the promise before tuning the database: which transaction classes may lose acknowledged data, which failure domains must be survived, whether read-after-write from a standby is required, and when writes should stop instead of weakening the promise.

Then choose `synchronous_commit`, quorum membership, topology, and degradation automation to implement that contract. “Synchronous” is not the outcome. An explicit, tested acknowledgement boundary is.

## References

- [PostgreSQL: Log-shipping standby servers](https://www.postgresql.org/docs/current/warm-standby.html)
- [PostgreSQL: Write-ahead log configuration](https://www.postgresql.org/docs/current/runtime-config-wal.html)
- [PostgreSQL: Replication configuration](https://www.postgresql.org/docs/current/runtime-config-replication.html)
- [Related: Backups do not prove recoverability](/posts/backups-do-not-prove-recoverability)
- [Related: PostgreSQL logical-replication failover](/posts/postgresql-logical-replication-failover)


---

# PostgreSQL Vacuum Is Concurrency Control, Not Housekeeping

> A mechanical guide to MVCC dead tuples, visibility horizons, autovacuum thresholds, transaction-ID freezing, wraparound, and the sessions that prevent cleanup.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-vacuum-is-concurrency-control
Author: Ayush Basak
Last modified: 2026-08-31
Topics: postgresql, database-engineering, mvcc, reliability

An `UPDATE` in PostgreSQL usually does not overwrite a row. It creates a new tuple version and leaves the old version behind until no active snapshot can need it. `DELETE` similarly marks a version obsolete without immediately removing its storage.

That behavior is the foundation of MVCC: readers and writers avoid blocking one another because they can observe different versions. Vacuum is the process that eventually proves an old version is no longer visible to anybody and makes its space reusable.

The invariant is:

> A tuple may be removed only after every transaction that could legally observe it has ended, and every surviving tuple must remain comparable across transaction-ID wraparound.

## The cleanup horizon is global

Suppose transaction 100 begins a report and holds its snapshot. Transactions 101 through 500 update the same account rows. Those old tuple versions cannot be removed if transaction 100 might still see them.

```text
T100: snapshot opens ────────────────────────────────┐
T101..T500: updates create dead versions            │
VACUUM: sees versions, but cannot remove them        │
T100: commits ───────────────────────────────────────┘
next VACUUM: cleanup can progress
```

The long transaction may be “only reading,” yet it expands storage, index work, cache pressure, replica WAL, and future vacuum cost. Idle transactions are worse: they retain a snapshot while delivering no product value.

Old prepared transactions and replication slots can also hold horizons. A logical slot's `catalog_xmin` protects catalog rows required to decode its stream. Dropping an abandoned slot may release cleanup, but a real consumer may then require rebuilding. This is an operational decision, not a routine delete.

## Autovacuum thresholds are workload policy

Autovacuum decides whether a table needs vacuuming using a base threshold plus a scale factor derived from table size. A large table can therefore accumulate many dead tuples before the percentage threshold is crossed. A small, intensely updated table may need much more frequent vacuuming than global defaults provide.

Tune per table when the workload demands it:

```sql
ALTER TABLE job_queue SET (
  autovacuum_vacuum_scale_factor = 0.01,
  autovacuum_vacuum_threshold = 500,
  autovacuum_analyze_scale_factor = 0.02
);
```

Lower thresholds trade background I/O for a smaller dead-tuple backlog. Increasing workers without checking I/O capacity can turn maintenance into foreground latency. The target is not “vacuum constantly”; it is “complete useful cleanup faster than the workload creates debt.”

## Vacuum and vacuum full solve different problems

Ordinary `VACUUM` marks space reusable inside the relation and can run alongside normal reads and writes. It generally does not return the file's space to the operating system. `VACUUM FULL` rewrites the table into a compact file and requires an `ACCESS EXCLUSIVE` lock.

If the only plan for bloat is regular `VACUUM FULL`, the maintenance policy has already failed. Prefer preventing the backlog, controlling transaction horizons, and scheduling rewrites only when returning disk space justifies the lock and rewrite cost.

## Transaction IDs create a safety deadline

PostgreSQL's internal `xid` is 32 bits and wraps after roughly four billion assignments. Visibility comparisons treat about two billion IDs as the past and two billion as the future. A tuple left with an ancient normal XID can eventually appear to belong to the future.

Vacuum freezes sufficiently old tuple versions so they remain visible regardless of future XID movement. PostgreSQL tracks the oldest remaining unfrozen XID through `relfrozenxid` and `datfrozenxid`. Anti-wraparound vacuum is therefore correctness work, not storage optimization.

Near exhaustion, PostgreSQL protects data by refusing commands that assign new XIDs. The database becoming effectively read-only is a safety mechanism after maintenance has been ignored.

```sql
SELECT datname, age(datfrozenxid) AS xid_age
FROM pg_database
ORDER BY xid_age DESC;

SELECT relname, n_live_tup, n_dead_tup,
       last_autovacuum, autovacuum_count
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC;
```

## What to monitor

- oldest `backend_xmin` and transaction age;
- oldest prepared transaction;
- replication-slot `xmin`, `catalog_xmin`, and retained WAL;
- `age(datfrozenxid)` and per-table `age(relfrozenxid)`;
- dead tuples created versus removed per interval;
- autovacuum duration, cancellations, and worker saturation;
- table and index growth relative to live rows;
- commit-to-replica lag during heavy maintenance.

An alert on disk percentage alone arrives too late and explains too little.

## The CTO decision

Vacuum policy belongs in capacity planning. Set limits for transaction age, prohibit idle-in-transaction sessions, size autovacuum for the write rate, and give high-churn tables explicit settings. Treat abandoned slots and prepared transactions as incidents because they can pin global cleanup.

MVCC moves contention out of the foreground path. Vacuum is where the deferred cost becomes visible.

## References

- [PostgreSQL: Routine Vacuuming](https://www.postgresql.org/docs/current/routine-vacuuming.html)
- [PostgreSQL: VACUUM](https://www.postgresql.org/docs/current/sql-vacuum.html)
- [PostgreSQL: Transactions and Identifiers](https://www.postgresql.org/docs/current/transaction-id.html)
- [Related: PostgreSQL connections are a capacity budget](/posts/postgresql-connections-are-capacity-budget)


---

# PostgreSQL Zero-Downtime Schema Migrations: A Production Guide

> A production playbook for expand-backfill-contract migrations, PostgreSQL lock control, concurrent indexes, and safe constraint validation.

Canonical URL: https://www.ayushworks.xyz/posts/postgresql-zero-downtime-schema-migrations
Author: Ayush Basak
Last modified: 2026-09-11
Topics: postgresql, database-engineering, system-design, reliability

A database migration is not safe because the SQL finishes quickly on staging. It is safe when old application instances, new application instances, background jobs, replicas, and rollback procedures can coexist while the change is in flight.

That makes zero-downtime migration a **compatibility protocol**, not a DDL trick. PostgreSQL gives us useful primitives—transactional DDL, `NOT VALID`, `VALIDATE CONSTRAINT`, and `CREATE INDEX CONCURRENTLY`—but it cannot decide whether two application versions understand the same data.

## The invariant

During every deployment phase, all live readers must understand the stored representation and every live writer must preserve it. A useful deployment model is:

```text
expand schema -> deploy compatible code -> backfill -> enforce -> remove legacy path
```

Each arrow is a checkpoint. If rollback crosses more than one checkpoint, the plan is too coupled.

Consider renaming `customers.name` to `display_name`. A direct rename makes old binaries fail immediately. The compatible sequence is:

1. Add nullable `display_name`.
2. Deploy code that can read either column and writes both.
3. Backfill old rows in bounded batches.
4. Verify parity and switch reads to `display_name`.
5. Stop writing `name`, wait for old binaries and jobs to disappear, then drop it in a later release.

The apparent duplication is not waste. It buys a rollback window.

## Lock time is the first budget

“Without downtime” means preserving the application’s availability target throughout the rollout. It does not mean every DDL operation is lock-free. Confirm the behavior against the documentation for your deployed PostgreSQL version, and test with representative concurrent traffic.

Many `ALTER TABLE` forms acquire `ACCESS EXCLUSIVE`, PostgreSQL's strongest table lock. Even a metadata-only operation can sit behind a long transaction and then block new traffic once it reaches the front of the lock queue. The dangerous variable is often **wait time**, not execution time.

Set a short lock timeout for online migrations and fail visibly instead of waiting indefinitely:

```sql
BEGIN;
SET LOCAL lock_timeout = '2s';
SET LOCAL statement_timeout = '15s';

ALTER TABLE customers
  ADD COLUMN display_name text;
COMMIT;
```

Retry the migration under controlled automation. Do not raise the timeout until it eventually succeeds; inspect blockers in `pg_stat_activity` and `pg_locks`, then remove the operational cause.

## Backfill without becoming the incident

A single `UPDATE` over a large table creates a large transaction, increases WAL, retains dead tuples until commit, delays replicas, and competes with customer traffic. Backfill by a stable key, commit each batch, and make the operation restartable:

```sql
UPDATE customers
SET display_name = name
WHERE id > $1
  AND id <= $2
  AND display_name IS NULL;
```

The worker should persist its cursor, cap rows or wall-clock time per batch, sleep under load, and expose rows processed, WAL volume, replica lag, lock wait, and error rate. The correct batch size is an operating measurement—not a constant copied from another company.

Avoid offset pagination. Concurrent inserts and updates make offsets expensive and ambiguous; a monotonic primary-key cursor gives deterministic progress. If the transform is not naturally idempotent, record a migration version so retries cannot apply it twice.

## Add constraints in two phases

Validating a new foreign key or check constraint against every historical row can hold disruptive locks longer than expected. PostgreSQL supports a safer sequence:

```sql
ALTER TABLE orders
  ADD CONSTRAINT orders_total_nonnegative
  CHECK (total_cents >= 0) NOT VALID;

ALTER TABLE orders
  VALIDATE CONSTRAINT orders_total_nonnegative;
```

`NOT VALID` avoids the initial table scan while still enforcing the constraint for new or changed rows. `VALIDATE CONSTRAINT` later checks historical rows with a less disruptive lock. PostgreSQL documents that validation uses `SHARE UPDATE EXCLUSIVE`, allowing ordinary reads and writes to continue.

For a uniqueness constraint on a large table, build the index first:

```sql
CREATE UNIQUE INDEX CONCURRENTLY customers_tenant_email_uq
  ON customers (tenant_id, lower(email));
```

Concurrent index construction permits writes but performs more work, takes longer, and cannot run inside a transaction block. If it fails, it can leave an `INVALID` index that must be inspected and removed or rebuilt. “Concurrent” means lower write blocking, not zero operational cost.

## The hidden contracts

Schema dependencies extend beyond the request-serving application:

- CDC connectors may identify columns by name and ordinal position.
- BI queries and exports may bypass the service abstraction.
- old queue messages may be replayed by new consumers.
- read replicas may lag behind the migration checkpoint.
- caches may hold objects encoded with the old shape.
- rollback may reintroduce a writer that no longer maintains the new field.

Inventory these consumers before DDL. A schema registry helps only if the unregistered paths are found.

## A migration control record

Every material migration should have a small control document:

```yaml
owner: customer-platform
invariant: name and display_name remain equal during dual-write
abort_if:
  lock_wait_seconds: 2
  replica_lag_seconds: 10
  api_error_rate: 1%
rollback: deploy previous reader; retain both columns
destructive_after: 2026-09-20
verification: parity query plus sampled application reads
```

This is more valuable than a hundred-line migration script with no operating boundary.

## Are PostgreSQL migrations reversible?

Application rollback and data rollback are different operations. While both name columns are maintained, an older reader can be redeployed. After the old column is dropped or information is irreversibly transformed, a reverse SQL script cannot reconstruct missing values.

Record the last compatible application version at each phase. Before a destructive step, prove that old writers, scheduled jobs, and replay consumers have retired. If recovery requires a backup, test the restore and reconciliation process first; see [why backups do not prove recoverability](/posts/backups-do-not-prove-recoverability).

## Common migration mistakes

**Deploying DDL and dependent code together.** Rolling deployments guarantee a period with mixed versions. Separate expansion from adoption.

**Treating a backfill as maintenance work.** It is a production workload with its own rate limit and SLO impact.

**Dropping compatibility immediately.** The absence of errors for ten minutes does not prove that cron jobs, replay workers, or rollback images are gone.

**Assuming an ORM migration is online.** Inspect the exact SQL and the lock mode for the deployed PostgreSQL version.

**Calling the plan zero-downtime without an abort threshold.** Safety requires a measurable point at which automation stops.

## CTO review checklist

Before approving the change, ask:

1. Which old and new binaries coexist at every phase?
2. What is the maximum lock-wait budget, and does the migration fail closed?
3. Is the backfill bounded, observable, throttled, and restartable?
4. Can we roll back code without reversing a destructive schema operation?
5. Which replicas, connectors, jobs, exports, and caches consume the field?
6. What evidence permits the final destructive step?

The mature posture is simple: schema changes are distributed-system changes. Expand first, preserve compatibility, measure the backfill, enforce separately, and delete only after the evidence says the old world is gone.

## References

- [PostgreSQL: ALTER TABLE](https://www.postgresql.org/docs/current/sql-altertable.html)
- [PostgreSQL: CREATE INDEX and concurrent builds](https://www.postgresql.org/docs/current/sql-createindex.html)
- [PostgreSQL: explicit locking](https://www.postgresql.org/docs/current/explicit-locking.html)
- [Related: Thinking Through Offline-First Payments](/posts/offline-first-payments)
- [Related: Building Microservices That Fail Gracefully](/posts/building-reliable-microservices)


---

# Product Quantization Is a Recall–Memory Contract

> Reason about IVFPQ through memory, candidate coverage, quantization error, reranking, and measured retrieval recall.

Canonical URL: https://www.ayushworks.xyz/posts/product-quantization-is-recall-memory-contract
Author: Ayush Basak
Last modified: 2026-09-05
Topics: algorithms, vector-search, ai-infrastructure, performance

At large vector counts, storing every coordinate as a 32-bit float becomes an architecture decision. One billion 768-dimensional vectors require about 3.1 TB for the raw coordinates alone, before IDs, indexes, replicas, and allocator overhead.

Product quantization (PQ) changes that equation by replacing each full vector with a compact learned code. The price is approximation error. The system is no longer buying “a smaller index”; it is choosing a recall–memory contract.

## The mechanism

Split a `d`-dimensional vector into `M` sub-vectors. Train a small codebook for each subspace. Store the nearest codeword identifier rather than every floating-point component.

If every sub-quantizer uses 256 centroids, each identifier fits in one byte. With `M = 96`, a vector can be represented by a 96-byte PQ code rather than 3,072 bytes of float32 data. This comparison excludes surrounding index state, but shows why the technique matters.

At query time, the engine builds distance lookup tables between query sub-vectors and codewords, then sums approximate distances for candidate codes.

```text
query
  -> coarse centroid search
  -> probe selected inverted lists
  -> approximate distance over PQ codes
  -> optional exact rerank
  -> top-k
```

Faiss `IndexIVFPQ` combines an inverted file (IVF) with PQ encoding of residual vectors. That introduces two distinct ways to miss a relevant result:

1. The correct vector is in a list that was not probed.
2. Quantization changes the relative order of candidates.

Do not compress those into a single tuning knob.

## Measure against an exact baseline

Create a representative, versioned evaluation set of queries. Compute exact top-k neighbors on a manageable corpus, then compare approximate results.

```python
def recall_at_k(exact_ids, approx_ids, k):
    exact = set(exact_ids[:k])
    approx = set(approx_ids[:k])
    return len(exact & approx) / k
```

Measure at least:

- recall@k by query cohort;
- p50, p95, and p99 search latency;
- resident memory and index build time;
- candidates scanned and lists probed;
- quality after filters are applied;
- degradation after embedding-distribution changes.

Average recall can conceal failure for short queries, rare languages, or a high-value tenant. Product metrics should identify which misses actually change user outcomes.

## Tune the whole retrieval path

Increasing `nprobe` explores more coarse lists and usually improves candidate coverage at a latency cost. Increasing PQ code size reduces compression and can reduce distance error. Reranking a larger shortlist with original vectors improves final ordering but requires access to those vectors and more bandwidth.

| Lever | Likely gain | Cost |
| --- | --- | --- |
| more coarse lists probed | candidate recall | latency and reads |
| larger PQ code | distance fidelity | memory |
| larger rerank set | final precision | compute and original-vector access |
| better training sample | representative codebooks | retraining pipeline |
| smaller index shards | cache locality | routing and operational complexity |

## Common mistakes

- Training codebooks on an unrepresentative sample.
- Reporting only index memory while keeping a full-vector copy on every replica.
- Tuning on random vectors instead of production query–document pairs.
- Changing the embedding model without retraining and re-evaluating the index.
- Treating top-k recall as identical to answer quality in a RAG system.

## Decision

Use PQ when exact-vector memory or bandwidth is the binding constraint and the product can define an acceptable measured loss. Keep exact search when the corpus is small enough, errors are extremely expensive, or operational simplicity is worth the RAM.

Compression is not free capacity. It is an explicit exchange of representation fidelity for memory, evaluated at the product boundary.

## Further reading

- [Faiss IndexIVFPQ reference](https://faiss.ai/cpp_api/file/IndexIVFPQ_8h.html)
- [Billion-scale similarity search with GPUs](https://arxiv.org/abs/1702.08734)
- [Filtered vector search is a recall budget](/posts/filtered-vector-search-is-a-recall-budget)


---

# Production AI Needs Release Gates, Not a Trust Score

> A practical operating model for AI risk: use-case boundaries, eval gates, authority limits, observability, degradation, and accountable production release.

Canonical URL: https://www.ayushworks.xyz/posts/production-ai-risk-gates
Author: Ayush Basak
Last modified: 2026-08-28
Topics: ai-infrastructure, production-ai, technical-leadership, reliability, security

Teams often ask whether a model is “good enough for production.” That question is too broad to answer and too vague to govern a release.

A model is not deployed in isolation. It is placed inside a use case, given data, connected to tools, shown to users, and allowed to influence decisions. Production readiness belongs to that entire system.

NIST’s AI Risk Management Framework organises work around four continuing functions: govern, map, measure, and manage. The useful engineering translation is a set of release gates tied to a named use case and an explicit risk tolerance.

## Gate 1: define the decision boundary

Write down what the AI component does and, equally important, what it cannot do.

```text
input: customer email and permitted account context
output: proposed reply and cited account facts
authority: draft only
human decision: approve, edit, or discard
prohibited: send message, change price, alter account state
```

“Sales assistant” is not a boundary. “Draft a response using these authorised records without sending it” is testable.

Classify consequences. A bad internal summary is different from an incorrect medical recommendation or an agent that can execute a refund. As consequence rises, require stronger evidence, narrower authority, deterministic validation, and human approval.

## Gate 2: build an evaluation contract

An evaluation set should represent the distribution the product expects, including edge cases and adversarial inputs. Separate dimensions instead of compressing them into one score:

- task correctness;
- factual support and citation validity;
- policy compliance;
- refusal behaviour;
- tool-selection correctness;
- sensitive-data handling;
- latency and cost;
- consistency across relevant languages or customer segments.

Define blocking thresholds before running the release candidate. Averages can hide catastrophic classes, so add zero-tolerance or near-zero-tolerance gates for severe outcomes.

```yaml
release_gate:
  task_success: ">= 0.90"
  unsupported_account_claim: "<= 0.005"
  unauthorised_tool_attempt: "0"
  p95_latency_ms: "<= 2500"
  cost_per_completed_task_usd: "<= 0.04"
```

The example values are illustrative; production thresholds must follow the use case and evidence. Version the dataset, rubric, evaluator, prompt, model, retrieval configuration, and tool schema so results are reproducible.

## Gate 3: constrain authority in code

Prompts are instructions to a probabilistic component, not access control. Enforce permissions outside the model.

Every tool call should pass through a policy layer that validates identity, tenant, requested action, resource scope, and budget. Use narrow tools such as `draft_follow_up(account_id)` instead of generic database or HTTP access.

For consequential writes:

1. produce a typed proposal;
2. validate it deterministically;
3. show material effects to an authorised human;
4. record approval with the proposal hash;
5. execute idempotently;
6. preserve an audit event.

Authority should be revocable without redeploying the model. Feature flags or policy configuration can disable one tool, tenant, or workflow while preserving safe read-only capability.

## Gate 4: design degradation before launch

Providers throttle, models change, retrieval becomes stale, and latency spikes. Define the product behaviour for each dependency failure:

| Failure | Safe behaviour |
|---|---|
| model timeout | deterministic template or manual workflow |
| retrieval unavailable | do not make account-specific claims |
| evaluator/policy unavailable | block consequential action |
| primary model unavailable | approved fallback only if it passes the same gates |
| cost budget exhausted | queue, reduce optional context, or disable non-critical generation |

A fallback model is not safe because it returns text. It must satisfy the relevant evaluation contract and tool semantics.

## Gate 5: make production observable

Capture enough evidence to reconstruct a decision without indiscriminately logging sensitive prompts. Useful fields include:

- use-case and policy version;
- model and configuration;
- retrieval document identifiers and freshness;
- requested and permitted tools;
- validation and approval outcomes;
- latency, token usage, and cost;
- user correction, rejection, or escalation;
- final business outcome where measurable.

Apply retention, redaction, tenant isolation, and access controls to telemetry. An audit trail that creates a new data leak is not a control.

Monitor both system and product drift. Latency and errors detect infrastructure regressions; correction rate, groundedness, task completion, and policy violations detect behavioural regressions. Sample production cases into a reviewed evaluation pipeline rather than treating the launch dataset as permanent truth.

## Gate 6: assign release accountability

The release record should name:

- product owner for the intended outcome;
- engineering owner for the system and rollback;
- risk owner for prohibited outcomes;
- dataset/evaluation owner;
- incident escalation path;
- expiry or review date for the approval.

High-risk exceptions need written reasoning, compensating controls, and an expiration. “Leadership accepted the risk” without a named risk, duration, or evidence is not governance.

## Common mistakes

**One benchmark score.** It hides severe failure classes and rarely matches the production workflow.

**Prompt-only safety.** The same model interpreting the request cannot be the only authority deciding whether that request is permitted.

**Unversioned retrieval.** A model release can appear unchanged while its knowledge source silently changes.

**Invisible fallback.** Switching provider or model without recording it breaks incident analysis and user expectations.

**No kill path.** If an unsafe tool cannot be disabled quickly and independently, the control plane is incomplete.

## The production standard

Do not ask whether the organisation trusts the model. Ask whether this version of the system, for this bounded use case, has passed named tests; whether its authority is constrained; whether failure becomes safe degradation; and whether the team can observe, stop, and investigate it.

That is a release decision a CTO can defend.

## References

- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
- [NIST Generative AI Profile](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf)
- [NIST AI RMF Core](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/)
- [Related: AI Evals Are Release Engineering](/posts/ai-evals-are-release-engineering)
- [Related: A Bigger Prompt Will Not Make Your Agent Safe](/posts/ai-agent-control-plane-mcp-security)


---

# Prompt Caching Is a Context-Layout Problem

> How stable prefixes, tenant boundaries, invalidation, observability, and privacy turn prompt caching into AI infrastructure design.

Canonical URL: https://www.ayushworks.xyz/posts/prompt-caching-is-context-layout
Author: Ayush Basak
Last modified: 2026-09-01
Topics: ai-infrastructure, llm, ai-agents, performance

Prompt caching is often presented as a switch that makes repeated LLM calls cheaper. In production it is a context-layout problem.

Caches benefit from shared, stable prefixes. Agent prompts often do the opposite: inject a timestamp, request ID, user state, retrieved documents, and tool output near the beginning. One changing token can move the reusable boundary and reduce the value of the cache.

The operating invariant is:

> Put stable, broadly reusable instructions before volatile, narrowly scoped state—without weakening isolation or correctness.

## Context has a memory hierarchy

An agent request mixes several classes of information:

```text
stable:   system policy, tool schemas, output contract
versioned: product rules, workflow definition, tenant policy
session:  compact conversation state, user preferences
volatile: current request, retrieval results, tool responses, timestamp
```

Arrange them in roughly that order. Stable prefixes maximize reuse; volatile suffixes preserve request specificity. This is similar to laying out data for locality: the model can accept the same semantic content in many orders, but the infrastructure cost differs.

Do not hide semantic changes to protect cache hits. A cached old policy is cheap and wrong.

## Version the prefix deliberately

Treat the reusable prefix as a deployable artifact:

```ts
const promptVersion = 'sales-agent/decision-policy/v7'

const input = [
  stableSystemPolicy,
  stableToolSchemas,
  `Policy-Version: ${promptVersion}`,
  tenantPolicy,
  sessionSummary,
  retrievedEvidence,
  userRequest,
]
```

The version should change when behaviorally relevant content changes. Record it with model, tool versions, retrieval snapshot, cache usage, latency, token counts, and final policy outcome. That makes a regression traceable.

## Cache keys are isolation boundaries

Provider features differ, but the architecture question is universal: which requests are allowed to share cached computation?

Never optimize cross-tenant reuse by placing private tenant data in a supposedly shared prefix. Separate public product instructions from tenant policy and user data. If an API exposes a prompt cache key, derive it from a bounded workload identity—not raw personal data—and understand the provider's retention and routing behavior.

| Context class | Reuse scope | Invalidation trigger |
|---|---|---|
| Public tool schema | application | tool contract release |
| Product policy | policy version | approved policy change |
| Tenant instructions | tenant | tenant configuration revision |
| Session summary | session | summary replacement |
| Retrieved evidence | request or short-lived | source/version change |

Extended caching can have data-retention implications. Privacy and compliance policy must be reviewed before latency wins are counted.

## Measure outcome, not hit rate

A high hit rate is not the goal. Track:

- cached versus uncached input tokens;
- time to first token and end-to-end latency;
- cost per completed business operation;
- correctness and policy-violation rate by prompt version;
- cache reuse by tenant and workload class;
- miss reasons: version churn, volatile prefixes, model routing, truncation.

Conversation truncation can also destroy prefix reuse by dropping messages from the beginning. Compact old state into a versioned summary before the window is full, and test whether the summary preserves decisions and unresolved obligations.

## Failure boundaries

Prompt caching must be an optimization, not durable memory. If the cache disappears, the request should remain correct—only slower or more expensive. If business correctness depends on remembered facts, store them in an authoritative system and reconstruct context.

Do not log complete prompts merely to debug caching. Capture structured fingerprints, versions, token ranges, and approved redacted excerpts.

## Roll out layout changes like code

Changing prompt order can affect both cache behavior and model behavior, even when the words are unchanged. Evaluate the new layout against a frozen task set, then canary it by prompt version. Compare task success, tool-call correctness, latency distribution, cached-token ratio, and cost per successful outcome.

Route fallback models through explicit compatibility tests. Tokenization, cache thresholds, tool-schema handling, and context limits vary. A layout optimized for one provider should degrade safely on another instead of silently dropping policy or evidence to fit a smaller window.

## Conclusion

Efficient AI systems do not merely choose a model; they arrange context. Stable prefixes, explicit versions, narrow sharing scopes, and observable invalidation convert prompt caching from a vendor feature into an engineering discipline.

The cache should accelerate a correct request. It must never become the place where policy, tenant state, or product truth secretly lives.

---

Further reading: [OpenAI prompt caching](https://platform.openai.com/docs/guides/prompt-caching), [API reference](https://platform.openai.com/docs/api-reference/chat/create), [model context specifications](https://platform.openai.com/docs/models), [LLM KV-cache capacity planning](/posts/llm-serving-is-kv-cache-capacity-planning), and [AI inference routing](/posts/ai-inference-routing-is-capacity-control).


---

# Queues Hide Overload Until Recovery Becomes Impossible

> How CTOs should reason about bounded queues, admission control, backpressure, load shedding, and recovery time before asynchronous architecture becomes an outage amplifier.

Canonical URL: https://www.ayushworks.xyz/posts/queues-backpressure-overload-control
Author: Ayush Basak
Last modified: 2026-08-24
Topics: distributed-systems, reliability, queues, platform-engineering

Queues make systems look healthy after capacity has already been lost.

Requests are accepted, producers receive acknowledgements, dashboards stay green, and the backlog quietly converts a capacity problem into a time problem. By the time queue depth triggers an alert, the oldest work may already be too stale to matter and recovery may require hours of perfect operation.

The dangerous architecture is not “using a queue.” It is accepting work without a declared limit on **how much delay, memory, and recovery debt the business is willing to own**.

## A queue stores promises

Every queued item is a promise to spend future capacity. If arrival rate is `λ` and sustainable completion rate is `μ`, then while `λ > μ`, backlog grows at approximately:

```text
backlog growth per second = λ - μ
```

If 1,500 jobs arrive per second and workers sustainably finish 1,200, ten minutes creates 180,000 pending jobs. After traffic returns to 1,000 per second, only 200 jobs per second of spare capacity remain. Catch-up takes 900 seconds—another fifteen minutes—assuming no retries, poison messages, or reduced efficiency under load.

This is why depth alone is a weak signal. The more useful metrics are:

- age of the oldest useful item;
- estimated drain time at current spare capacity;
- arrival and completion rate separately;
- percentage of work already beyond its business deadline;
- retry and redelivery rate;
- hot partition or tenant skew.

A backlog of one million thumbnail jobs may be harmless. A backlog of 200 password-reset emails may already violate the product promise.

## Bound the queue with a business deadline

“Unbounded” is simply a capacity decision delegated to memory, disk, or the broker administrator.

Define queue limits in three dimensions:

1. **Count or bytes** — the resource ceiling.
2. **Age** — how long work remains useful.
3. **Recovery time** — how long the organization accepts degraded service after load normalizes.

Attach an expiry or logical deadline to each item. A consumer should reject stale work before performing an expensive side effect:

```json
{
  "jobId": "quote-72a9",
  "createdAt": "2026-08-24T06:29:12Z",
  "executeBefore": "2026-08-24T06:29:42Z",
  "priority": "interactive"
}
```

Discarding work can be the most correct behavior. Recomputing a 20-minute-old recommendation or sending an expired one-time password does not restore correctness; it consumes recovery capacity.

## Admission control belongs before expensive work

Overload control is most effective where the system still has a choice. Once a request has acquired a database connection, loaded a model, or produced five downstream messages, rejecting it saves less.

Admission can be based on:

- concurrency rather than requests per second;
- estimated cost classes;
- tenant quotas;
- queue age and projected drain time;
- downstream saturation;
- priority and business criticality.

Concurrency is often the stronger primitive because it tracks work retained in the system. One service may handle 5,000 fast requests per second but collapse with 300 slow concurrent requests.

```text
admit if:
  inflight_cost + request_cost <= capacity_budget
  and tenant_inflight < tenant_limit
  and oldest_queue_age < recovery_threshold
```

The cost estimate does not need to be perfect. Separating a one-row lookup from a report export is already better than pretending all requests cost one token.

## Backpressure must cross ownership boundaries

Slowing one consumer does not create backpressure if upstream producers keep accepting demand. The pressure signal must travel far enough to change admission.

Possible signals include:

- synchronous `429` or `503` responses with retry guidance;
- reduced consumer credit or pull rate;
- bounded channel writes that block producers;
- explicit broker quota errors;
- a control-plane signal that lowers per-tenant concurrency;
- UI feedback that disables or defers nonessential actions.

Backpressure is not “the queue is getting longer.” It is a closed loop in which downstream scarcity causes upstream demand to reduce.

Be careful when automatically scaling consumers. More workers may improve throughput, or they may move the bottleneck into the database and multiply connection pressure. Scale against the constrained resource, not queue depth alone.

## Load shedding protects useful work

Google’s SRE guidance treats overload as normal and recommends rejecting some work so the system can continue serving the rest. That requires a policy before the incident.

A practical priority order might be:

```text
1. authentication and safety controls
2. committed financial or state transitions
3. interactive reads
4. customer notifications
5. indexing and analytics
6. speculative enrichment
```

The exact order is a product decision, not an infrastructure default. During overload, pause speculative enrichment before delaying a committed payment. Reserve capacity for control operations so operators can still cancel, drain, or reconfigure work.

Do not silently drop accepted work. Either reject before acceptance, persist a visible terminal outcome, or expose reconciliation. The user must be able to distinguish “not accepted” from “accepted but delayed.”

## Retry queues can become permanent storage

Retries need a bounded attempt count, a next-attempt time, and a terminal owner. Immediate redelivery of a poison event can consume an entire partition.

```text
attempt 1 -> short jittered delay
attempt 2 -> longer delay
attempt 3 -> quarantine with reason and owner
```

A dead-letter queue is not a solution if nobody drains it. Track oldest age, volume by failure class, and resolution time. Store enough context to reproduce the failure without leaking secrets into an operational dumping ground.

For ordered streams, moving one event aside may violate aggregate order. Quarantine the affected key or partition while allowing unrelated keys to progress; then make the business impact visible.

## Run the recovery calculation before launch

Capacity tests usually ask, “What peak can we process?” Add three harder tests:

1. What happens when a dependency runs at 40% capacity for fifteen minutes?
2. How long does the backlog take to drain after recovery?
3. Can critical traffic meet its objective while background work drains?

Inject slow consumers, skew one tenant, expire work, and restart workers during redelivery. Confirm that autoscaling does not exhaust database connections and that overload responses do not trigger aggressive client retries.

The target is not an infinite buffer. It is controlled degradation with a calculable path back to normal.

## CTO review questions

1. What promise does accepting a queued item make?
2. At what age does the item lose business value?
3. What resource ultimately limits consumer throughput?
4. How is downstream pressure communicated to the original producer?
5. Which work is shed first, and who approved that priority?
6. How long will recovery take after a realistic partial-capacity incident?
7. Who owns quarantined work?

Queues separate arrival from execution. They do not create capacity. A production design makes the hidden debt visible, bounds it, and refuses new promises before the old ones become impossible to keep.

## References

- [Google SRE: Handling Overload](https://sre.google/sre-book/handling-overload/)
- [AWS Builders’ Library: Timeouts, retries, and backoff with jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/)
- [Apache Kafka documentation](https://kafka.apache.org/documentation/)
- [Related: Your Timeout Budget Is an Architecture Decision](/posts/deadline-budgets-retry-amplification)
- [Related: The Transactional Outbox Is Not the Delivery Guarantee](/posts/transactional-outbox-delivery-guarantees)


---

# QUIC Migration Is Not Session Continuity

> Understand what QUIC connection migration preserves—and what application sessions, authorization, and idempotency must still guarantee.

Canonical URL: https://www.ayushworks.xyz/posts/quic-migration-is-not-session-continuity
Author: Ayush Basak
Last modified: 2026-09-06
Topics: networking, quic, distributed-systems, security

A phone moves from Wi-Fi to cellular. Its source IP address and port change, but a QUIC connection may continue because the protocol identifies connections independently of one network four-tuple.

That does not mean the application session is automatically safe, authorized, or exactly once. QUIC preserves transport state across a validated path change. Product continuity remains an application contract.

## What the transport preserves

QUIC uses connection IDs so packets can remain associated with a connection after an address change. Before sending substantial traffic on a new path, an endpoint validates peer reachability using `PATH_CHALLENGE` and `PATH_RESPONSE`. Anti-amplification limits constrain what a server can send to an unvalidated address.

Migration can preserve cryptographic and stream state, avoiding a complete new handshake. The new path still has different latency, loss, maximum packet size, and congestion characteristics. RFC 9002 treats congestion control as path-specific.

```text
old path: client CID <-> Wi-Fi address <-> server
                         network changes
new path: client CID <-> cellular address <-> server
                         path validation
```

The connection survived. A request inside it might still be incomplete or have an ambiguous outcome.

## Separate four identities

1. **Connection identity** associates packets with QUIC transport state.
2. **User session identity** associates requests with an authenticated principal.
3. **Operation identity** deduplicates a business effect.
4. **Device or client-instance identity** may support risk policy and token binding.

None should be silently substituted for another. A connection ID is not an authorization credential. An access token remains subject to expiry, audience, scope, and revocation. A stream identifier is not an idempotency key that survives application retries.

## Design the migration boundary

For read requests, retrying after a path transition is often safe if the caller’s deadline remains. For writes, carry an operation key and make the authoritative side effect converge:

```http
POST /transfers
Authorization: Bearer ...
Idempotency-Key: 5f0e...
```

The server persists the key with the accepted result in the same transactional boundary as the business mutation. A second request receives the stored result or a conflict if its payload differs.

Streaming applications need explicit resume semantics. Record a durable sequence or offset so a reconnect can request `after=8412`. Transport stream offsets help QUIC deliver bytes within the current connection; they are not a permanent product cursor.

## Failure policy

| Event | System response |
| --- | --- |
| address changes | validate new path before substantial sending |
| validation fails | keep viable old path or close connection |
| latency jumps | recompute deadline viability; shed stale work |
| write response is lost | resolve using operation identity |
| auth token expires | reauthorize at application layer |
| server instance disappears | resume from durable application offset |
| suspicious repeated migration | apply risk controls without trusting IP as identity |

## Observe migration as its own cohort

Measure path-validation success, migration count, old-to-new RTT delta, post-migration loss, request completion rate, resumed-stream gaps, and authentication failures. Split mobile and desktop cohorts. A globally healthy p95 can conceal a poor handover experience.

Avoid logging raw connection IDs as permanent user identifiers. They are protocol state and can become sensitive correlation data.

## Common mistakes

- Treating client IP as a durable session identity.
- Assuming migration prevents every request retry.
- Reusing transport identifiers as business deduplication keys.
- Ignoring the new path’s congestion and MTU behavior.
- Testing only a clean interface switch, not loss during an active write.

## Trade-offs

Connection migration improves continuity on changing networks but requires more endpoint state and privacy-aware connection-ID management. Disabling migration simplifies some deployments but turns ordinary network changes into reconnects. Aggressive resumption reduces visible interruption while increasing the need for strong application cursors and idempotency.

QUIC can carry continuity across networks. The application must still define what continuity means.

## Further reading

- [RFC 9000: QUIC Transport](https://www.rfc-editor.org/rfc/rfc9000.html)
- [RFC 9002: QUIC Loss Detection and Congestion Control](https://www.rfc-editor.org/rfc/rfc9002.html)
- [HTTP early data is a replay boundary](/posts/http-early-data-is-a-replay-boundary)


---

# A Raft Write Is Not Committed Just Because Most Replicas Have It

> A mechanical deep dive into Raft terms, elections, log replication, the current-term commit rule, linearizable reads, persistence ordering, and ambiguous client outcomes.

Canonical URL: https://www.ayushworks.xyz/posts/raft-commit-is-not-replica-count
Author: Ayush Basak
Last modified: 2026-08-30
Topics: distributed-systems, consensus, raft, system-design, reliability

Raft is often summarized as: send a write to the leader, copy it to a majority, and call it committed. That sentence is useful for orientation and dangerous for implementation.

A real system must answer harder questions. What if the leader dies after replication but before replying? Can a new leader overwrite the entry? Why can a leader safely commit an older entry only after committing something from its own term? Can the leader serve a read without writing to the log? Which bytes must reach stable storage before a message leaves the process?

This article follows one command through those boundaries.

The central invariant is:

> Once an entry is committed, every future leader contains that entry, and the state machine applies committed entries in log order.

Everything in Raft—terms, voting restrictions, log matching, quorum intersection, and the commit rule—exists to preserve that sentence.

## The replicated state machine underneath

Raft does not directly replicate a database. It replicates an ordered log of deterministic commands. Each node applies the same committed commands in the same order to its local state machine.

```text
client command
      │
      ▼
┌─────────────┐     AppendEntries      ┌─────────────┐
│   leader    │ ─────────────────────► │  follower   │
│ log + state │                        │ log + state │
└─────────────┘ ◄───────────────────── └─────────────┘
      │              acknowledgement
      ▼
advance commit index → apply command → return result
```

Each log entry contains an index, a term, and a command:

```text
index:   1    2    3    4    5
term:    1    1    2    2    4
command: A    B    C    D    E
```

The index establishes position. The term identifies the leadership epoch in which the entry was created. The pair `(index, term)` is the entry's identity for consistency checks.

## Terms are logical epochs, not wall-clock time

Every server stores a monotonically increasing `currentTerm`. A follower that stops hearing from a leader waits for a randomized election timeout, increments its term, becomes a candidate, votes for itself, and requests votes.

A term can have at most one elected leader because winning requires a majority and a server grants at most one vote per term. Two different majorities in the same cluster must intersect at at least one voter.

The candidate does not win merely by having the largest term. A voter also checks whether the candidate's log is at least as up to date as its own. Raft compares the term of the final log entry first, then its index. This voting restriction is what links leader election to committed history.

If any server receives a valid message with a higher term, it updates its term and steps down to follower. A leader is therefore not a permanent role. It is a claim scoped to one logical epoch.

## AppendEntries is both replication and consistency repair

The leader sends `AppendEntries` containing:

- its current term;
- the index and term immediately before the new entries;
- zero or more log entries;
- the leader's commit index.

The follower accepts new entries only if it has the preceding `(index, term)` pair. If that check fails, the leader moves backward to find the last shared prefix. Conflicting suffix entries are removed and replaced by the leader's suffix.

This produces the log-matching property:

> If two logs contain an entry with the same index and term, they contain identical entries through that index.

Heartbeats are simply `AppendEntries` calls with no new log entries. They assert leadership, carry the commit index, and continue consistency checks.

## Majority replication is necessary, not always sufficient

For an entry created in the leader's current term, the leader can advance `commitIndex` when that entry is stored on a majority. Because election majorities intersect replication majorities, some voter in every future election has that entry. The voting up-to-date rule prevents a candidate with an older log from replacing it.

The subtle case is an entry from an earlier term.

Imagine five nodes. In term 2, leader A replicates entry `x` to A and B, then crashes. In term 3, another leader can be elected and create a different suffix on C, D, and E. After another election, `x` may appear on a majority of the currently visible logs through a combination of old copies and partial repair, yet a future leader whose last-log term is newer can still be elected without `x`.

The Raft paper's Figure 8 demonstrates this counterintuitive history. Counting copies of an old-term entry does not prove it is committed.

Raft's rule is stricter:

```text
commit N when:
  a majority has matchIndex >= N
  AND log[N].term == currentTerm
```

Once the leader commits one entry from its own term, every preceding entry in its log becomes committed indirectly. The current-term entry anchors the entire prefix to Leader Completeness: any future leader must contain it, and because of log matching, must also contain everything before it.

Many implementations append a no-op entry when a leader is elected. Committing that no-op establishes the current-term anchor even when there is no immediate client write.

## Commitment and application are different positions

Three indexes matter:

```text
last log index ≥ commit index ≥ applied index
```

- `last log index`: the newest entry stored locally;
- `commit index`: the highest entry known to be committed;
- `applied index`: the highest entry already executed by the state machine.

A follower can possess an uncommitted entry. A node can know an entry is committed but not yet have applied it. Returning a read from state at `appliedIndex = 90` after proving `commitIndex = 95` is still stale until the application catches up through 95.

This distinction becomes critical for snapshots. A snapshot may compact the log only through state that has actually been applied. Its metadata must preserve the last included index and term so future consistency checks can bridge the compacted prefix.

## Why a leader cannot simply read local memory

A node may believe it is leader after it has been partitioned from the majority. Meanwhile, the majority can elect a new leader and commit newer writes. If the old leader serves a local read, the result violates linearizability even though its state machine is internally consistent.

A safe linearizable read needs two proofs:

1. this node is still the leader for the relevant term;
2. its state machine has applied every write committed before the read began.

One simple method is to place every read through the replicated log. That is safe but expensive. Raft's ReadIndex approach confirms leadership with a quorum without appending a new log entry, obtains a safe read index, then waits until the local applied index reaches it.

etcd exposes the trade-off explicitly. Linearizable reads coordinate through consensus. Its “serializable” read mode can be served by one member for lower latency and higher throughput, but may be stale relative to the quorum. In this API, “serializable” does not mean the same thing as SQL serializable isolation; names must be interpreted from their documented contract.

Lease-based reads avoid a quorum exchange by relying on a leader lease and bounded clock behavior. The etcd Raft implementation warns that unbounded clock drift can make this unsafe. A lease is a timing assumption added to the protocol, not free consensus.

## Persistence must happen before dependent messages

Consensus safety can be broken by the order of disk and network operations even when the state machine logic is correct.

Suppose a follower grants a vote in term 7, sends the response, crashes before persisting `currentTerm` and `votedFor`, then restarts and grants another vote in term 7. The one-vote-per-term property has been violated.

Likewise, acknowledging an appended entry before stable persistence allows the leader to count a copy that disappears after restart.

The etcd Raft library deliberately leaves storage and transport to its caller, but specifies the ordering: persist entries, hard state, and snapshot appropriately before sending messages that depend on them. Deterministic protocol code does not remove the integration's durability responsibility.

The production boundary is:

```text
state transition
  → durable write of required term/vote/log state
  → network acknowledgement
```

Batching and parallel disk writes may optimize this path, but they must preserve the happens-before relationship required by the protocol.

## A successful client response is later than commitment

Consider one write:

```text
client → leader: put(order-42, paid)
leader → quorum: replicate
quorum → leader: persisted
leader: commit and apply
leader → client: success
```

The leader may crash after commit but before the response reaches the client. The client sees a timeout, yet the command is durable and may already be visible.

Consensus decides the log. It does not make the client transport exactly once.

Clients need a stable command identifier. The state machine should store the outcome associated with that identifier and return the same result when a retry reaches the current leader:

```text
if command_id already applied:
    return recorded_result
else:
    apply command
    record command_id → result
```

Deduplication state is part of the replicated state machine and needs a retention policy aligned with the maximum retry horizon. Expire it too early and an old retry can create a second business effect.

## What a partition actually does

In a five-voter cluster, the majority side with three connected voters can elect a leader and continue. The minority side cannot commit new entries. An isolated old leader may accept client requests into its local log, but cannot commit them without a quorum; those entries may later be overwritten.

This is the availability boundary of quorum consensus. A five-node cluster does not “survive any two failures” if the remaining three cannot communicate with one another. Failure count is shorthand; quorum connectivity and latency are the real conditions.

Adding voters can reduce availability if it moves the quorum across unreliable or high-latency links. Learners and non-voting replicas may improve read locality, backup, or replacement workflows without changing the commit quorum, but their semantics must be explicit.

## The operational metrics that explain Raft

CPU and request rate will not tell you why consensus is slow. Track the protocol positions and the costs between them:

- leader changes and election duration;
- current term and leader identity;
- per-follower match-index lag;
- commit index minus applied index;
- proposal-to-commit and commit-to-apply latency;
- fsync latency and batching size;
- rejected proposals while leaderless;
- snapshot creation, transfer, and application time;
- ReadIndex latency and quorum failures;
- uncommitted log growth during quorum loss.

Alert on sustained gaps, not momentary differences. A brief apply lag during a batch may be normal; an increasing commit-to-apply gap means the state machine cannot keep up with consensus.

## Common implementation mistakes

| Mistake | Broken boundary |
|---|---|
| commit any entry found on a majority | old-term entries are not safely anchored |
| serve a leader's local state directly | leadership may be stale |
| return after ReadIndex but before apply catches up | the proof is newer than the state |
| send vote or append acknowledgement before persistence | acknowledged protocol state can disappear |
| treat proposal receipt as commitment | uncommitted suffixes may be overwritten |
| retry client commands without identity | ambiguous responses become duplicate effects |
| change membership in one unsafe step | old and new quorums may not intersect correctly |
| compact beyond applied state | snapshot claims commands the state machine has not executed |

## The CTO decision

Do not choose Raft because “we need high availability.” Choose a replicated system whose quorum geography, durability path, read semantics, recovery procedures, and operational ownership fit the business invariant.

For most teams, implementing consensus is the wrong product investment. Operating an established implementation still requires understanding its contract. You need to know whether a read is linearizable, when a write response can be ambiguous, what storage acknowledgements mean, how membership changes are performed, and what happens when the quorum spans regions.

The memorable rule is not “majority means committed.” It is this:

> A current-term quorum commits an ordered prefix that every future leader must preserve; safe reads and client effects require additional proofs around that prefix.

That is the mechanism underneath the abstraction.

## References

- [Ongaro and Ousterhout: In Search of an Understandable Consensus Algorithm](https://raft.github.io/raft.pdf)
- [etcd-io/raft: implementation contract and features](https://github.com/etcd-io/raft/blob/main/README.md)
- [etcd API guarantees](https://etcd.io/docs/v3.5/learning/api_guarantees/)
- [etcd-io/raft read-safety configuration](https://github.com/etcd-io/raft/blob/main/raft.go)
- [Papershelf: Raft and other systems papers](/papershelf)
- [Related: External consistency has a latency budget](/posts/external-consistency-has-a-latency-budget)
- [Related: Durable workflows do not remove idempotency](/posts/durable-workflows-do-not-remove-idempotency)


---

# RAG Access Control: What Happens After Document Permission Is Revoked?

> Design permission-aware RAG around query-time authorization, chunk provenance, answer caches, revocation windows, and adversarial tests.

Canonical URL: https://www.ayushworks.xyz/posts/rag-permission-revocation-is-a-serving-contract
Author: Ayush Basak
Last modified: 2026-09-11
Topics: ai-infrastructure, rag, security, system-design

A document is removed from an employee's access group at 10:00. At 10:01, an assistant answers their question using a chunk indexed yesterday. The retriever found relevant text. The product disclosed information the person should no longer receive.

This failure is easy to miss when a retrieval evaluation measures relevance but never changes permissions. The question is not only whether access control exists. It is when an access change becomes effective across documents, chunks, retrieval, generated answers, caches, and conversation history.

This guide proposes a serving contract for permission-sensitive RAG. The product must choose its acceptable revocation window. No vector similarity score can make that decision.

## Identify every authorization boundary

Start with the complete path:

~~~text
identity -> entitlement resolution -> filtered retrieval
         -> source authorization -> prompt construction
         -> generation -> answer delivery -> answer history
~~~

A permission change can race with any stage. Filtering the initial query does not automatically authorize a stored answer replayed an hour later. Deleting a search record does not necessarily remove a cached response or an attachment already delivered to the client.

The system should preserve provenance through these stages. For each chunk, retain a stable source identifier, tenant boundary, content version, and relevant authorization version. For each answer, record which source identities contributed to it.

Provenance supports enforcement and investigation. It is not permission by itself.

## Distinguish filters from an authorization service

Microsoft documents a [security-filter pattern](https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search) in which indexed principal identifiers are matched against the caller's authorized identities. Such a pattern depends on the application supplying the correct filter and identity context.

Build that context on the trusted server. Do not accept a tenant ID or group list from a browser and assume it is authoritative. Require every retrieval path to use the same authorization boundary, including fallback lexical search, related-document lookup, and citation expansion.

Microsoft also documents [query-time ACL and RBAC enforcement](https://learn.microsoft.com/en-us/azure/search/search-query-access-control-rbac-enforcement). Availability and prerequisites depend on the specific service feature and data source. Verify those details before treating a provider capability as a general guarantee for every index.

The design review should name the component that actually enforces access and the state it consults.

## Give revocation a measurable contract

Suppose entitlement caches can remain stale for E seconds and indexed ACL updates can lag by I seconds. The effective exposure window depends on the architecture: one path may require both to be current, another may recheck the authoritative source before using retrieved text.

Do not blindly add these values or take their maximum. Draw the sequence and identify which stale view can independently authorize disclosure. Then test the worst permitted propagation path.

For sensitive material, an authoritative check immediately before prompt construction can reduce reliance on stale indexed ACLs. It also adds latency and a dependency to the serving path. If the authority is unavailable, a fail-closed policy may reduce answer availability.

For less sensitive material, a documented bounded-staleness policy may be acceptable. State the window to the product and security owners. “Eventually consistent permissions” without a bound is not a useful operating contract.

## The generation race needs a decision

Imagine authorization succeeds, generation begins, and access is revoked before the answer is delivered. There are at least two coherent product policies.

One permits completion under the authorization decision made at request start. Another requires authorization to remain valid at delivery. The second policy may need revalidation, output buffering, or cancellation support.

Streaming complicates the second policy because already delivered tokens cannot be recalled. If the product needs a strict delivery boundary, buffer the response until the final check or define exactly what ongoing streaming is allowed to disclose.

Label this as a product decision with an implementation cost. Do not advertise immediate revocation if the serving architecture knowingly permits already authorized generations to continue.

## Design answer caches around provenance

A cache entry keyed only by the question can bypass all the retrieval protections. Another user's answer may contain restricted information, even when both people ask identical questions.

A proposed cache record might include:

~~~json
{
  "answerId": "answer-example-28",
  "tenantId": "tenant-4",
  "authorizationScope": "scope-version-19",
  "sources": [
    {
      "documentId": "document-8",
      "contentVersion": "v6",
      "authorizationVersion": "acl12"
    }
  ],
  "requiresRevalidation": true
}
~~~

This is an application design example, not a vendor schema.

Before serving a cached answer, establish that the current principal may receive every source-derived part of it under your policy. If any source has become unauthorized, regenerating from the permitted set is often simpler than trying to surgically remove leaked facts from an already generated answer.

Permission-scoped cache keys reduce accidental sharing, but stale scopes still need expiry or invalidation. The broader cache boundary is discussed in [shared cache keys and data isolation](/posts/shared-cache-keys-are-data-isolation-boundaries).

## Retrieval quality must be measured after authorization

Permissions change the candidate population. A query that retrieves excellent global neighbors can perform poorly for a person allowed to see only a small subset.

Azure documents different [vector filtering modes](https://learn.microsoft.com/en-us/azure/search/vector-search-filters), including effects on recall. Choose and evaluate the mode using the deployment's supported features and the actual authorized population.

Post-filtering unauthorized results is not permission to put those results into the prompt first. The enforcement boundary must precede model exposure. If the permitted result set lacks evidence, return an insufficient-evidence response rather than quietly broadening access.

## Failure policy and tests

| Scenario | Expected outcome |
| --- | --- |
| Caller forges a group or tenant identifier | Ignore untrusted claims; derive scope from verified identity |
| Source authorization cannot be established | Exclude the source or fail according to the documented policy |
| Cached answer references a revoked document | Revalidate and withhold or regenerate |
| Citation endpoint is called directly | Apply the same access policy as answer generation |
| Permission changes during streaming | Follow the explicitly chosen in-flight policy |

Create a test corpus with two tenants, shared documents, group changes, and a restricted source containing a distinctive phrase. Test retrieval, answers, citations, cache hits, exports, and resumed conversations. Observe whether the phrase appears anywhere outside its authorized context.

Measure revocation propagation time, denied-source retrieval attempts, cache revalidation failures, and answers with incomplete provenance. Keep document contents and sensitive identifiers out of broad operational logs.

## The architecture decision

Permission-aware RAG is a continuing authorization workflow. Indexing-time checks are one input, and the model is not the policy engine.

A review is ready when the team can state who enforces access, which copies may be stale, how long revocation can take, and what happens to an answer already in flight. Those answers are part of the product's security contract, alongside its relevance and latency targets.

## References

- [Azure AI Search security filters](https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search)
- [Query-time ACL and RBAC enforcement](https://learn.microsoft.com/en-us/azure/search/search-query-access-control-rbac-enforcement)
- [Vector search filtering modes](https://learn.microsoft.com/en-us/azure/search/vector-search-filters)
- [Related: retrieval evidence](/posts/rag-quality-starts-with-retrieval-evidence)


---

# RAG Quality Starts with Retrieval Evidence

> Debug retrieval-augmented generation by separating corpus, retrieval, context assembly, generation, and citation failures.

Canonical URL: https://www.ayushworks.xyz/posts/rag-quality-starts-with-retrieval-evidence
Author: Ayush Basak
Last modified: 2026-09-05
Topics: ai-systems, rag, evaluation, search

When a retrieval-augmented generation system gives a wrong answer, “the model hallucinated” is often an incomplete diagnosis. The correct evidence may be missing from the corpus, missed by retrieval, removed during context assembly, ignored by generation, or cited incorrectly.

RAG is a pipeline. Quality has to be attributed to a stage.

```text
source -> parse -> chunk -> index -> retrieve -> rerank
       -> assemble context -> generate -> verify citation
```

## Define a traceable evaluation unit

For each test case, store:

- user question and relevant cohort;
- expected answer or decision criteria;
- authoritative source document and passage;
- corpus/index version;
- retrieved document and chunk IDs with scores;
- final context after truncation;
- model, prompt, and generation parameters;
- answer and citation verdict.

Without this lineage, two runs that look comparable may have used different evidence.

## Evaluate stages independently

### Corpus coverage

Can the authoritative source be found in the indexed corpus? Parsing failures, access-control filters, stale synchronization, and poor chunk boundaries are corpus problems. A better language model cannot recover evidence it never receives.

### Retrieval

Measure whether relevant passages appear in top-k candidates. Use recall@k and rank-aware metrics, then slice them by query class, language, document age, and filter combination. A global average can hide a complete failure for one tenant or content type.

### Context assembly

Record which retrieved passages survive reranking, deduplication, token limits, and policy filtering. “Retrieved” does not mean “present in the prompt.”

### Generation

Given a context that definitely contains the answer, does the model answer faithfully, express uncertainty, and refuse unsupported conclusions? This isolates generation from retrieval.

### Citation

Check that each material claim is entailed by the cited passage and that links resolve to a source the user may access. Citation formatting alone is not groundedness.

## Use a failure taxonomy

```text
NO_SOURCE        authoritative material absent
STALE_SOURCE     indexed version too old
RETRIEVAL_MISS   relevant passage outside candidates
RERANK_DROP      candidate removed before context
CONTEXT_TRUNCATE relevant text exceeded budget
GENERATION_DRIFT answer contradicts supplied evidence
CITATION_MISS    claim lacks supporting passage
ACL_LEAK         retrieved content violates access policy
```

Assign one primary failure and optional contributing failures. Weekly counts then tell the team whether to invest in ingestion, search, prompts, models, or authorization.

## Production gates

Before release, require thresholds for high-risk cohorts, not only aggregate scores. Validate source freshness, tenant isolation, latency, cost, and abstention behavior. In production, sample traces for human review and maintain canary questions whose answers should change when source content changes.

## Common mistakes

- Using answer similarity as the only metric.
- Evaluating with synthetic questions that mirror chunk wording.
- Logging prompts without corpus and index versions.
- Letting the model cite a document ID it never received.
- Increasing context size before measuring retrieval misses.
- Mixing permission filtering after retrieval with a shared candidate cache.

## Trade-offs

Larger `k` can improve recall while increasing reranking cost and irrelevant context. Smaller chunks improve passage precision but lose surrounding meaning and expand index size. Human labels are expensive but expose failures that model-based graders can reproduce or amplify. More detailed traces improve diagnosis while increasing privacy and retention obligations.

The useful question is not “Is our RAG accurate?” It is “Which stage failed, for which cohort, under which version, and what evidence proves it?”

## Further reading

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)
- [Dense Passage Retrieval](https://aclanthology.org/2020.emnlp-main.550/)
- [AI evals are release engineering](/posts/ai-evals-are-release-engineering)


---

# Rate Limiting Is Admission Control

> How identity, fairness, concurrency, Retry-After, distributed counters, and overload policy turn rate limits into a product reliability boundary.

Canonical URL: https://www.ayushworks.xyz/posts/rate-limiting-is-admission-control
Author: Ayush Basak
Last modified: 2026-09-03
Topics: backend-engineering, reliability, api-design, distributed-systems, platform-engineering

Rate limiting is usually presented as a counter: allow 100 requests per minute. Production systems need a harder question—**which work should enter when demand exceeds the capacity to finish it usefully?**

The invariant is:

> Reject work before it consumes the scarce resource, using an identity and budget aligned with the business promise.

A gateway request counter cannot protect a database if one accepted export query consumes 10,000 times more work than one cached read. Requests, concurrency, bytes, tokens, rows scanned, and downstream calls are different currencies.

## Put the limit near the bottleneck

Use layered admission:

```text
edge: abusive IP and gross tenant rate
API: authenticated plan and operation cost
worker: queue depth and deadline
database: pool/concurrency budget
AI gateway: token and accelerator budget
```

The outer layer sheds cheaply; the inner layer protects the real resource. Do not let each layer retry the next one independently or rejection becomes amplification.

## Identity determines fairness

Per-IP limits punish offices behind NAT and are weak against distributed clients. Per-user limits can let one organization consume the fleet through many users. Per-tenant limits need endpoint or cost weighting. Most platforms need a hierarchy with a global safety ceiling plus tenant and principal budgets.

Reserve capacity for recovery and control operations. If bulk exports consume every worker, cancellation, status checks, and incident tools must still run.

## Rate and concurrency solve different failures

A token bucket controls arrivals over time and permits bounded bursts. A concurrency limit controls in-flight work. When dependency latency rises, the same arrival rate creates more concurrency:

```text
in_flight ~= arrival_rate × service_time
```

That is why a fixed requests-per-second limit can fail during a slowdown. Adaptive concurrency or queue-deadline admission protects finite sockets, threads, database connections, and memory.

## Rejection is an API contract

RFC 6585 defines `429 Too Many Requests`; `Retry-After` tells a cooperative client when to try again. Return a stable machine-readable error, the budget scope, and a retry time when known. Add jitter client-side and preserve the original operation deadline.

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 7
Content-Type: application/problem+json

{"type":"rate-limit","scope":"tenant","retryable":true}
```

Do not promise a precise reset if distributed counters provide only approximate agreement. A conservative hint is better than false certainty.

## Distribution has a consistency price

Local buckets are fast and available but allow a tenant to multiply its rate across replicas. A global service enforces fairness more closely but adds latency and another dependency. Hybrid designs allocate leases of budget to each region, accepting bounded overshoot in exchange for local decisions.

Write the maximum overshoot:

```text
overshoot <= regions × local lease size
```

If that bound can exhaust the protected resource, reduce leases or add a hard global circuit breaker.

Measure accepted and rejected work by scope, limiter latency, concurrency, queue age, completion before deadline, distributed overshoot, and customer success—not only 429 count. Load-test skewed tenants and slow dependencies.

## The CTO decision

Define capacity in the units that drive cost and failure. Allocate it by customer promise, protect control traffic, combine arrival and concurrency limits, and make rejection actionable.

A rate limiter is successful when admitted work completes predictably—not when a counter is perfectly accurate.

## References

- [RFC 6585: Additional HTTP Status Codes](https://www.rfc-editor.org/rfc/rfc6585)
- [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110)
- [Envoy: Global rate limiting](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_features/global_rate_limiting)
- [Related: Queues hide overload](/posts/queues-backpressure-overload-control)
- [Related: Your timeout budget is an architecture decision](/posts/deadline-budgets-retry-amplification)


---

# A Read Replica Needs a Freshness Contract

> Design PostgreSQL replica routing around observable staleness, read-your-writes guarantees, recovery conflicts, and explicit fallback policy.

Canonical URL: https://www.ayushworks.xyz/posts/read-replicas-need-freshness-contracts
Author: Ayush Basak
Last modified: 2026-09-05
Topics: postgresql, databases, replication, reliability, system-design

Adding a PostgreSQL read replica looks like a capacity decision: send reads elsewhere and give the primary room to breathe. In production it is a semantics decision. Asynchronous replication permits a committed write to be temporarily absent from the replica. A router that ignores that fact silently changes what the product means.

> A replica is safe only for reads whose tolerated staleness is explicit.

“Eventually consistent” is not a usable contract. The useful questions are: how stale, for which operation, and what happens when the bound is exceeded?

## Classify reads before routing them

Use product behavior, not SQL verbs, to classify a read.

| Read | Required behavior | Default route |
| --- | --- | --- |
| User reloads a just-saved profile | Read your writes | Primary or session fence |
| Permission check before mutation | Latest authoritative policy | Primary |
| Dashboard aggregate | Bounded staleness | Replica if inside budget |
| Historical export | Stable snapshot, latency tolerant | Dedicated replica |
| Search suggestions | Best effort | Replica with degradation |

The same `SELECT` can belong to different classes. A balance shown before authorizing a payment is not equivalent to the balance displayed in a weekly report.

## Measure replay position, not replica health

A green TCP check proves that the server accepts connections. It does not prove that replay is current. Track both time and byte distance where possible: last WAL receive position, last replay position, replay timestamp, and primary WAL position.

Time lag alone can mislead during a quiet period because there may be no new commit timestamp to compare. Byte lag alone does not translate directly to user-visible time. Together they explain more.

```sql
SELECT
  pg_is_in_recovery() AS is_replica,
  pg_last_wal_receive_lsn() AS received,
  pg_last_wal_replay_lsn() AS replayed,
  now() - pg_last_xact_replay_timestamp() AS replay_time_lag;
```

Treat this as an operational probe, not a universal routing query. Cache the result briefly in the router and alert on the underlying series.

## Preserve read your writes deliberately

There are three practical patterns.

1. **Primary stickiness:** after a successful write, route that user or session to the primary for a bounded interval. It is simple but conservative.
2. **Commit-position fence:** return a commit/WAL position and permit replica reads only after replay reaches it. This is precise but couples the application to database progress semantics.
3. **Version-aware response:** include a domain version in writes and refuse to present an older object version. This works across more storage topologies but requires product-level reconciliation.

Do not use a fixed 500 ms sleep. Replication delay is workload- and incident-dependent; sleeping converts uncertainty into latency without establishing correctness.

## Long reads compete with recovery

Hot standby queries can conflict with WAL replay. PostgreSQL may wait and then cancel a conflicting query according to `max_standby_streaming_delay`. The setting is a cumulative allowance for applying received WAL, not a per-query execution timeout. One query can consume most of the allowance available to later queries.

Increasing the delay protects analytics queries but permits replay lag to grow. Enabling `hot_standby_feedback` can reduce cleanup conflicts, while allowing dead row versions to remain longer on the primary and contribute to bloat. Neither knob is free.

Separate latency-sensitive replica traffic from unbounded analytical work. Apply statement timeouts and workload-specific connection pools. A reporting query should not consume the freshness budget of customer-facing reads.

## Define overload and failure behavior

Automatic fallback to the primary can turn a replica incident into a primary incident. Decide ahead of time:

- critical correctness reads may fall back within a protected primary capacity budget;
- stale-tolerant views may serve a marked stale result;
- optional panels may disappear;
- batch work may pause;
- no class may create unbounded primary failover traffic.

Protect the fallback path with admission control and a separate pool. The [rate-limiting guide](/posts/rate-limiting-is-admission-control) explains why rejecting work can preserve the critical path.

## Test the contract

Continuously exercise a write followed by reads through the real router. Inject replay delay, disconnect the replica, run a conflicting long query, and exhaust the fallback pool. Verify user-visible behavior and alarms—not just infrastructure metrics.

The architecture review should end with four numbers: staleness budget by read class, fallback capacity, query deadline, and maximum analytical concurrency. Without them, “reads go to replicas” is not an architecture. It is an unmeasured correctness change.

## References

- [PostgreSQL: High Availability, Load Balancing, and Replication](https://www.postgresql.org/docs/18/high-availability.html)
- [PostgreSQL: Replication configuration](https://www.postgresql.org/docs/18/runtime-config-replication.html)
- [PostgreSQL: Log-shipping standby servers](https://www.postgresql.org/docs/current/warm-standby.html)


---

# Refresh-Token Rotation Is a Replay Detector

> Implement OAuth refresh-token families with atomic rotation, reuse detection, bounded grace, and grant-level revocation.

Canonical URL: https://www.ayushworks.xyz/posts/refresh-token-rotation-is-a-replay-detector
Author: Ayush Basak
Last modified: 2026-09-06
Topics: security, oauth, identity, backend-engineering

Refresh-token rotation is often described as “issue a new token and invalidate the old one.” Its security value comes from the history retained between them.

If both a legitimate client and an attacker possess one refresh token, one will eventually present a token that has already been consumed. The authorization server detects replay and revokes the active token family. Rotation converts otherwise invisible credential theft into an observable conflict.

## Store a token family, not isolated strings

Persist only a cryptographic hash of each opaque token. Associate tokens with one authorization grant and a generation:

```sql
CREATE TABLE refresh_token (
  token_hash bytea PRIMARY KEY,
  family_id uuid NOT NULL,
  generation integer NOT NULL,
  client_id text NOT NULL,
  subject_id text NOT NULL,
  status text NOT NULL,
  expires_at timestamptz NOT NULL,
  consumed_at timestamptz,
  replaced_by_hash bytea,
  UNIQUE (family_id, generation)
);
```

The family lets the server revoke the currently active descendant when an ancestor is replayed. The client binding, scope, and resource-server audience must remain constrained to the original grant.

## Rotation must be atomic

Two browser tabs or mobile requests can refresh concurrently. A naive read-then-write flow may allow both to observe the same active token and mint two children.

Use one transaction and lock the presented token row:

```text
BEGIN
  load token FOR UPDATE
  verify client, expiry, grant, status
  if ACTIVE:
    mark CONSUMED
    insert generation + 1
    link replacement
    COMMIT and return new pair
  if CONSUMED:
    revoke family
    COMMIT and reject
```

The access token and replacement refresh token should be derived only after all policy checks. If the transaction fails, do not return credentials the database did not record.

## Handle lost responses deliberately

The authorization server can commit rotation while the response is lost. The legitimate client still holds the consumed parent and retries; strict reuse detection then revokes its own family.

There are three defensible approaches:

| Policy | Benefit | Cost |
| --- | --- | --- |
| strict one-time use | strongest immediate replay signal | lost response forces login |
| very short retry grace returning same child | tolerates network retry | requires safely recoverable response/token material |
| sender-constrained tokens | stolen value alone is insufficient | key management and client support |

RFC 9700 requires public clients receiving refresh tokens to use rotation or sender-constraining. DPoP and mutual TLS are standardized ways to bind tokens to key possession in applicable deployments.

A grace window must not mint multiple children. It should recognize the same client instance and return the already committed outcome, or deliberately accept the reauthentication cost.

## Security response

On confirmed reuse, revoke the active family, emit a security event, invalidate relevant server-side sessions where policy requires it, and require a new authorization grant. Do not attempt to guess whether the first or second presenter was the attacker.

Revoke on password change, logout, client compromise, or other high-confidence security events according to product policy. Apply inactivity and maximum lifetimes so continuous rotation does not create an immortal credential.

## Observability without leakage

Log family and generation identifiers, client ID, policy outcome, and coarse risk signals. Never log raw tokens. Monitor reuse rate, concurrent refresh conflicts, family revocations, refresh latency, and login recovery rate. Sudden reuse spikes may indicate a client concurrency bug rather than a new attacker campaign; the response must still preserve security.

## Common mistakes

- Storing refresh tokens in plaintext.
- Invalidating the parent without retaining family history.
- Performing rotation outside one transaction.
- Extending the absolute grant lifetime on every refresh.
- Putting bearer tokens in URLs or analytics logs.
- Treating a device label as cryptographic sender constraint.

## Trade-offs

Strict rotation detects replay but introduces a distributed race between client persistence and server commit. Retry grace improves availability while expanding state and replay analysis. Sender-constrained tokens reduce the value of theft, but require protected client keys.

Rotation is not token housekeeping. It is a replay-detection protocol whose failure behavior must be designed before the happy path.

## Further reading

- [RFC 9700: OAuth 2.0 Security Best Current Practice](https://www.rfc-editor.org/rfc/rfc9700.html)
- [RFC 9449: Demonstrating Proof of Possession](https://www.rfc-editor.org/rfc/rfc9449.html)
- [JWT verification is a key-distribution system](/posts/jwt-verification-is-key-distribution)


---

# Why Rust Fits Infrastructure Work

> What Rust changes when reliability and resource use both matter.

Canonical URL: https://www.ayushworks.xyz/posts/rust-for-infrastructure
Author: Ayush Basak
Last modified: 2026-06-30
Topics: rust, cloud-infrastructure, reliability

Infrastructure software often lives for a long time, handles untrusted inputs, and runs close to resource limits. Rust makes many of the dangerous states visible while the code is still being written.

## Constraints become design feedback

Ownership can feel strict at first, but it pushes lifecycle decisions into the architecture. Who owns a connection? How long may a buffer live? Can two tasks mutate this state? Clear answers usually produce clearer systems.

## Errors deserve types

Using `Result` throughout a boundary keeps failure paths explicit. A good error type separates invalid input, unavailable dependencies, timeouts, and internal faults so callers can respond appropriately.

## Performance without mystery

Predictable memory use and zero-cost abstractions make Rust a strong option for collectors, gateways, and data-processing services. The benefit is not benchmark theatre; it is confidence under sustained load.

Rust is not necessary everywhere. It earns its place where correctness, concurrency, and efficiency are all first-class requirements.


---

# S3 Multipart Uploads Need Garbage Collection

> Operate multipart uploads with idempotent completion, bounded retries, checksums, lifecycle cleanup, and explicit ownership.

Canonical URL: https://www.ayushworks.xyz/posts/s3-multipart-uploads-need-garbage-collection
Author: Ayush Basak
Last modified: 2026-09-06
Topics: cloud-infrastructure, object-storage, aws, reliability

A multipart upload is not an object under construction. It is a durable upload session containing separately stored parts until the client explicitly completes or aborts it.

If the client disappears after sending parts, those bytes remain chargeable. Ordinary object expiration rules do not remove incomplete multipart uploads. The storage system needs an explicit garbage-collection policy.

## Model the upload as a state machine

```text
initiated -> uploading -> completing -> complete
                 |             |
                 +-----------> aborting -> aborted
```

Persist the provider upload ID, destination key, expected content identity, initiator, creation time, and application status. The upload ID is authority to add parts to one session; it should not be reconstructed from an object key.

A minimal control table might look like:

```sql
CREATE TABLE object_upload (
  operation_id uuid PRIMARY KEY,
  bucket text NOT NULL,
  object_key text NOT NULL,
  provider_upload_id text UNIQUE NOT NULL,
  expected_size bigint,
  content_sha256 text,
  status text NOT NULL,
  created_at timestamptz NOT NULL,
  completed_at timestamptz
);
```

`operation_id` is the business idempotency key. Retrying initiation with the same operation should return the same active session or the already completed outcome, rather than create another set of orphaned parts.

## Completion is the commit boundary

Uploading the final part does not publish the object. The client completes the upload by supplying the part numbers and returned identifiers in order. A timeout during completion creates an ambiguous outcome: the server may have committed even though the response did not arrive.

Resolve ambiguity by reading authoritative object or upload state. Do not blindly initiate a replacement upload. Completion retries must use the stored session and exact part manifest.

Checksums prove transfer integrity; they do not prove that the object corresponds to the intended business operation. Bind bucket, key, expected content hash, tenant, and operation identity at the application boundary.

## Add a lifecycle safety net

AWS provides `AbortIncompleteMultipartUpload` as a lifecycle action. A rule can make sessions older than a chosen number of days eligible for abort and deletion of their stored parts.

```xml
<Rule>
  <ID>abort-incomplete-uploads</ID>
  <Status>Enabled</Status>
  <Filter><Prefix>uploads/</Prefix></Filter>
  <AbortIncompleteMultipartUpload>
    <DaysAfterInitiation>7</DaysAfterInitiation>
  </AbortIncompleteMultipartUpload>
</Rule>
```

The lifecycle rule is a backstop, not the normal control path. Applications should abort known cancellations promptly. The lifecycle window must exceed legitimate slow uploads, paused mobile transfers, and recovery time for downstream incidents.

## Operational contract

| Failure | Safe response |
| --- | --- |
| one part fails | retry that numbered part |
| client loses local state | recover manifest from durable control state |
| completion times out | inspect object/session before retrying |
| user cancels | stop active part work, then abort session |
| worker crashes | lease expires; another worker reconciles |
| session exceeds maximum age | lifecycle policy aborts it |
| checksum mismatches | do not publish business reference |

Monitor incomplete multipart bytes, upload count, oldest active session, completion latency, abort failures, and sessions per tenant. Alert on rate of growth, not only a large absolute total.

## Security boundaries

Presigned part URLs should be short-lived and scoped to one upload. Enforce allowed object prefixes and content limits before initiation. Do not let a client choose arbitrary buckets or overwrite another tenant’s keys. Treat object metadata as untrusted input when it later becomes an HTTP header or download filename.

## Trade-offs

Larger parts reduce request count but increase the cost of retrying one failed part. More parallelism shortens ideal upload time while increasing client memory, network bursts, and server request rate. Longer lifecycle windows protect slow uploads but retain garbage longer. Shorter windows save cost but can terminate valid resumable work.

Multipart upload is a distributed transaction without an automatic rollback. Give it durable state, a reconciliation loop, and garbage collection.

## Further reading

- [Amazon S3 multipart upload overview](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html)
- [Abort incomplete uploads with lifecycle rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html)
- [Strongly consistent object storage still needs concurrency control](/posts/object-storage-needs-concurrency-control)


---

# S3 Presigned Uploads: Ownership, Overwrites, and Safe Finalization

> Design direct-to-S3 uploads with server-owned object keys, constrained capabilities, quarantine, object verification, and idempotent finalization.

Canonical URL: https://www.ayushworks.xyz/posts/s3-presigned-uploads-need-a-finalization-protocol
Author: Ayush Basak
Last modified: 2026-09-11
Topics: aws, object-storage, security, backend-engineering

Direct uploads to S3 remove file bytes from the application server's request path. They do not remove the server's responsibility for ownership, validation, publication, or cleanup.

A common implementation creates a presigned URL, lets the browser upload, and accepts a completion request containing an object key. That completion request is an assertion from the client. It is not proof that the right object exists, belongs to the caller, satisfies the product's limits, or is safe to publish.

The missing component is a finalization protocol: a controlled transition from an authorized upload attempt to a usable product asset.

## Treat the URL as a capability

A presigned URL allows a request to exercise permissions represented by its signature within applicable validity constraints. Anyone possessing it may be able to use that capability. AWS describes expiration, credential lifetime, and policy restrictions in its [presigned URL guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html).

Do not put the full URL in analytics, support logs, or third-party error reports. Treat query-string redaction as part of the upload feature.

The relevant product question is: what exactly can the holder do, to which key, for how long? A URL for one object should not become an excuse to trust arbitrary object names later.

## Allocate identity on the server

When a user requests an upload, create a server-owned record that binds the authenticated principal, tenant, purpose, allowed constraints, and object location.

~~~json
{
  "uploadId": "upload-example-42",
  "tenantId": "tenant-7",
  "ownerId": "user-12",
  "objectKey": "staging/tenant-7/upload-example-42",
  "purpose": "project-attachment",
  "maximumBytes": 10485760,
  "state": "issued"
}
~~~

These values illustrate an application record. The maximum size is an example policy, not a universal recommendation.

Use an opaque identifier in the completion API. Resolve the object key from the server record rather than accepting a replacement key supplied by the browser. Recheck that the caller can finalize this upload for the intended tenant and parent resource.

A unique key reduces accidental collisions. It does not by itself prove ownership, prevent replay, or make the uploaded contents immutable.

## Separate upload success from publication

Model the lifecycle explicitly:

~~~text
issued -> uploaded -> verification pending -> accepted -> published
                    -> rejected
issued or uploaded -> expired -> cleanup
~~~

The browser can report that its request completed, but the server owns the authoritative state transitions. A completion request should be idempotent: repeating it returns the existing outcome or advances one valid transition, without creating a second attachment or firing the same business effect twice.

Before acceptance, inspect the expected object using trusted storage APIs. Verify the relevant size, metadata, encryption policy, and object identity. A client-provided MIME type does not establish the file's actual type. A checksum can establish byte integrity under its documented semantics, but cannot establish that the bytes are safe.

When malware scanning or format validation is required, keep the object in quarantine until that work succeeds. Downloads should resolve only accepted assets.

## Close the overwrite window

AWS documents that a presigned upload to an existing key can replace the object. It also documents [conditional writes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html), including conditions that can prevent overwriting an existing object.

This matters after verification. Suppose the server scans an object, marks it accepted, and then the original upload capability writes different bytes to the same key. If downloads resolve the latest object, the published asset may differ from the scanned asset.

One approach is write-once upload semantics using an appropriate conditional request included in the signed request. Another uses versioning and pins verification and subsequent reads to a specific version. A third publishes a verified copy to a location that the upload capability cannot modify.

Each approach requires end-to-end testing. Pinning a version in the database is ineffective if the download service silently reads the latest version. Copying is ineffective if the source can change and the copy is not bound to the verified identity.

## Design finalization as a compare-and-set transition

A proposed finalizer proceeds as follows:

~~~text
load upload by ID and authenticate its owner
verify the expected object or pinned version
run required checks and retain their object identity
atomically transition verification-pending -> accepted
create the attachment using a unique upload ID
make downloads resolve the accepted object identity
~~~

This is protocol pseudocode. Database uniqueness, transaction boundaries, and storage conditions must be implemented for your stack.

A competing finalizer should observe the winner's result, not create another attachment. A failed check should preserve the reason without making the object publicly retrievable. If verification is asynchronous, expose pending as a real user-visible state rather than reporting success prematurely.

For the storage race itself, see [object-storage concurrency control](/posts/object-storage-needs-concurrency-control).

## Failure policy

| Event | Required behavior |
| --- | --- |
| URL expires before upload begins | Issue a new authorized attempt under product policy |
| Client reports completion but object is absent | Keep the upload unaccepted |
| Object identity changes during verification | Reject or restart verification against the new identity |
| Finalization is repeated | Return the same asset or pending result |
| Scanner is unavailable | Keep quarantine closed; surface delayed processing |
| User loses access before finalization | Recheck authorization and deny publication when required |

Do not assume that hiding a completion button prevents misuse. The server must enforce the protocol for direct API calls and delayed retries.

## Cleanup is part of the cost model

Some users abandon uploads. Some uploads succeed but never finalize. Some objects are rejected and retained for investigation. These states need explicit retention and cleanup policies.

Track bytes in staging, oldest unfinalized object, rejection rate, verification latency, and orphan cleanup failures. Separate legitimate slow uploads from abandoned records using an operational window appropriate to the product.

Multipart uploads add another retention path: incomplete parts need their own lifecycle handling. See the [multipart cleanup guide](/posts/s3-multipart-uploads-need-garbage-collection).

## What to prove before launch

Attempt finalization from another tenant. Repeat it concurrently. Replace the object after scanning. Expire the capability. Interrupt verification after the storage check but before the database transition. Confirm that every accepted asset still points to exactly the object identity that passed verification.

The product contract should be simple for the user: upload, wait if necessary, then use the asset. The internal protocol earns that simplicity by treating publication as a verified state transition.

## References

- [S3 presigned URL permissions and expiration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html)
- [Uploading objects with presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)
- [S3 conditional writes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html)


---

# Saga Compensation Is Not Rollback

> Designing distributed workflows when completed side effects cannot be erased: semantic compensation, idempotency, orchestration, and manual resolution.

Canonical URL: https://www.ayushworks.xyz/posts/saga-compensation-is-not-rollback
Author: Ayush Basak
Last modified: 2026-09-03
Topics: distributed-systems, backend-engineering, sagas, workflow, system-design

A database rollback erases uncommitted local changes. A saga compensation creates new business facts after earlier facts have already become visible. Confusing the two produces workflows that look atomic in diagrams and fail messily in production.

The invariant is:

> Every committed saga step must have a defined forward recovery, semantic compensation, or explicit manual-resolution state.

Consider order creation, inventory reservation, payment authorization, and shipment. If shipment fails, “undo payment” might mean voiding an authorization, issuing a refund, or creating a receivable. Those outcomes are financially and temporally different.

## Model facts, not inverse API calls

```text
reserve inventory -> authorize payment -> create shipment
                                      X failure
release inventory <- void/refund payment <- mark order resolution_pending
```

Compensation can fail too. Inventory may already be sold, a refund provider may be unavailable, or a parcel may have left the warehouse. The workflow therefore needs durable state, retry policy, deadlines, and an owner for terminal exceptions.

A useful step record includes:

```sql
CREATE TABLE workflow_step (
  workflow_id uuid NOT NULL,
  step_name text NOT NULL,
  operation_id uuid NOT NULL,
  state text NOT NULL,
  attempt int NOT NULL,
  result jsonb,
  updated_at timestamptz NOT NULL,
  PRIMARY KEY (workflow_id, step_name)
);
```

Each participant must accept a stable operation ID. A timeout creates an ambiguous outcome; the orchestrator should query status or retry idempotently, not assume failure.

## Separate technical from business failure

A transient network error usually calls for bounded forward retry. “Card declined” is a business decision and should not be retried as infrastructure noise. An unknown payment outcome needs reconciliation. Treating all three as exceptions guarantees duplicate effects or stuck workflows.

| Outcome | Action |
| --- | --- |
| transient dependency failure | bounded retry with deadline |
| deterministic rejection | compensate completed steps |
| ambiguous write | look up by operation ID |
| compensation exhausted | manual-resolution queue |
| deadline exceeded | move to explicit expired state |

## Orchestration buys visibility

Choreography works for a small number of participants, but the workflow becomes implicit across event handlers as branches grow. An orchestrator centralizes state transitions, deadlines, and recovery evidence. It must itself be durable and horizontally safe; “central” should not mean one process with in-memory progress.

Store state transitions before dispatching the next effect, use an outbox to couple state and message creation, and make consumers idempotent. Exactly-once business execution is not supplied by the broker.

## Design the human path

Some effects cannot be compensated automatically. Build an operator view containing the workflow timeline, operation IDs, provider references, current invariant violation, safe next actions, and audit trail. Manual intervention should issue the same idempotent commands as automation rather than editing databases.

Measure age by state, retry and compensation rates, ambiguous outcomes, manual queue size, time to resolution, and money or inventory trapped in intermediate states.

## The CTO decision

Choose sagas only when the business accepts temporary inconsistency and can define recovery for every committed step. Name irreversible boundaries, cap automation, and fund the operator path.

Compensation does not restore history. It moves the business into a new, acceptable state with evidence of how it got there.

## References

- [AWS: Saga patterns](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-patterns.html)
- [AWS: Saga pattern for data persistence](https://docs.aws.amazon.com/prescriptive-guidance/latest/modernization-data-persistence/saga-pattern.html)
- [Related: Durable workflows do not remove idempotency](/posts/durable-workflows-do-not-remove-idempotency)
- [Related: The transactional outbox is not the delivery guarantee](/posts/transactional-outbox-delivery-guarantees)


---

# Kiro Flock Explained: Multi-Agent Coordination with S3

> How kiro-flock coordinates Kiro CLI agents through shared S3 logs, and how to choose between peer coordination and a supervisor.

Canonical URL: https://www.ayushworks.xyz/posts/scaling-patterns-for-self-organizing-multi-agent-clusters-with-kiro
Author: Ayush Basak
Last modified: 2026-09-11
Topics: aws, ai-agents, distributed-systems

**Kiro Flock (`kiro-flock`) is an open-source reference implementation for coordinating Kiro CLI agents through shared Amazon S3 state.** The AWS example runs agents on EC2; agents exchange progress through logs rather than sending every assignment through a supervisor.

This article examines that particular pattern. Kiro supports other coordination approaches too. Use the [AWS implementation walkthrough](https://aws.amazon.com/blogs/architecture/scaling-patterns-for-self-organizing-multi-agent-clusters-with-kiro/) as the reference, and evaluate the verification and coordination requirements of your own workload.

## Architectural Decoupling with Kiro and S3

In traditional agent systems, a central control plane tracks active agents, assigns work items, and manages global state transitions. By contrast, Kiro coordinates AI agents by storing shared state directly in Amazon S3 rather than depending on a central orchestrator service ([AWS Architecture Blog](https://aws.amazon.com/blogs/architecture/scaling-patterns-for-self-organizing-multi-agent-clusters-with-kiro/)).

To deploy and observe these decentralized workloads, engineers can study the open-source `kiro-flock` reference implementation on Amazon EC2 ([AWS Architecture Blog](https://aws.amazon.com/blogs/architecture/scaling-patterns-for-self-organizing-multi-agent-clusters-with-kiro/)). Each agent reads a direction file and a bounded set of peer logs, writes artifacts, and appends its latest action and intent to its own log. There is no master assigning tasks.

```
+-------------------------------------------------------+
|                 Amazon S3 (Shared State)              |
+-------------------------------------------------------+
       ^                       ^                       ^
       | state sync            | state sync            | state sync
       v                       v                       v
+---------------+       +---------------+       +---------------+
| EC2 Node 1    |       | EC2 Node 2    |       | EC2 Node 3    |
| (kiro-flock)  |       | (kiro-flock)  |       | (kiro-flock)  |
+---------------+       +---------------+       +---------------+
```

This stateless control layer aligns with core principles of [building reliable microservices](/posts/building-reliable-microservices), where decoupling coordinator dependencies improves overall fault tolerance.

## Implementation Pattern: S3 State Synchronization

The coordination record is intentionally small and append-only. A log entry communicates what an agent did, what it produced, and what it intends to inspect next:

Because each agent owns its log, peers do not contend on a shared mutable task row. Coordination emerges from reading traces rather than acquiring a global lock.

```json
{
  "ts": "2026-07-21T14:14:31Z",
  "iteration": 1,
  "action": "wrote discussion-scaling-laws.md",
  "result": "Compared communication cost with useful diversity",
  "next_intent": "Read neighbour updates and look for uncovered angles"
}
```

The bounded peer set is the scaling control. In the amorphous ring topology, an agent reads a fixed number of neighbours instead of every agent. Per-agent context therefore stays bounded as the cluster grows, while information needs multiple iterations to propagate. Mesh visibility converges faster for small groups but encourages early consensus; recency-based swarm visibility follows the active part of the work.

## Operational Shape

The reference stack uses EC2 for headless Kiro CLI agents and S3 for direction, logs, and artifacts. API Gateway, Lambda, and Cognito provide the optional dashboard control plane, while CloudWatch receives metrics. Treat the sample as a system to study and adapt, not a production-ready agent platform. Scope each agent's IAM permissions, constrain network egress, and set budget alerts before experimentation. The same discipline applies to [reliable microservice boundaries](/posts/building-reliable-microservices) and [declarative platform changes](/posts/kubernetes-kyaml-production-workflow).

## Common Misconceptions

- **Misconception: A central master orchestrator is mandatory for multi-agent clusters.**
  Architectures built with Kiro prove that shared object state in Amazon S3 allows fully self-organizing clusters to function autonomously without a central control node.
- **Misconception: every agent should read every other agent.**
  Full visibility speeds alignment but can collapse diversity. Bounded neighbourhoods make signals travel more slowly and leave room for competing approaches.

## Tradeoffs and Constraints

1. **State Propagation Latency**: S3 object writes exhibit higher latency compared to in-memory key-value stores like Redis or Raft clusters.
2. **Polling and Context Cost**: Short loop intervals increase S3 requests and model usage; large peer sets expand every agent's context.
3. **No Intermediate Arbiter**: Bad signals can spread before correction, so this pattern is unsuitable when every step needs central verification.

## When NOT to Use This Pattern

Avoid relying on S3-backed self-organizing clusters if your agent system demands sub-millisecond inter-agent IPC or real-time synchronous state barriers. For microsecond-level synchronization, in-memory distributed caches or direct messaging queues are significantly better suited.

## Conclusion

By moving coordination into shared append-only state, `kiro-flock` removes the supervisor from the critical path. The gain is parallel exploration and tolerance of individual agent failure; the cost is slower convergence and weaker intermediate verification. Use the pattern for decomposable work where diversity matters, not as a universal replacement for supervised agents.

## References

- [AWS Architecture Blog: Scaling patterns for self-organizing multi-agent clusters with Kiro](https://aws.amazon.com/blogs/architecture/scaling-patterns-for-self-organizing-multi-agent-clusters-with-kiro/)


---

# Serializable Isolation Is a Retry Protocol, Not a Checkbox

> How PostgreSQL Serializable Snapshot Isolation prevents write skew, why serialization failures are correct behavior, and how to build bounded transaction retries.

Canonical URL: https://www.ayushworks.xyz/posts/serializable-isolation-is-a-retry-protocol
Author: Ayush Basak
Last modified: 2026-08-30
Topics: postgresql, database-engineering, concurrency-control, reliability

Most teams reach for a row lock when concurrent requests break an invariant. That works when the rows that must be locked are already known. It fails for predicates such as “at least one doctor must remain on call” because two transactions can update different rows while jointly violating the rule.

The production invariant is:

> Every committed transaction must be explainable as part of one serial execution, and any rejected transaction must be safe to run again from the beginning.

PostgreSQL's `SERIALIZABLE` level provides that first property through Serializable Snapshot Isolation. The application must provide the second.

## The anomaly is between decisions, not rows

Assume two doctors are on call. Each transaction checks that another doctor remains and then removes its own doctor:

```sql
BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT count(*) FROM doctor_shift WHERE on_call = true;
UPDATE doctor_shift SET on_call = false WHERE doctor_id = $1;
COMMIT;
```

The transactions update different rows, so ordinary row-level conflict detection does not connect them. Both can observe a count of two and commit, leaving zero doctors.

Under `SERIALIZABLE`, PostgreSQL tracks read/write dependencies, including predicate reads. If the concurrent history cannot be serialized safely, one transaction aborts with SQLSTATE `40001`. That abort is not database instability. It is the database refusing to certify an impossible history.

## Retry the decision, not the final statement

A serialization failure invalidates every decision made from the old snapshot. Retrying only `COMMIT` or the last `UPDATE` preserves stale reasoning.

```ts
for (let attempt = 1; attempt <= 4; attempt++) {
  const client = await pool.connect()
  try {
    await client.query('BEGIN ISOLATION LEVEL SERIALIZABLE')
    const result = await changeOnCallState(client, command)
    await client.query('COMMIT')
    return result
  }
  catch (error) {
    await client.query('ROLLBACK').catch(() => {})
    if (sqlState(error) !== '40001' || attempt === 4)
      throw error
    await sleep(randomBetween(5, 25) * attempt)
  }
  finally {
    client.release()
  }
}
```

The callback must contain the complete read-decide-write unit. It should not send email, call a payment provider, or publish a message directly: a retry could repeat those effects. Commit an outbox record in the same transaction and deliver it separately with stable event identity.

## Retrying forever hides overload

Serializable transactions can abort more often when they are long, touch broad predicates, or contend on hot data. A retry loop converts some aborts into latency and additional load. Bound it by both attempts and the request deadline.

Measure at least:

- serialization failure rate by transaction type;
- attempts required before success;
- time spent waiting before retry;
- exhausted retries and caller deadlines;
- the predicates or aggregates responsible for contention.

When failures become routine, the answer is rarely a larger retry limit. Shorten the transaction, reduce the rows it examines, partition the hot invariant, or deliberately serialize access with a lock or queue.

## Know the failure classes

PostgreSQL recommends retrying serialization failures (`40001`) and, in some applications, deadlocks (`40P01`). A unique violation is not automatically retryable: it may be the correct business result. A connection failure during commit is ambiguous and requires idempotency or reconciliation, not blind repetition.

| Outcome | Policy |
|---|---|
| `40001` serialization failure | retry the complete transaction with jitter |
| `40P01` deadlock | retry only if the operation is safe and bounded |
| unique violation | usually return a domain conflict |
| statement timeout | retry only if the deadline and operation semantics permit |
| connection lost during commit | reconcile using command identity |

## The CTO decision

Use serializable isolation for invariants spanning a predicate or multiple rows when the cost of an invalid state exceeds the cost of occasional aborts. Do not apply it as a global badge of correctness. Establish a transaction boundary, idempotency model, retry budget, and contention SLO together.

The database can reject unsafe histories. Only the application can decide whether repeating the business command is safe.

## References

- [PostgreSQL: Transaction Isolation](https://www.postgresql.org/docs/current/transaction-iso.html)
- [PostgreSQL: Serialization Failure Handling](https://www.postgresql.org/docs/current/mvcc-serialization-failure-handling.html)
- [PostgreSQL: Explicit Locking](https://www.postgresql.org/docs/current/explicit-locking.html)
- [Snippet: Retry a PostgreSQL serializable transaction](/snippets/serializable-transaction)


---

# Design Service Identity Before Buying a Service Mesh

> A production design for workload identity, SPIFFE trust domains, short-lived credentials, authorization, rotation, and migration away from shared service secrets.

Canonical URL: https://www.ayushworks.xyz/posts/service-identity-before-service-mesh
Author: Ayush Basak
Last modified: 2026-08-25
Topics: security, distributed-systems, cloud-infrastructure, service-identity

Mutual TLS answers one question: did both endpoints present credentials signed by a trusted authority? It does not decide what a workload is, why it received that identity, or which operation it may perform.

Those decisions form the service-identity architecture. A mesh can transport and rotate credentials, but it cannot invent the trust model for you.

## Replace location with identity

IP allowlists assume that network location is a useful proxy for authority. In dynamic infrastructure, addresses are reused, workloads move, and several applications share nodes. The policy “10.0.4.0/24 may write invoices” grants power to a subnet, not to the invoice writer.

A workload identity should be stable across instances but specific to one operational purpose:

```text
spiffe://prod.example.com/payments/authorizer
spiffe://prod.example.com/orders/checkout
spiffe://prod.example.com/data/report-reader
```

SPIFFE defines this URI shape and calls the root namespace a trust domain. Its SVIDs are verifiable identity documents represented as X.509 certificates or JWTs. The Workload API allows a local process to obtain identities and rotating trust bundles without embedding a long-lived secret in an image.

The important separation is:

```text
attestation → identity issuance → authentication → authorization
```

Collapsing these steps creates systems that are encrypted but over-authorized.

## Attest what the platform can prove

An identity issuer needs evidence connecting a running process to the declared workload. Depending on the platform, selectors may include Kubernetes namespace and service account, cloud instance identity, Unix process attributes, or a signed deployment artifact.

Prefer properties controlled by the delivery platform over labels the application can choose itself. A process should not become the payment service because it sends `X-Service-Name: payments`.

Treat registration as production code:

```yaml
identity: spiffe://prod.example.com/orders/checkout
selectors:
  kubernetes_namespace: commerce
  kubernetes_service_account: checkout
owner: team-commerce
expires_with: service-retirement
```

Review broad selectors like broad IAM roles: they increase the population able to obtain the credential.

## Separate trust domains deliberately

A trust domain is a cryptographic and administrative boundary. Do not put development and production under the same roots because the names look tidy. SPIFFE guidance recommends distinct domains where environments have different security practices or physical locations.

Federation should be explicit. If a vendor or acquired system needs access, exchange trust bundles and authorize a narrow foreign identity. Do not copy the production root or issue local identities on another organization’s behalf.

Trust-domain decisions should answer:

- who can issue an identity;
- how issuer compromise is contained;
- which environments may federate;
- how roots and bundles rotate;
- how federation is revoked.

## Authenticate identity; authorize intent

After mTLS verifies a caller, the receiving service still needs an authorization decision:

```text
caller = spiffe://prod.example.com/orders/checkout
action = payments.authorize
resource = merchant/acme
context = amount, region, risk state
```

Keep policy near the resource that owns the invariant. A central policy engine may distribute rules or evaluate decisions, but the payment service must fail closed when it cannot prove authority for a mutation.

Avoid role strings that grow into opaque superpowers. Model capabilities around operations and resources:

```text
allow if {
  input.caller == "spiffe://prod.example.com/orders/checkout"
  input.action == "payments.authorize"
  input.resource.merchant_id == input.claims.merchant_id
}
```

Log the identity, action, resource, policy version, and decision. A TLS handshake log alone cannot explain why a destructive request was allowed.

## Prefer short-lived, automatically rotated credentials

Static API keys create a difficult contradiction: rotate often enough to limit exposure, but not so often that deployments break. Platform-issued short-lived credentials remove most application coordination from rotation.

Rotation still needs testing:

- clients reload certificates without restart;
- servers accept overlapping trust bundles during root rotation;
- clock skew is bounded;
- cached authorization does not outlive identity validity;
- revocation has an emergency path shorter than normal expiry.

For service-to-service communication, SPIFFE recommends X.509-SVIDs where practical because bearer JWTs can be replayed if stolen. When using OAuth, follow the current OAuth Security BCP: bind flows correctly, validate redirect URIs, avoid deprecated grants, and make token audience explicit.

## Migrate without a flag day

Move from shared secrets in stages:

1. Inventory service credentials and observed callers.
2. Issue workload identities while keeping the old path.
3. Log identity-based decisions in shadow mode.
4. Compare expected and observed callers.
5. Enforce identity on one low-risk operation.
6. Expand enforcement and remove shared credentials.
7. Delete secret-distribution code and rotate the old secret.

Dual mode must have an expiry date. Otherwise the fallback becomes the permanent bypass.

## Operate the identity plane

Measure issuance failures, certificate age, time to expiry, rotation latency, authorization denies by policy version, unknown caller identities, and trust-bundle convergence. Alert before expiry causes a fleet-wide outage.

Plan for control-plane impairment. Existing workloads should continue within a bounded credential lifetime; new workloads may fail to start. Decide whether that failure mode is safer than issuing unverifiable identity. Security and availability tradeoffs must be explicit.

## CTO review

1. What platform evidence binds a process to its identity?
2. Are production, development, and partners separate trust domains?
3. Does authorization name operations and resources, not only services?
4. Can every credential rotate without an application deploy?
5. How quickly can one workload or issuer be revoked?
6. What remains available when the identity control plane is degraded?
7. Which shared secrets disappear after migration?

Buy a mesh for capabilities you need. Design identity first, because the mesh will faithfully automate whatever trust model—good or bad—you give it.

## References

- [SPIFFE Standard](https://spiffe.io/docs/latest/spiffe-specs/)
- [SPIFFE Concepts: identities, trust domains, and SVIDs](https://spiffe.io/docs/latest/spiffe/concepts/)
- [RFC 9700: OAuth 2.0 Security Best Current Practice](https://datatracker.ietf.org/doc/html/rfc9700)
- [Related: Multi-Tenant SaaS Isolation](/posts/multi-tenant-saas-isolation)
- [Related: AI Agent Control Plane Security](/posts/ai-agent-control-plane-mcp-security)


---

# A Shared Cache Key Is a Data-Isolation Boundary

> Audit authenticated HTTP caching through representation keys, private/no-store semantics, revocation windows, and two-user cache-hit tests.

Canonical URL: https://www.ayushworks.xyz/posts/shared-cache-keys-are-data-isolation-boundaries
Author: Ayush Basak
Last modified: 2026-09-04
Topics: security, caching, http, multi-tenancy, system-design

The origin checks authorization correctly. The first user receives the right invoice. The second user receives the first user's cached response without the origin running at all.

This is not necessarily a broken authentication library. It is a mismatch between the application's representation boundaries and the cache's reuse boundaries.

> Every request allowed to reuse one cached representation must be entitled to receive that exact representation.

A cache hit is a data-access decision, even when the component making it has no concept of tenants or permissions.

## Enumerate what changes the response

Start with one route and list all inputs that affect its bytes: tenant, principal, role, locale, query parameters, entitlement, feature configuration, and selected resource version.

Then inspect the actual cache key at every layer. A correct application cache does not compensate for an unsafe reverse proxy. A correct CDN policy does not compensate for a browser or service worker that reuses account-specific content after a user switch.

```text
browser -> service worker -> CDN -> reverse proxy -> application cache -> origin
```

Ask which layers can answer without re-running authorization and what evidence makes that safe.

## Understand the directives you deploy

RFC 9111 distinguishes shared and private caches. A private response is not for shared-cache storage. No-store prohibits storing the response under the protocol's rules. No-cache permits storage but requires validation before reuse.

For a sensitive account endpoint, a conservative response policy is:

```http
Cache-Control: private, no-store
Content-Type: application/json
```

That is a deployment policy choice, not encryption or a substitute for access control. It does not erase copies already saved elsewhere, and custom application caches need their own equivalent enforcement.

The RFC also specifies restrictions for shared reuse of responses to requests containing Authorization, with explicit exceptions. Do not depend on a header being present if authentication actually uses cookies or a proxy has transformed the request. See [RFC 9111](https://www.rfc-editor.org/rfc/rfc9111.html).

## Vary is not an authorization system

Vary tells a conforming cache which request headers distinguish representations. It does not prove the header values are trustworthy, establish a tenant relationship, or guarantee a particular CDN supports the desired key policy.

Putting a raw bearer token into an application cache key can create unbounded cardinality and credential-handling risks. Prefer bypassing shared response caching for personalized data unless a reviewed design can establish a bounded, verified identity partition.

Never vary on an attacker-controlled tenant header while the application authenticates using a different source. Identity normalization must happen before the key is selected.

## Split public data from private overlays

A product page may contain public descriptions and customer-specific discounts. Caching the whole response forces the public representation to inherit the strictest private constraints.

An alternative is to cache a public product representation and fetch the private price or account overlay separately. This adds a request and requires careful loading behavior, but it creates an auditable reuse boundary.

Another option is an application cache behind authorization, keyed by verified tenant and policy version. That can reduce expensive computation while keeping authorization in every request path. It is not equivalent to serving a shared CDN response before checking identity.

## Freshness is also a permission question

Suppose a user loses access after a response enters a private or identity-partitioned cache. If future hits bypass authorization, the TTL becomes an access-revocation delay.

Choose the permitted window deliberately. Sensitive administrative data may require reauthorization on every request. Lower-risk personalized content may accept a short window. If permissions change, invalidate every derived representation that relied on them—or version the permission context and prove old keys cannot be reached.

Do not add stale serving to an authenticated route solely to improve availability. Stale content can mean stale authorization.

## Test cache hits with different identities

A security test must warm the cache; testing only cold responses misses the failure.

1. User A requests a tenant-specific resource and populates the cache.
2. User B requests the same URL with different credentials.
3. An anonymous client requests it.
4. A's permissions are revoked, then A requests it again.
5. Repeat through every public hostname and relevant proxy path.

Assert bodies, headers, status codes, and cache-hit evidence—not just whether a login redirect occurred. Include error responses: a cached denial can lock out an authorized user, while a cached success can disclose data.

Also test query normalization, trailing slashes, alternate encodings, and resource identifiers that look like static filenames. Security depends on every layer agreeing on route identity.

## The CTO decision

Treat cache policy as part of data classification and authorization review. Require an owner for each shared representation, documented key inputs, a revocation window, and cross-identity hit tests before rollout.

The performance win is valuable only when reuse is safe. A high cache-hit ratio cannot compensate for returning the right data to the wrong person.

## Further reading

- [RFC 9110: HTTP semantics](https://www.rfc-editor.org/rfc/rfc9110.html)
- [Cache invalidation is a consistency protocol](/posts/cache-invalidation-is-consistency-protocol)
- [Tenant isolation is a data model](/posts/multi-tenant-saas-isolation)


---

# From Repository to Real Product

> A practical checklist for turning a full-stack build into a product people can rely on.

Canonical URL: https://www.ayushworks.xyz/posts/shipping-full-stack-products
Author: Ayush Basak
Last modified: 2026-05-24
Topics: product-engineering, backend-engineering, system-design

A working demo proves an idea. A product must also survive confusing input, slow networks, repeated requests, changing requirements, and users returning months later.

## Start with the critical path

Identify the smallest end-to-end journey that creates value. Build and instrument that path before expanding the feature list. It reveals bad assumptions earlier than isolated frontend or backend work.

## Model the domain explicitly

Names such as `Deal`, `Wishlist`, `Subscription`, and `PriceAlert` should mean the same thing in the UI, API, and database. Clear domain boundaries reduce translation bugs and make later features easier to place.

## Make operations boring

Validate configuration at startup, apply database migrations predictably, expose health checks, and log useful context without leaking secrets. Add a rollback plan before the risky release, not after it.

Shipping is a loop: release a narrow slice, observe real behavior, remove friction, and repeat. Reliability is part of the user experience.


---

# Valid JSON Is Not Permission to Execute

> Separate AI output shape from factual evidence, authorization, freshness, and transactional execution using a concrete proposal-to-command boundary.

Canonical URL: https://www.ayushworks.xyz/posts/structured-ai-output-is-not-an-authorized-command
Author: Ayush Basak
Last modified: 2026-09-04
Topics: ai-infrastructure, ai-agents, security, backend-engineering

An AI system returns a perfectly shaped refund proposal. Every field has the right type. The customer identifier exists. The amount is positive. The proposal can still be unauthorized, based on an outdated balance, or justified by evidence that says the opposite.

Structured output reduces one category of integration failure. It does not turn model output into trusted business state.

> Model output is a proposal until deterministic systems establish evidence, authority, freshness, and execution safety.

That boundary should exist even when the model has performed well on every demonstration.

## Validate shape without overstating the guarantee

A standard JSON Schema can define a bounded proposal:

```json
{
  "type": "object",
  "additionalProperties": false,
  "required": ["orderId", "amountMinor", "evidenceIds"],
  "properties": {
    "orderId": { "type": "string", "minLength": 1, "maxLength": 100 },
    "amountMinor": { "type": "integer", "minimum": 1 },
    "evidenceIds": {
      "type": "array",
      "minItems": 1,
      "maxItems": 10,
      "uniqueItems": true,
      "items": { "type": "string", "minLength": 1, "maxLength": 100 }
    }
  }
}
```

Required fields and additional-property restrictions have distinct jobs, as the [JSON Schema object reference](https://json-schema.org/understanding-json-schema/reference/object) explains. Provider-supported schema subsets differ; this is a standard-schema illustration, not a promise that every generation API accepts it unchanged.

Validate again server-side. Bound total response bytes before parsing, preserve a refusal or incomplete-response path, and reject malformed output rather than silently manufacturing missing fields.

## Keep annotations out of the security model

A description saying “only authorized refunds” helps communicate intent. It does not enforce authorization. JSON Schema's [annotation documentation](https://json-schema.org/understanding-json-schema/reference/metadata) distinguishes descriptive metadata from validation.

Even format requires care: the [type reference](https://json-schema.org/understanding-json-schema/reference/type) notes that format is an annotation by default. A plausible identifier or date is not proof that the referenced object exists or is usable.

Business constraints belong in code and authoritative data checks, not prose attached to schema properties.

## Build a proposal-to-command boundary

```text
model proposal
 -> parse and shape validation
 -> authenticated principal and tenant binding
 -> evidence lookup and policy checks
 -> fresh state/version check
 -> durable command with idempotency
 -> execution and reconciliation
```

The server derives tenant and principal from the authenticated request. It does not let the model supply a privileged identity.

For each evidence ID, verify that it was available to this request, belongs to the correct tenant, and refers to a known source revision. These checks establish provenance, not that the evidence semantically supports the proposal. High-risk decisions may need explicit deterministic rules or human review in addition.

## Close the time-of-check gap

Suppose the proposal is valid against order version 17. Another process issues a partial refund before execution. Reusing the earlier eligibility decision can over-refund.

Bind approval to an operation identity, relevant inputs, policy version, and state version. At execution, recheck the mutable constraints transactionally or reserve the eligible amount through an owned workflow. If the version changes, reevaluate; do not ask the model to reinterpret a concurrency failure as permission.

Money should use a currency-aware representation and authoritative limits. An integer field alone does not establish currency scale, maximum refundable amount, or settlement eligibility.

## Different failures need different actions

| Failure | Response |
| --- | --- |
| invalid shape | reject; optionally bounded regeneration |
| unauthorized order | deny, without widening tenant scope |
| missing or stale evidence | retrieve again or require review |
| policy rejection | record a business rejection |
| changed order version | reevaluate against fresh state |
| ambiguous external effect | reconcile by operation ID |

Repeatedly prompting the model until it returns an allowed answer can turn a policy gate into a negotiation. Limit retries and preserve the original failure reason in the audit record.

## Evaluate execution safety separately

Track schema validity, evidence provenance, unsupported claims, policy violations, and actual business outcomes as separate measurements. A higher valid-JSON rate can coexist with worse decisions.

Build adversarial fixtures: cross-tenant identifiers, fabricated evidence IDs, previously refunded orders, conflicting sources, excessive amounts, duplicate command delivery, and a timeout after the provider accepted the refund. Use known expected decisions so failures are actionable.

Log proposal IDs, policy decisions, source revisions, execution versions, and provider references. Avoid placing raw customer documents or full prompts in unrestricted observability systems.

## The CTO decision

Keep the generative component where ambiguity is useful: extraction, explanation, and proposing options. Put irreversible authority behind narrow commands with deterministic eligibility checks, bounded approval scope, and durable operation identity.

The goal is not to make every response executable. It is to make inappropriate execution impossible at a boundary the model cannot talk its way around.

## Further reading

- [AI agents need a control plane](/posts/ai-agent-control-plane-mcp-security)
- [Production AI needs release gates](/posts/production-ai-risk-gates)
- [Durable workflows still require idempotency](/posts/durable-workflows-do-not-remove-idempotency)


---

# SynchDB 1.4: Oracle CDB Support and Secured FDW Snapshots

> SynchDB 1.4 introduces Oracle Container Database support across CDC, FDW, and OLR paths, plus TLS and Oracle Wallet encryption for initial snapshot data transfers.

Canonical URL: https://www.ayushworks.xyz/posts/synchdb-1-4-oracle-cdb-tls-fdw-snapshots
Author: Ayush Basak
Last modified: 2026-08-19
Topics: postgresql, database-replication, oracle, synchdb, change-data-capture

Heterogeneous database replication is an architectural staple for teams offloading analytics or migrating production workloads to PostgreSQL. The release of [SynchDB 1.4](https://www.postgresql.org/about/news/synchdb-14-released-oracle-container-database-support-and-tls-secured-fdw-snapshots-3362/) addresses two critical operational friction points in enterprise replication pipelines: first-class support for Oracle Container Databases (CDB/PDB) across all ingestion mechanisms, and end-to-end network encryption for initial Foreign Data Wrapper (FDW) snapshot transfers.

This update enhances resilience and security for enterprise deployments where streaming data into PostgreSQL or IvorySQL must meet strict compliance and network isolation requirements.

## Architecture Overview

SynchDB operates as a native PostgreSQL extension, orchestrating initial bulk data ingestion alongside continuous Change Data Capture (CDC).

```text
+-------------------------------------------------------------+
|                    Source Data Layer                        |
|  +--------------------+             +--------------------+  |
|  | Oracle CDB/PDB     |             | MySQL / PostgreSQL |  |
|  | (e.g. FREE/PDB1)   |             | Sources            |  |
|  +---------+----------+             +---------+----------+  |
+--------|---|----------------------------------|-------------+
         |   |                                  |
Snapshot |   | CDC Logs                Snapshot | TLS
(Wallet) |   | (Debezium / OLR)                 | (FDW)
         v   v                                  v
+-------------------------------------------------------------+
|                     SynchDB 1.4 Engine                      |
|  +-------------------------------------------------------+  |
|  |  FDW Snapshot Module (Secured via TLS / Wallet)       |  |
|  +-------------------------------------------------------+  |
|  +-------------------------------------------------------+  |
|  |  Embedded Debezium Engine / Openlog Replicator        |  |
|  +-------------------------------------------------------+  |
+------------------------------+------------------------------+
                               |
                               v
+-------------------------------------------------------------+
|               Target PostgreSQL / IvorySQL                  |
+-------------------------------------------------------------+
```

Building resilient data pipelines requires careful operational isolation and streaming guarantees, as discussed in our articles on [Building Reliable Microservices](/posts/building-reliable-microservices) and backend architecture patterns in [Engineering Notes](/engineering-notes).

## Multi-Tenant Oracle Container Database (CDB/PDB) Support

Oracle Container Databases isolate tenant pluggable databases (PDBs) inside a parent container (CDB). Previous database replication tools required complex custom TNS routing or forced unified administrative credentials across containers.

SynchDB 1.4 expands Oracle CDB/PDB support across all three of its Oracle replication pathways:
1. **Change Data Capture (CDC)** via the embedded Debezium engine.
2. **Initial Snapshots** using `oracle_fdw`.
3. **Log-Based Replication** using Openlog Replicator (OLR).

Targeting a pluggable database is straightforward: supply the target service in standard `CDB/PDB` notation (for example, `FREE/FREEPDB1`). SynchDB automatically handles authentication and connects directly to the underlying PDB service. This path is validated against modern Oracle engines including Oracle 23ai.

```sql
-- Registering an Oracle CDB/PDB source in SynchDB 1.4
SELECT synchdb_add_conninfo(
    'oracle_tenant_connector', 'oracle-host.internal', 1521,
    'c##synch_user', 'read-from-a-secret-manager',
    'FREE/FREEPDB1', 'postgres', 'null', 'null', 'oracle'
);
```

## TLS-Secured FDW Snapshots via `synchdb_add_fdw_conninfo()`

Prior to SynchDB 1.4, initial snapshot loading performed via foreign data wrappers lacked native encryption parameter configuration. While continuous streaming CDC channels could be secured, historical bulk table loads over FDW ran over unencrypted plaintext connections unless wrapped in external IPSec or SSH tunnels.

SynchDB 1.4 introduces `synchdb_add_fdw_conninfo()`, allowing backend engineers to attach encryption connection parameters directly to foreign wrappers:
* **TLS Encryption** for MySQL and PostgreSQL snapshot sources.
* **Oracle Wallet** security parameters for Oracle and Openlog Replicator snapshot sources.

The extension exposes `synchdb_add_fdw_conninfo()` for this configuration. Its installed SQL signature accepts a connector name plus four text parameters; use the version-matched SynchDB documentation to map those parameters to TLS mode, certificate, key, root certificate, or Oracle Wallet settings. Avoid copying positional calls across releases without checking the installed extension definition.

This ensures full in-transit confidentiality across the entire lifecycle of a replication pipeline—from initial bulk snapshot ingestion to real-time delta propagation.

## Runtime Observability and Performance Adjustments

SynchDB 1.4 adds the ability to adjust the Debezium engine log level at runtime without interrupting active CDC streaming connectors. Previously, increasing verbosity to diagnose schema drift or lag spikes required restarting the connector service, risking backlog build-up.

```sql
-- Dynamically change log level to DEBUG for troubleshooting
SELECT synchdb_set_dbz_loglevel('oracle_tenant_connector', 'DEBUG');
```

The release also upgrades the embedded Debezium engine from 2.6.2.Final to 3.5.2.Final and reports fixes for crashes under sustained replication load, a data-converter crash, and an Oracle parser startup conflict. Treat those as reasons to test the upgrade under your own workload, not as a guarantee that backpressure is solved.

## Operational Tradeoffs

While SynchDB 1.4 simplifies heterogeneous replication, engineers should evaluate key operational tradeoffs:

* **Oracle Wallet Maintenance**: Utilizing Oracle Wallet for encrypted snapshots avoids inline credentials, but requires persistent filesystem certificate mounting and key rotation procedures across database nodes.
* **Snapshot Resource Pressure**: Initial snapshots compete with live CDC for source, network, and target capacity. Measure throughput and replication lag during a production-sized rehearsal.
* **PDB Service Privileges**: Grant only the Oracle permissions required by the chosen Debezium, FDW, or OLR path; the three paths do not have identical privilege requirements.

## When NOT to Use SynchDB

* **Single-Database Homogeneous Replication**: If you are replicating strictly between PostgreSQL instances, native logical replication or physical streaming replication ([PostgreSQL High Availability](https://www.postgresql.org/docs/current/high-availability.html)) provides better native integration and lower system overhead.
* **Stateless Event Ingestion**: If downstream consumer applications only require event notifications rather than strict relational state synchronization in PostgreSQL, standard event streaming brokers (e.g., Apache Kafka or NATS) are better suited.

## Common Mistakes

* **Assuming Historical FDW Snapshots Were Encrypted**: Relying on SynchDB versions prior to 1.4 for initial snapshot loading across public networks without external VPN tunnels left snapshot data unencrypted.
* **Omitted PDB Service Specifications**: Specifying only the container database name instead of the combined CDB/PDB string (e.g., using `FREE` instead of `FREE/FREEPDB1`) causes connector initialization failures during service resolution.

## Conclusion

SynchDB 1.4 closes an important encryption gap for initial snapshots and extends CDB/PDB handling across its Oracle paths. It is still an extension with native, JVM, FDW, and source-database dependencies, so production adoption should follow a staged snapshot-and-CDC rehearsal with observable lag, restart, and certificate-rotation tests.

## References

* [SynchDB 1.4 Release Announcement](https://www.postgresql.org/about/news/synchdb-14-released-oracle-container-database-support-and-tls-secured-fdw-snapshots-3362/)
* [PostgreSQL Documentation: High Availability and Replication](https://www.postgresql.org/docs/current/high-availability.html)


---

# Tail Sampling Is a Memory Budget Disguised as an Observability Feature

> How tail-based trace sampling buffers incomplete traces, routes spans consistently, makes late decisions, and fails under cardinality, bursts, and collector loss.

Canonical URL: https://www.ayushworks.xyz/posts/tail-sampling-is-a-memory-budget
Author: Ayush Basak
Last modified: 2026-08-31
Topics: observability, opentelemetry, distributed-tracing, capacity-planning

Head sampling decides whether to keep a trace when it begins. It is cheap and cannot know the outcome. Tail sampling waits for spans to arrive, then can retain errors, slow traces, or rare attributes. That power requires buffering an unknown amount of unfinished distributed work.

The invariant is:

> Every span for one trace reaches the same decision maker, and the decision is made only after a bounded wait with bounded memory.

## The collector does not know when a trace is complete

Distributed tracing has no universal “final span” message. Services report asynchronously; retries and queues can create late spans. A tail sampler groups spans by trace ID and waits for a configured decision delay.

```text
t=0ms    root span begins
t=80ms   database span arrives
t=200ms  HTTP response completes
t=900ms  async child arrives
t=10s    sampling decision fires
```

If the delay is too short, the decision sees an incomplete trace. If it is too long, memory and export latency grow. Tail sampling is therefore an approximation over a time window, not omniscient post-processing.

## Trace affinity is mandatory

If spans from one trace are load-balanced across several independent tail samplers, each sees a fragment and may make a different decision. Route by trace ID to a stable collector shard before tail sampling.

```text
agents/gateways → load-balancing exporter(hash trace_id)
                → tail-sampler shard
                → backend
```

Changing shard membership can move in-flight trace IDs. Plan rolling updates and failures knowing that buffered traces are ephemeral unless an external durable layer exists. Most collector deployments trade perfect preservation for bounded cost.

## Memory follows arrival rate and delay

A first planning estimate is:

```text
buffer bytes ≈ spans/second × decision delay × average span bytes × overhead
```

Bursts, large attributes, and long traces dominate the tail. Size from measured high percentiles, not average payloads. The collector also needs memory for receivers, processors, queues, and exporters; giving the tail sampler the entire container limit turns pressure into process death.

Configure expected trace capacity, a memory limiter, exporter queues, and refusal/drop metrics together. A sampler that preserves every error until its own OOM kills all buffered errors has not improved observability.

## Policies compose in surprising ways

Useful policies include:

- always keep error status;
- keep latency above a service-specific threshold;
- keep security-sensitive routes;
- probabilistically sample the healthy remainder;
- rate-limit high-volume categories.

Policy order and combination semantics matter. An inverted or broad string/attribute rule can sample nearly everything. A global “over 1 second” rule over-samples naturally long batch jobs and under-samples a 400 ms endpoint whose SLO is 100 ms.

Attach service criticality and route class as governed resource/span attributes, then test policies against recorded distributions before rollout.

## Sampling cannot repair bad instrumentation

An error trace without causal context remains unhelpful. High-cardinality attributes increase memory and backend cost. Secrets copied into spans become a larger exposure when tail policy deliberately preserves exceptional requests.

The schema must define stable service identity, operation name, status, tenant classification where permitted, and links across asynchronous boundaries. Redaction belongs before buffering and export.

## Failure policy

| Failure | Expected behavior |
|---|---|
| sampler at trace capacity | reject/drop with an explicit metric, not silent growth |
| exporter unavailable | queue within disk/memory budget, then shed |
| collector shard dies | accept loss of its buffered traces or add a durable hop |
| late span after decision | apply documented late-span behavior; measure it |
| policy configuration expands sample rate | protect backend with rate and memory limits |

Keep a small independent head-sampled stream if losing every trace during a tail-sampler failure is unacceptable. Redundant evidence is often cheaper than pretending one pipeline is lossless.

## What to monitor

Track traces in memory, spans per trace, decision latency, sampled and dropped traces by policy, late spans, collector RSS versus limit, refused spans, exporter queue occupancy, shard balance, and backend ingestion rate. Alert on sample-rate changes even when collector health appears green.

## The CTO decision

Use tail sampling when outcome-aware selection materially improves diagnosis and its buffering cost is understood. Keep head sampling when low latency, simple scaling, or predictable loss is more valuable. Many platforms need both: a small unbiased baseline plus targeted tail retention.

Sampling is a data-loss policy. Make the loss observable.

## References

- [OpenTelemetry: Sampling](https://opentelemetry.io/docs/concepts/sampling/)
- [OpenTelemetry Collector: Tail Sampling Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor)
- [OpenTelemetry sample configurations](https://opentelemetry.io/docs/demo/sample-configurations/)
- [Related: Observability is a data contract](/posts/opentelemetry-observability-data-contract)


---

# Technical Decision Models Compound

> Build lasting engineering leverage by recording constraints, invariants, evidence, and reversibility—not memorizing tools.

Canonical URL: https://www.ayushworks.xyz/posts/technical-decision-models-compound
Author: Ayush Basak
Last modified: 2026-09-05
Topics: engineering-leadership, architecture, career-growth, decision-making

Framework knowledge decays quickly. Decision quality compounds.

An engineer who memorizes one queue, database, or cloud platform can move fast inside familiar boundaries. An architect who can expose constraints, identify invariants, compare failure modes, and design reversible experiments can make progress when the tools change.

That capability is built through a repeatable decision record—not through longer technology lists.

## Use a compact decision model

For consequential choices, capture seven fields:

1. **Outcome** — what user or business result must improve?
2. **Constraints** — latency, cost, regulation, team skill, deadline, and scale.
3. **Invariants** — what must remain true during failure and change?
4. **Options** — include the current design and doing less.
5. **Evidence** — measurements, prototypes, incidents, and authoritative references.
6. **Reversibility** — migration and exit cost.
7. **Review trigger** — the condition that makes the decision stale.

```md
# Decision: isolate analytics writes from checkout

Outcome: protect checkout p99 during reporting spikes.
Invariant: accepted orders remain durable and queryable.
Evidence: 38% of peak DB CPU is unbounded report scans.
Choice: CDC into a separate analytical store.
Guardrail: source lag < 5 minutes for operational reports.
Rollback: retain existing report endpoint behind a flag.
Review: when peak source CPU stays below 45% for 30 days.
```

This is an architecture decision record (ADR) with operational teeth. The review trigger prevents a correct decision from becoming permanent folklore after its assumptions change.

## Prefer invariants over products

“Use Kafka” does not transfer across contexts. “A committed event must survive one broker loss, consumers may process it more than once, and effects must converge” is reusable.

“Use microservices” is similarly weak. “Deploy these capabilities independently because their change cadence and failure budget differ” exposes the reason and the cost.

Tool choices become easier after the system promise is explicit.

## Grow judgment through prediction

Before shipping a decision, write what you expect:

- which metric will move;
- which failure becomes more likely;
- where the new bottleneck will appear;
- what operators will see first;
- when the option stops being economical.

Review the prediction after an incident or one operating quarter. The delta between expected and observed behavior is the learning asset. Without the prediction, teams rewrite history and call every outcome inevitable.

## Leadership application

Decision records are not permission documents. Keep them short, discoverable, and close to the code or service they govern. Assign a directly responsible owner, invite dissent before commitment, and separate reversible choices from one-way migrations.

| Decision shape | Governance |
| --- | --- |
| reversible and low blast radius | owner decides; observe |
| reversible but expensive | written options and staged experiment |
| hard to reverse | multi-discipline review and migration rehearsal |
| safety or compliance boundary | independent evidence and approval |

An error-budget policy supplies another useful model: reliability is not “as much as possible.” It is a product commitment that should alter release behavior when consumed.

## Common mistakes

- Recording the chosen tool but not rejected alternatives.
- Inventing scale requirements with no workload evidence.
- Treating a benchmark as production behavior.
- Hiding organizational constraints as technical necessities.
- Never revisiting decisions after the triggering constraint disappears.

## Trade-offs

Writing decisions consumes time and can become ceremony. Record only choices with meaningful future cost, keep the format proportional to risk, and automate links from services to relevant records. The alternative—reconstructing intent during an incident or migration—is usually more expensive.

Durable expertise is a library of tested decision models. Tools are examples inside that library, not the library itself.

## Further reading

- [Architecture Decision Records](https://adr.github.io/)
- [Google SRE: Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)
- [Architecture decisions need expiry dates](/posts/architecture-decisions-need-expiry-dates)


---

# The Cost of Complexity

> Why simple systems fail less often than complex ones, and how to think about architectural decisions.

Canonical URL: https://www.ayushworks.xyz/posts/the-cost-of-complexity
Author: Ayush Basak
Last modified: 2026-08-16
Topics: system-design, technical-leadership, architecture

<span id="the-cost-of-complexity"></span>

Most engineers overestimate how much complexity they need to solve today's problems.

We've been taught to prepare for scale. To architect for flexibility. To abstract away the details that might change later.

But there's a hidden cost to every abstraction layer, every parameterized configuration, every "future-proofing" decision we make.

## Complexity Has a Cost

Every line of code you write:

- **Takes time to understand** when someone (or you, six months later) reads it
- **Can introduce bugs** in subtle and unexpected ways
- **Requires maintenance** as requirements change
- **Multiplies context** that someone needs to keep in their head

The more complex your system, the higher the cognitive load on every person who touches it.

And cognitive load is the real bottleneck in engineering—not CPU or memory or network latency.

## Why We Build Complex Systems

We usually build complexity for one of three reasons:

**1. Premature optimization**

We optimize for a scale that doesn't exist yet. We shave milliseconds off hot paths in code that runs once per day. We architect for millions of users before we have thousands.

**2. Over-generalization**

We build abstractions that handle "all possible cases" instead of the cases we actually have. We parameterize configuration that will never change.

**3. Defensive programming**

We add error handling for exceptions that shouldn't happen. We validate inputs that can only come from trusted sources. We add layers of redundancy everywhere.

All of these have their place—but most projects suffer from too much, not too little.

## The Simplicity Test

Before you add a layer of abstraction or a configuration option, ask:

- **Do I need this today?** If not, don't build it.
- **Will this be easier to add later?** For most things, yes. Abstraction layers are easier to add when you understand the problem better.
- **Who will understand this?** If only you can reason about it, it's too complex.

## What Simple Systems Look Like

The best systems I've worked on had a few things in common:

- **Clear boundaries** - Each component has one job
- **Obvious data flow** - You can follow how data moves through the system
- **Minimal indirection** - If something happens, you can trace why
- **Explicit over implicit** - The code shows you what it does, not what it *might* do

None of these require less code. Sometimes they require more. But they're easier to reason about.

## The Reality

The most expensive systems aren't the ones with the most features. They're the ones nobody understands.

Before you write your next architectural decision, ask yourself:

*Am I solving a real problem today, or preparing for a problem that might never come?*

Often, the answer will surprise you.

Start simple. Add complexity when you feel pain, not when you anticipate it.

Your future self will thank you.


---

# The Transactional Outbox Is Not the Delivery Guarantee

> A CTO-level design for atomic event creation, idempotent relays and consumers, ordering, replay, and end-to-end delivery evidence.

Canonical URL: https://www.ayushworks.xyz/posts/transactional-outbox-delivery-guarantees
Author: Ayush Basak
Last modified: 2026-08-20
Topics: distributed-systems, backend-engineering, event-driven, reliability

The transactional outbox solves one precise problem: it makes a business-state change and the creation of its integration event part of the same database transaction. That is important. It is not the same as guaranteeing that every downstream effect happens exactly once.

The production question is not “do we have an outbox?” It is: **what invariant survives every crash boundary from command to consumer side effect?**

## Name the dual-write failure

Suppose checkout must update an order and publish `OrderConfirmed`.

```text
write order -> crash -> publish event       (missing event)
publish event -> crash -> write order       (phantom event)
```

No ordering of two independent writes removes both windows. A distributed transaction can coordinate some systems, but it adds support and availability constraints most service architectures do not want.

The outbox places both records in one local transaction:

```sql
BEGIN;

UPDATE orders
SET status = 'confirmed', version = version + 1
WHERE id = $1 AND status = 'pending';

INSERT INTO outbox (
  event_id, aggregate_type, aggregate_id,
  aggregate_version, event_type, payload, created_at
) VALUES (
  $2, 'order', $1, $3,
  'OrderConfirmed', $4::jsonb, now()
);

COMMIT;
```

Now the order and event exist together or not at all. AWS Prescriptive Guidance describes this as the core purpose of the pattern and explicitly warns that duplicate delivery still requires idempotent consumers.

## Follow every crash boundary

The relay reads committed outbox rows and publishes them to a broker:

```text
database transaction
      -> outbox row
      -> relay publish
      -> broker acknowledge
      -> mark row delivered
```

If the relay crashes after the broker accepts the event but before `delivered_at` is stored, it publishes again. If it marks delivered before the broker accepts, the event can be lost. Therefore a conventional relay is **at-least-once**, and duplicate publication is the correct failure behavior.

Process rows concurrently without letting workers claim the same work:

```sql
SELECT event_id, payload
FROM outbox
WHERE delivered_at IS NULL
ORDER BY created_at, event_id
FOR UPDATE SKIP LOCKED
LIMIT 100;
```

Keep claims short-lived. Do not hold a database transaction open during an unbounded network call. A practical design records a lease, commits, publishes, then marks success; an expired lease makes work recoverable after a worker dies.

## Idempotency belongs at the effect

A consumer that “checks then acts” can still race:

```text
check event unseen -> charge card -> crash -> record event seen
```

The retry charges again. The deduplication record must be atomic with the side effect where possible. For a database-owned effect:

```sql
BEGIN;

INSERT INTO consumed_events (consumer, event_id)
VALUES ('invoice-projector', $1)
ON CONFLICT DO NOTHING;

-- Continue only when one row was inserted.
UPDATE invoice_summary
SET paid_cents = paid_cents + $2
WHERE customer_id = $3;

COMMIT;
```

For an external API, pass a stable idempotency key if the provider supports one. If it does not, “exactly once” is not an honest promise. Design a reconciliation process that detects ambiguous outcomes.

Broker-level transactions can provide strong guarantees inside the broker's boundary. Kafka's idempotent producer and transactions, for example, can atomically write Kafka records and offsets in supported workflows. They do not make an unrelated payment processor or email provider transactional with Kafka. State the boundary whenever using the phrase exactly once.

## Ordering needs a domain, not a global queue

Most systems do not need total order across every event. They need order per aggregate—events for one order, account, or device.

Include `aggregate_id` and a monotonically increasing `aggregate_version`. Partition the broker by aggregate ID. Consumers then reject stale versions and detect gaps:

```text
received version 41, stored version 40 -> apply
received version 40, stored version 41 -> duplicate/stale
received version 43, stored version 40 -> gap; defer and investigate
```

Global ordering reduces parallelism and still does not define business semantics across unrelated entities. Buy only the ordering the invariant requires.

## Event payloads are long-lived APIs

An outbox couples the business transaction to an immutable historical record. Treat payload evolution as API evolution:

- include an event type and schema version;
- prefer facts (`OrderConfirmed`) over imperative commands (`SendConfirmationEmail`);
- include the minimum context needed for a stable contract;
- avoid leaking a full internal row that changes whenever the database changes;
- make consumers tolerate additive fields;
- retain fixtures for older payload versions.

Replays expose every lazy schema decision. If rebuilding a read model requires bespoke transformations for undocumented events, the event log is not an operational asset.

## Operate lag, not just queue depth

Measure the system at each boundary:

- age of the oldest unpublished outbox row;
- rows created versus successfully published;
- relay lease expirations and retry counts;
- broker consumer lag by partition;
- duplicate rate at each consumer;
- dead-letter age and owner;
- end-to-end time from business commit to visible side effect.

Queue depth alone cannot distinguish a healthy burst from a poison event blocking one aggregate.

Keep outbox cleanup separate from delivery. Delete or archive only rows whose retention window has elapsed and whose delivery evidence is durable. A sudden cleanup job should not compete with the relay on the same hot index.

## Common mistakes

**Publishing from an ORM hook after commit.** It recreates the dual-write window.

**Using a random event ID on every retry.** The retry becomes a new event and defeats deduplication. Derive or persist the ID with the originating command.

**Marking published before acknowledgement.** This optimizes for silent loss.

**Assuming consumers are idempotent because handlers are short.** Test the crash after every external effect.

**No replay or poison-event policy.** Eventually one malformed payload will stop progress or loop forever.

## CTO review checklist

1. Where is event creation atomic with domain state?
2. Which crash boundary produces duplicates, and how is each effect deduplicated?
3. What is the required ordering scope?
4. How are schema versions evolved and replayed?
5. What end-to-end metric proves a committed fact became a downstream effect?
6. Who owns reconciliation when the external outcome is ambiguous?

The outbox is a valuable first link. Reliability comes from connecting the rest: stable identity, recoverable relay state, bounded ordering, idempotent effects, replayable schemas, and observable convergence.

## References

- [AWS Prescriptive Guidance: Transactional Outbox Pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html)
- [Apache Kafka documentation](https://kafka.apache.org/documentation/)
- [PostgreSQL: SELECT locking and SKIP LOCKED](https://www.postgresql.org/docs/current/sql-select.html)
- [Related: Building Microservices That Fail Gracefully](/posts/building-reliable-microservices)
- [Related: Delivery Semantics Shape the Data Model](/engineering-notes#delivery-semantics)


---

# Verify What You Deploy

> A CTO-level software supply-chain design for immutable artifacts, SLSA provenance, Sigstore verification, admission policy, builder trust, and incident response.

Canonical URL: https://www.ayushworks.xyz/posts/verify-what-you-deploy
Author: Ayush Basak
Last modified: 2026-08-26
Topics: security, software-supply-chain, platform-engineering, cloud-infrastructure

“Deployed from main” is not evidence about the bytes running in production. Between source review and runtime sits a supply chain: dependencies, build runners, scripts, registries, credentials, promotion jobs, and admission controls.

The production invariant should be:

> Every running artifact is identified by digest and admitted only when trusted evidence connects that digest to an approved source revision and build process.

## Stop deploying mutable names

Tags are convenient references. Digests identify content.

```text
unsafe identity: registry.example.com/payments:latest
stable identity: registry.example.com/payments@sha256:8c4…e21
```

A tag can move after approval. A digest changes when content changes. Build once, scan and test that artifact, then promote the same digest through environments. Rebuilding for production creates different bytes and breaks the evidence chain.

Record a release manifest:

```yaml
artifact_digest: sha256:8c4...e21
source_repository: github.com/acme/payments
source_revision: 61c9d08
builder_identity: github-actions://acme/release
workflow: .github/workflows/release.yml@61c9d08
provenance_digest: sha256:2f1...a09
```

## Provenance is a claim, not automatic trust

SLSA defines progressive guarantees for source and builds. At Build L1, provenance exists. At L2, a hosted build platform generates and signs it. L3 requires a hardened build platform with stronger isolation and protection of provenance generation.

Provenance answers:

- which artifact digest was produced;
- which builder produced it;
- which source and top-level inputs were used;
- which build type and parameters applied.

It does not prove the source code is safe. It proves facts about how an artifact came to exist—if the attesting builder is trusted.

Your verification policy therefore needs expectations:

```text
artifact subject digest matches candidate
builder identity is approved
source repository is approved
source revision belongs to protected branch
build type and parameters match policy
provenance signature is valid
```

SLSA’s verification guidance explicitly requires checking provenance against configured roots of trust and expected builder properties. Merely storing an attestation beside an image changes nothing.

## Keep signing authority out of build steps

If user-controlled build code can read the long-lived signing key, compromised source can sign its own malicious artifact. Prefer a build platform that issues provenance from its trusted control plane.

Sigstore supports identity-based, keyless signing using short-lived certificates. Cosign verification can require both a certificate identity and OIDC issuer, while also checking the artifact digest in the signature claims.

```bash
cosign verify \
  --certificate-identity-regexp '^https://github.com/acme/payments/' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  registry.example.com/payments@sha256:8c4...e21
```

Pin the expected workflow identity narrowly. Trusting any workflow in the organization is equivalent to giving every repository release authority.

## Enforce at admission

CI verification catches mistakes before deployment. Runtime admission prevents bypasses from other paths.

An admission policy should reject:

- mutable tags where digests are required;
- unsigned or unverifiable artifacts;
- unapproved builders or repositories;
- provenance whose subject does not match the artifact;
- builds older than a policy window where required;
- known critical vulnerabilities under a documented exception process.

Failing closed needs an emergency process. Define who can grant a time-bounded exception, what evidence is recorded, which scope it affects, and how automatic expiry is enforced. A permanent “break glass” allow-all policy becomes the primary path during pressure.

## Dependencies belong in the chain

A signed application built from a compromised dependency is still compromised. Generate an SBOM and preserve lockfiles, but distinguish inventory from integrity.

For high-risk components:

- pin by immutable version or digest;
- verify package provenance or signatures where available;
- proxy through an approved registry;
- retain downloaded artifacts for reproducible investigation;
- alert when a dependency disappears or changes unexpectedly;
- rebuild after base-image and toolchain fixes.

Provenance should expose build inputs sufficiently to answer “which releases used this compromised base image?” within minutes.

## Design the incident query

During a supply-chain incident, leadership needs bidirectional answers:

```text
source revision → artifacts → environments → tenants
running digest → provenance → builder → source → dependencies
```

Index release manifests. Retain attestations beyond artifact cleanup. Log admission decisions with policy version and evidence digest.

Test revocation: can you block one builder, repository, certificate identity, or artifact digest without stopping every deployment?

## CTO review

1. Are production workloads pinned by digest?
2. Is the tested artifact exactly the promoted artifact?
3. Who generates provenance, and can build steps forge it?
4. Which builder and source identities does admission trust?
5. Is provenance verified or merely published?
6. How are emergency exceptions scoped and expired?
7. Can one compromised dependency be mapped to running releases?
8. Can a compromised builder be revoked independently?

Signing says who made a claim. Provenance says what the claim is about. Admission policy turns both into a production control.

## References

- [SLSA Specification v1.2](https://slsa.dev/spec/v1.2/)
- [SLSA: Verifying Artifacts](https://slsa.dev/spec/v1.2/verifying-artifacts)
- [Sigstore Cosign: Verifying Signatures](https://docs.sigstore.dev/cosign/verifying/verify/)
- [Related: Service Identity Before a Service Mesh](/posts/service-identity-before-service-mesh)
- [Related: From Repository to Real Product](/posts/shipping-full-stack-products)


---

# A Webhook Endpoint Is a Durable Inbox

> Design webhook ingestion around raw-body authentication, durable acceptance, idempotent effects, unordered delivery, replay defense, and reconciliation.

Canonical URL: https://www.ayushworks.xyz/posts/webhook-endpoints-are-durable-inboxes
Author: Ayush Basak
Last modified: 2026-09-05
Topics: backend-engineering, webhooks, reliability, security, distributed-systems

A webhook is often implemented as a controller that parses JSON, runs business logic, and returns `200`. That works until the sender retries, events arrive out of order, the handler times out after committing, or a framework mutates the request body before signature verification.

The robust model is different:

> A webhook endpoint authenticates and durably accepts a message. Processing happens after acceptance.

This separates a short network acknowledgement from a potentially long, failure-prone business workflow.

## Define the acknowledgement boundary

The receiver should perform only the work required to decide whether it can safely take responsibility:

1. read the exact raw request bytes;
2. identify the provider and endpoint configuration;
3. verify the signature and timestamp according to that provider’s scheme;
4. validate basic size and event-envelope limits;
5. insert the delivery into a durable inbox with a unique key;
6. return success only after the insert commits.

```text
provider -> authenticate -> durable inbox -> 2xx
                                  |
                                  v
                         async processor -> domain effects
```

Returning success before durable storage risks losing the event. Returning success only after all business work completes couples provider retry behavior to internal dependency latency.

Stripe recommends returning a successful response before complex logic and handling events asynchronously. GitHub expects a `2xx` promptly and likewise recommends queued processing.

## Verify the bytes you received

Signature schemes authenticate the payload as delivered, not a parsed-and-reserialized equivalent. Stripe explicitly warns that whitespace changes, key reordering, encoding changes, or body parsing can break verification.

Capture raw bytes before generic JSON middleware. Associate secrets with a specific provider, environment, and endpoint. During secret rotation, accept the documented overlap set rather than trying every secret ever issued.

A valid signature proves possession of the configured secret for those bytes. It does not prove that processing the same valid delivery twice is safe.

## Deduplicate at two levels

Providers may retry the same delivery. Store the provider’s delivery or event identifier behind a uniqueness constraint.

```sql
CREATE TABLE webhook_inbox (
  provider text NOT NULL,
  delivery_id text NOT NULL,
  event_type text NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  payload jsonb NOT NULL,
  status text NOT NULL DEFAULT 'pending',
  PRIMARY KEY (provider, delivery_id)
);
```

This makes repeated receipt idempotent. But Stripe notes that separate Event objects can sometimes represent the same underlying object and event type. Business effects therefore need their own idempotency boundary: invoice transition, subscription version, or `(object_id, event_type, semantic_version)`.

Transport deduplication answers “have I accepted this message?” Domain idempotency answers “has this effect already happened?”

## Assume unordered delivery

Stripe does not guarantee event order, and GitHub documents that deliveries may arrive in a different order from the underlying events. An event handler must not infer current truth solely from arrival order.

Prefer one of these policies:

- retrieve current authoritative state from the provider;
- apply a provider-supplied monotonic version when available;
- use a domain state machine that rejects invalid backward transitions;
- record ambiguous events for reconciliation.

Timestamps are weak ordering tokens when clocks, generation, and redelivery semantics differ. “Latest message received” is not necessarily “latest state.”

## Separate replay defense from retry support

Stripe signs a timestamp and recommends a recency tolerance to limit captured-request replay. GitHub recommends tracking `X-GitHub-Delivery`, which remains the same for a redelivery.

These mechanisms differ. Implement the provider’s documented verification protocol rather than a generic webhook abstraction that guesses. Clock-based freshness requires synchronized server time. Delivery-ID retention must cover the period in which redelivery is possible plus an operational margin.

Do not reject a legitimate provider retry merely because it is a retry. Accept it idempotently, record it, and return the response that stops unnecessary future delivery.

## Build reconciliation, not faith

Webhook delivery is a notification channel, not always the complete ledger. GitHub documents that failed deliveries are not automatically redelivered in all cases. Network errors, disabled endpoints, expired retry windows, and operator mistakes create gaps.

Run a reconciliation job against the provider’s authoritative API when supported. Track checkpoints, compare expected object state, and enqueue repairs through the same idempotent processor. Alert on oldest pending inbox item, failure rate, retry count, signature failures, and reconciliation drift.

## Test the ambiguous outcomes

Exercise duplicate delivery, out-of-order delivery, invalid signatures, valid old signatures, rotated secrets, storage failure before acknowledgement, worker crash after side effect but before status update, and provider API outage during reconciliation.

The hardest test is the ambiguous write: the downstream action succeeds, but the worker loses the response. The retry must resolve the outcome through an idempotency key or read-before-retry—not hope.

The reliable webhook system is intentionally boring at the edge. Authenticate, commit, acknowledge, and let a replayable state machine do the dangerous work.

## References

- [Stripe: Receive webhook events](https://docs.stripe.com/webhooks)
- [Stripe: Resolve signature verification errors](https://docs.stripe.com/webhooks/signature)
- [GitHub: Webhook best practices](https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks)
- [GitHub: Handling failed webhook deliveries](https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries)


---

# Securing Multi-Tenant Systems: From Request Identity to Data Deletion

> A practical tenant-isolation guide covering authorization, PostgreSQL RLS, pooled connections, caches, queues, storage, resource limits, and offboarding.

Canonical URL: https://www.ayushworks.xyz/security/multi-tenant-systems
Author: Ayush Basak
Last modified: 2026-09-13
Topics: security-engineering, multi-tenancy, authorization, postgresql, backend-engineering

[Security Engineering](/security) / 01

A customer opens an invoice URL, changes the invoice ID, and sees another company's bill. The request had a valid login. The SQL was parameterized. Neither control answered the question that mattered: is this person allowed to read this invoice?

Multi-tenant security starts with that distinction. Authentication establishes who is calling. Authorization decides what they may do within a tenant. Isolation keeps the decision intact as work moves through databases, caches, workers, storage, and administrative tools.

This guide develops the supplied [OWASP multi-tenant security checklist](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html) into an implementation and review path. It is not a claim that one example makes an application secure.

## On this page

- [Choose the boundary](#choose-the-boundary)
- [Verify tenant context](#verify-tenant-context)
- [Keep database context inside the transaction](#keep-database-context-inside-the-transaction)
- [Authorize objects and caches](#authorize-objects-and-caches)
- [Carry the boundary through workers](#carry-the-boundary-through-workers)
- [Protect files and shared capacity](#protect-files-and-shared-capacity)
- [Close access during offboarding](#close-access-during-offboarding)
- [Release checklist](#release-checklist)

## Choose the boundary

Assume a tenant member can alter every client-controlled identifier, replay requests, and submit work faster than expected. Also account for ordinary bugs: a forgotten query filter, a stale permission cache, or a worker that inherits the previous job's context.

Write down the allowed exceptions. A public catalogue, a deliberately shared document, and an authorized support operation are not isolation failures. They need explicit policies so that “shared” does not become a convenient explanation for any unscoped access.

| Data layout | What can enforce isolation | What still needs attention |
| --- | --- | --- |
| Separate databases | Credentials, database access, network boundaries | Shared admin credentials, backup access, migrations and operating cost |
| Separate schemas | Database roles and grants as well as namespaces | Search paths, role permissions and migration discipline |
| Shared tables | Tenant ownership, constrained roles and row policies | Every access path and every tenant-owned table |
| Hybrid | A documented boundary for each data class | Inconsistent assumptions between workloads |

A database per tenant does not isolate customers if every request uses credentials that can read every database. A schema name alone is not a permission boundary either.

Record which data is tenant-owned, intentionally global, user-private, or shared by policy. Include exports, audit records, search indexes and object versions—not just the tables behind your main API.

## Verify tenant context

A tenant ID in a header is a selector. So is a subdomain, request parameter, or a field in a JSON payload. Check the authenticated caller's current membership or explicit service authorization before accepting that selection.

The request path should look like this:

~~~text
untrusted tenant selector + verified caller identity
                      |
              membership / scope check
                      |
              server-owned tenant context
                      |
          resource and operation authorization
                      |
               tenant-scoped data access
~~~

Reject a missing or invalid context on tenant-owned routes. Do not recover by issuing a query without a tenant filter. Public endpoints and genuinely global jobs can use their own explicit scope.

Pass verified context to downstream components that need it. If a service accepts a new header value and replaces the verified tenant, the boundary has moved back into the caller's hands. Reset request-local state when the request ends.

A service identity may legitimately act across several tenants. Give it an explicit tenant set and operation scope. “Internal service” is not an authorization rule.

## Keep database context inside the transaction

For shared-table PostgreSQL deployments, row-level security can provide another check when an application query forgets its tenant predicate. It works only under the roles and policies you actually deploy.

Here is a small policy example for a **disposable database**. Run the setup as an authorized administrator. The non-login role illustrates a constrained request role; configuring production login credentials, role membership, and connection pooling is separate work.

~~~sql
CREATE ROLE tenant_request NOLOGIN NOSUPERUSER NOBYPASSRLS;

CREATE TABLE public.tenant_notes (
  tenant_id uuid NOT NULL,
  note_id uuid NOT NULL,
  body text NOT NULL,
  PRIMARY KEY (tenant_id, note_id)
);

ALTER TABLE public.tenant_notes ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.tenant_notes FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_notes_scope
ON public.tenant_notes
TO tenant_request
USING (
  tenant_id = current_setting('app.current_tenant')::uuid
)
WITH CHECK (
  tenant_id = current_setting('app.current_tenant')::uuid
);

GRANT USAGE ON SCHEMA public TO tenant_request;
GRANT SELECT, INSERT, UPDATE, DELETE
ON public.tenant_notes TO tenant_request;
~~~

The read predicate limits visible rows. The write check constrains the tenant identity of inserted and updated rows. This is one table, not policy coverage for the rest of an application.

Set the verified tenant for each transaction on the **same checked-out connection** that runs the queries. This demonstration uses a literal tenant UUID; application code should bind a server-verified value as a parameter.

~~~sql
BEGIN;
SET LOCAL ROLE tenant_request;

SELECT set_config(
  'app.current_tenant',
  '11111111-1111-4111-8111-111111111111',
  true
);

SELECT note_id, body FROM public.tenant_notes;
COMMIT;
~~~

The final argument to `set_config` makes the setting transaction-local. Re-establish it for the next transaction and commit or roll back before releasing the connection. A session-level setting can survive a committed transaction and leak into a later pooled request. [PostgreSQL documents both setting scopes](https://www.postgresql.org/docs/current/functions-admin.html).

This policy uses `current_setting` without the missing-value option. An absent setting raises an error; a malformed or empty UUID also fails rather than selecting all tenants. Do not catch those failures and retry an unscoped query.

There are two important limits:

- Superusers and roles with `BYPASSRLS` bypass row security. `FORCE ROW LEVEL SECURITY` subjects a table owner to policies; it does not constrain those privileged roles. Check `rolsuper` and `rolbypassrls` in `pg_roles` for the deployed request identity. See [PostgreSQL's row-security rules](https://www.postgresql.org/docs/current/ddl-rowsecurity.html).
- A custom tenant setting is not proof of membership. This example trusts the application to verify and set it. Arbitrary SQL execution through that same role can undermine a setting-based policy. Keep parameterized SQL and application authorization, and do not advertise this as isolation from a compromised application.

Keep privileged migrations and cross-tenant administration off the normal request path. Review other grants and policies too: a permissive policy added later can broaden access. Tenant-owned relationships also need constraints that prevent a row from pointing into another tenant.

An ORM filter is useful defense in depth, but raw SQL, bulk operations, alternate sessions and Core connections may not traverse it. Test those paths rather than assuming every query goes through your helper.

## Authorize objects and caches

Resolve tenant-owned resources through the verified scope. An invoice lookup should establish both the invoice identity and the tenant in which the caller may perform the requested action. Random IDs make guessing harder; a leaked random ID still needs authorization.

Caches need the same care. Classify the value before choosing the key:

~~~text
global:currency-codes:v1
tenant:<verified-tenant>:invoice:<invoice-id>:v3
tenant:<verified-tenant>:user:<user-id>:permissions:<version>
~~~

These are naming examples, not a complete key design. Include every dimension that changes the result: permissions, user, locale or feature set where applicable. Authorize access before serving a protected cache hit. A well-scoped key cannot make stale membership valid.

Choose expiry and invalidation from the freshness and permission contract. Not every immutable global entry needs a TTL, but cached authorization must have a deliberate revocation policy. The [cache-isolation essay](/posts/shared-cache-keys-are-data-isolation-boundaries) goes deeper into these choices.

## Carry the boundary through workers

An authenticated HTTP request may enqueue a job that runs after the user loses access. Decide which permissions must still hold at execution time.

For tenant-owned jobs, bind tenant context to an authenticated producer path, trusted routing, or integrity-protected metadata. At consumption, verify the path, establish context again, and authorize the target operation. A `tenant_id` inside an untrusted message proves nothing.

Scope deduplication keys, retry records, dead-letter access and ordering state whenever their effects vary by tenant. Reusing another tenant's idempotency key must not return their response or suppress their work.

Classify global and authorized cross-tenant jobs explicitly. A fabricated tenant ID does not make a platform maintenance job safe. If jobs are delayed, test revocation during the delay. See [Kafka quarantine and replay boundaries](/posts/kafka-poison-messages-need-an-ordering-policy).

## Protect files and shared capacity

For a tenant-owned object, authorize the exact object and operation before issuing a signed URL. Pick an object key, bucket, account or storage policy that enforces the intended boundary. A tenant prefix is useful organization, but without enforcement it remains a naming convention.

Signed URLs are capabilities. Limit the operation and validity window to the task. Do not assume that removing application membership immediately invalidates a URL already issued. The tenant name need not appear in that URL if authorization occurred before signing.

Where the threat or compliance model requires it, use tenant-specific encryption keys. Separate keys do not replace object-access checks. For upload validation and overwrite races, use the [S3 finalization guide](/posts/s3-presigned-uploads-need-a-finalization-protocol).

Availability also belongs in the isolation review. A tenant can remain within an HTTP request-rate limit while creating expensive fan-out, a deep queue, or too many concurrent queries.

Apply tenant-aware budgets at the actual bottleneck: work in flight, queued jobs, connections, CPU or memory. Retain global caps so many individually compliant tenants cannot exhaust the fleet. Dedicated worker pools can reduce the affected population, at the cost of capacity and operations. Choose limits from measured workloads and commitments—not copied constants.

## Close access during offboarding

Deleting a tenant row is not a deletion policy. Inventory active stores, caches, object versions, replicas, search copies, exports and backups. Record what is deleted immediately, what expires later, and what must be retained under a documented legal or contractual basis.

Revoke tenant credentials and access, stop new work, and handle queued jobs before declaring offboarding complete. A restored backup must not silently restore access that was revoked after the backup was taken.

Keep provisioning and deletion evidence in an access-controlled audit trail. Tenant-scoped audit events should carry server-verified tenant context. Restrict both tenant reads and platform-wide investigations; centralized logging does not justify unrestricted access. Do not log credentials, signed URLs or sensitive payloads just to make debugging easier.

High-entropy generated API tokens and user-chosen passwords need different storage treatment. Do not generalize a fast-digest pattern for random tokens into a password-storage design.

## Release checklist

Use two tenants, two ordinary users, and an explicitly privileged identity. Exercise the deployed request role, network path and pooling mode—not just an administrator's test connection.

| Test | Evidence to keep |
| --- | --- |
| Substitute another tenant's object ID in read, update, delete and export paths | Denied access without another tenant's data or effects |
| Omit context or supply a malformed value | Failure without an unscoped fallback |
| Reuse a database connection across tenants, including rollback paths | No inherited context or cross-tenant rows |
| Attempt to insert or move a row into another tenant | Rejected write under the request role |
| Add a tenant-owned table | Classification, policy and negative tests are required |
| Read a cache entry with another tenant's scope or stale permissions | No unauthorized cache hit |
| Tamper with a job's tenant, then replay it | Rejection or authorized handling, with scoped deduplication |
| Revoke membership before a delayed job or cached answer is served | The documented revocation rule is enforced |
| Saturate one tenant's queue or expensive endpoint | Other tenants retain the promised service budget |
| Offboard, restore a backup, and retry old access paths | Retention and revocation rules still hold |

Discover table coverage from the schema or enforce classification during migrations. A manually maintained list can omit the very table whose missing policy you need to catch. Compare classified tables with `pg_class.relrowsecurity`, `relforcerowsecurity`, and `pg_policies`.

Also test the successful paths. A system that denies every request has not demonstrated correct authorization. Intentional sharing and support access need positive tests that prove exactly what is permitted, alongside negative tests for everything else.

## When a boundary fails

For a suspected cross-tenant disclosure, contain the affected path and preserve access-controlled evidence. Investigate whether the same flaw exists in exports, cache hits, jobs and administrative tools. A repaired endpoint is not proof that every copy or access path is repaired.

Assign an owner to each isolation control and its test. The useful outcome of this guide is a review someone can rerun after a schema change, new worker, permission feature or storage migration.

## Sources and attribution

Adapted and expanded from the [OWASP Multi-Tenant Application Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html), contributed by the OWASP Cheat Sheet Series community. Changes include the walkthrough, tenant-notes SQL example, cross-layer scenarios and release-test matrix. This adapted guide is shared under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/); it does not imply OWASP endorsement.

- [PostgreSQL row-security policies](https://www.postgresql.org/docs/current/ddl-rowsecurity.html)
- [PostgreSQL configuration-setting functions](https://www.postgresql.org/docs/current/functions-admin.html)
- [Further reading: RAG permission revocation](/posts/rag-permission-revocation-is-a-serving-contract)

[Back to Security Engineering](/security)


---