Database Credential Rotation: How to Replace Secrets Without Breaking Connection Pools

A credential can be rotated successfully in a secret store while applications continue using the previous value. The control plane reports success, but a worker with a long-lived cache may fail the next time its pool creates a connection.

The useful unit of rotation is therefore a deployed population of clients, not a string. A complete rollout needs to account for the secret version, credential accepted by the database, process configuration, pool creation, existing sessions, and rollback behavior.

This guide proposes an application rollout protocol. Exact session behavior depends on the database, authentication method, driver, and proxy. Test those boundaries instead of assuming password replacement terminates or refreshes existing connections.

Map the copies of the credential #

Start with an inventory of consumers: request-serving instances, queue workers, scheduled jobs, migration runners, reporting processes, and administrative tools. For each, record how the secret arrives and what event refreshes it.

secret version
    |
    +-- process configuration
    |       |
    |       +-- pool factory -> new authenticated sessions
    |
    +-- worker cache
    |
    +-- scheduled job launched from an older image

Changing the source does not necessarily update any of these copies. An environment variable is a process snapshot. A cache has its own refresh policy. A connection pool may retain sessions while creating replacements with credentials captured during initialization.

AWS documents client-side caching separately from secret retrieval and rotation. Its Python caching guide includes refresh behavior and limitations; do not infer that every SDK, wrapper, or application cache uses the same policy.

Define the invariant in terms of new sessions #

For a planned rotation, a useful availability invariant is: every active workload can establish an authorized connection using the intended current credential before the old credential is retired.

A query over an already open session is insufficient evidence. It proves that session can still execute a query. It may say nothing about the next pool expansion, restart, autoscaling event, or failover.

Use a fresh-connection probe through the same driver, network path, TLS settings, and authentication mode as the workload. Keep the probe read-only and narrow. It should not require broader privileges than the application already has.

Record the credential version identifier and probe outcome, never the secret value. Version identifiers also need care if they encode confidential deployment information.

Choose a rotation strategy deliberately #

AWS Secrets Manager describes single-user and alternating-users strategies for supported Lambda-based database rotation. Alternating users maintains two database identities; it adds privilege and identity-management considerations. Single-user rotation updates one identity and requires clients to handle the transition appropriately. These are documented strategies, not a guarantee that a custom application is disruption-free. See the rotation strategy guide.

For a provider that supports overlapping credentials or identities, the rollout can make acceptance of the new value precede retirement of the old value. If the backend accepts only one current password, that overlap may not exist. Your client recovery behavior becomes more important, and your availability target may require a different authentication mechanism.

Do not promise zero downtime until the chosen backend and client path have passed the rotation test.

A proposed state machine #

Use observable stages rather than one boolean called rotated:

{
  "rotationId": "rotation-example-17",
  "targetVersion": "credential-version-b",
  "phase": "client-adoption",
  "requiredConsumerClasses": [
    "api",
    "queue-workers",
    "scheduled-jobs"
  ],
  "retirePreviousAfter": "all-required-checks-pass"
}

This is an example control record, not an AWS API payload.

First prepare the new credential with the intended privileges. Next verify new authentication from a canary client. Then roll the pool factory to the new version in a small part of the fleet. Expand only while fresh connections succeed and business error rates remain acceptable.

Drain old pools through driver-supported behavior. Decide what happens to checked-out sessions and long transactions. Closing all connections at once can turn a routine security operation into a reconnect storm.

Finally, retire the prior credential and prove that a fresh authentication attempt with it is rejected. Securely control that negative test and avoid logging credentials in command arguments or diagnostics.

Bound the reconnect workload #

Credential replacement can synchronize thousands of clients. Even with valid authentication, they may overwhelm connection admission, TLS establishment, or an intermediate proxy.

As an illustrative estimate, replacing 1,200 sessions over 60 seconds creates an average of 20 new sessions per second before normal churn. This is a planning calculation, not a benchmark or a safe database limit. Measure the actual bottleneck and account for bursts.

Roll by workload group, cap new-connection concurrency, and add jitter where clients would otherwise reconnect together. Preserve enough working capacity while each group changes. The relevant limit is discussed further in database connection budgets.

Failure policy #

Emergency rotation has a different objective from planned maintenance. An overlap window that is useful for availability can be unacceptable after compromise. Decide that exception before an incident.

Also separate future authentication from active-session revocation. Disabling a credential may not terminate sessions already established with it. Verify database-specific behavior and define who is authorized to terminate sessions when containment requires it.

Test the least convenient clients #

The most informative tests often involve infrequent consumers. Start a scheduled job from an older deployment template. Trigger scale-out halfway through rotation. Restart a worker whose secret cache has not refreshed. Exercise database failover after client adoption but before retirement.

For each test, record the expected credential version, fresh-connection result, and recovery time. Avoid treating aggregate success rate as proof that every consumer class works: a broken nightly job can disappear inside millions of healthy API queries.

The engineering decision #

Managed rotation reduces secret-management work, but the application still owns adoption and evidence. A reasonable review asks who inventories consumers, who controls retirement, how new authentication is verified, and what recovery is possible after the old value stops working.

The process is complete when the fleet uses the intended version, stale clients are accounted for, and the old access path behaves according to the security policy. A successful secret-store update is one checkpoint in that process.

References #

• AWS Secrets Manager rotation overview
• Single-user and alternating-users rotation
• Client-side secret caching
• Related: service identity