Design Service Identity Before Buying a Service Mesh

Aug 25

By

securitydistributed systemscloud infrastructureservice identity

Mutual TLS answers one question: did both endpoints present credentials signed by a trusted authority? It does not decide what a workload is, why it received that identity, or which operation it may perform.

Those decisions form the service-identity architecture. A mesh can transport and rotate credentials, but it cannot invent the trust model for you.

Replace location with identity

IP allowlists assume that network location is a useful proxy for authority. In dynamic infrastructure, addresses are reused, workloads move, and several applications share nodes. The policy “10.0.4.0/24 may write invoices” grants power to a subnet, not to the invoice writer.

A workload identity should be stable across instances but specific to one operational purpose:

spiffe://prod.example.com/payments/authorizer
spiffe://prod.example.com/orders/checkout
spiffe://prod.example.com/data/report-reader

SPIFFE defines this URI shape and calls the root namespace a trust domain. Its SVIDs are verifiable identity documents represented as X.509 certificates or JWTs. The Workload API allows a local process to obtain identities and rotating trust bundles without embedding a long-lived secret in an image.

The important separation is:

attestation → identity issuance → authentication → authorization

Collapsing these steps creates systems that are encrypted but over-authorized.

Attest what the platform can prove

An identity issuer needs evidence connecting a running process to the declared workload. Depending on the platform, selectors may include Kubernetes namespace and service account, cloud instance identity, Unix process attributes, or a signed deployment artifact.

Prefer properties controlled by the delivery platform over labels the application can choose itself. A process should not become the payment service because it sends X-Service-Name: payments.

Treat registration as production code:

identity: spiffe://prod.example.com/orders/checkout
selectors:
  kubernetes_namespace: commerce
  kubernetes_service_account: checkout
owner: team-commerce
expires_with: service-retirement

Review broad selectors like broad IAM roles: they increase the population able to obtain the credential.

Separate trust domains deliberately

A trust domain is a cryptographic and administrative boundary. Do not put development and production under the same roots because the names look tidy. SPIFFE guidance recommends distinct domains where environments have different security practices or physical locations.

Federation should be explicit. If a vendor or acquired system needs access, exchange trust bundles and authorize a narrow foreign identity. Do not copy the production root or issue local identities on another organization’s behalf.

Trust-domain decisions should answer:

  • who can issue an identity;
  • how issuer compromise is contained;
  • which environments may federate;
  • how roots and bundles rotate;
  • how federation is revoked.

Authenticate identity; authorize intent

After mTLS verifies a caller, the receiving service still needs an authorization decision:

caller = spiffe://prod.example.com/orders/checkout
action = payments.authorize
resource = merchant/acme
context = amount, region, risk state

Keep policy near the resource that owns the invariant. A central policy engine may distribute rules or evaluate decisions, but the payment service must fail closed when it cannot prove authority for a mutation.

Avoid role strings that grow into opaque superpowers. Model capabilities around operations and resources:

allow if {
  input.caller == "spiffe://prod.example.com/orders/checkout"
  input.action == "payments.authorize"
  input.resource.merchant_id == input.claims.merchant_id
}

Log the identity, action, resource, policy version, and decision. A TLS handshake log alone cannot explain why a destructive request was allowed.

Prefer short-lived, automatically rotated credentials

Static API keys create a difficult contradiction: rotate often enough to limit exposure, but not so often that deployments break. Platform-issued short-lived credentials remove most application coordination from rotation.

Rotation still needs testing:

  • clients reload certificates without restart;
  • servers accept overlapping trust bundles during root rotation;
  • clock skew is bounded;
  • cached authorization does not outlive identity validity;
  • revocation has an emergency path shorter than normal expiry.

For service-to-service communication, SPIFFE recommends X.509-SVIDs where practical because bearer JWTs can be replayed if stolen. When using OAuth, follow the current OAuth Security BCP: bind flows correctly, validate redirect URIs, avoid deprecated grants, and make token audience explicit.

Migrate without a flag day

Move from shared secrets in stages:

  1. Inventory service credentials and observed callers.
  2. Issue workload identities while keeping the old path.
  3. Log identity-based decisions in shadow mode.
  4. Compare expected and observed callers.
  5. Enforce identity on one low-risk operation.
  6. Expand enforcement and remove shared credentials.
  7. Delete secret-distribution code and rotate the old secret.

Dual mode must have an expiry date. Otherwise the fallback becomes the permanent bypass.

Operate the identity plane

Measure issuance failures, certificate age, time to expiry, rotation latency, authorization denies by policy version, unknown caller identities, and trust-bundle convergence. Alert before expiry causes a fleet-wide outage.

Plan for control-plane impairment. Existing workloads should continue within a bounded credential lifetime; new workloads may fail to start. Decide whether that failure mode is safer than issuing unverifiable identity. Security and availability tradeoffs must be explicit.

CTO review

  1. What platform evidence binds a process to its identity?
  2. Are production, development, and partners separate trust domains?
  3. Does authorization name operations and resources, not only services?
  4. Can every credential rotate without an application deploy?
  5. How quickly can one workload or issuer be revoked?
  6. What remains available when the identity control plane is degraded?
  7. Which shared secrets disappear after migration?

Buy a mesh for capabilities you need. Design identity first, because the mesh will faithfully automate whatever trust model—good or bad—you give it.

References

>