Verify What You Deploy

Aug 26

By

securitysoftware supply chainplatform engineeringcloud infrastructure

“Deployed from main” is not evidence about the bytes running in production. Between source review and runtime sits a supply chain: dependencies, build runners, scripts, registries, credentials, promotion jobs, and admission controls.

The production invariant should be:

Every running artifact is identified by digest and admitted only when trusted evidence connects that digest to an approved source revision and build process.

Stop deploying mutable names

Tags are convenient references. Digests identify content.

unsafe identity: registry.example.com/payments:latest
stable identity: registry.example.com/payments@sha256:8c4…e21

A tag can move after approval. A digest changes when content changes. Build once, scan and test that artifact, then promote the same digest through environments. Rebuilding for production creates different bytes and breaks the evidence chain.

Record a release manifest:

artifact_digest: sha256:8c4...e21
source_repository: github.com/acme/payments
source_revision: 61c9d08
builder_identity: github-actions://acme/release
workflow: .github/workflows/release.yml@61c9d08
provenance_digest: sha256:2f1...a09

Provenance is a claim, not automatic trust

SLSA defines progressive guarantees for source and builds. At Build L1, provenance exists. At L2, a hosted build platform generates and signs it. L3 requires a hardened build platform with stronger isolation and protection of provenance generation.

Provenance answers:

  • which artifact digest was produced;
  • which builder produced it;
  • which source and top-level inputs were used;
  • which build type and parameters applied.

It does not prove the source code is safe. It proves facts about how an artifact came to exist—if the attesting builder is trusted.

Your verification policy therefore needs expectations:

artifact subject digest matches candidate
builder identity is approved
source repository is approved
source revision belongs to protected branch
build type and parameters match policy
provenance signature is valid

SLSA’s verification guidance explicitly requires checking provenance against configured roots of trust and expected builder properties. Merely storing an attestation beside an image changes nothing.

Keep signing authority out of build steps

If user-controlled build code can read the long-lived signing key, compromised source can sign its own malicious artifact. Prefer a build platform that issues provenance from its trusted control plane.

Sigstore supports identity-based, keyless signing using short-lived certificates. Cosign verification can require both a certificate identity and OIDC issuer, while also checking the artifact digest in the signature claims.

cosign verify \
  --certificate-identity-regexp '^https://github.com/acme/payments/' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  registry.example.com/payments@sha256:8c4...e21

Pin the expected workflow identity narrowly. Trusting any workflow in the organization is equivalent to giving every repository release authority.

Enforce at admission

CI verification catches mistakes before deployment. Runtime admission prevents bypasses from other paths.

An admission policy should reject:

  • mutable tags where digests are required;
  • unsigned or unverifiable artifacts;
  • unapproved builders or repositories;
  • provenance whose subject does not match the artifact;
  • builds older than a policy window where required;
  • known critical vulnerabilities under a documented exception process.

Failing closed needs an emergency process. Define who can grant a time-bounded exception, what evidence is recorded, which scope it affects, and how automatic expiry is enforced. A permanent “break glass” allow-all policy becomes the primary path during pressure.

Dependencies belong in the chain

A signed application built from a compromised dependency is still compromised. Generate an SBOM and preserve lockfiles, but distinguish inventory from integrity.

For high-risk components:

  • pin by immutable version or digest;
  • verify package provenance or signatures where available;
  • proxy through an approved registry;
  • retain downloaded artifacts for reproducible investigation;
  • alert when a dependency disappears or changes unexpectedly;
  • rebuild after base-image and toolchain fixes.

Provenance should expose build inputs sufficiently to answer “which releases used this compromised base image?” within minutes.

Design the incident query

During a supply-chain incident, leadership needs bidirectional answers:

source revision → artifacts → environments → tenants
running digest → provenance → builder → source → dependencies

Index release manifests. Retain attestations beyond artifact cleanup. Log admission decisions with policy version and evidence digest.

Test revocation: can you block one builder, repository, certificate identity, or artifact digest without stopping every deployment?

CTO review

  1. Are production workloads pinned by digest?
  2. Is the tested artifact exactly the promoted artifact?
  3. Who generates provenance, and can build steps forge it?
  4. Which builder and source identities does admission trust?
  5. Is provenance verified or merely published?
  6. How are emergency exceptions scoped and expired?
  7. Can one compromised dependency be mapped to running releases?
  8. Can a compromised builder be revoked independently?

Signing says who made a claim. Provenance says what the claim is about. Admission policy turns both into a production control.

References

>