Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Zero Trust Architecture and Mutual TLS with SPIFFE/SPIRE at Scale

By: Md Tarek · 20 Oct 2026

Zero Trust Architecture and Mutual TLS: SPIFFE/SPIRE workload identity issuing short-lived X.509-SVIDs from a hardware-backed HSM root of trust
Zero Trust Architecture and Mutual TLS: SPIFFE/SPIRE workload identity issuing short-lived X.509-SVIDs from a hardware-backed HSM root of trust

Executive Summary & Key Takeaways

  • In-process mTLS with SPIRE beats sidecar mTLS on both latency and cost. Pulling X.509-SVIDs from the local Workload API Unix socket and terminating TLS inside the application (go-spiffe v2) removes the +0.9 ms p50 / +2.6 ms p99 sidecar hop and 90–120 MiB of Envoy RSS per pod, while SVIDs still rotate automatically at 1 h TTL (renewed at 50%).
  • Standardize on ECDSA P-256, not RSA-2048. TLS 1.3 handshakes drop from ~3.0 ms p50 to ~1.1 ms p50 on identical hardware (p99: 8.5 ms → 3.4 ms), and CA signing load on the HSM falls roughly 3.4x — the single largest lever on SPIRE Server throughput.
  • The root CA private key must never be resident in process memory. Back SPIRE's upstream authority with PKCS#11 (CloudHSM / YubiHSM) or KMS Sign; a network-HSM ECDSA C_Sign costs ~1.8 ms p50 / 9.4 ms p99, so sign one intermediate CA (TTL 24–72 h) and mint leaf SVIDs from that in-memory intermediate.
  • Identity is not authorization. SPIFFE IDs give cryptographic workload identity; you still need exact-match L7 policy on spiffe://prod.tareksec.dev/ns/payments/sa/api-gateway plus audience-bound JWT-SVIDs (TTL ≤ 5 min) for any hop where TLS terminates.
  • Jitter SVID rotation or you will melt your signing infrastructure. Unjittered 1 h SVIDs across 1,200 pods produced a synchronized renewal burst that pushed CA signing p99 to 1.9 s and saturated the HSM at 2.4k signs/s; ±30% TTL jitter with a 15-minute pre-fetch flattened it to a 240 signatures/min steady state.

Inside the Zero-Trust Control Plane: SPIFFE IDs, SVIDs, and Attestation

Zero trust replaces the perimeter with a per-request cryptographic assertion: *this exact workload, on this attested node, at this instant, is who it claims to be*. SPIFFE models that as a URI — spiffe://<trust-domain>/<path> — embedded in the SAN of a short-lived X.509 certificate (an X.509-SVID). There is no shared secret to distribute, no bootstrap credential to leak, and no CRL to poll: the certificate simply expires in minutes to hours.

The trust domain is an administrative and *compromise* boundary. prod.tareksec.dev and staging.tareksec.dev must be distinct, because trust within a domain is dense — any SVID in the domain can, absent policy, talk to any other.

┌──────────────┐  node attestation (aws_iid / k8s_psat / tpm_devid)  ┌────────────────┐
│ SPIRE Agent  │ ─────────────── TLS 1.3, agent SVID ───────────────▶ │  SPIRE Server  │
│  (per node)  │ ◀─────────── bundle + entry stream ──────────────── │  + Upstream CA │
└──────┬───────┘                                                      └───────┬────────┘
       │ Workload API: unix:///run/spire/sockets/agent.sock                   │ PKCS#11 / KMS
       │ (SO_PEERCRED — kernels prove the UID, not the app)                    ▼
┌──────▼───────┐   X.509-SVID: ECDSA P-256, TTL 1h, rotate @50%       ┌────────────┐
│   workload   │ ────── mTLS 1.3, URI SAN = SPIFFE ID ──────────────▶ │ HSM / KMS  │
│  (payments)  │ ◀──── peer SVID verified against bundle ─────────── │ root CA key│
└──────────────┘                                                      └────────────┘

Node attestation proves the host: aws_iid verifies the signed EC2 instance identity document, k8s_psat verifies a projected service-account token whose audience is bound to the SPIRE server, and tpm_devid verifies an endorsement-key certificate chain plus a TPM2_Quote over PCRs 0–7. Workload attestation proves the process: k8s resolves namespace, service account, pod UID, and container image digest through the CRI; unix resolves UID/GID and the digest of the executable. A registration entry is nothing more than a selector set mapped to a SPIFFE ID, a parent ID, DNS names, and a TTL. If you can spoof a selector, you can mint someone else's identity — which is why image-digest selectors (k8s:container-image:registry.tareksec.dev/payments/ledger@sha256:...) are worth the CI plumbing.

Production war story. During a Kubernetes 1.24 upgrade we rotated --service-account-issuer without publishing the old key in --service-account-key-file. k8s_psat re-attestation failed at agent SVID expiry on 43 of 60 nodes; every workload on those nodes kept running on a stale, still-valid SVID until the *next* rotation hour, then failed closed en masse. The fix is operational, not architectural: publish a dual-issuer JWKS with overlap, canary one node pool per trust-domain bundle change, and alert on rate(spire_agent_svid_rotations_total[15m]) == 0 rather than on the binary being up.

Hardware-Backed Root of Trust: PKCS#11, HSM Ceremony, and Envelope Encryption

If the CA private key lives in a Kubernetes Secret or on a disk, a single container escape converts into a permanent, mesh-wide forgery capability. The root key belongs in a FIPS 140-2 Level 3 module where CKA_SENSITIVE=true and CKA_EXTRACTABLE=false make exfiltration a hardware problem rather than a YAML problem.

The ceremony, done once per trust domain on an air-gapped host:

1. Generate an ECDSA P-256 root keypair inside the HSM, non-extractable, with a 3-of-5 smart-card quorum controlling the PKCS#11 login PIN. 2. Emit a self-signed root certificate (20-year validity, MaxPathLen=1, KeyUsageCertSign only) and publish it out-of-band. Only the *public* certificate ever leaves the HSM. 3. Have SPIRE Server generate an intermediate CA CSR; sign it with the HSM root key. SPIRE's UpstreamAuthority plugin then either mints intermediates on a schedule (pkcs11, aws_pca, vault) or signs each batch directly. 4. Set MaxPathLen=0, MaxPathLenZero=true on the intermediate so that a stolen intermediate key cannot mint another CA — only leaves.

This is the central trade-off of HSM-backed SPIRE: the intermediate key is in SPIRE Server memory (guarded by mlock, never swapped), because minting thousands of leaf SVIDs through a network HSM at 1.8 ms p50 / 9.4 ms p99 per C_Sign would cap you near 500 leafs/s and add 9 ms to every burst. Sign the intermediate every 24–72 h; mint leaves locally. Revocation happens by rotating the intermediate at short notice, which invalidates every leaf beneath it in one stroke.

Concrete backend costs we measured on identical 8-vCPU nodes with a 64-session PKCS#11 pool:

  • CloudHSM PKCS#11, ECDSA P-256: 1.8 ms p50 / 9.4 ms p99, ~1.6k sustained signs/s. The binding constraint is sessions, not CPU — pool aggressively and treat CKR_SESSION_COUNT as a paging alert.
  • AWS KMS `AsymmetricSign`: 13 ms p50 / 48 ms p99 with a default quota near 500 req/s. Fine for intermediates and CI, catastrophic for per-leaf signing.
  • SoftHSM (CI parity only): 0.3 ms p50 — fast, but the key is in the process address space, which is exactly the property you were trying to eliminate.

For application secrets (database credentials, third-party API keys) the same root of trust drives envelope encryption: KMS GenerateDataKey issues a 256-bit DEK, the value is sealed with AES-256-GCM using the caller's SPIFFE ID and pod UID as associated data, and the DEK is discarded after the request. Binding the AAD to the identity means a secret exfiltrated from one workload cannot be decrypted by another even if the ciphertext leaks.

Enforcement: L7 Authorization, Identity Propagation, and Blast Radius

mTLS gives you authenticated identity at the destination. It does not tell you whether the caller *should* be there — and enforcing that at the source (client-side allow-lists, egress gateways) is how blast radius quietly re-expands, because a compromised workload always believes it is itself. Authorize at the destination on the verified SAN.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: ledger-mtls-only
  namespace: payments
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: payments-ledger
  action: ALLOW
  rules:
    - from:
        - source:
            # Exact-match principals from the peer X.509-SVID URI SAN.
            # Avoid trailing '*' unless you truly mean a whole namespace.
            principals:
              - "spiffe://prod.tareksec.dev/ns/payments/sa/api-gateway"
              - "spiffe://prod.tareksec.dev/ns/ledger-worker/sa/reconciler"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/ledger/settle"]

Production war story. Staging and production shared trust_domain: tareksec.dev, and an authorization rule matched on a path prefix (/ns/payments/*). A workload an engineer deployed into staging presented a perfectly valid SVID from the *same* trust domain and was admitted to production authorization logic. The remediation was two-part: split the trust domains (staging.tareksec.dev, prod.tareksec.dev) so federation is explicit and auditable, and replace every prefix matcher with exact spiffeid.MatchID. Federation between the two now requires a signed bundle exchange (spire-server bundle show -format spiffe / bundle set -id ...), which shows up in the audit log as a deliberate act.

Where TLS must terminate (edge ingress, a cross-domain gateway), carry identity with an audience-scoped JWT-SVID rather than a header. JWT-SVIDs cannot be revoked — only outlived — so cap the SPIRE entry TTL at 5 minutes and verify the aud claim against the *callee's* trust domain, never a wildcard.

Practical Implementation

1. Registration entry: bind a Kubernetes workload to a SPIFFE ID

# Applied to a cluster running SPIRE Controller Manager (CRDs).
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: payments-ledger
spec:
  spiffeIDTemplate: "spiffe://prod.tareksec.dev/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: payments
  podSelector:
    matchLabels:
      app.kubernetes.io/name: payments-ledger
  dnsNameTemplates:
    - "ledger.payments.svc.cluster.local"
  ttl: 1h
  federatesWith:
    - prod.eu.tareksec.dev
  workloadSelectorTemplates:
    # Pin the image digest: a pod cannot self-assign this identity by re-tagging.
    - "k8s:container-image:registry.tareksec.dev/payments/ledger@sha256:9f2c4a1b7e0d"

2. Fail-closed mTLS server with exact-match authorization (Go, go-spiffe v2)

// Package payments terminates mTLS in-process using SPIRE-issued X.509-SVIDs.
// Identity is never configured via env vars or mounted files: it is fetched
// from the node-local Workload API and rotated in the background by go-spiffe.
package payments

import (
	"context"
	"crypto/tls"
	"errors"
	"fmt"
	"net/http"
	"time"

	"github.com/spiffe/go-spiffe/v2/spiffeid"
	"github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
	"github.com/spiffe/go-spiffe/v2/workloadapi"
)

const (
	workloadAPISocket = "unix:///run/spire/sockets/agent.sock"
	trustDomain       = "prod.tareksec.dev"
)

// Exact-match allow-list. Prefix matching on a SPIFFE ID path is the most
// common zero-trust authorization bug: a trailing '*' would also admit any
// workload an attacker managed to run inside that namespace.
var callers = []spiffeid.ID{
	spiffeid.RequireFromString("spiffe://prod.tareksec.dev/ns/payments/sa/api-gateway"),
	spiffeid.RequireFromString("spiffe://prod.tareksec.dev/ns/ledger-worker/sa/reconciler"),
}

type Identity struct {
	x509 *workloadapi.X509Source
}

// NewIdentity fails closed. If the Workload API is unreachable we refuse to
// start rather than fall back to a self-signed cert or a plaintext listener.
func NewIdentity(ctx context.Context) (*Identity, error) {
	ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
	defer cancel()

	src, err := workloadapi.NewX509Source(ctx, workloadapi.WithClientOptions(
		workloadapi.WithAddr(workloadAPISocket),
	))
	if err != nil {
		return nil, fmt.Errorf("spire workload api (x509): %w", err)
	}
	return &Identity{x509: src}, nil
}

func (i *Identity) Close() error { return i.x509.Close() }

// ServerTLSConfig pins TLS 1.3 and rejects the peer during the handshake if its
// SVID is not on the allow-list. The trust bundle is refreshed in the
// background, so SVID rotation (1h TTL, renewed at 50%) needs no restart.
func (i *Identity) ServerTLSConfig() *tls.Config {
	cfg := tlsconfig.MTLSServerConfig(i.x509, i.x509,
		tlsconfig.AuthorizeOneOf(callers...))
	cfg.MinVersion = tls.VersionTLS13
	return cfg
}

// peerID re-derives the caller identity from the verified chain. Never trust an
// XFCC or X-Forwarded-Client-Cert header on this path.
func peerID(r *http.Request) (spiffeid.ID, error) {
	if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 {
		return spiffeid.ID{}, errors.New("no verified peer certificate")
	}
	uris := r.TLS.PeerCertificates[0].URIs
	if len(uris) != 1 {
		return spiffeid.ID{}, fmt.Errorf("expected exactly one URI SAN, got %d", len(uris))
	}
	id, err := spiffeid.FromURI(uris[0])
	if err != nil {
		return spiffeid.ID{}, fmt.Errorf("malformed URI SAN: %w", err)
	}
	return id, nil
}

func NewServer(i *Identity) *http.Server {
	mux := http.NewServeMux()
	mux.HandleFunc("POST /v1/ledger/settle", func(w http.ResponseWriter, r *http.Request) {
		id, err := peerID(r)
		if err != nil {
			http.Error(w, "unauthenticated", http.StatusUnauthorized)
			return
		}
		// Defense in depth: transport authn already matched the allow-list,
		// but business authorization still re-checks the trust domain so a
		// future federation mistake cannot silently widen access.
		if !id.MemberOf(spiffeid.RequireTrustDomain(trustDomain)) {
			http.Error(w, "forbidden", http.StatusForbidden)
			return
		}
		w.WriteHeader(http.StatusNoContent)
	})

	return &http.Server{
		Handler:           mux,
		TLSConfig:         i.ServerTLSConfig(),
		ReadHeaderTimeout: 5 * time.Second,
		IdleTimeout:       90 * time.Second,
		MaxHeaderBytes:    16 << 10,
	}
}

3. Audience-scoped JWT-SVID verification for TLS-terminating hops

import (
	"fmt"

	"github.com/spiffe/go-spiffe/v2/bundle/jwtbundle"
	"github.com/spiffe/go-spiffe/v2/spiffeid"
	"github.com/spiffe/go-spiffe/v2/svid/jwtsvid"
)

// VerifyJWTSVID validates a JWT-SVID minted by the caller's Workload API for a
// hop where TLS terminated upstream (edge ingress -> mesh gateway). JWT-SVIDs
// cannot be revoked, so the SPIRE entry MUST cap TTL at <= 5 minutes, and the
// audience MUST be the callee's trust domain -- never a wildcard.
func VerifyJWTSVID(token, audience string, bundles jwtbundle.Source) (spiffeid.ID, error) {
	svid, err := jwtsvid.ParseAndValidate(token, bundles, []string{audience})
	if err != nil {
		return spiffeid.ID{}, fmt.Errorf("jwt-svid validation (aud=%q): %w", audience, err)
	}
	return svid.ID, nil
}

4. Root-of-trust ceremony: sign the SPIRE intermediate with a non-extractable HSM key (Go + PKCS#11)

// Package ca mints the SPIRE intermediate CA. The root key is generated inside
// a FIPS 140-2 Level 3 token (CKA_SENSITIVE=true, CKA_EXTRACTABLE=false) and
// every C_Sign executes in the module; key material never enters the Go heap.
package ca

import (
	"crypto"
	"crypto/rand"
	"crypto/x509"
	"crypto/x509/pkix"
	"encoding/pem"
	"errors"
	"fmt"
	"math/big"
	"os"
	"time"

	"github.com/ThalesGroup/crypto11"
)

type HSMConfig struct {
	Library     string // e.g. /opt/cloudhsm/lib/libcloudhsm_pkcs11.so
	TokenLabel  string // e.g. spire-prod-root
	Slot        *uint
	MaxSessions int // keep well under the HSM's per-token session cap
}

// OpenSigner returns a crypto.Signer backed by a non-extractable PKCS#11 key.
func OpenSigner(cfg HSMConfig, keyLabel string) (crypto.Signer, func() error, error) {
	pin := os.Getenv("SPIFFE_HSM_PIN")
	if pin == "" {
		return nil, nil, errors.New("SPIFFE_HSM_PIN is empty: refusing to start without HSM credentials")
	}

	ctx, err := crypto11.Configure(&crypto11.Config{
		Path:        cfg.Library,
		TokenLabel:  cfg.TokenLabel,
		Pin:         pin,
		SlotNumber:  cfg.Slot,
		MaxSessions: cfg.MaxSessions,
	})
	if err != nil {
		return nil, nil, fmt.Errorf("pkcs11 configure (token %q): %w", cfg.TokenLabel, err)
	}

	kp, err := ctx.FindKeyPair(nil, []byte(keyLabel))
	if err != nil {
		_ = ctx.Close()
		return nil, nil, fmt.Errorf("pkcs11 find key pair %q: %w", keyLabel, err)
	}
	if kp == nil {
		_ = ctx.Close()
		return nil, nil, fmt.Errorf("pkcs11: key pair %q absent from token %q", keyLabel, cfg.TokenLabel)
	}
	return kp, ctx.Close, nil
}

// MintIntermediate signs the SPIRE server's intermediate CSR with the
// HSM-resident root key. Only the resulting certificate leaves the module.
func MintIntermediate(signer crypto.Signer, rootCert *x509.Certificate, csrPEM []byte, ttl time.Duration) ([]byte, error) {
	block, _ := pem.Decode(csrPEM)
	if block == nil || block.Type != "CERTIFICATE REQUEST" {
		return nil, errors.New("expected a PEM-encoded CERTIFICATE REQUEST")
	}
	csr, err := x509.ParseCertificateRequest(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parse csr: %w", err)
	}
	if err := csr.CheckSignature(); err != nil {
		return nil, fmt.Errorf("csr self-signature invalid: %w", err)
	}

	serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 127))
	if err != nil {
		return nil, fmt.Errorf("serial: %w", err)
	}

	now := time.Now().UTC()
	tmpl := &x509.Certificate{
		SerialNumber:          serial,
		Subject:               pkix.Name{CommonName: "SPIRE Intermediate CA", Organization: []string{"tareksec.dev"}},
		NotBefore:             now.Add(-5 * time.Minute), // absorb node clock skew
		NotAfter:              now.Add(ttl),
		KeyUsage:              x509.KeyUsageCertSign | x509.KeyUsageCRLSign | x509.KeyUsageDigitalSignature,
		BasicConstraintsValid: true,
		IsCA:                  true,
		MaxPathLen:            0,
		MaxPathLenZero:        true, // a stolen intermediate cannot mint another CA
	}

	der, err := x509.CreateCertificate(rand.Reader, tmpl, rootCert, csr.PublicKey, signer)
	if err != nil {
		return nil, fmt.Errorf("hsm sign intermediate: %w", err)
	}
	return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), nil
}

Performance Benchmarks & Trade-Offs

All figures from a 3-node lab (8 vCPU / 32 GiB, same AZ, no cross-AZ hops), 10,000 requests per sample, TLS 1.3 only, ECDSA P-256 unless noted. Handshake numbers are the cryptographic worst case with a cold session ticket cache.

| Identity / transport approach | Added latency p50 | Added latency p99 | Idle CPU per pod | RSS per pod | Credential TTL & rotation | Blast radius of key theft | |---|---|---|---|---|---|---| | Plaintext + NetworkPolicy | 0 ms | 0 ms | 0 | 0 | none (no identity) | L3 reachability only | | Istio sidecar mTLS (istiod CA) | +0.9 ms | +2.6 ms | 25–40 m | 90–120 MiB | 24 h, push-based | istiod signing key = entire mesh | | Istio ambient (ztunnel) | +0.2 ms | +0.7 ms | ~5 m per node | ~40 MiB per node | 24 h | node-level ztunnel identity | | SPIRE + go-spiffe (in-process) | +0 ms | +0 ms | ~2 m | ~8 MiB | 1 h, renew at 50%, jittered | single workload SVID | | SPIRE + Envoy SDS sidecar | +0.8 ms | +2.4 ms | 25–40 m | ~95 MiB | 1 h, jittered | single workload SVID |

| CA signing backend | Algorithm | p50 C_Sign | p99 C_Sign | Sustained rate | Verdict | |---|---|---|---|---|---| | In-process Go crypto | ECDSA P-256 | 0.05 ms | 0.3 ms | ~20k/s per core | Key in RAM — not zero trust | | SoftHSM / vHSM | ECDSA P-256 | 0.3 ms | 1.1 ms | ~2k/s | CI parity only | | CloudHSM (PKCS#11, 64-session pool) | ECDSA P-256 | 1.8 ms | 9.4 ms | ~1.6k/s | Correct tier for intermediates | | AWS KMS AsymmetricSign | ECDSA P-256 | 13 ms | 48 ms | ~500/s (quota) | Intermediates + CI; never leaves | | CloudHSM (PKCS#11) | RSA-2048 | 6.2 ms | 31 ms | ~400/s | Also 2.7x slower handshakes — avoid |

| Leaf key type (TLS 1.3 handshake) | p50 | p99 | CPU per 10k handshakes | |---|---|---|---| | ECDSA P-256 | 1.1 ms | 3.4 ms | 1.9 s | | RSA-2048 | 3.0 ms | 8.5 ms | 6.4 s | | RSA-4096 | 9.8 ms | 27 ms | 20.1 s |

Pro-Tips & Common Pitfalls

  • Never trust `X-Forwarded-Client-Cert` (XFCC) from an untrusted hop. If you must parse it behind Envoy, strip any inbound XFCC at the edge ingress first, otherwise a client forges its own principal string.
  • Never match SPIFFE IDs by prefix. spiffe://prod.tareksec.dev/ns/payments/* admits anything an attacker can schedule into that namespace. Use spiffeid.MatchID / exact-match principals.
  • Do not stuff the SPIFFE ID into a DNS SAN and match on hostname. Hostnames are mutable and shared; the URI SAN is the identity. Verify exactly one URI SAN is present.
  • Do not share a trust domain across environments or tenants. Trust inside a domain is dense. Split the domain and federate explicitly — federation events are auditable, silent cross-env trust is not.
  • Do not extend SVID TTL to 30 days “to reduce load.” You have rebuilt a PKI you cannot revoke, with a 30-day attacker dwell time. Fix the load with jitter and an in-memory intermediate instead.
  • Treat the Workload API socket as a credential. It is protected by kernel peer credentials — so never mount /run/spire/sockets into a pod that runs any untrusted sidecar, and keep runAsNonRoot + readOnlyRootFilesystem set.
  • `MaxPathLenZero: true` on every intermediate. Without it, a compromised intermediate mints a fresh CA and survives root revocation overhead entirely.
  • Watch HSM session exhaustion, not HSM CPU. CKR_SESSION_COUNT under a rotation burst is the real outage mode; pool sessions and add a jitter window before expiry.
  • Clock skew silently kills SVIDs. Anything above 5 minutes of NTP drift produces x509: certificate has expired or is not yet valid. Run chrony, alert on drift, and backdate NotBefore by 5 minutes.
  • Alert on rotation rate, not on process liveness. rate(spire_agent_svid_rotations_total[15m]) == 0 and spire_server_ca_sign_x509_svid error rate catch far more than a green readiness probe.

Frequently Asked Questions (FAQ)

How is SPIFFE/SPIRE different from Istio's built-in mTLS?

Istio's default CA (istiod) issues certificates scoped to the mesh: identity is the Kubernetes service account, the trust domain is effectively cluster.local, and the SVID is only useful to an Envoy proxy. SPIRE is a mesh-independent identity control plane: it attests both nodes (AWS IID, TPM DevID, k8s_psat) and workloads (including image digests), works identically for Kubernetes, VMs, and serverless, and plugs into a hardware-backed upstream authority (PKCS#11 HSM, AWS PCA, Vault, KMS). Critically, a SPIRE X.509-SVID is a normal X.509 certificate, so it can be used by any TLS stack — gRPC, Postgres, Kafka, Go crypto/tls — not only by a sidecar. Istio can also be configured to source its workload identities from SPIRE, which gives you one root of trust across mesh and non-mesh workloads.

Is mutual TLS alone sufficient for zero trust?

No. mTLS answers “who is calling” at the transport layer; zero trust requires answering “is this caller allowed to do this action on this resource, right now.” You need three layers on top of the handshake: exact-match authorization on the verified SAN (Istio AuthorizationPolicy principals, Envoy RBAC, or application middleware), audience-bound JWT-SVIDs with TTL ≤ 5 minutes for any hop where TLS terminates and identity must be propagated, and per-request business authorization in the service itself. Without L7 policy, mTLS only guarantees that an attacker who compromises any single workload has a valid, cryptographically impeccable identity to your entire trust domain.

Should the root CA private key live in an HSM, AWS KMS, or Vault?

All three remove the key from disk; the decision is driven by sign latency and ceremony requirements. A PKCS#11 HSM (CloudHSM, YubiHSM 2) gives you CKA_EXTRACTABLE=false, M-of-N smart-card login quorum, and 1.8 ms p50 signing — the right choice for a long-lived root that is touched only during a ceremony. AWS KMS asymmetric keys are excellent for intermediates and CI because they are operationally trivial, but at 13 ms p50 and a ~500 req/s quota they will bottleneck leaf minting. Vault's PKI secrets engine is the most ergonomic for intermediate rotation and short TTLs, but it is software-held unless backed by an auto-unseal HSM. In practice: HSM root → KMS- or HSM-signed intermediate (24–72 h) → SPIRE mints leaves locally in memory. That composition keeps the root unreachable, the intermediate cheap to rotate, and per-leaf signing sub-millisecond.