Zero Trust Architecture and Mutual TLS with SPIFFE/SPIRE at Scale
By: Md Tarek · 20 Oct 2026
Executive Summary & Key Takeaways
- In-process mTLS with SPIRE beats sidecar mTLS on both latency and cost. Pulling X.509-SVIDs from the local Workload API Unix socket and terminating TLS inside the application (go-spiffe v2) removes the +0.9 ms p50 / +2.6 ms p99 sidecar hop and 90–120 MiB of Envoy RSS per pod, while SVIDs still rotate automatically at 1 h TTL (renewed at 50%).
- Standardize on ECDSA P-256, not RSA-2048. TLS 1.3 handshakes drop from ~3.0 ms p50 to ~1.1 ms p50 on identical hardware (p99: 8.5 ms → 3.4 ms), and CA signing load on the HSM falls roughly 3.4x — the single largest lever on SPIRE Server throughput.
- The root CA private key must never be resident in process memory. Back SPIRE's upstream authority with PKCS#11 (CloudHSM / YubiHSM) or KMS
Sign; a network-HSM ECDSAC_Signcosts ~1.8 ms p50 / 9.4 ms p99, so sign one intermediate CA (TTL 24–72 h) and mint leaf SVIDs from that in-memory intermediate. - Identity is not authorization. SPIFFE IDs give cryptographic workload identity; you still need exact-match L7 policy on
spiffe://prod.tareksec.dev/ns/payments/sa/api-gatewayplus audience-bound JWT-SVIDs (TTL ≤ 5 min) for any hop where TLS terminates. - Jitter SVID rotation or you will melt your signing infrastructure. Unjittered 1 h SVIDs across 1,200 pods produced a synchronized renewal burst that pushed CA signing p99 to 1.9 s and saturated the HSM at 2.4k signs/s; ±30% TTL jitter with a 15-minute pre-fetch flattened it to a 240 signatures/min steady state.
Inside the Zero-Trust Control Plane: SPIFFE IDs, SVIDs, and Attestation
Zero trust replaces the perimeter with a per-request cryptographic assertion: *this exact workload, on this attested node, at this instant, is who it claims to be*. SPIFFE models that as a URI — spiffe://<trust-domain>/<path> — embedded in the SAN of a short-lived X.509 certificate (an X.509-SVID). There is no shared secret to distribute, no bootstrap credential to leak, and no CRL to poll: the certificate simply expires in minutes to hours.
The trust domain is an administrative and *compromise* boundary. prod.tareksec.dev and staging.tareksec.dev must be distinct, because trust within a domain is dense — any SVID in the domain can, absent policy, talk to any other.
┌──────────────┐ node attestation (aws_iid / k8s_psat / tpm_devid) ┌────────────────┐
│ SPIRE Agent │ ─────────────── TLS 1.3, agent SVID ───────────────▶ │ SPIRE Server │
│ (per node) │ ◀─────────── bundle + entry stream ──────────────── │ + Upstream CA │
└──────┬───────┘ └───────┬────────┘
│ Workload API: unix:///run/spire/sockets/agent.sock │ PKCS#11 / KMS
│ (SO_PEERCRED — kernels prove the UID, not the app) ▼
┌──────▼───────┐ X.509-SVID: ECDSA P-256, TTL 1h, rotate @50% ┌────────────┐
│ workload │ ────── mTLS 1.3, URI SAN = SPIFFE ID ──────────────▶ │ HSM / KMS │
│ (payments) │ ◀──── peer SVID verified against bundle ─────────── │ root CA key│
└──────────────┘ └────────────┘Node attestation proves the host: aws_iid verifies the signed EC2 instance identity document, k8s_psat verifies a projected service-account token whose audience is bound to the SPIRE server, and tpm_devid verifies an endorsement-key certificate chain plus a TPM2_Quote over PCRs 0–7. Workload attestation proves the process: k8s resolves namespace, service account, pod UID, and container image digest through the CRI; unix resolves UID/GID and the digest of the executable. A registration entry is nothing more than a selector set mapped to a SPIFFE ID, a parent ID, DNS names, and a TTL. If you can spoof a selector, you can mint someone else's identity — which is why image-digest selectors (k8s:container-image:registry.tareksec.dev/payments/ledger@sha256:...) are worth the CI plumbing.
Production war story. During a Kubernetes 1.24 upgrade we rotated --service-account-issuer without publishing the old key in --service-account-key-file. k8s_psat re-attestation failed at agent SVID expiry on 43 of 60 nodes; every workload on those nodes kept running on a stale, still-valid SVID until the *next* rotation hour, then failed closed en masse. The fix is operational, not architectural: publish a dual-issuer JWKS with overlap, canary one node pool per trust-domain bundle change, and alert on rate(spire_agent_svid_rotations_total[15m]) == 0 rather than on the binary being up.
Hardware-Backed Root of Trust: PKCS#11, HSM Ceremony, and Envelope Encryption
If the CA private key lives in a Kubernetes Secret or on a disk, a single container escape converts into a permanent, mesh-wide forgery capability. The root key belongs in a FIPS 140-2 Level 3 module where CKA_SENSITIVE=true and CKA_EXTRACTABLE=false make exfiltration a hardware problem rather than a YAML problem.
The ceremony, done once per trust domain on an air-gapped host:
1. Generate an ECDSA P-256 root keypair inside the HSM, non-extractable, with a 3-of-5 smart-card quorum controlling the PKCS#11 login PIN. 2. Emit a self-signed root certificate (20-year validity, MaxPathLen=1, KeyUsageCertSign only) and publish it out-of-band. Only the *public* certificate ever leaves the HSM. 3. Have SPIRE Server generate an intermediate CA CSR; sign it with the HSM root key. SPIRE's UpstreamAuthority plugin then either mints intermediates on a schedule (pkcs11, aws_pca, vault) or signs each batch directly. 4. Set MaxPathLen=0, MaxPathLenZero=true on the intermediate so that a stolen intermediate key cannot mint another CA — only leaves.
This is the central trade-off of HSM-backed SPIRE: the intermediate key is in SPIRE Server memory (guarded by mlock, never swapped), because minting thousands of leaf SVIDs through a network HSM at 1.8 ms p50 / 9.4 ms p99 per C_Sign would cap you near 500 leafs/s and add 9 ms to every burst. Sign the intermediate every 24–72 h; mint leaves locally. Revocation happens by rotating the intermediate at short notice, which invalidates every leaf beneath it in one stroke.
Concrete backend costs we measured on identical 8-vCPU nodes with a 64-session PKCS#11 pool:
- CloudHSM PKCS#11, ECDSA P-256: 1.8 ms p50 / 9.4 ms p99, ~1.6k sustained signs/s. The binding constraint is sessions, not CPU — pool aggressively and treat
CKR_SESSION_COUNTas a paging alert. - AWS KMS `AsymmetricSign`: 13 ms p50 / 48 ms p99 with a default quota near 500 req/s. Fine for intermediates and CI, catastrophic for per-leaf signing.
- SoftHSM (CI parity only): 0.3 ms p50 — fast, but the key is in the process address space, which is exactly the property you were trying to eliminate.
For application secrets (database credentials, third-party API keys) the same root of trust drives envelope encryption: KMS GenerateDataKey issues a 256-bit DEK, the value is sealed with AES-256-GCM using the caller's SPIFFE ID and pod UID as associated data, and the DEK is discarded after the request. Binding the AAD to the identity means a secret exfiltrated from one workload cannot be decrypted by another even if the ciphertext leaks.
Enforcement: L7 Authorization, Identity Propagation, and Blast Radius
mTLS gives you authenticated identity at the destination. It does not tell you whether the caller *should* be there — and enforcing that at the source (client-side allow-lists, egress gateways) is how blast radius quietly re-expands, because a compromised workload always believes it is itself. Authorize at the destination on the verified SAN.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: ledger-mtls-only
namespace: payments
spec:
selector:
matchLabels:
app.kubernetes.io/name: payments-ledger
action: ALLOW
rules:
- from:
- source:
# Exact-match principals from the peer X.509-SVID URI SAN.
# Avoid trailing '*' unless you truly mean a whole namespace.
principals:
- "spiffe://prod.tareksec.dev/ns/payments/sa/api-gateway"
- "spiffe://prod.tareksec.dev/ns/ledger-worker/sa/reconciler"
to:
- operation:
methods: ["POST"]
paths: ["/v1/ledger/settle"]Production war story. Staging and production shared trust_domain: tareksec.dev, and an authorization rule matched on a path prefix (/ns/payments/*). A workload an engineer deployed into staging presented a perfectly valid SVID from the *same* trust domain and was admitted to production authorization logic. The remediation was two-part: split the trust domains (staging.tareksec.dev, prod.tareksec.dev) so federation is explicit and auditable, and replace every prefix matcher with exact spiffeid.MatchID. Federation between the two now requires a signed bundle exchange (spire-server bundle show -format spiffe / bundle set -id ...), which shows up in the audit log as a deliberate act.
Where TLS must terminate (edge ingress, a cross-domain gateway), carry identity with an audience-scoped JWT-SVID rather than a header. JWT-SVIDs cannot be revoked — only outlived — so cap the SPIRE entry TTL at 5 minutes and verify the aud claim against the *callee's* trust domain, never a wildcard.
Practical Implementation
1. Registration entry: bind a Kubernetes workload to a SPIFFE ID
# Applied to a cluster running SPIRE Controller Manager (CRDs).
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
name: payments-ledger
spec:
spiffeIDTemplate: "spiffe://prod.tareksec.dev/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: payments
podSelector:
matchLabels:
app.kubernetes.io/name: payments-ledger
dnsNameTemplates:
- "ledger.payments.svc.cluster.local"
ttl: 1h
federatesWith:
- prod.eu.tareksec.dev
workloadSelectorTemplates:
# Pin the image digest: a pod cannot self-assign this identity by re-tagging.
- "k8s:container-image:registry.tareksec.dev/payments/ledger@sha256:9f2c4a1b7e0d"2. Fail-closed mTLS server with exact-match authorization (Go, go-spiffe v2)
// Package payments terminates mTLS in-process using SPIRE-issued X.509-SVIDs.
// Identity is never configured via env vars or mounted files: it is fetched
// from the node-local Workload API and rotated in the background by go-spiffe.
package payments
import (
"context"
"crypto/tls"
"errors"
"fmt"
"net/http"
"time"
"github.com/spiffe/go-spiffe/v2/spiffeid"
"github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
"github.com/spiffe/go-spiffe/v2/workloadapi"
)
const (
workloadAPISocket = "unix:///run/spire/sockets/agent.sock"
trustDomain = "prod.tareksec.dev"
)
// Exact-match allow-list. Prefix matching on a SPIFFE ID path is the most
// common zero-trust authorization bug: a trailing '*' would also admit any
// workload an attacker managed to run inside that namespace.
var callers = []spiffeid.ID{
spiffeid.RequireFromString("spiffe://prod.tareksec.dev/ns/payments/sa/api-gateway"),
spiffeid.RequireFromString("spiffe://prod.tareksec.dev/ns/ledger-worker/sa/reconciler"),
}
type Identity struct {
x509 *workloadapi.X509Source
}
// NewIdentity fails closed. If the Workload API is unreachable we refuse to
// start rather than fall back to a self-signed cert or a plaintext listener.
func NewIdentity(ctx context.Context) (*Identity, error) {
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
defer cancel()
src, err := workloadapi.NewX509Source(ctx, workloadapi.WithClientOptions(
workloadapi.WithAddr(workloadAPISocket),
))
if err != nil {
return nil, fmt.Errorf("spire workload api (x509): %w", err)
}
return &Identity{x509: src}, nil
}
func (i *Identity) Close() error { return i.x509.Close() }
// ServerTLSConfig pins TLS 1.3 and rejects the peer during the handshake if its
// SVID is not on the allow-list. The trust bundle is refreshed in the
// background, so SVID rotation (1h TTL, renewed at 50%) needs no restart.
func (i *Identity) ServerTLSConfig() *tls.Config {
cfg := tlsconfig.MTLSServerConfig(i.x509, i.x509,
tlsconfig.AuthorizeOneOf(callers...))
cfg.MinVersion = tls.VersionTLS13
return cfg
}
// peerID re-derives the caller identity from the verified chain. Never trust an
// XFCC or X-Forwarded-Client-Cert header on this path.
func peerID(r *http.Request) (spiffeid.ID, error) {
if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 {
return spiffeid.ID{}, errors.New("no verified peer certificate")
}
uris := r.TLS.PeerCertificates[0].URIs
if len(uris) != 1 {
return spiffeid.ID{}, fmt.Errorf("expected exactly one URI SAN, got %d", len(uris))
}
id, err := spiffeid.FromURI(uris[0])
if err != nil {
return spiffeid.ID{}, fmt.Errorf("malformed URI SAN: %w", err)
}
return id, nil
}
func NewServer(i *Identity) *http.Server {
mux := http.NewServeMux()
mux.HandleFunc("POST /v1/ledger/settle", func(w http.ResponseWriter, r *http.Request) {
id, err := peerID(r)
if err != nil {
http.Error(w, "unauthenticated", http.StatusUnauthorized)
return
}
// Defense in depth: transport authn already matched the allow-list,
// but business authorization still re-checks the trust domain so a
// future federation mistake cannot silently widen access.
if !id.MemberOf(spiffeid.RequireTrustDomain(trustDomain)) {
http.Error(w, "forbidden", http.StatusForbidden)
return
}
w.WriteHeader(http.StatusNoContent)
})
return &http.Server{
Handler: mux,
TLSConfig: i.ServerTLSConfig(),
ReadHeaderTimeout: 5 * time.Second,
IdleTimeout: 90 * time.Second,
MaxHeaderBytes: 16 << 10,
}
}3. Audience-scoped JWT-SVID verification for TLS-terminating hops
import (
"fmt"
"github.com/spiffe/go-spiffe/v2/bundle/jwtbundle"
"github.com/spiffe/go-spiffe/v2/spiffeid"
"github.com/spiffe/go-spiffe/v2/svid/jwtsvid"
)
// VerifyJWTSVID validates a JWT-SVID minted by the caller's Workload API for a
// hop where TLS terminated upstream (edge ingress -> mesh gateway). JWT-SVIDs
// cannot be revoked, so the SPIRE entry MUST cap TTL at <= 5 minutes, and the
// audience MUST be the callee's trust domain -- never a wildcard.
func VerifyJWTSVID(token, audience string, bundles jwtbundle.Source) (spiffeid.ID, error) {
svid, err := jwtsvid.ParseAndValidate(token, bundles, []string{audience})
if err != nil {
return spiffeid.ID{}, fmt.Errorf("jwt-svid validation (aud=%q): %w", audience, err)
}
return svid.ID, nil
}4. Root-of-trust ceremony: sign the SPIRE intermediate with a non-extractable HSM key (Go + PKCS#11)
// Package ca mints the SPIRE intermediate CA. The root key is generated inside
// a FIPS 140-2 Level 3 token (CKA_SENSITIVE=true, CKA_EXTRACTABLE=false) and
// every C_Sign executes in the module; key material never enters the Go heap.
package ca
import (
"crypto"
"crypto/rand"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"errors"
"fmt"
"math/big"
"os"
"time"
"github.com/ThalesGroup/crypto11"
)
type HSMConfig struct {
Library string // e.g. /opt/cloudhsm/lib/libcloudhsm_pkcs11.so
TokenLabel string // e.g. spire-prod-root
Slot *uint
MaxSessions int // keep well under the HSM's per-token session cap
}
// OpenSigner returns a crypto.Signer backed by a non-extractable PKCS#11 key.
func OpenSigner(cfg HSMConfig, keyLabel string) (crypto.Signer, func() error, error) {
pin := os.Getenv("SPIFFE_HSM_PIN")
if pin == "" {
return nil, nil, errors.New("SPIFFE_HSM_PIN is empty: refusing to start without HSM credentials")
}
ctx, err := crypto11.Configure(&crypto11.Config{
Path: cfg.Library,
TokenLabel: cfg.TokenLabel,
Pin: pin,
SlotNumber: cfg.Slot,
MaxSessions: cfg.MaxSessions,
})
if err != nil {
return nil, nil, fmt.Errorf("pkcs11 configure (token %q): %w", cfg.TokenLabel, err)
}
kp, err := ctx.FindKeyPair(nil, []byte(keyLabel))
if err != nil {
_ = ctx.Close()
return nil, nil, fmt.Errorf("pkcs11 find key pair %q: %w", keyLabel, err)
}
if kp == nil {
_ = ctx.Close()
return nil, nil, fmt.Errorf("pkcs11: key pair %q absent from token %q", keyLabel, cfg.TokenLabel)
}
return kp, ctx.Close, nil
}
// MintIntermediate signs the SPIRE server's intermediate CSR with the
// HSM-resident root key. Only the resulting certificate leaves the module.
func MintIntermediate(signer crypto.Signer, rootCert *x509.Certificate, csrPEM []byte, ttl time.Duration) ([]byte, error) {
block, _ := pem.Decode(csrPEM)
if block == nil || block.Type != "CERTIFICATE REQUEST" {
return nil, errors.New("expected a PEM-encoded CERTIFICATE REQUEST")
}
csr, err := x509.ParseCertificateRequest(block.Bytes)
if err != nil {
return nil, fmt.Errorf("parse csr: %w", err)
}
if err := csr.CheckSignature(); err != nil {
return nil, fmt.Errorf("csr self-signature invalid: %w", err)
}
serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 127))
if err != nil {
return nil, fmt.Errorf("serial: %w", err)
}
now := time.Now().UTC()
tmpl := &x509.Certificate{
SerialNumber: serial,
Subject: pkix.Name{CommonName: "SPIRE Intermediate CA", Organization: []string{"tareksec.dev"}},
NotBefore: now.Add(-5 * time.Minute), // absorb node clock skew
NotAfter: now.Add(ttl),
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign | x509.KeyUsageDigitalSignature,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: 0,
MaxPathLenZero: true, // a stolen intermediate cannot mint another CA
}
der, err := x509.CreateCertificate(rand.Reader, tmpl, rootCert, csr.PublicKey, signer)
if err != nil {
return nil, fmt.Errorf("hsm sign intermediate: %w", err)
}
return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), nil
}Performance Benchmarks & Trade-Offs
All figures from a 3-node lab (8 vCPU / 32 GiB, same AZ, no cross-AZ hops), 10,000 requests per sample, TLS 1.3 only, ECDSA P-256 unless noted. Handshake numbers are the cryptographic worst case with a cold session ticket cache.
| Identity / transport approach | Added latency p50 | Added latency p99 | Idle CPU per pod | RSS per pod | Credential TTL & rotation | Blast radius of key theft | |---|---|---|---|---|---|---| | Plaintext + NetworkPolicy | 0 ms | 0 ms | 0 | 0 | none (no identity) | L3 reachability only | | Istio sidecar mTLS (istiod CA) | +0.9 ms | +2.6 ms | 25–40 m | 90–120 MiB | 24 h, push-based | istiod signing key = entire mesh | | Istio ambient (ztunnel) | +0.2 ms | +0.7 ms | ~5 m per node | ~40 MiB per node | 24 h | node-level ztunnel identity | | SPIRE + go-spiffe (in-process) | +0 ms | +0 ms | ~2 m | ~8 MiB | 1 h, renew at 50%, jittered | single workload SVID | | SPIRE + Envoy SDS sidecar | +0.8 ms | +2.4 ms | 25–40 m | ~95 MiB | 1 h, jittered | single workload SVID |
| CA signing backend | Algorithm | p50 C_Sign | p99 C_Sign | Sustained rate | Verdict | |---|---|---|---|---|---| | In-process Go crypto | ECDSA P-256 | 0.05 ms | 0.3 ms | ~20k/s per core | Key in RAM — not zero trust | | SoftHSM / vHSM | ECDSA P-256 | 0.3 ms | 1.1 ms | ~2k/s | CI parity only | | CloudHSM (PKCS#11, 64-session pool) | ECDSA P-256 | 1.8 ms | 9.4 ms | ~1.6k/s | Correct tier for intermediates | | AWS KMS AsymmetricSign | ECDSA P-256 | 13 ms | 48 ms | ~500/s (quota) | Intermediates + CI; never leaves | | CloudHSM (PKCS#11) | RSA-2048 | 6.2 ms | 31 ms | ~400/s | Also 2.7x slower handshakes — avoid |
| Leaf key type (TLS 1.3 handshake) | p50 | p99 | CPU per 10k handshakes | |---|---|---|---| | ECDSA P-256 | 1.1 ms | 3.4 ms | 1.9 s | | RSA-2048 | 3.0 ms | 8.5 ms | 6.4 s | | RSA-4096 | 9.8 ms | 27 ms | 20.1 s |
Pro-Tips & Common Pitfalls
- Never trust `X-Forwarded-Client-Cert` (XFCC) from an untrusted hop. If you must parse it behind Envoy, strip any inbound XFCC at the edge ingress first, otherwise a client forges its own principal string.
- Never match SPIFFE IDs by prefix.
spiffe://prod.tareksec.dev/ns/payments/*admits anything an attacker can schedule into that namespace. Usespiffeid.MatchID/ exact-match principals. - Do not stuff the SPIFFE ID into a DNS SAN and match on hostname. Hostnames are mutable and shared; the URI SAN is the identity. Verify exactly one URI SAN is present.
- Do not share a trust domain across environments or tenants. Trust inside a domain is dense. Split the domain and federate explicitly — federation events are auditable, silent cross-env trust is not.
- Do not extend SVID TTL to 30 days “to reduce load.” You have rebuilt a PKI you cannot revoke, with a 30-day attacker dwell time. Fix the load with jitter and an in-memory intermediate instead.
- Treat the Workload API socket as a credential. It is protected by kernel peer credentials — so never mount
/run/spire/socketsinto a pod that runs any untrusted sidecar, and keeprunAsNonRoot+readOnlyRootFilesystemset. - `MaxPathLenZero: true` on every intermediate. Without it, a compromised intermediate mints a fresh CA and survives root revocation overhead entirely.
- Watch HSM session exhaustion, not HSM CPU.
CKR_SESSION_COUNTunder a rotation burst is the real outage mode; pool sessions and add a jitter window before expiry. - Clock skew silently kills SVIDs. Anything above 5 minutes of NTP drift produces
x509: certificate has expired or is not yet valid. Run chrony, alert on drift, and backdateNotBeforeby 5 minutes. - Alert on rotation rate, not on process liveness.
rate(spire_agent_svid_rotations_total[15m]) == 0andspire_server_ca_sign_x509_sviderror rate catch far more than a green readiness probe.
Frequently Asked Questions (FAQ)
How is SPIFFE/SPIRE different from Istio's built-in mTLS?
Istio's default CA (istiod) issues certificates scoped to the mesh: identity is the Kubernetes service account, the trust domain is effectively cluster.local, and the SVID is only useful to an Envoy proxy. SPIRE is a mesh-independent identity control plane: it attests both nodes (AWS IID, TPM DevID, k8s_psat) and workloads (including image digests), works identically for Kubernetes, VMs, and serverless, and plugs into a hardware-backed upstream authority (PKCS#11 HSM, AWS PCA, Vault, KMS). Critically, a SPIRE X.509-SVID is a normal X.509 certificate, so it can be used by any TLS stack — gRPC, Postgres, Kafka, Go crypto/tls — not only by a sidecar. Istio can also be configured to source its workload identities from SPIRE, which gives you one root of trust across mesh and non-mesh workloads.
Is mutual TLS alone sufficient for zero trust?
No. mTLS answers “who is calling” at the transport layer; zero trust requires answering “is this caller allowed to do this action on this resource, right now.” You need three layers on top of the handshake: exact-match authorization on the verified SAN (Istio AuthorizationPolicy principals, Envoy RBAC, or application middleware), audience-bound JWT-SVIDs with TTL ≤ 5 minutes for any hop where TLS terminates and identity must be propagated, and per-request business authorization in the service itself. Without L7 policy, mTLS only guarantees that an attacker who compromises any single workload has a valid, cryptographically impeccable identity to your entire trust domain.
Should the root CA private key live in an HSM, AWS KMS, or Vault?
All three remove the key from disk; the decision is driven by sign latency and ceremony requirements. A PKCS#11 HSM (CloudHSM, YubiHSM 2) gives you CKA_EXTRACTABLE=false, M-of-N smart-card login quorum, and 1.8 ms p50 signing — the right choice for a long-lived root that is touched only during a ceremony. AWS KMS asymmetric keys are excellent for intermediates and CI because they are operationally trivial, but at 13 ms p50 and a ~500 req/s quota they will bottleneck leaf minting. Vault's PKI secrets engine is the most ergonomic for intermediate rotation and short TTLs, but it is software-held unless backed by an auto-unseal HSM. In practice: HSM root → KMS- or HSM-signed intermediate (24–72 h) → SPIRE mints leaves locally in memory. That composition keeps the root unreachable, the intermediate cheap to rotate, and per-leaf signing sub-millisecond.
