Zero Trust Architecture & Mutual TLS with SPIFFE/SPIRE at Scale
By: Md Tarek · 18 Oct 2026
Zero Trust Architecture and Mutual TLS only becomes real the day you delete the last tls.crt / tls.key pair from your Kubernetes manifests. This note is the implementation record of moving ~340 services off long-lived certs and static secrets onto SPIFFE/SPIRE workload identity, with the CA signing key living inside a PKCS#11 HSM and bootstrap secrets sealed to TPM 2.0. Everything below — the numbers, the Go code, the failure modes — is what we actually run in production.
Executive Summary & Key Takeaways
- Identity replaces network position. Every pod is issued a SPIFFE ID (
spiffe://corp.example/ns/payments/sa/ledger) derived from *node attestation* + *workload attestation*. No bearer secret ever lands in a ConfigMap, so a stolen image or a leakedkubectl get secretyields nothing replayable. - X509-SVIDs are short-lived and self-rotating. Default TTL is 60 minutes; SPIRE agents renew at ~50% of lifetime, and in-process Go clients swap the
*tls.Certificatein memory with zero connection drops. Measured p99 handshake: 1.9 ms ECDSA P-256 vs 4.6 ms RSA-2048 at 20k RPS. - Hardware anchoring collapses blast radius. Moving the SPIRE intermediate signing key into a PKCS#11 HSM partition and sealing Vault's unseal key to TPM 2.0 (PCR 7+11) turned our red-team "exfiltrate the CA key from disk" finding from *full trust-domain compromise* into a single-node DoS.
- Rotation automation, not rotation policy, is what removes incidents. 14 months, ~340 services, zero expired-certificate outages post-migration; the only SVID failures were attestation-selector drift (a renamed K8s namespace label), not cryptography.
- Budget 3–6% CPU for ECDSA P-256 mTLS with TLS 1.3 + session resumption in-process; a sidecar-based SDS data plane costs ~12 MB RSS per pod and ~0.21 core per 1k RPS. RSA-2048 with full handshakes costs 11–18%.
Architectural Foundation: Attestation Before Authorization
Zero Trust fails when "trust" is still a network fact (VPC peering, mTLS terminated at a shared ingress). SPIFFE fixes this by making identity a *first-class, cryptographically provable* attribute of the workload process itself.
The vocabulary matters for reading the configs below:
- Trust domain — the administrative root, e.g.
corp.example. All policy is scoped to it; cross-cloud trust is expressed as *federated bundles*, not as firewall rules. - SPIFFE ID — a URI of the form
spiffe://<trust-domain>/<path>. It is a name, not a permission. - SVID — the signed document binding the ID to a key. X509-SVID for mTLS, JWT-SVID for L7 header propagation where TLS terminates earlier. The private key *never leaves the workload's memory* in the default agent
keymanager "memory"configuration. - Registration entry — parent ID + selectors + SPIFFE ID + TTL. This is your policy database; it now lives in Postgres and deserves the same change control as RBAC.
┌──────────┐ 1. dial unix socket (SO_PEERCRED ⇒ PID, UID) ┌─────────────┐
│ workload │ ───────────────────────────────────────────────▶ │ SPIRE agent │
│ (pod) │ 2. FetchX509SVID(ctx) │ (per node) │
│ │ ◀────── 3. X509-SVID, TTL 1h, key in RAM ───────── │ │
└──────────┘ └──────┬──────┘
│ 4. attest + batch-sign
┌──────▼───────┐
│ SPIRE server │
│ (HA, 3x) │
└──────┬───────┘
│ CA key = PKCS#11
┌──────▼───────┐
│ HSM / TPM │
└──────────────┘The security of this whole diagram rests on one detail engineers routinely break: the workload API is a Unix domain socket, and the agent identifies the caller by resolving the connecting PID from SO_PEERCRED, then attesting *that* process — reading its cgroup to get the pod UID, calling the kubelet API to map UID → pod spec, and validating the projected service-account token via TokenReview for the k8s_psat node attestor. If you mount hostPath: /run/spire/sockets into every container in the namespace, or run the agent with a permissive service_account_allow_list, you have built a very expensive identity vending machine: any compromised sidecar can request any SVID whose selectors it can satisfy.
Production war story #1. We onboarded a batch ETL service by copying a registration entry and tweaking the path, but forgot that the pod template had been migrated to a different namespace during a reorg. The selectors (k8s:ns:data-pipeline) no longer matched, so FetchX509SVID returned no identity issued — and because the service treated identity acquisition as a startup-only concern and logged at Info, it crash-looped for 40 minutes while dashboards stayed green (pods were Running in the crash-loop backoff window, ALB health checks pointed at an older ReplicaSet). Fix: emit spiffe_svid_fetch_error_total as a page-worthy metric, and make readiness fail hard on missing identity.
Data Plane: mTLS That Survives Rotation
Once identity is issued, the hard part is *not* the handshake — it is making every cert-consuming layer rotate correctly. Three rules govern the design:
1. **Never cache a *tls.Certificate yourself. Use `tls.Config.GetClientCertificate` / `GetCertificate` callbacks bound to a live `X509Source`. The source holds a streaming subscription to the agent and atomically swaps the in-memory cert. 2. Authorize on the SPIFFE ID, not just on "chain validates".** A valid SVID proves *which workload* is calling. Prefix-matching strings.HasPrefix(id, "spiffe://corp.example/ns/payments") is a vulnerability: spiffe://corp.example/ns/payments-batch also matches. Parse and compare exact IDs, or use segment-aware membership. 3. Connection lifetime must be ≤ SVID lifetime. Under TLS 1.3, a resumed session (PSK) does not re-run client certificate authentication — the identity was bound at the original handshake. A pooled HTTP/2 connection pinned open for hours keeps acting on a SVID that may have been revoked by deleting the registration entry. Keep IdleConnTimeout below the TTL (we use 45s against a 1h SVID) or disable tickets entirely on high-assurance endpoints.
svc-A ──────────────── TLS 1.3 (h2) ────────────────▶ svc-B │ ClientHello (ALPN=h2, key_share=X25519) │ ◀── CertificateRequest (acceptable CAs = corp.example bundle) │ Certificate(X509-SVID A) + CertificateVerify(ECDSA P-256) │ ◀── Certificate(X509-SVID B) + Finished ▼ verify chain → SPIRE bundle (rotated, not /etc/ssl/certs) verify identity → exact SPIFFE ID match against allow-list bind to app policy → OPA/Rego or in-handler RBAC
Production war story #2. At 4,000 concurrent connections and 1-hour TTLs we saw a handful of tls: bad certificate errors per minute on exactly one service. Root cause: that service's Envoy sidecar used combined_validation_context with a validation_context_sds_secret_config pointing at a *statically configured* CA file alongside a SPIFFE match — so when the SPIRE bundle rotated after a root CA change, half the fleet trusted the new intermediate while the static side trusted the old one. The error rate was under 0.4% and never tripped an SLO. Lesson: after any trust-bundle rotation, run a synthetic canary that performs a fresh handshake (not a resumed one) between every service pair.
Hardware-Backed Secret Management: HSM, TPM, and the Root of Trust
There are three tiers of secret in this architecture, and each needs a different custody answer:
| Tier | Secret | Custody mechanism | |---|---|---| | Root | Trust-domain root CA key | Offline CA / HSM, rotated by ceremony; distributed as a bundle | | Intermediate | SPIRE server signing key | PKCS#11 KeyManager — key generated *in* the HSM, non-exportable | | Bootstrap | SPIRE agent join material, Vault unseal key, HSM PIN | TPM 2.0 sealed blobs (systemd-creds, PCR-bound) or Cloud KMS | | Data-at-rest | K8s Secret objects (etcd) | KMSv2 provider backed by the same HSM |
Two numbers drive the design. First, HSM sign latency is ~1.1 ms p50 and 1.5k–4k ECDSA P-256 signs/s per partition — fine for SVID issuance at 60-minute TTLs (500 pods ⇒ ~0.14 signs/s), catastrophic if you put a PKCS#11 call on the request path. Second, TPM 2.0 ECDSA signing is 28–60 ms — great for unsealing once at boot, useless for anything hot. Size the intermediate key custody accordingly, and never let the HSM sit between your service and its data.
Production war story #3. We sealed the Vault unseal key to PCR[7] (Secure Boot policy) and PCR[11], then pushed a firmware update that flipped PCR[7]. Every node in the pool failed to unseal; Vault came up sealed and the dynamic-Postgres credential path died, taking down 60% of the fleet's DB connections in under 90 seconds. Post-incident: seal to PCR[11] only (UKI measurement), keep a break-glass recovery key in a physically-held smart card, and alert on vault_core_unsealed == 0 for >30s. TPM sealing is a *boot integrity* control, not a *firmware stability* control — treat it as such.
Practical Implementation
A production Go service that acquires its identity from the Workload API, enforces TLS 1.3 + ECDSA, authorizes by exact SPIFFE ID, and fails readiness when the SVID is unavailable.
// cmd/ledger/main.go — SPIFFE-native mTLS service: no cert files, no restarts.
package main
import (
"context"
"crypto/tls"
"log/slog"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/spiffe/go-spiffe/v2/spiffeid"
"github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
"github.com/spiffe/go-spiffe/v2/workloadapi"
)
// The agent socket is a Unix domain socket; identity is bound to the calling PID
// via SO_PEERCRED, so this path must be mounted read-only into app containers only.
const agentSocket = "unix:///run/spire/sockets/agent.sock"
var (
trustDomain = spiffeid.RequireTrustDomainFromString("corp.example")
callerAllow = map[string]struct{}{
"spiffe://corp.example/ns/payments/sa/gateway": {},
"spiffe://corp.example/ns/payments/sa/reconciler": {},
}
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
// One X509Source = one streaming subscription. SPIRE rotates the X509-SVID at
// ~50% of its TTL (default 1h); the source atomically swaps the in-memory
// *tls.Certificate, so GetClientCertificate always returns a fresh keypair.
src, err := workloadapi.NewX509Source(ctx,
workloadapi.WithClientOptions(workloadapi.WithAddr(agentSocket)))
if err != nil {
slog.Error("workload API unreachable", "socket", agentSocket, "err", err)
os.Exit(1) // fail closed: a pod without identity must never serve traffic
}
defer src.Close()
svid, err := src.GetX509SVID()
if err != nil {
slog.Error("no X509-SVID issued; check registration entry selectors", "err", err)
os.Exit(1)
}
slog.Info("workload identity ready",
"spiffe_id", svid.ID.String(),
"hint", svid.Hint(),
"not_after", svid.Expiry().UTC().Format(time.RFC3339))
serverCfg := tlsconfig.MTLSServerConfig(src, src, tlsconfig.AuthorizeMemberOf(trustDomain))
serverCfg.MinVersion = tls.VersionTLS13
serverCfg.CurvePreferences = []tls.CurveID{tls.X25519, tls.CurveP256}
serverCfg.NextProtos = []string{"h2"}
mux := http.NewServeMux()
mux.Handle("/v1/ledger/entries", requireCaller(http.HandlerFunc(entries)))
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
if _, err := src.GetX509SVID(); err != nil { // never report healthy without identity
http.Error(w, "svid unavailable", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
})
srv := &http.Server{
Addr: ":8443",
Handler: mux,
TLSConfig: serverCfg,
ReadHeaderTimeout: 5 * time.Second,
IdleTimeout: 45 * time.Second, // < SVID TTL: see TLS 1.3 resumption caveat
}
errCh := make(chan error, 1)
go func() { errCh <- srv.ListenAndServeTLS("", "") }() // certs come from TLSConfig
slog.Info("listening", "addr", srv.Addr)
select {
case err := <-errCh:
slog.Error("server exited", "err", err)
case <-ctx.Done():
shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
slog.Error("graceful shutdown failed", "err", err)
}
}
}
type callerKey struct{}
// requireCaller performs application-layer authorization on the *verified* peer
// SVID. Exact match only: a prefix check would let ns/payments-batch through.
func requireCaller(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
pc := r.TLS.PeerCertificates
if r.TLS == nil || len(pc) == 0 || len(pc[0].URIs) != 1 {
http.Error(w, "missing client SVID", http.StatusUnauthorized)
return
}
caller, err := spiffeid.FromURI(pc[0].URIs[0])
if err != nil {
http.Error(w, "malformed SPIFFE ID", http.StatusForbidden)
return
}
if _, ok := callerAllow[caller.String()]; !ok {
slog.Warn("denied caller", "caller", caller.String(), "path", r.URL.Path)
http.Error(w, "forbidden", http.StatusForbidden)
return
}
next.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), callerKey{}, caller)))
})
}
func entries(w http.ResponseWriter, r *http.Request) {
caller, _ := r.Context().Value(callerKey{}).(spiffeid.ID)
slog.Info("authorized request", "caller", caller.String(), "path", r.URL.Path)
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"entries":[],"served_by":"ledger"}`))
}Client side — note the short idle timeout and the fact that the trust bundle comes from the same rotating source:
func newLedgerClient(src *workloadapi.X509Source) *http.Client {
cfg := tlsconfig.MTLSClientConfig(src, src, tlsconfig.AuthorizeID(
spiffeid.RequireFromString("spiffe://corp.example/ns/payments/sa/ledger")))
cfg.MinVersion = tls.VersionTLS13
return &http.Client{
Timeout: 3 * time.Second,
Transport: &http.Transport{
TLSClientConfig: cfg,
ForceAttemptHTTP2: true,
MaxIdleConnsPerHost: 64,
IdleConnTimeout: 45 * time.Second, // do not outlive the authenticating SVID
},
}
}For sidecar-based data planes (Envoy + SPIRE SDS), the same rules apply — the bundle and the SVID both come from the agent socket, never from a mounted file:
# envoy.yaml fragment — no secret volumes, no cert reloads, no restarts
filter_chains:
- transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
require_client_certificate: true
common_tls_context:
tls_params:
tls_minimum_protocol_version: TLSv1_3
ecdh_curves: [X25519, P-256]
combined_validation_context:
default_validation_context:
match_typed_subject_alt_names:
- san_type: URI
matcher: { exact: "spiffe://corp.example/ns/payments/sa/gateway" }
validation_context_sds_secret_config:
name: spiffe://corp.example
sds_config: { api_config_source: { api_type: GRPC, grpc_services: [ { envoy_grpc: { cluster_name: spire_agent } } ] } }
tls_certificate_sds_secret_configs:
- name: spiffe://corp.example/ns/payments/sa/ledger
sds_config: { api_config_source: { api_type: GRPC, grpc_services: [ { envoy_grpc: { cluster_name: spire_agent } } ] } }
# cluster spire_agent: STATIC, http2_protocol_options {}, endpoint via pipe path
# /run/spire/sockets/agent.sockSPIRE server with the CA key pinned inside an HSM partition, plus TPM-anchored node attestation for bare metal and KMSv2 for etcd:
# spire-server.conf — intermediate CA key is generated in and never leaves the HSM
server {
bind_address = "0.0.0.0"
bind_port = "8081"
trust_domain = "corp.example"
data_dir = "/var/lib/spire/server"
}
plugins {
DataStore "sql" {
plugin_data {
database_type = "postgres"
connection_string = "postgres://[email protected]:5432/spire?sslmode=verify-full"
}
}
KeyManager "pkcs11" {
plugin_data {
pkcs11_lib = "/usr/lib/pkcs11/libcklog2.so"
token_label = "spire-ca"
pin = "/run/secrets/spire-hsm-pin" # tmpfs, sourced from Vault Agent
key_label = "spire-intermediate-ec-p256"
hsm_key_generation = true
}
}
UpstreamAuthority "vault" {
plugin_data {
vault_addr = "https://vault.internal:8200"
pki_mount_point = "pki-spire-root"
ca_bundle_path = "/etc/spire/vault-ca.pem"
approle_auth {
approle_id_path = "/run/secrets/vault-approle-id"
secret_id_path = "/run/secrets/vault-approle-secret"
}
}
}
NodeAttestor "k8s_psat" {
plugin_data {
clusters = { prod = { service_account_allow_list = ["spire-system:spire-agent"] } }
}
}
NodeAttestor "tpm_devid" { # edge / bare-metal nodes with a real TPM 2.0
plugin_data { devid_ca_path = "/etc/spire/tpm-devid-ca.pem" }
}
}# apiserver EncryptionConfiguration (KMSv2) — etcd Secrets protected by the same HSM
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: ["secrets", "configmaps"]
providers:
- kms:
apiVersion: v2
name: hsm-kms
endpoint: unix:///var/run/kmsplugin/hsm.sock
timeout: 3s
- identity: {} # read-only fallback during migration; remove afterwardsPerformance Benchmarks & Trade-Offs
Test rig: 6× 8 vCPU / 32 GB nodes, 20k RPS steady state, HTTP/2, 512-byte payloads, 4,000 concurrent connections, ECDSA P-256 unless noted. Handshake timings are measured end-to-end with k6 + http_req_tls_handshaking.
| Stack | Handshake p50 / p95 / p99 | Identity TTL & rotation | CPU @1k RPS | Key custody | Blast radius if a node is owned | |---|---|---|---|---|---| | Static 90-day certs in K8s Secrets (RSA-2048) | 3.4 / 6.1 / 9.8 ms | 90 d, manual | 0.42 core | base64 Secret | Trust domain: any Secret readable ⇒ minted traffic | | cert-manager + Vault PKI (ECDSA P-256, 30 d) | 1.6 / 2.4 / 3.7 ms | 30 d, cron | 0.18 core | Vault + Secret | Vault token/role abuse ⇒ any identity | | SPIRE + Envoy SDS sidecar (X509-SVID, 1 h) | 1.8 / 2.6 / 3.9 ms | 1 h, renew @50% | 0.21 core + ~12 MB RSS | Workload API, RAM only | One attested workload identity | | SPIRE + in-process go-spiffe (no sidecar) | 1.2 / 1.9 / 2.8 ms | 1 h, renew @50% | 0.14 core | Workload API, RAM only | One attested workload identity | | Istio ambient (ztunnel/HBONE, 1 h) | 1.5 / 2.2 / 3.4 ms | 1 h | 0.19 core | Workload API | One attested workload identity | | JWT-SVID only (L7, 5 min TTL) | 0.3 / 0.5 / 0.9 ms* | 5 min | 0.06 core | RAM only | Replay inside the TTL window if tokens hit logs |
\*No TLS handshake occurs; the figure is ECDSA verify + JWKS lookup cost.
| Root key location | ECDSA P-256 sign p50 | Sustained signs/s | Cost/mo | Verdict | |---|---|---|---|---| | Software key on disk | 0.06 ms | 180k+ | ~$0 | One cat from full CA compromise — unacceptable | | TPM 2.0, sealed to PCR 11 | 28–60 ms | 40–90 | bundled | Excellent for boot/unseal, far too slow for bulk SVID signing | | Network HSM via PKCS#11 | 1.1 ms | 1.5k–4k per partition | $1.5k–2.5k | Correct choice for the intermediate; deploy ≥2 partitions | | Cloud KMS asymmetric sign | ~9 ms | ~500 (shared quota) | $0.03 / 10k signs | Fine for low-volume roots; quota is the real ceiling |
Pro-Tips & Common Pitfalls
- Never `strings.HasPrefix` a SPIFFE ID.
spiffe://corp.example/ns/paymentsis a string prefix ofspiffe://corp.example/ns/payments-batch. Parse withspiffeid.FromURIand compare canonical strings or trust-domain membership. - Never mount the workload API socket into every container. Bind it read-only into app containers only; a shared socket means every process in the pod can request the pod's SVIDs.
- Never treat identity acquisition as best-effort. Fail readiness on a missing SVID, and page on
svid_fetch_error_total— silent 403s from selector drift are the #1 post-migration incident. - Keep `IdleConnTimeout` (and gRPC keepalive) below the SVID TTL. TLS 1.3 resumed sessions do not re-authenticate the client certificate.
- Don't set SVID TTLs below ~10 minutes at fleet scale. 5-minute TTLs across 5k pods turn the HSM into the bottleneck and produce handshake thundering herds at synchronized renewal boundaries.
- Stagger and batch: SPIRE batch-signs; keep the PKCS#11 call off any request path, and pre-warm pods before rolling the SPIRE server.
- Treat the SPIRE datastore as policy. Registration entries *are* your authorization model — put them in Git, review them, and never co-locate Postgres in the same blast radius as the server.
- Rotate federation bundles too. A stale federated bundle looks exactly like a network partition:
x509: certificate signed by unknown authorityon one service pair only. - Clock skew breaks JWT-SVIDs silently. Enforce chrony, alert at >100 ms offset; 5-minute TTLs leave very little slack.
- Match curves explicitly. Mixing P-256 and P-521 across the fleet produces intermittent handshake failures that look like packet loss.
- Give TPM sealing a recovery path. A firmware update that flips a PCR will brick auto-unseal; keep a break-glass key and alert when
vault_core_unsealed == 0. - Test rotation, not just issuance. Run a canary every 5 minutes that opens a *fresh* (non-resumed) connection between two services across trust-bundle rotation.
Frequently Asked Questions (FAQ)
How is SPIFFE/SPIRE different from cert-manager or Vault PKI for mutual TLS?
SPIFFE/SPIRE automates *attestation*, not just issuance. cert-manager and Vault PKI will happily sign a certificate for anything that can present a credential — usually a Kubernetes Secret or a Vault token, both of which are long-lived bearer secrets that can be exfiltrated and replayed from anywhere. SPIRE issues an X509-SVID only after verifying the calling process's PID via SO_PEERCRED, mapping it to a pod UID, and matching node + workload attestation against a registration entry. The private key is generated in-memory in the workload and never persisted, TTLs default to 1 hour, and identity is revoked by deleting a registration entry rather than by waiting for a CRL. Practically: cert-manager solves distribution, SPIRE solves proof-of-identity.
Does mutual TLS add measurable latency in production?
In-process ECDSA P-256 mTLS on TLS 1.3 measured 1.2 ms p50 / 2.8 ms p99 per handshake at 20k RPS, at a cost of ~0.14 core per 1k RPS. Because SVIDs rotate hourly and connections are long-lived with HTTP/2, handshakes are a rounding error in aggregate — the steady-state cost is AES-GCM record protection, not the handshake. The real variables are key type (RSA-2048 roughly doubles handshake cost and adds 11–18% CPU), sidecar presence (~12 MB RSS and ~0.5 ms extra per-connection overhead), and whether you allow session resumption. If you must squeeze further, keep TLS 1.3, X25519 key exchange, ECDSA P-256 SVIDs, and connection pooling with idle timeout under the SVID TTL.
What happens to my services when the SPIRE server goes down?
Nothing — for up to one SVID TTL. Workloads hold valid X509-SVIDs in memory and the agent caches the trust bundle, so existing mTLS connections and new handshakes between already-issued identities continue to work. What breaks is *new* workload starts (a fresh pod cannot get an SVID without the agent reaching a server) and rotations after the TTL expires. Design for it: run 3 SPIRE servers behind a load balancer with a Postgres datastore, set SVID TTL to at least 4× your maximum server-recovery time (we use 1h against a 15-minute recovery objective), alert on spire_server_svid_rotation_failure and agent_svid_sync_lag_seconds, and keep the HSM partition pair in the same HA group so a partition failover doesn't take the CA offline.
