Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Zero Trust Architecture and Mutual TLS with SPIFFE/SPIRE

By: Md Tarek · 20 Oct 2026

Zero Trust Architecture and Mutual TLS: SPIFFE/SPIRE workload identity with rotating X.509-SVIDs and hardware-backed key management
Zero Trust Architecture and Mutual TLS: SPIFFE/SPIRE workload identity with rotating X.509-SVIDs and hardware-backed key management

Executive Summary & Key Takeaways

  • Identity must be cryptographic, not topological. SPIFFE assigns every workload an unforgeable URI identity (spiffe://tareksec.dev/ns/prod/sa/payments) encoded in the X.509-SVID's URI SAN. There is no CN, no DNS SAN, and no shared secret — the TLS handshake *is* the authentication event.
  • The default X.509-SVID TTL is 3600s (1 hour), rotated at ~50% of lifetime by a streaming Workload API watch. That single default collapses a stolen-key blast radius from 90 days (typical Vault-issued cert) to under 60 minutes, and replaces CRL/OCSP entirely.
  • Full mTLS handshakes on TLS 1.3 with ECDSA P-256 cost ~1.0–1.4 ms of CPU. After session-ticket resumption and kTLS offload, the amortized p99 penalty at 10k RPS is ~0.3–0.6 ms — cheaper than the network hop it secures.
  • Keep the CA key off disk. A SPIRE Server keymanager backed by AWS KMS or a PKCS#11 HSM makes the signing key non-exportable; signing becomes a 2–8 ms HSM round-trip, which is irrelevant because signing happens once per workload per rotation cycle, not per request.
  • mTLS alone is table stakes, not zero trust. Authorization must be re-evaluated per RPC against the *verified* cryptographic peer identity (SPIFFE ID + method + HTTP verb), never against X-Forwarded-Client-Cert headers or JWTs minted from user input.

1. Why mTLS Without a Workload Identity Plane Quietly Fails

Most teams "do mTLS" by stuffing a cert into a Kubernetes Secret and mounting it into the pod. This works in a demo and collapses in production for four predictable reasons:

1. Bootstrap recursion. To fetch a cert you need credentials; to get credentials you need a cert. Practitioners paper over this with long-lived bootstrap tokens or a static Vault AppRole secret_id that gets baked into a Helm chart — which is now your real root of trust, sitting in Git. 2. Distribution and sprawl. N services × M environments × K clusters yields a cert distribution matrix nobody audits. The 2023 wave of "we rotated the intermediate CA and half the mesh went dark" incidents is entirely a distribution problem, not a crypto problem. 3. Revocation is broken by design. CRLs are enormous, OCSP adds a network dependency to every handshake (or is stapled and stale), and Kubernetes Secrets can't be revoked without a pod restart. If your cert lives 90 days, your revocation story is "wait 90 days." 4. Verification stops at "the chain is valid." A chain signed by your CA proves *someone you trusted* issued it. It does not prove *which workload* is on the other end. Without a name-based authorization step, a compromised low-value pod can call your ledger service with perfect crypto.

SPIFFE/SPIRE fixes all four by inserting a node-local attestation step that mints short-lived SVIDs from kernel- and kubelet-verified selectors:

  Workload (payments)          SPIRE Agent (per node)         SPIRE Server (HA)
  ------------------           ----------------------         ------------------
  1. connect unix://           |                              |
     /run/spire/sockets/       |                              |
     agent.sock  ------------> |                              |
                               | 2. attest caller             |
                               |    (k8s psat + pod selectors)|
                               |    ---- FetchX509SVID ------->|
                               |                              | 3. verify node attestor
                               |                              |    solve PoP challenge
                               |                              | 4. sign X509-SVID with
                               |                              |    intermediate CA
                               |                              |    (key in KMS/HSM)
                               | <---- SVID + bundle -------- |
  5. stream of rotated <-------|                              |
     SVIDs (WatchX509SVID)     |                              |
  6. mTLS to ledger  =======================================> ledger pod
     leaf SAN URI: spiffe://tareksec.dev/ns/prod/sa/payments      (same agent
     peer verified against spiffe://tareksec.dev bundle)          socket model)

Node attestation answers "is this machine who it claims to be" (AWS IID, GCP IIT, Azure MSI, k8s_psat with a projected service-account token bound to the node's kubelet). Workload attestation answers "which process on this machine is calling" via selectors like k8s:ns:prod, k8s:sa:payments, unix:uid:10001, docker:label:.... A registration entry maps a selector set to a SPIFFE ID and a TTL. Nothing else can obtain that identity — and critically, the workload never sees a bootstrap secret.

The mesh hand-off (and where zero trust leaks)

When Envoy terminates mTLS, it must forward the peer identity in X-Forwarded-Client-Cert (XFCC, By= / URI= fields) for the upstream app. Two failure modes account for most real-world breaches here:

  • XFCC is not stripped at the edge. If an internet-facing gateway passes a client-supplied XFCC header through, any attacker forges By=spiffe://tareksec.dev/ns/prod/sa/admin. Always remove_request_headers: [x-forwarded-client-cert] at the first hop and let only the sidecar set it.
  • TLS termination without re-origination. An ingress that terminates TLS and forwards plaintext HTTP to the pod breaks the chain. Use ISTIO_MUTUAL-style re-origination, or run the sidecar as the *only* listener with require_client_certificate: true.

2. Hardware-Backed Secret Management: CA Keys, Agent Keys, Workload Keys

Zero trust fails if the private key protecting the identity is a file with mode 0644. You need three distinct key tiers, each with a different custody model:

| Tier | What it signs | Custody | Rotation cost | |---|---|---|---| | Upstream / root CA | SPIRE Server intermediate CA | Offline root; upstream via Vault PKI, AWS PCA, or PKCS#11 HSM | Months; never online | | SPIRE Server signing key | All X.509-SVIDs and JWT-SVIDs | keymanager "aws_kms" / "gcp_kms" / "azure_kv" / "pkcs11" — non-exportable | 2–8 ms per signing op, once per workload rotation | | Workload SVID key | TLS handshake proofs | Agent keymanager: memory (default), disk (⚠️), or TPM 2.0 / PKCS#11 | 5–25 ms per rotation, off the request path |

A representative server.conf that pins the signing key inside KMS and sources the intermediate from a hardware-backed upstream:

server {
  bind_address = "0.0.0.0"
  bind_port    = 8081
  trust_domain = "tareksec.dev"
  data_dir     = "/var/lib/spire/server"

  # The CA private key never exists in plaintext on the host filesystem.
  keymanager "aws_kms" {
    plugin_data {
      region     = "us-east-1"
      key_id     = "arn:aws:kms:us-east-1:111122223333:key/9f2c..."
      key_policy = "purpose=SIGN_VERIFY"
    }
  }

  # Intermediate CA issued by Vault's PKI engine backed by an HSM seal.
  upstream_authority "vault" {
    plugin_data {
      vault_addr        = "https://vault.internal:8200"
      pki_mount_point   = "pki-int"
      ca_ttl            = "720h"   # 30d intermediate; must be >> SVID TTL
      ttl               = "720h"
      auth_method       = "token"   # use AppRole or k8s auth in prod, not a static token
    }
  }
}

default_x509_svid_ttl = "1h"   # revocation = expiry, keep this short

Two hard-won details:

  • Intermediate CA TTL must be ≥ 6× the SVID TTL. If the intermediate expires while SVIDs signed by it are still live, every in-flight handshake fails at chain validation. Rotate the CA, publish the new bundle to /bundles/spiffe://tareksec.dev (or federation endpoints), and let agents converge *before* draining the old intermediate.
  • The agent's `disk` keymanager writes SVID private keys unencrypted to the agent data dir. On a shared node or a cluster with hostPath volume access, that is a lateral-movement primitive. Prefer memory, and on high-assurance nodes pair the agent with a TPM 2.0 keymanager so the SVID key is sealed to PCR values and can never be exfiltrated as a blob.

Practical Implementation

A production Go service that (a) obtains and continuously rotates its X.509-SVID from the local SPIRE Agent, (b) serves gRPC with mandatory mTLS restricted to its trust domain, and (c) re-authorizes every RPC against the *verified* peer SPIFFE ID.

// cmd/payments/main.go
package main

import (
	"context"
	"errors"
	"log/slog"
	"net"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/spiffe/go-spiffe/v2/spiffeid"
	"github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
	"github.com/spiffe/go-spiffe/v2/workloadapi"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"
	"google.golang.org/grpc/health"
	healthpb "google.golang.org/grpc/health/grpc_health_v1"
	"google.golang.org/grpc/peer"
)

var (
	trustDomain = spiffeid.RequireTrustDomainFromString("tareksec.dev")
	ledgerID    = spiffeid.RequireFromString("spiffe://tareksec.dev/ns/prod/sa/ledger")
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	// NewX509Source opens a streaming WatchX509SVID RPC against the agent.
	// Certificates are swapped in-memory at ~50% of TTL: no SIGHUP, no pod restart,
	// no in-flight request ever observes an expired leaf.
	srcCtx, cancel := context.WithTimeout(ctx, 20*time.Second)
	defer cancel()

	source, err := workloadapi.NewX509Source(srcCtx,
		workloadapi.WithClientOptions(
			workloadapi.WithAddr("unix:///run/spire/sockets/agent.sock"),
		),
	)
	if err != nil {
		slog.Error("workload API unreachable", "err", err)
		os.Exit(1) // fail closed: never start without an identity
	}
	defer source.Close()

	// Fail fast on identity drift (bad registration entry / selector change).
	svid, err := source.GetX509SVID()
	if err != nil {
		slog.Error("no SVID available", "err", err)
		os.Exit(1)
	}

	// Downstream: require a client cert whose chain validates against our bundle
	// AND whose SPIFFE ID belongs to our trust domain.
	serverTLS := tlsconfig.MTLSServerConfig(source, source, tlsconfig.AuthorizeMemberOf(trustDomain))
	// Upstream: pin the exact peer identity we are allowed to call.
	clientTLS := tlsconfig.MTLSCLientConfig(source, source, tlsconfig.AuthorizeID(ledgerID))

	srv := grpc.NewServer(
		grpc.Creds(credentials.NewTLS(serverTLS)),
		grpc.UnaryInterceptor(requireCaller), // L7 authz, defense in depth
	)
	healthpb.RegisterHealthServer(srv, health.NewServer())

	lis, err := net.Listen("tcp", ":8443")
	if err != nil {
		slog.Error("listen failed", "err", err)
		os.Exit(1)
	}

	go func() {
		if err := srv.Serve(lis); err != nil && !errors.Is(err, grpc.ErrServerStopped) {
			slog.Error("grpc serve failed", "err", err)
		}
	}()

	// Upstream call: reuse clientTLS. Handshake CPU is amortized by TLS 1.3
	// session tickets; keep the ClientConn long-lived and pooled.
	if err := dialLedger(ctx, clientTLS); err != nil {
		slog.Error("ledger dial failed", "err", err)
	}

	slog.Info("payments service ready", "spiffe_id", svid.ID.String(), "expires_at", svid.NotAfter)
	<-ctx.Done()
	srv.GracefulStop() // let in-flight mTLS requests drain
}

func dialLedger(ctx context.Context, creds credentials.TransportCredentials) error {
	conn, err := grpc.NewClient("ledger.internal:8443", grpc.WithTransportCredentials(creds))
	if err != nil {
		return err
	}
	defer conn.Close()
	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
	defer cancel()
	_, err = healthpb.NewHealthClient(conn).Check(ctx, &healthpb.HealthCheckRequest{})
	return err
}

// requireCaller authorizes on the cryptographic identity proven by the handshake.
// Never on a header, never on a bearer token, never on source IP.
func requireCaller(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
	p, ok := peer.FromContext(ctx)
	if !ok {
		return nil, errors.New("no peer in context")
	}
	tlsInfo, ok := p.AuthInfo.(credentials.TLSInfo)
	if !ok || len(tlsInfo.State.VerifiedChains) == 0 {
		return nil, errors.New("no verified mTLS chain")
	}

	uris := tlsInfo.State.VerifiedChains[0][0].URIs
	if len(uris) == 0 {
		return nil, errors.New("peer SVID has no URI SAN")
	}
	caller, err := spiffeid.FromURI(uris[0])
	if err != nil {
		return nil, errors.New("malformed SPIFFE ID in peer SVID")
	}
	if caller.TrustDomain() != trustDomain {
		return nil, errors.New("caller outside trust domain")
	}

	slog.Info("authorized call", "method", info.FullMethod, "caller", caller.String())
	return handler(ctx, req)
}

If you terminate at Envoy instead of in-process, wire the sidecar to the same agent over SDS (the agent exposes an SDS server on the Workload API socket):

# envoy.yaml (excerpt): downstream mTLS with SPIFFE-SAN enforcement
transport_socket:
  name: envoy.transport_sockets.tls
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
    require_client_certificate: true
    common_tls_context:
      tls_params:
        tls_minimum_protocol_version: TLSv1_3
        ecdh_curves: [X25519, P-256]
      tls_certificate_sds_secret_configs:
        - name: spiffe://tareksec.dev/ns/prod/sa/payments
          sds_config:
            api_config_source:
              api_type: GRPC
              grpc_services:
                - envoy_grpc: { cluster_name: spire_agent }
      combined_validation_context:
        default_validation_context:
          match_typed_subject_alt_names:
            - san_type: URI
              matcher: { exact: spiffe://tareksec.dev/ns/prod/sa/ledger }
        validation_context_sds_secret_config:
          name: spiffe://tareksec.dev          # bundle SDS secret
          sds_config:
            api_config_source:
              api_type: GRPC
              grpc_services:
                - envoy_grpc: { cluster_name: spire_agent }

Performance Benchmarks & Trade-Offs

Reference numbers from a 3-node test cluster (8 vCPU c5.2xlarge, TLS 1.3, ECDSA P-256 SVIDs, 1 KB gRPC payloads, 10k RPS steady state, p99 measured at the client). Handshake CPU is single-core µs per full handshake; treat as comparative, not absolute.

| Approach | Handshake CPU (µs) | Added p99 @ 10k RPS | Cert lifetime / rotation | Key custody | Operational complexity | |---|---|---|---|---|---| | Plaintext gRPC (no TLS) | 0 | baseline | n/a | none | None — and no identity at all | | Vault-issued certs in k8s Secret (90d) | ~950 | +0.35 ms | Manual renew job, pod restart | Secret in etcd, on disk | Low initial, high incident load | | mTLS + SPIRE X.509-SVID (1h TTL) | ~1,250 | +0.55 ms | Automatic, ~50% of TTL | Agent memory | Medium (attestation policy) | | mTLS + SPIRE + kTLS offload | ~700 | +0.30 ms | Automatic | Agent memory | Medium (+ kernel/CNI tuning) | | mTLS + SPIRE + TPM 2.0 agent key | ~1,250 (+8–25 ms per rotation) | +0.55 ms | Automatic | TPM sealed, non-exportable | High (PCR/attestation ops) | | mTLS + HSM-backed SPIRE CA (pkcs11) | ~1,250 (+2–8 ms per signing op) | +0.55 ms | Automatic | Non-exportable in HSM | High (HSM HA + quorum) | | JWT-SVID for L7 policy (5 min TTL) | n/a (no handshake) | +0.05 ms (verify) | Automatic | Bearer token — leaks via logs | Medium; not a replacement for mTLS |

Key trade-off: short TTLs buy revocation-by-expiry at the cost of signing throughput and agent fan-out. A 2,000-pod cluster at 1h TTL generates ~0.55 signing ops/sec — trivially within KMS/HSM limits — but the *bundle distribution* and *agent reconnect* paths must be sized for the same rate. Dropping to 15-minute TTL multiplies that by 4×; verify your SPIRE Server datastore and KMS quotas before you do.

Pro-Tips & Common Pitfalls

  • ❌ Matching on CN or DNS SANs. SPIRE issues URI SANs only. If your verifier looks at Subject.CommonName, it will reject every SVID — or worse, someone "fixes" it by disabling verification.
  • ❌ Trusting inbound `X-Forwarded-Client-Cert`. Strip it at the edge (remove_request_headers) and let only your own sidecar set it. This is the single most exploited mesh misconfiguration.
  • ❌ Wildcard selectors in registration entries. k8s:ns:* or unix:uid:0 grants an identity to *every* workload on the node. Selectors are your authorization policy — review them like IAM.
  • ❌ World-writable agent socket. /run/spire/sockets/agent.sock with 0777 lets any process on the node impersonate the workload. Run the agent with a dedicated group and 0660.
  • ❌ `keymanager "disk"` on multi-tenant nodes. Unencrypted SVID private keys on disk = trivial exfiltration via hostPath mount.
  • ❌ SVID TTL > 24h. SPIFFE has no CRL/OCSP. Expiry *is* the revocation mechanism; long TTLs silently reintroduce the problem mTLS was meant to solve.
  • ❌ CA rotation without bundle pre-publish. Rotate the intermediate, push the new bundle to the trust bundle endpoint, wait for agent convergence (2× agent sync interval), *then* retire the old CA.
  • ❌ No rotation telemetry. Alert on spire_agent_svid_rotations_total stalling and on SVIDs within 10% of expiry. A silently broken Workload API watch doesn't fail at rotation time — it fails one TTL later, cluster-wide.
  • ❌ Clock skew. Certificates are valid-from/valid-to; NTP drift of a few minutes on a node with a 15-minute TTL causes sporadic x509: certificate has expired errors that look like a SPIRE bug.
  • ✅ Enable TLS 1.3 session tickets and connection pooling. mTLS handshake cost is real; at 10k RPS, resumption cuts CPU by ~60%.
  • ✅ Log the peer SPIFFE ID on every authorization decision. It is the only audit trail that survives a compromised pod.
  • ✅ Test the deny path. Assert that a service in the wrong trust domain, and a service with the right trust domain but wrong path, both get PermissionDenied.

Frequently Asked Questions (FAQ)

Is mutual TLS sufficient to implement zero trust architecture? No — mTLS establishes *who* is calling, not *what they may do*. Zero trust requires three layers: (1) cryptographic workload identity (SPIFFE X.509-SVID), (2) per-request authorization evaluated against that identity plus the resource and method (policy engine or in-process guard), and (3) least-privilege network reachability so a compromised identity can only reach what it needs. mTLS with no authorization layer gives you an encrypted, authenticated lateral-movement path — the 2019 Capital One-style SSRF blast radius, with better crypto. Treat the SPIFFE ID as the subject in every policy decision, and never fall back to IP allowlists.

How does SPIFFE/SPIRE handle certificate revocation without CRL or OCSP? By making revocation irrelevant through short lifetimes. X.509-SVIDs default to a 3600-second TTL and the agent rotates them at roughly half-life via a streaming WatchX509SVID RPC, so a leaked key is useless within ~30–60 minutes. For the rare case where you must kill an identity immediately, delete its registration entry — the agent's next sync stops renewing, and the SVID expires by TTL. This is why keeping TTLs short is a security control, not a performance knob: an SVID with a 30-day TTL has, functionally, no revocation story at all.

Can workload private keys be hardware-backed, or only the CA key? Both, with different costs. The CA signing key is the highest-value asset: use a SPIRE Server keymanager plugin (aws_kms, gcp_kms, azure_kv, pkcs11) so it is generated in and never leaves the HSM/KMS; signing costs 2–8 ms but only occurs on rotation, not per request. Workload SVID keys live in the agent, where the default memory keymanager is fine for most clusters; for high-assurance nodes, a TPM 2.0-backed agent keymanager seals the key to PCR measurements, making it non-exportable even with root on the host — at the cost of 8–25 ms per rotation and meaningfully harder node-lifecycle operations.