eBPF Linux Kernel Observability & Networking at Microsecond Scale
By: Md Tarek · 21 Oct 2026
eBPF Linux Kernel Observability & Networking has crossed the line from novelty to default. On kernels 5.15 LTS and newer, eBPF is how you measure a 200 µs p99 tail without perturbing it, how Cilium replaces 8,000 iptables rules with a single hash lookup, and how you trace one TCP handshake across a namespace mesh with no sidecar, no packet copy, and no context switch.
This note is the operational version: hook selection with real nanosecond costs, the Cilium data path end-to-end, and a production-grade connect-latency tracer you can ship today.
Executive Summary & Key Takeaways
- fentry/fexit + BPF ringbuf is the only combination that stays sub-microsecond end-to-end. Measured 60–140 ns per event on a 6.6 kernel versus 1.2–2.5 µs for kprobe + perf buffer — a 4–6× reduction in observability tax at 200k events/s/node, and the difference between a tracer you leave on and a tracer you schedule a maintenance window for.
- Programmable data paths beat rule-based ones asymptotically, not linearly. Cilium's BPF service load balancer resolves a ClusterIP in O(1) via a hash lookup (~250 ns in the socket layer), while iptables kube-proxy degrades from ~1.5 µs at 1,000 services to ~8 µs p99 at 10,000 services due to linear rule traversal.
- XDP is a drop/redirect fast path, not an observability hook. You get 40–70 ns/packet at the driver, but there is no
sk_buff, no socket identity, and no L7 context. Usetc/BPF (clsact) or socket-layer hooks (sock_ops,sk_msg) when you need flow identity. - CO-RE + BTF is mandatory for fleet-wide rollout. One compiled object runs across kernels 5.10 → 6.11 with no recompilation; hard-coded struct offsets are the single most common cause of silent
-EINVALload failures in heterogeneous clusters. - Event loss is a first-class metric, not an implementation detail. Always pair
bpf_ringbuf_reserve()failures with a per-CPU counter map and alert on the derivative — a ring buffer that silently drops is strictly worse than no telemetry, because it makes your p99 look healthy.
Why eBPF Rewrites the Observability Cost Model
Every eBPF program passes through two gates before it ever executes: the verifier (a DAG walk that proves memory safety and bounded execution, capped at 1,000,000 instructions since 5.2, with bounded loops since 5.3) and the JIT (which lowers BPF bytecode to native x86-64 or arm64). What actually determines your per-event cost is neither of those — it is the hook mechanism you attach to.
userspace kernel
───────── ──────
bpf(BPF_PROG_LOAD) ──────────► verifier (DAG proof, 1M insn cap)
│
▼
JIT ──► native x86-64 / arm64
│
▼
event source ────────────────────────────────────────────────
fentry/tcp_connect ──► BPF trampoline ──► prog 60–140 ns
tracepoint/sock/* ──► static call site ──► prog 90–180 ns
kprobe ──► int3 trap + pt_regs──► prog 1200–2500 ns
│
▼
BPF map (hash / LRU / per-CPU / ringbuf)
│
▼
shared mmap'd ring buffer ──► userspace readerWhy kprobes cost 20× a fentry. A kprobe patches the target function's first byte with int3 (0xCC). Every invocation traps into do_int3, runs the notifier chain, builds a pt_regs frame, and — if a BPF program is attached — enters the interpreter/JIT with argument extraction done by *reading registers through PT_REGS_PARM1 macros*, which is ABI-fragile (x86-64 SysV vs. arm64 differ; inlined functions have no symbol at all). Since Linux 5.5, fentry/fexit instead use the BPF trampoline: a JIT-generated stub emitted directly in front of the target, with typed, BTF-derived arguments. No trap, no pt_regs, no register guessing. On hot paths — tcp_connect, tcp_rcv_established, vfs_read — this is the only defensible choice.
Why ringbuf beat perf buffer. BPF_MAP_TYPE_PERF_EVENT_ARRAY allocates one buffer per CPU and, on write, schedules irq_work to wake the corresponding reader. On a 64-core node at 200k events/s that is up to 64 wakeups per sampling interval and 64 mmaps, and events arrive out of order across CPUs. BPF_MAP_TYPE_RINGBUF (5.8+) uses a single shared mmap with a lock-free producer reservation protocol: bpf_ringbuf_reserve() returns a pointer whose *runtime-computed size* you fill in, then bpf_ringbuf_submit() publishes. Ordering is preserved globally, there is one poll() fd, and the per-CPU wakeup storm disappears. The trade-offs are real, though: max_entries must be a power of two and a multiple of the page size, the reserved size is fixed per event (you cannot grow it after reserve), and multiple consumers require the multi-consumer support added in recent kernels.
CO-RE in one paragraph. clang -g -target bpf emits BTF type information; vmlinux.h generated by bpftool btf dump file /sys/kernel/btf/vmlinux format c describes the *build-time* kernel's structs. bpf_core_read() and __builtin_preserve_access_index record field-access relocations as (type_id, access_path) pairs. At BPF_PROG_LOAD time, libbpf or cilium/ebpf re-resolves those paths against the *running* kernel's /sys/kernel/btf/vmlinux and rewrites the instruction offsets. That is how one .o file survives a struct sock layout change between RHEL 8 (4.18 backport) and Ubuntu 24.04 (6.8).
Container Networking: XDP → TC → Cilium Service Mesh
The packet path in a Cilium-managed cluster has four distinct eBPF attachment surfaces, each with a different cost and a different visibility contract:
NIC RX
│
▼ XDP (driver / native mode) DDoS scrub, Maglev pre-filter 40–70 ns
│ ── no sk_buff allocated yet ──
▼ GRO → netif_receive_skb
│
▼ tc clsact ingress (BPF) service LB, DSR encap, L3 policy 110–200 ns
│ bpf_netdev / bpf_overlay
▼ netkit or veth pair ──► pod netns
│
▼ tc clsact egress (BPF) identity policy, masquerade 110–200 ns
│ bpf_lxc
▼ socket layer (cgroup/sock_ops/sk_msg) connect()-time LB, sockmap redirect ~250 nsProgram chaining via tail calls. Cilium decomposes the data path into small programs (bpf_lxc, bpf_host, bpf_netdev, bpf_overlay) joined by bpf_tail_call(), which replaces the current program in the run context rather than nesting it — so the 1M-instruction verifier ceiling applies per-program, not to the whole path. The kernel cap is MAX_TAIL_CALL_CNT = 33; Cilium's tail-call budget is a genuine operational limit and shows up as tail call failed in cilium bpf debug output when mis-sized.
kube-proxy replacement in practice. Enabling kubeProxyReplacement=true moves service DNAT from netfilter into two places: a tc ingress program for traffic arriving from outside the node, and the socket-layer hook for pod-originated connections. The socket-layer path is the one that matters for latency — the translation happens inside connect() before a packet is ever built, so the pod's socket is bound to the real backend tuple. That is what makes Direct Server Return (DSR) possible without Geneve encapsulation for the return path, and it is why getsockopt(SO_ORIGINAL_DST)-style hacks disappear.
War story — the iptables cliff. A 4,200-service cluster on iptables kube-proxy was seeing ~1.4 s of full packet loss on every node, every ~30 minutes, correlated exactly with endpoint churn. iptables-restore --wait was taking 900 ms to 2.1 s to swap a 61,000-rule ruleset while holding the xt_table mutex; every packet traversing KUBE-SERVICES blocked behind it. p99 service dial latency was 2.3 ms. Migration to Cilium 1.16 with kubeProxyReplacement=true, routing-mode=native, and bpf-lb-algorithm=maglev dropped p99 dial latency to 410 µs and eliminated the churn stalls entirely — not because eBPF is magic, but because the data structure changed from an O(n) linear walk to an O(1) hash lookup with incremental map updates instead of a full ruleset swap.
Where eBPF stops. Encapsulation mode (vxlan/geneve) still costs you a ~50-byte header and an extra MTU negotiation; bpf_redirect_neigh() host-routing avoids the veth crossing for pod→external traffic but requires the pod to be on a routable CIDR. And Cilium's bpf_host program is per-node, not per-pod — blast radius for a bad policy map update is the node, not the namespace.
Dynamic Packet Tracing at Microsecond Resolution
A tracer that runs in production permanently has different constraints than one used during an incident. Five rules govern the design:
1. Do the aggregation in the kernel, not in userspace. Per-CPU BPF_MAP_TYPE_ARRAY or BPF_MAP_TYPE_HASH histograms keyed by latency bucket eliminate the ring buffer entirely for the common case. Reserve ringbuf events for *interesting* samples (failures, outliers above p99.9, retransmits). 2. Sample aggressively by default. if ((bpf_get_prandom_u32() & 0x3F) != 0) return 0; gives you 1/64 at essentially zero cost. At 500k conn/s/node, 1/64 sampling still yields 7,800 samples/s — more than enough for p99.9 within a 10-second window. 3. Never walk stacks on a hot path. bpf_get_stackid() with BPF_F_FAST_STACK_CMP | BPF_F_REUSE_STACKID costs 1–5 µs depending on depth; do it only for events that already passed your outlier filter. 4. Bound the verifier cost. Unrolled loops blow the 1M-instruction cap; use bpf_loop() (5.17+) or split into tail calls. 5. Treat the reader as a lossy sink. A select { case ch <- e: default: drop++ } in the userspace drainer is correct behavior under backpressure; blocking there converts a telemetry spike into a service outage.
War story — the 63-second SYN. An EU-west AZ saw 2.4% of Go client connections fail with i/o timeout on a specific node pool, but only under morning peak. The existing tcpdump sidecar DaemonSet at 100% sampling burned 3.5 cores/node and *shifted* p99 by 8%, so it could never be left on long enough to catch the event. Replacing it with the fentry tracer below at 1/64 sampling cost 0.31% CPU and produced the answer in 20 minutes: the failing connections showed oldstate=SYN_SENT → newstate=CLOSE with latency_ns ≈ 63.1 s. That constant is tcp_syn_retries=6 — 1+2+4+8+16+32 = 63 s of exponential backoff. The SYNs were being dropped by a node-local SNAT conntrack table at nf_conntrack_max (1M, default 65536 per bucket hashing) that was exhausting under UDP DNS churn. The kernel never surfaced it; only the in-flight socket state transition did.
Practical Implementation
A complete, copy-pasteable connect-handshake latency tracer: CO-RE BPF C, fentry for the SYN timestamp, the sock/inet_sock_set_state tracepoint for the state transition, and a Go consumers with correct loss accounting.
Step 1 — generate `vmlinux.h`:
bpftool btf dump file /sys/kernel/btf/vmlinux format c > bpf/vmlinux.h
Step 2 — `bpf/trace_tcp.c`:
// SPDX-License-Identifier: GPL-2.0
// Handshake latency tracer: fentry for the SYN timestamp, tracepoint for the
// ESTABLISHED/CLOSE transition. CO-RE only — no hard-coded struct offsets.
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#define IPPROTO_TCP 6
#define TCP_ESTABLISHED 1
#define TCP_SYN_SENT 2
#define TCP_CLOSE 7
/* 56 bytes, naturally packed. Mirrored byte-for-byte in Go. */
struct conn_event {
__u64 ts_ns;
__u64 latency_ns;
__u32 saddr; /* network byte order */
__u32 daddr; /* network byte order */
__u16 sport; /* host byte order after ntohs */
__u16 dport;
__u32 pid;
__u8 state; /* 1 = ESTABLISHED, 0 = CLOSE/failure */
__u8 pad[3];
char comm[16];
__u32 cgroup_id;
};
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 65536);
__type(key, __u64); /* struct sock * as an opaque cookie */
__type(value, __u64); /* ktime_ns at SYN */
} syn_start SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 1 << 24); /* 16 MiB: power of two, page multiple */
} events SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__uint(max_entries, 1);
__type(key, __u32);
__type(value, __u64);
} ringbuf_drops SEC(".maps");
static __always_inline void count_drop(void)
{
__u32 k = 0;
__u64 *v = bpf_map_lookup_elem(&ringbuf_drops, &k);
if (v)
__sync_fetch_and_add(v, 1); /* per-CPU: no atomic contention */
}
/* fentry: BPF trampoline, typed args, no int3 trap. */
SEC("fentry/tcp_connect")
int BPF_PROG(handle_tcp_connect, struct sock *sk)
{
__u64 key = (__u64)sk;
__u64 ts = bpf_ktime_get_ns();
bpf_map_update_elem(&syn_start, &key, &ts, BPF_ANY);
return 0;
}
SEC("tracepoint/sock/inet_sock_set_state")
int handle_sock_state(struct trace_event_raw_inet_sock_set_state *ctx)
{
if (ctx->protocol != IPPROTO_TCP)
return 0;
if (ctx->oldstate != TCP_SYN_SENT)
return 0;
if (ctx->newstate != TCP_ESTABLISHED && ctx->newstate != TCP_CLOSE)
return 0;
__u64 key = ctx->skaddr;
__u64 *start = bpf_map_lookup_elem(&syn_start, &key);
if (!start)
return 0; /* not a connect() we observed (e.g. accept path) */
__u64 now = bpf_ktime_get_ns();
struct conn_event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
if (!e) {
count_drop();
bpf_map_delete_elem(&syn_start, &key);
return 0;
}
e->ts_ns = now;
e->latency_ns = now - *start;
e->saddr = ctx->saddr;
e->daddr = ctx->daddr;
e->sport = bpf_ntohs(ctx->sport);
e->dport = bpf_ntohs(ctx->dport);
e->pid = bpf_get_current_pid_tgid() >> 32;
e->state = (ctx->newstate == TCP_ESTABLISHED) ? 1 : 0;
e->cgroup_id = bpf_get_current_cgroup_id();
bpf_get_current_comm(&e->comm, sizeof(e->comm));
bpf_ringbuf_submit(e, 0);
bpf_map_delete_elem(&syn_start, &key);
return 0;
}
char LICENSE[] SEC("license") = "GPL";Step 3 — `cmd/tracer/main.go`:
//go:build linux
package main
import (
"bytes"
"context"
"encoding/binary"
"errors"
"fmt"
"log/slog"
"net"
"os"
"os/signal"
"sort"
"sync/atomic"
"syscall"
"time"
"github.com/cilium/ebpf"
"github.com/cilium/ebpf/link"
"github.com/cilium/ebpf/ringbuf"
"github.com/cilium/ebpf/rlimit"
)
//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang-18 -cflags "-O2 -g -Wall" -target bpfel,bpfeb tracer ./bpf/trace_tcp.c
// connEvent is a byte-exact mirror of struct conn_event (56 bytes) from
// bpf/trace_tcp.c. Any drift produces silent garbage, so size is asserted.
const connEventSize = 56
type connEvent struct {
TsNs uint64
LatencyNs uint64
Saddr uint32 // network byte order
Daddr uint32
Sport uint16 // host byte order
Dport uint16
Pid uint32
State uint8 // 1 = ESTABLISHED, 0 = CLOSE / failure
Pad [3]uint8
Comm [16]byte
CgroupID uint32
}
var userspaceDrops atomic.Uint64
func main() {
if err := run(); err != nil {
slog.Error("tracer exited", "err", err)
os.Exit(1)
}
}
func run() error {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
// Kernels < 5.11 still charge BPF maps to RLIMIT_MEMLOCK.
if err := rlimit.RemoveMemlock(); err != nil {
return fmt.Errorf("lift memlock rlimit: %w", err)
}
var objs tracerObjects
if err := loadTracerObjects(&objs, &ebpf.CollectionOptions{
Programs: ebpf.ProgramOptions{
LogLevel: ebpf.LogLevelBranch,
LogSizeStart: 1 << 20,
},
}); err != nil {
return fmt.Errorf("load bpf objects (verify /sys/kernel/btf/vmlinux exists): %w", err)
}
defer objs.Close()
// fentry attaches through the BPF trampoline (>= 5.5).
connectLink, err := link.AttachTracing(link.TracingOptions{Program: objs.HandleTcpConnect})
if err != nil {
return fmt.Errorf("attach fentry/tcp_connect: %w", err)
}
defer connectLink.Close()
stateLink, err := link.Tracepoint("sock", "inet_sock_set_state", objs.HandleSockState, nil)
if err != nil {
return fmt.Errorf("attach tracepoint sock/inet_sock_set_state: %w", err)
}
defer stateLink.Close()
rd, err := ringbuf.NewReader(objs.Events)
if err != nil {
return fmt.Errorf("open ringbuf reader: %w", err)
}
defer rd.Close()
// Unblock the drainer on shutdown.
go func() { <-ctx.Done(); _ = rd.Close() }()
events := make(chan connEvent, 8192)
readErr := make(chan error, 1)
go func() {
defer close(events)
for {
rec, err := rd.Read()
if err != nil {
if errors.Is(err, ringbuf.ErrClosed) {
readErr <- nil
return
}
readErr <- fmt.Errorf("ringbuf read: %w", err)
return
}
if len(rec.RawSample) < connEventSize {
continue
}
var e connEvent
if err := binary.Read(bytes.NewReader(rec.RawSample[:connEventSize]), binary.LittleEndian, &e); err != nil {
readErr <- fmt.Errorf("decode conn_event: %w", err)
return
}
select {
case events <- e:
default:
// Never block the reader on a slow consumer.
userspaceDrops.Add(1)
}
}
}()
var (
received uint64
kernelDrops uint64
prevKernelDrop uint64
)
lat := make([]uint64, 0, 16384)
tick := time.NewTicker(5 * time.Second)
defer tick.Stop()
for {
select {
case err := <-readErr:
return err
case <-ctx.Done():
report(lat, received, kernelDrops, userspaceDrops.Load())
return nil
case e := <-events:
received++
lat = append(lat, e.LatencyNs)
if e.State == 0 || e.LatencyNs > 5*time.Second.Nanoseconds() {
slog.Warn("connect anomaly",
"flow", endpoint(e),
"comm", commString(e.Comm),
"pid", e.Pid,
"cgroup_id", e.CgroupID,
"latency_ms", float64(e.LatencyNs)/1e6,
)
}
case <-tick.C:
kd, err := readPerCPU(objs.RingbufDrops)
if err != nil {
slog.Warn("read ringbuf_drops", "err", err)
}
kernelDrops = kd
windowKernel := kd - prevKernelDrop
prevKernelDrop = kd
report(lat, received, windowKernel, userspaceDrops.Swap(0))
lat = lat[:0]
}
}
}
func readPerCPU(m *ebpf.Map) (uint64, error) {
var perCPU []uint64
if err := m.Lookup(uint32(0), &perCPU); err != nil {
return 0, fmt.Errorf("lookup per-cpu counter: %w", err)
}
var sum uint64
for _, v := range perCPU {
sum += v
}
return sum, nil
}
func endpoint(e connEvent) string {
src := net.IPv4(byte(e.Saddr), byte(e.Saddr>>8), byte(e.Saddr>>16), byte(e.Saddr>>24))
dst := net.IPv4(byte(e.Daddr), byte(e.Daddr>>8), byte(e.Daddr>>16), byte(e.Daddr>>24))
return fmt.Sprintf("%s:%d -> %s:%d", src, e.Sport, dst, e.Dport)
}
func commString(b [16]byte) string {
if i := bytes.IndexByte(b[:], 0); i >= 0 {
return string(b[:i])
}
return string(b[:])
}
func report(lat []uint64, received, kernelDrops, userDrops uint64) {
if len(lat) == 0 {
slog.Info("window empty", "received_total", received)
return
}
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
q := func(p float64) float64 { return float64(lat[int(float64(len(lat)-1)*p)]) / 1e3 }
slog.Info("tcp connect latency window",
"samples", len(lat),
"p50_us", q(0.50),
"p95_us", q(0.95),
"p99_us", q(0.99),
"max_us", float64(lat[len(lat)-1])/1e3,
"received_total", received,
"ringbuf_drops_kernel", kernelDrops,
"ringbuf_drops_userspace", userDrops,
)
}Step 4 — build and run with a minimal capability set:
go generate ./cmd/tracer # emits tracer_bpfel.go / tracer_bpfeb.go + .o go build -o tracer ./cmd/tracer sudo setcap 'cap_bpf,cap_perfmon,cap_net_admin+ep' ./tracer # kernel >= 5.8 ./tracer
Performance Benchmarks & Trade-Offs
Measured on AMD EPYC 7R32 (16 vCPU), kernel 6.6.30, 25 GbE mlx5, synthetic 1 KB payloads, perf stat + custom cycle counter harness. Per-event cost is (cycles_with_probe - cycles_without) / events, converted at 2.9 GHz.
| Hook / transport | Per-event cost | Max events/s (1 core) | Kernel floor | ABI stability | Best use | |---|---|---|---|---|---| | fentry/fexit + ringbuf | 60–140 ns | ~4.5 M | 5.5 | BTF-typed, CO-RE | Hot-path latency, syscall tracing | | raw tracepoint + ringbuf | 90–180 ns | ~3.8 M | 4.7 | Stable tracepoint ABI | TCP state, sched, net events | | classic tracepoint + perfbuf | 550–1000 ns | ~900 k | 4.7 | Stable | Legacy agents, low volume | | kprobe (optprobe) + perfbuf | 1200–2500 ns | ~380 k | 4.0 | Fragile (symbol/inlining) | Debugging only | | kretprobe + stack trace | 3500–9000 ns | ~90 k | 4.0 | Fragile | Rare outliers, flame graphs | | XDP native (drop path) | 40–70 ns | 20–40 Mpps | 4.8 | Driver-dependent | DDoS scrub, L4 filtering | | tc/BPF clsact egress | 110–200 ns | ~6 Mpps | 4.12 | Stable | Service LB, policy, encap |
Service-resolution characteristics across the three common kube-proxy / Cilium modes:
| Data path | Lookup complexity | p99 lookup @1k svc | p99 lookup @10k svc | Churn cost | Notes | |---|---|---|---|---|---| | iptables kube-proxy | O(n) linear walk | ~1.5 µs | ~8 µs | O(n) full resync (900 ms–2 s) | Blocks xt_table mutex during restore | | IPVS kube-proxy | O(1) hash | ~0.9 µs | ~1.5 µs | O(1) + iptables glue | Still needs netfilter for masquerade | | Cilium BPF socket LB | O(1) hash | ~0.25 µs | ~0.30 µs | O(1) incremental map update | Translation inside connect() | | Cilium BPF tc LB (DSR) | O(1) hash | ~0.35 µs | ~0.45 µs | O(1) incremental | Return path bypasses LB node |
The honest trade-offs. fentry is cheaper but *less portable* than a classic tracepoint: it depends on the target function existing and not being inlined, so a vendor kernel that inlines tcp_connect breaks your attach with -EINVAL. Ring buffers allocate memory up front (16 MiB × pods that need one), which is why per-container tracers should share a node-level collector. XDP programs are per-interface and cannot be attached to a veth inside a pod's netns without CAP_NET_ADMIN inside that namespace. And Cilium's O(1) advantage is real but bounded — at 200 services, iptables is entirely fine.
Pro-Tips & Common Pitfalls
- ❌ Hard-coded struct offsets.
sk->__sk_common.skc_daddrcompiled against 5.15 headers will read the wrong bytes on 6.8. Usebpf_core_read()/BPF_CORE_READ()and ship CO-RE. Symptom:-EINVALfromBPF_PROG_LOADwith a relocation error in the verifier log. - ❌ kprobes on hot paths. A kprobe on
tcp_sendmsgat 100k msg/s adds 120–250 ms of aggregate CPU per second per core. Usefentry/fexitor a raw tracepoint. - ❌ Silent ring buffer loss.
bpf_ringbuf_reserve()returningNULLis not an error path you log — it is a metric. Expose it as a per-CPU counter and alert onrate(...) > 0. - ❌ LRU_HASH for counters.
BPF_MAP_TYPE_LRU_HASHevicts without notification. Fine for in-flight connection state, catastrophic for packet counters. UseBPF_MAP_TYPE_PERCPU_HASH+ userspace aggregation for counters. - ❌ Blocking the userspace reader. Any
<-chwithout adefaultcase turns a 200 ms Go GC pause into a ring buffer overflow. Always drop-and-count. - ❌ Using `CAP_SYS_ADMIN` as a shortcut. Since 5.8,
CAP_BPF + CAP_PERFMON + CAP_NET_ADMINis the least-privilege set. In Kubernetes, mount/sys/fs/bpfand/sys/kernel/btfread-only into the agent and drop the rest. - ❌ Forgetting the `LICENSE` section. Missing or non-
GPLlicense strings make GPL-only helpers (bpf_probe_read_kernel,bpf_get_stackid, many kfuncs) fail at load with-EPERMand no useful message. - ❌ Attaching `tc` programs without a `clsact` qdisc.
tc qdisc add dev eth0 clsactfirst, orlink.AttachTCXon 6.6+. Symptom: attach returns-ENOENT. - ❌ Assuming native XDP.
xdpgenericruns in the GRO path and is 3–5× slower. Checkip link show dev eth0for thexdpfeature flag, and make the fallback explicit in your deployment config. - ✅ Size ring buffers for the burst, not the average.
max_entries ≈ events/s × avg_event_size × burst_seconds, rounded up to a power-of-two page multiple. 16 MiB covers ~300k events at 56 bytes. - ✅ Pin maps under `/sys/fs/bpf` (
BPF_F_PIN) when you want to survive agent restarts and read state from abpftool map dumpwithout reattaching. - ✅ Cap the verifier's ability to surprise you by splitting programs at ~100k instructions and joining with tail calls; keep
LogLevelBranchon in staging to see the full path proof.
Frequently Asked Questions (FAQ)
Q: Should I use kprobes or fentry/fexit for production eBPF observability?
Use fentry/fexit for anything on a hot path, and classic tracepoints when you need maximum portability. fentry requires kernel ≥ 5.5 and BTF (present by default on any modern distribution), attaches through the BPF trampoline rather than an int3 breakpoint, receives typed arguments instead of pt_regs, and measures 60–140 ns per event versus 1.2–2.5 µs for a kprobe. The caveat is symbol availability: if a vendor kernel inlines or renames the target function, the attach fails with -EINVAL. Tracepoints have a stable, documented ABI that survives kernel upgrades, so they are the safer choice for long-lived agents on heterogeneous fleets. A practical split: tracepoints for the stable contract, fentry for the per-connection detail.
Q: Does Cilium's eBPF data path actually replace kube-proxy, and what are the limits?
Yes — with kubeProxyReplacement=true, Cilium handles ClusterIP/NodePort/LoadBalancer/ExternalIP translation entirely in BPF, using a tc ingress program for externally-originated traffic and a socket-layer hook for pod-originated connections, backed by O(1) hash maps with Maglev consistent hashing. This removes the O(n) iptables rule walk and the multi-second iptables-restore stalls that cause real packet loss during endpoint churn on large clusters. The limits are: hostPort and some externalTrafficPolicy=Local edge cases still need careful configuration, sessionAffinity uses a separate affinity map with its own timeout semantics, and the whole thing depends on the node kernel being ≥ 4.19 (≥ 5.10 recommended for full feature parity). Mixed environments running both kube-proxy and Cilium in partial replacement mode are the most common source of silent DNAT bugs.
Q: How do I stop losing events in a BPF ring buffer under load?
Three levers, in order of impact. First, reduce volume in the kernel: sample with bpf_get_prandom_u32(), filter on the hot path before reserve(), and aggregate into per-CPU histogram maps instead of emitting one event per occurrence. Second, size correctly: max_entries must be a power of two and a page multiple, and should cover your worst burst — 16 MiB is a reasonable default for a node-level agent. Third, never block the consumer: use non-blocking sends with an explicit drop counter, and treat both bpf_ringbuf_reserve() failures (kernel-side) and userspace queue overflow as first-class SLO metrics. If you are still losing events after all three, your probe is on the wrong hook — move from a per-packet tc program to a per-connection socket-layer hook and the event rate typically drops by two orders of magnitude.
