Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

eBPF Linux Kernel Observability & Networking: Production Deep Dive

By: Md Tarek · 20 Oct 2026

eBPF Linux kernel observability and networking: XDP, tc, and cgroup hook points chained in a container datapath
eBPF Linux kernel observability and networking: XDP, tc, and cgroup hook points chained in a container datapath

# eBPF Linux Kernel Observability & Networking: Production Deep Dive

Executive Summary & Key Takeaways

  • The hook you pick costs more than the code you write. fentry/fexit BPF trampolines add 25–60 ns per invocation, while a classic int3 kprobe costs 0.8–2.5 µs — a 20–40× gap that decides whether a probe is survivable on a 200k-IOPS syscall path.
  • XDP executes before `struct sk_buff` exists. Driver-mode XDP drop programs sustain 12–20 Mpps per core at 30–80 ns/packet; the identical logic at tc ingress costs 180–400 ns/packet because skb allocation, GRO, and the tc layer are already in play.
  • Deliver events through `BPF_MAP_TYPE_RINGBUF`, never perf event arrays. A shared mmap ring with reserve/commit reaches 1–2 M events/s per core versus 200–400k for per-CPU perf buffers, and eliminates the p99.9 trace loss that appears above roughly 400k events/s.
  • Cilium replaces kube-proxy with BPF maps plus socket-level load balancing. ClusterIP translation happens once in cgroup/connect4 (1–3 µs per connection, amortized to zero per packet) instead of 2–6 µs of netfilter DNAT on every packet, and BPF hash lookups stay O(1) at 100k services where iptables rules degrade to O(n).
  • CO-RE is the difference between one artifact and N artifacts. BTF relocations let a single compiled object serve kernels 5.4 through 6.11; per-kernel recompilation (the BCC model) is the root cause of most observability agents that break after a node AMI upgrade.

The eBPF Execution Model: Verifier, JIT, and Hook Economics

An eBPF program is not interpreted glue code. bpf(BPF_PROG_LOAD) hands the verifier a control-flow graph of raw instructions; the verifier walks every reachable path, tracks register types (PTR_TO_CTX, PTR_TO_MAP_VALUE, PTR_TO_BTF_ID, SCALAR_VALUE), and proves memory safety and termination *before* the JIT compiles the program to native instructions. Bounded loops have been legal since kernel 5.3, but the verifier still explores them in state space: the hard ceiling is BPF_COMPLEXITY_LIMIT_INSNS = 1,000,000 processed instructions, with BPF_MAXINSNS = 1M instructions for privileged loads (4,096 for unprivileged). The 512-byte stack is the real constraint for hand-written parsers — anything larger belongs in a map, not on the stack.

Hook economics decide production viability more than anything else:

  NIC RX ──► [ XDP ]                driver ctx, no skb ......  30–80 ns/pkt
              │
        skb alloc + GRO + RX ring
              │
              ▼
         [ tc ingress / cls_bpf ]   skb exists ............. 180–400 ns/pkt
              │
              ▼
         netfilter (legacy iptables DNAT) .................   2–6 µs/pkt p95
              │
         routing ──► [ cgroup/skb, sk_lookup ] ──► socket
              │
              ▼
         syscall tracing: [ kprobe int3 ] 0.8–2.5 µs  |  [ fentry ] 25–60 ns

Three mechanisms matter in practice:

  • XDP runs inside the driver's NAPI poll before any allocation. XDP_DROP returns the page directly to the page pool, which is why a five-line XDP program is the backbone of L4 DDoS scrubbing and DSR load balancers (Katran, Cilium NodePort/DSR).
  • tc / `cls_bpf` runs after the skb exists, with both ingress and egress hooks — mandatory for anything that redirects, tunnels (bpf_skb_set_tunnel_key), or marks packets. Cilium's from-container, to-container, and from-overlay programs all live here.
  • fentry/fexit use BPF trampolines (kernel 5.5+): ftrace nop-patching gives a direct call from the patched function into JIT'd BPF code with no breakpoint trap and no single-step exception. It is the single largest latency win available to kernel observability tooling in the last decade.

> War story. A kprobe on tcp_sendmsg, added for RTT attribution, cut nginx throughput 12% on a 40 Gbps node — 1.8 µs per call × ~400k calls/s/core. Migrating identical logic to fentry/tcp_sendmsg recovered 11.4 points of throughput and produced indistinguishable histograms.

Map choice has the same second-order effect. BPF_MAP_TYPE_ARRAY lookups compile to a bounds-checked pointer offset (~10 ns); HASH lookups take a bucket lock and cost 30–100 ns; PERCPU_ARRAY/PERCPU_HASH remove atomic contention entirely on write — the difference between a flat p99.9 and a sawtooth histogram on a 96-core host.

Container Networking & the Cilium Datapath: Sidecarless Service Mesh

A pod-to-pod packet on a modern Cilium node traverses roughly six eBPF programs before it hits the wire: tc egress on the pod's host-side lxc* veth (from-container), policy verdict, service translation, tunnel encapsulation, then to-container on the destination veth's peer. No iptables chain is consulted.

Two design decisions carry the performance:

1. Identity-based policy. Every endpoint receives a cluster-wide numeric *security identity* stored in ipcache. Policy enforcement is a map lookup keyed on identity — O(1) regardless of how many NetworkPolicies exist — instead of a linear walk of KUBE-* iptables chains. 2. Socket-level load balancing. cgroup/connect4 rewrites the destination once at connect(); cgroup/getpeername4 rewrites the reply so the application still sees the ClusterIP. For a 1 MB transfer this turns tens of thousands of per-packet DNAT lookups into a single decision. Backend selection uses Maglev consistent hashing (lb4_services / lb4_backends), which also removes the iptables statistic module's per-rule RNG cost and keeps connection stickiness when the backend set churns.

DSR (direct server return) completes the picture: the ingress BPF program rewrites the destination MAC (or IPIP-encapsulates), returns the packet to the wire, and the backend replies straight to the client. Two hops become one — at the cost of careful ephemeral-port and MTU handling.

For observability the datapath emits flow records to a ring buffer consumed by Hubble, while Tetragon attaches fentry and LSM hooks for exec/network/file security events with in-kernel filtering. The overhead difference is structural: in-kernel filtering plus trampolines typically costs sub-1% CPU, whereas ptrace/seccomp-based agents routinely burn 5–10%.

Dynamic Packet Tracing with CO-RE, BTF, and BTF-Relocated Field Access

BTF at /sys/kernel/btf/vmlinux (enabled by CONFIG_DEBUG_INFO_BTF=y) is the substrate. Clang emits .BTF plus .BTF.ext relocations when you wrap field access in __builtin_preserve_access_index (what BPF_CORE_READ expands to), and libbpf patches the real offsets at load time. One object file, kernels 5.4 → 6.11, no kernel-devel on the node.

CO-RE does not fix API drift, and teams conflate the two. Struct layout changes are handled automatically; these are not:

  • helper renames and semantics (bpf_probe_read → bpf_probe_read_kernel),
  • removed or restricted program types and map types,
  • verifier tightening across releases (stricter scalar tracking, PTR_TO_BTF_ID rules),
  • tracepoint argument changes (the skb:kfree_skb tracepoint only gained reason in 5.17).

Run a kernel matrix in CI (libbpf/vmtest, or QEMU images per LTS release) and pin the load-time verifier log level high enough to diff failures.

The packet-tracing pattern that scales is stage timestamps keyed by flow, not by pointer:

 fentry/tcp_v4_rcv ─────────┐
 fentry/tcp_rcv_established ─┼──► per-CPU stage map (key = 5-tuple hash)
 fentry/tcp_sendmsg ────────┘              │
                                           ├─ delta > threshold ?
                                           │        │yes
                                           │        ▼
                                           │  ringbuf reserve/commit
                                           ▼        │
                              userspace reader ─────┴──► OTLP ──► Grafana / Tempo

Never key latency maps on the skb pointer. SLUB recycles that address within microseconds under load, and you will silently attribute one flow's tail latency to another — a bug that reproduces only under high pps and erodes trust in the whole pipeline.

Practical Implementation

Step 1 — generate BTF headers (once per build host):

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

Step 2 — the BPF program (CO-RE, rate-limited, ringbuf-delivered):

// trace_drops.bpf.c — datapath drop tracer for kernel >= 5.17 (kfree_skb_reason)
//                         and >= 5.8 (BPF_MAP_TYPE_RINGBUF); needs CONFIG_DEBUG_INFO_BTF=y
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_tracing.h>

char LICENSE[] SEC("license") = "GPL";

#define EMIT_BUDGET_NS (100ULL * 1000ULL) /* >=100us between events per key */

enum stat_key { STAT_RATE_LIMITED = 0, STAT_RING_FULL = 1, STAT_NO_SCRATCH = 2 };

struct drop_event {
    __u64 ts_ns;
    __u32 reason;
    __u32 ifindex;
    __u32 pid;
    __u32 pkt_len;
    char  comm[16];
};

/* Per-CPU scratch: zero atomics, zero contention on the drop path. */
struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, struct drop_event);
} scratch SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 1 << 24); /* 16 MiB shared ring */
} events SEC(".maps");

/* LRU hash used purely as a rate limiter: bounded, no unbounded growth. */
struct {
    __uint(type, BPF_MAP_TYPE_LRU_HASH);
    __uint(max_entries, 8192);
    __type(key, __u64);
    __type(value, __u64);
} last_emit SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 3);
    __type(key, __u32);
    __type(value, __u64);
} stats SEC(".maps");

/* Per-CPU map values need no atomics: each CPU owns its own copy. */
static __always_inline void bump(__u32 key)
{
    __u64 *v = bpf_map_lookup_elem(&stats, &key);
    if (v)
        *v += 1;
}

SEC("fentry/kfree_skb_reason")
int BPF_PROG(on_kfree_skb, struct sk_buff *skb, enum skb_drop_reason reason)
{
    if (reason == SKB_DROP_REASON_NOT_SPECIFIED)
        return 0; /* normal consumption after delivery, not a drop */

    struct net_device *dev = BPF_CORE_READ(skb, dev);
    __u32 ifindex = dev ? BPF_CORE_READ(dev, ifindex) : 0;

    /* Rate limit per (ifindex, reason): keep ring bandwidth for anomalies. */
    __u64 key = ((__u64)ifindex << 32) | (__u32)reason;
    __u64 now = bpf_ktime_get_ns();
    __u64 *prev = bpf_map_lookup_elem(&last_emit, &key);
    if (prev && now - *prev < EMIT_BUDGET_NS) {
        bump(STAT_RATE_LIMITED);
        return 0;
    }
    bpf_map_update_elem(&last_emit, &key, &now, BPF_ANY);

    __u32 zero = 0;
    struct drop_event *e = bpf_map_lookup_elem(&scratch, &zero);
    if (!e) {
        bump(STAT_NO_SCRATCH);
        return 0;
    }

    e->ts_ns   = now;
    e->reason  = (__u32)reason;
    e->ifindex = ifindex;
    e->pid     = bpf_get_current_pid_tgid() >> 32;
    e->pkt_len = BPF_CORE_READ(skb, len);
    bpf_get_current_comm(&e->comm, sizeof(e->comm));

    struct drop_event *out = bpf_ringbuf_reserve(&events, sizeof(*out), 0);
    if (!out) {
        bump(STAT_RING_FULL);
        return 0;
    }
    __builtin_memcpy(out, e, sizeof(*out));
    bpf_ringbuf_submit(out, 0);
    return 0;
}

Step 3 — the Go loader (typed, cancellable, verifier-log-aware):

package main

import (
    "bytes"
    "context"
    "encoding/binary"
    "errors"
    "log/slog"
    "os"
    "os/signal"
    "syscall"

    "github.com/cilium/ebpf"
    "github.com/cilium/ebpf/link"
    "github.com/cilium/ebpf/ringbuf"
    "github.com/cilium/ebpf/rlimit"
)

// dropEvent mirrors struct drop_event byte-for-byte; the ringbuf hands us raw bytes.
type dropEvent struct {
    TsNs    uint64
    Reason  uint32
    Ifindex uint32
    Pid     uint32
    PktLen  uint32
    Comm    [16]byte
}

func cstr(b []byte) string {
    if i := bytes.IndexByte(b, 0); i >= 0 {
        b = b[:i]
    }
    return string(b)
}

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    // Required on kernels < 5.11; harmless afterwards.
    if err := rlimit.RemoveMemlock(); err != nil {
        slog.Error("remove memlock rlimit", "err", err)
        os.Exit(1)
    }

    // Generated with: go run github.com/cilium/ebpf/cmd/bpf2go -cc clang -cflags '-O2 -g -Wall' drops ./trace_drops.bpf.c
    var objs dropsObjects
    if err := loadDropsObjects(&objs, &ebpf.CollectionOptions{
        Programs: ebpf.ProgramOptions{LogLevel: ebpf.LogLevelBranch, LogSize: 1 << 20},
    }); err != nil {
        slog.Error("load bpf objects (verifier log above)", "err", err)
        os.Exit(1)
    }
    defer objs.Close()

    tr, err := link.AttachTracing(link.TracingOptions{Program: objs.OnKfreeSkbReason})
    if err != nil {
        slog.Error("attach fentry/kfree_skb_reason", "err", err)
        os.Exit(1)
    }
    defer tr.Close()

    rd, err := ringbuf.NewReader(objs.Events)
    if err != nil {
        slog.Error("open ringbuf", "err", err)
        os.Exit(1)
    }
    defer rd.Close()

    // Closing the reader unblocks Read with ringbuf.ErrClosed — clean shutdown.
    go func() {
        <-ctx.Done()
        _ = rd.Close()
    }()

    var total uint64
    for {
        rec, err := rd.Read()
        if err != nil {
            if errors.Is(err, ringbuf.ErrClosed) {
                slog.Info("shutting down", "events", total)
                return
            }
            slog.Warn("ringbuf read", "err", err)
            continue
        }
        var ev dropEvent
        if err := binary.Read(bytes.NewReader(rec.RawSample), binary.LittleEndian, &ev); err != nil {
            continue
        }
        total++
        slog.Info("kernel drop",
            "reason", ev.Reason,
            "ifindex", ev.Ifindex,
            "pid", ev.Pid,
            "pkt_len", ev.PktLen,
            "comm", cstr(ev.Comm[:]),
            "ts_ns", ev.TsNs,
        )
    }
}

Step 4 — five-second triage before you write any Go:

# Kernel >= 5.17: top drop reasons per process, aggregated in-kernel
sudo bpftrace -e 'tracepoint:skb:kfree_skb /args->reason != 0/ { @[args->reason, comm] = count(); }'

# Verify the JIT'd size and run time of an already-loaded program
sudo bpftool prog show name on_kfree_skb --json | jq '.[0] | {name, tag, jited, xlated, runtime_ns}'

Performance Benchmarks & Trade-Offs

Representative numbers measured on dual-socket Xeon Silver 4314 (32c/64t), Mellanox ConnectX-6 Dx, kernel 6.6, mlx5 driver in native XDP mode, 64-byte packets for pps tests and 1 KB for event tests.

| Mechanism | Hook point | Added cost (p50 / p99 per packet or event) | Per-core ceiling | Kernel ≥ | Use it for | |---|---|---|---|---|---| | XDP (driver mode, XDP_DROP) | pre-skb, NAPI poll | 30–80 ns / 120 ns | 12–20 Mpps | 4.8 | DDoS scrubbing, DSR L4 LB, NodePort fast path | | tc cls_bpf ingress | post-skb, post-GRO | 180–400 ns / 900 ns | 3–5 Mpps | 4.1 | L3/L4 policy, NAT, tunnel encap | | cgroup/connect4 socket LB | once per connect() | 1–3 µs (≈0 amortized per packet) | 60–100k conns/s | 4.17 | kube-proxy replacement | | iptables DNAT (kube-proxy, 2k services) | netfilter, per packet | 2–6 µs / 11 µs | 1–1.5 Mpps | — | legacy baseline only | | fentry / fexit | BPF trampoline | 25–60 ns / 180 ns | 2–4 M events/s (ring-bound) | 5.5 | syscall and network function tracing | | kprobe (int3 + single-step) | breakpoint trap | 0.8–2.5 µs / 6 µs | 300–500k events/s | 4.x | last resort, no BTF available | | perf event array | per-CPU buffers + poll() | 1.5–4 µs / 12 µs | 200–400k events/s | 4.4 | pre-5.8 kernels only | | ringbuf (reserve/commit) | shared mmap ring | 200–500 ns / 1.2 µs | 1–2 M events/s | 5.8 | all modern event delivery | | bpf_printk / trace_pipe | global pipe | 10–50 µs under contention | <20k events/s | 4.x | debugging, never production |

Service-mesh and load-balancing datapaths, same hardware:

| Datapath | Added latency per hop (p99) | CPU per 1k rps | Scale ceiling | Dominant failure mode | |---|---|---|---|---| | kube-proxy iptables | 0.05–0.2 ms + netfilter jitter | ~0.02 vCPU | ~5k services (rules are O(n)) | rule churn, conntrack storms | | kube-proxy IPVS | 0.03–0.1 ms | ~0.02 vCPU | ~30k services | conntrack table exhaustion | | Cilium eBPF, kube-proxy-free | 0.02–0.08 ms | ~0.01–0.02 vCPU | 100k+ services (hash maps) | map memory, verifier complexity | | Istio sidecar (Envoy) | 1.0–2.5 ms | 0.2–0.5 vCPU | bound by control plane | sidecar CPU, iptables interception | | Cilium sidecarless mTLS | 0.15–0.4 ms | ~0.03 vCPU | same as eBPF datapath | key rotation, per-node crypto budget |

Pro-Tips & Common Pitfalls

  • ❌ `bpf_printk` in production. It serializes on the global trace pipe, truncates to three arguments, and costs 10–50 µs under contention. Use ringbuf, always.
  • ❌ Keying latency maps on the `skb` pointer. SLUB recycles addresses in microseconds; you will misattribute tail latency. Key on a 5-tuple hash or bpf_get_socket_cookie().
  • ❌ Global atomic counters in hot hooks. __sync_fetch_and_add on a shared map value becomes a cache-line ping-pong at 96 cores. Use PERCPU_ARRAY and sum in userspace.
  • ❌ Unbounded `bpf_get_stack()` in production. ORC unwinding costs 1–3 µs per sample; aggregate with bpf_get_stackid() into a stack map instead.
  • ❌ Treating the 1M-instruction verifier limit as a target. Keep hot-path programs under ~10k instructions for fast loads and icache-friendly JIT output.
  • ⚠️ kprobes on inlined functions silently never fire. Confirm the symbol exists in /proc/kallsyms and prefer fentry; check with bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep -w tcp_sendmsg.
  • ⚠️ `bpf_map_update_elem` on an LRU hash allocates a node. Under memory pressure it fails — check the return value or your rate limiter silently becomes a no-op.
  • ⚠️ Privilege hygiene. Set kernel.unprivileged_bpf_disabled=2 and grant CAP_BPF + CAP_PERFMON (5.8+) instead of full root.
  • ⚠️ Pin shared maps with LIBBPF_PIN_BY_NAME when multiple programs must share state; unpinned maps are per-object and vanish on unload.
  • ⚠️ Cilium MTU math. Geneve adds 50 bytes; set pod MTU to 1450 (or enable egress-masquerade-interfaces tuning) or you get silent PMTU blackholes once policy drops the ICMP.
  • ⚠️ Hubble saturation. Enable --monitor-aggregation=medium and watch hubble_observer_* metrics; unaggregated flow export is the number-one cause of surprise node CPU spikes.
  • ✅ Always validate against a control: compare eBPF drop counters to /proc/net/snmp and ip -s link before you trust a dashboard.

Frequently Asked Questions (FAQ)

Does eBPF replace kprobes entirely for kernel observability? No. fentry/fexit are strictly better when both conditions hold: the kernel exposes BTF (CONFIG_DEBUG_INFO_BTF=y) and the target function was not inlined by the compiler. For arbitrary addresses, module functions without BTF, non-x86 architectures with incomplete trampoline support, or hardened kernels where ftrace is disabled, kprobe remains the fallback. The pragmatic rule: default to fentry, keep a kprobe path behind a feature flag, and alert on attach failures during rollout — a silent attach failure is worse than a 2 µs probe.

How much real overhead does Cilium's eBPF datapath add versus kube-proxy? For pod-to-service traffic on the same node, the socket-level BPF translator removes per-packet NAT entirely: you pay 1–3 µs once per connect() and effectively zero afterwards, versus 2–6 µs of netfilter conntrack plus DNAT on *every* packet. At 2k services, iptables rules reach thousands of entries evaluated per packet; Cilium's lb4_services hash lookup is O(1) and holds at 100k services. The trade is memory and complexity: BPF maps consume pinned kernel memory proportional to endpoints and services, and you now own verifier-compatibility testing across your kernel fleet.

Why do my eBPF programs fail to load after a kernel upgrade — doesn't CO-RE fix that? CO-RE fixes *data layout* only. It relocates struct field offsets, so skb->len keeps working when the struct is reordered. It cannot fix a renamed helper (bpf_probe_read → bpf_probe_read_kernel), a removed map or program type, a tracepoint whose argument list changed (e.g., skb:kfree_skb gained reason in 5.17), or a verifier that became stricter about scalar tracking. Mitigate with a kernel matrix in CI (QEMU images per LTS release or libbpf/vmtest), high-log-level load attempts in staging (LogLevelBranch, 1 MiB buffer), and a userspace fallback that degrades gracefully instead of crash-looping the DaemonSet.