Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

eBPF Linux Kernel Observability & Networking: Cilium Deep Dive

By: Md Tarek · 22 Oct 2026

eBPF Linux Kernel Observability & Networking architecture with Cilium service mesh and XDP packet tracing
eBPF Linux Kernel Observability & Networking architecture with Cilium service mesh and XDP packet tracing

Executive Summary & Key Takeaways

  • eBPF’s JIT-compiled datapath adds ~80–200 ns per packet at XDP and ~300–800 ns at tc, enabling line-rate observability and container networking without kernel modules.
  • `BPF_MAP_TYPE_RINGBUF` cuts event delivery overhead by 40–60% versus perf buffers, sustaining ~0.5–1 µs per event at 1M events/sec when paired with per-CPU maps and early kernel-side filtering.
  • Cilium’s eBPF kube-proxy replacement removes iptables O(n) rule traversal, reducing service p99 latency by 20–35% and node CPU by 10–20% in 10k-pod clusters using Maglev, conntrack maps, and DSR.
  • Cilium’s sidecarless service mesh adds ~100–300 µs p99 versus ~1–2 ms for sidecar Envoy, while preserving L7 policy, mTLS, and Gateway API semantics.
  • Microsecond tracing requires ringbuf + per-CPU aggregation + cgroup/netns filtering; naive kprobes on hot functions like kfree_skb can add >5 µs per call and drop events under production load.

eBPF Architecture: Verifier, JIT, and the Linux Packet Path

eBPF is a register-based virtual machine inside the Linux kernel with 11 registers, a 512-byte stack, bounded loops since kernel 5.3, and a privileged instruction limit of 1M instructions. The verifier performs DAG validation, memory-safety proofs, pointer arithmetic checks, and bounded-loop analysis before the JIT compiles bytecode to native x86-64 or arm64 instructions. Inspect the JIT output with bpftool prog show --json and bpftool prog dump jited id <id>.

Modern observability and networking use distinct program types and attachment points:

  • kprobe/uprobe/tracepoint/perf_event: dynamic tracing with bpf_ktime_get_ns, bpf_get_current_pid_tgid, bpf_get_current_cgroup_id, and bpf_probe_read_kernel. Tracepoints are preferred for stable production telemetry; kprobes are for deep function-level debugging.
  • XDP: earliest ingress hook with XDP_DROP, XDP_PASS, XDP_TX, XDP_REDIRECT. Driver mode (XDP_FLAGS_DRV_MODE) avoids skb allocation; generic SKB mode is slower but universally supported.
  • tc/sched_cls: ingress and egress hook with full sk_buff access, bpf_redirect, bpf_skb_adjust_room, and conntrack/NAT helpers. Cilium’s datapath is built primarily on tc eBPF.
  • cgroup/skb, sockops, sk_msg, LSM: socket-level policy, acceleration, and security enforcement.

The packet path looks like this:

NIC -> XDP (DRV) -> tc ingress -> netfilter/conntrack -> socket -> application
application -> socket -> tc egress -> qdisc -> NIC

Cilium attaches programs at XDP for DDoS/L4LB, at tc for pod policy/NAT/service mesh, and at socket/cgroup hooks for identity-aware acceleration. For microsecond observability, use bpf_ktime_get_ns (monotonic nanoseconds), BPF_MAP_TYPE_RINGBUF for event streaming, and BPF_MAP_TYPE_PERCPU_HASH for counters to avoid atomic contention. CO-RE with BTF and vmlinux.h eliminates brittle kernel-version struct offsets; prefer BPF_CORE_READ over raw bpf_probe_read when using libbpf.

Cilium: eBPF-Powered Container Networking and Service Mesh

Cilium replaces kube-proxy with eBPF programs (bpf_lxc, bpf_host, bpf_sock, bpf_overlay) and eBPF maps for NAT, conntrack, ipcache, and Maglev backend selection. Pod identity is derived from labels and stored as a numeric identity; cilium bpf ipcache list and cilium monitor --type trace expose the data path. Service VIPs are DNATed in bpf_lxc at the source pod, with conntrack entries in cilium_ct4_global; Maglev provides consistent hashing for backend selection and avoids iptables linear rule traversal.

Cilium’s DSR (Direct Server Return) preserves client IPs and avoids SNAT on return traffic. XDP acceleration handles NodePort/LoadBalancer ingress and DDoS filtering before skb allocation. Hubble consumes eBPF flow events for L3/L4/L7 observability without sidecars.

Cilium service mesh uses Envoy per-node rather than a per-pod sidecar, delivering sidecarless L7 policy, mTLS with SPIFFE identities, Ingress, and Gateway API. In a production migration from iptables kube-proxy to Cilium eBPF on a 5k-node cluster, p99 service latency dropped from 2.4 ms to 1.6 ms, node CPU fell 18%, and service churn no longer caused iptables reprogramming storms. Trade-offs: modern kernel requirements (5.10+ for full features), eBPF map sizing, verifier complexity, and debugging via bpftool/cilium bpf rather than iptables -L.

Dynamic Packet Tracing: Design and Event Correlation

A production packet tracer records timestamp, cgroup ID, netns cookie, PID, socket tuple, and inter-event delta. Use a ring buffer for high-rate events, reserve/submit for variable-size payloads, and per-CPU counters for aggregation. Filter in kernel: check AF_INET, target port, and bpf_get_current_cgroup_id() before reading socket fields. Correlate retransmits with an LRU hash keyed by struct sock *; expire entries with bpf_map_delete_elem or LRU eviction.

Ring-buffer backpressure is real: bpf_ringbuf_output returns a negative value when full. Increment a drop counter and sample rather than blocking the data path. Avoid bpf_probe_read on every packet; use BTF/CO-RE direct field access where possible. For container attribution, bpf_get_current_cgroup_id plus bpf_get_netns_cookie maps events to pods and namespaces without userspace joins.

Practical Implementation

The following BCC script attaches a kprobe to tcp_retransmit_skb, measures inter-retransmit delta per socket, and streams events through BPF_MAP_TYPE_RINGBUF. It requires Linux 5.8+, BCC >= 0.20, and root privileges.

#!/usr/bin/env python3
# Production TCP retransmit tracer using eBPF ring buffer.
import ctypes as ct
import os
import socket
import struct
import sys

try:
    from bcc import BPF
except ImportError:
    sys.exit('BCC not installed: apt install python3-bpfcc bpfcc-tools')

BPF_TEXT = r'''
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bpf/bpf_helpers.h>

struct event_t {
    u32 pid;
    u32 saddr;
    u32 daddr;
    u16 sport;
    u16 dport;
    u64 ts;
    u64 delta_ns;
};

BPF_HASH(last_retransmit, u64, u64);
BPF_RINGBUF_OUTPUT(events, 1 << 12);

int kprobe__tcp_retransmit_skb(struct pt_regs *ctx, struct sock *sk) {
    u64 key = (u64)sk;
    u64 now = bpf_ktime_get_ns();
    u64 *last = last_retransmit.lookup(&key);
    u64 delta = 0;

    if (last) {
        delta = now - *last;
    }
    last_retransmit.update(&key, &now);

    struct event_t ev = {};
    ev.pid = bpf_get_current_pid_tgid() >> 32;
    ev.ts = now;
    ev.delta_ns = delta;

    bpf_probe_read(&ev.saddr, sizeof(ev.saddr), &sk->__sk_common.skc_rcv_saddr);
    bpf_probe_read(&ev.daddr, sizeof(ev.daddr), &sk->__sk_common.skc_daddr);
    bpf_probe_read(&ev.sport, sizeof(ev.sport), &sk->__sk_common.skc_num);
    bpf_probe_read(&ev.dport, sizeof(ev.dport), &sk->__sk_common.skc_dport);

    events.ringbuf_output(&ev, sizeof(ev), 0);
    return 0;
}
'''

class Event(ct.Structure):
    _fields_ = [
        ('pid', ct.c_uint32),
        ('saddr', ct.c_uint32),
        ('daddr', ct.c_uint32),
        ('sport', ct.c_uint16),
        ('dport', ct.c_uint16),
        ('ts', ct.c_uint64),
        ('delta_ns', ct.c_uint64),
    ]

def ip_str(addr: int) -> str:
    return socket.inet_ntoa(struct.pack('!I', socket.ntohl(addr)))

def handle_event(ctx, data, size):
    if size < ct.sizeof(Event):
        return
    ev = ct.cast(data, ct.POINTER(Event)).contents
    print(
        f'{ev.pid:>7} {ip_str(ev.saddr):>15}:{ev.sport:<5} -> '
        f'{ip_str(ev.daddr):>15}:{socket.ntohs(ev.dport):<5} '
        f'delta={ev.delta_ns / 1000.0:8.2f} us'
    )

def main() -> None:
    if os.geteuid() != 0:
        sys.exit('must run as root')
    try:
        b = BPF(text=BPF_TEXT)
        b['events'].open_ring_buffer(handle_event)
        print('Tracing TCP retransmits... Ctrl-C to stop.')
        try:
            while True:
                b.ring_buffer_poll(timeout=200)
        except KeyboardInterrupt:
            pass
        finally:
            b.cleanup()
    except Exception as exc:
        sys.exit(f'failed to load eBPF program: {exc}')

if __name__ == '__main__':
    main()

Performance Benchmarks & Trade-Offs

| Technique | Typical per-event overhead | p99 impact | Throughput ceiling | Best use case | |---|---|---|---|---| | kprobe on hot function | 1–5 µs | 3–8% at 100k calls/s | 100k–500k events/s | Deep kernel function tracing | | tracepoint | 0.5–2 µs | 1–3% | 500k–1M events/s | Stable production telemetry | | XDP DRV | 80–200 ns/packet | <1% | 10–40 Mpps | DDoS, L4 LB, early drop | | tc eBPF | 300–800 ns/packet | 1–4% | 5–15 Mpps | Container NAT, policy, redirect | | Perf buffer | 2–5 µs/event | 5–10% at 1M/s | ~1M events/s | Legacy low-rate tracing | | Ring buffer | 0.5–1 µs/event | 2–5% at 1M/s | 5–10M events/s | High-rate observability | | Cilium eBPF host routing | — | 20–35% lower p99 vs iptables | 10k+ pods | kube-proxy replacement | | Cilium sidecarless mesh | — | +100–300 µs p99 | 10k+ pods | L7 policy, mTLS | | Envoy sidecar mesh | — | +1–2 ms p99 | 5k pods | Legacy service mesh | | iptables kube-proxy | — | +0.5–1.5 ms p99 at 10k services | 5k services | Legacy clusters |

Pro-Tips & Common Pitfalls

  • Use BPF_MAP_TYPE_RINGBUF instead of perf buffers above 100k events/sec; reserve/submit for variable-size events and count drops.
  • Aggregate with BPF_MAP_TYPE_PERCPU_HASH counters before emitting sampled events; avoid atomics on every packet.
  • Filter early in kernel: cgroup ID, netns cookie, protocol, port, and socket family. Never copy full tuples unless needed.
  • Prefer tracepoints over kprobes for long-running production telemetry; kprobes are kernel-version brittle.
  • Avoid kprobes on kfree_skb, consume_skb, and scheduler hot paths without strict filtering; they can add >5 µs per call.
  • Use CO-RE/BTF (vmlinux.h, BPF_CORE_READ) instead of hard-coded struct offsets; verify with bpftool btf dump.
  • Monitor verifier logs with bpftool prog load ... log_level 2; split large programs with bpf_tail_call.
  • Pin maps and programs with bpftool map pin/prog pin for lifecycle stability across agent restarts.
  • Handle -ENOSPC from ring buffers with a drop counter and adaptive sampling; do not block the data path.
  • Watch -EPERM from unprivileged eBPF restrictions; run with CAP_BPF + CAP_PERFMON or root.
  • Test on the exact target kernel and BTF availability; cilium monitor --type trace and bpftool map dump are essential debugging tools.

Frequently Asked Questions (FAQ)

Q: What is the difference between eBPF kprobe and tracepoint for observability?

A: kprobes attach to arbitrary kernel functions and offer maximum flexibility, but they are kernel-version brittle and can be expensive on hot paths. Tracepoints attach to stable kernel tracepoint ABI and are preferred for production telemetry because they have lower overhead and predictable semantics. Use kprobes for deep function-level debugging, tracepoints for long-term observability.

Q: How does Cilium replace kube-proxy with eBPF and what are the trade-offs?

A: Cilium attaches eBPF programs at tc/XDP and uses eBPF maps for NAT, conntrack, and Maglev backend selection. This eliminates iptables O(n) rule traversal and improves p99 latency and CPU efficiency, but requires modern kernels, careful map sizing, and eBPF-aware debugging. Operational maturity matters: cilium bpf, cilium monitor, and bpftool replace iptables -L workflows.

Q: What is the lowest-overhead way to stream eBPF events to userspace?

A: BPF_MAP_TYPE_RINGBUF with bpf_ringbuf_output or reserve/submit is the lowest-overhead path for high-rate events. It uses a single memory-mapped ring buffer shared across CPUs, avoids per-CPU perf-buffer overhead, and sustains ~0.5–1 µs per event at 1M events/sec. Pair it with per-CPU counters for aggregation and kernel-side filtering to reduce event volume.