eBPF and bpftrace: Modern Kernel Observability Without Kernel Modules

1. What eBPF is and why it changed kernel tracing

eBPF, short for Extended Berkeley Packet Filter, is a virtual machine inside the Linux kernel that runs small, verified programs directly in kernel context, without compiling and loading a dedicated kernel module. A verifier checks every eBPF program before execution to make sure it terminates, does not perform disallowed memory access, and cannot destabilize the kernel.

That combination of safety and kernel proximity was previously all but impossible: classic kernel modules run with full privileges and can crash the entire system on a bug, while userspace tools like strace have to hand every observed event back to userspace through an expensive context switch. eBPF programs evaluate events right where they occur and only return the relevant, already aggregated data to userspace.

2. eBPF use cases beyond tracing

eBPF has long since moved beyond pure observability. Network filtering and load balancing via XDP, container networking in Kubernetes implementations such as Cilium, and security monitoring through projects like Falco all build on the same kernel mechanism. For everyday admin work on a Magento hosting server, however, observability remains by far the most relevant application.

The key difference from classic tracing lies in the evaluation logic: instead of copying every raw event to userspace, an eBPF program can already filter, aggregate, and build histograms inside the kernel. Only the result of that evaluation leaves the kernel, cutting overhead by orders of magnitude compared to ptrace based tools.

3. bpftrace as a high-level language for eBPF

Writing raw eBPF usually means compiling C code against the BPF bytecode target and loading it via libbpf, an amount of effort that is impractical for a quick diagnosis on a production server. bpftrace closes that gap with an AWK-like scripting language that compiles one-liners and short scripts into eBPF bytecode and loads them automatically.

The core idea in bpftrace is probes, named attachment points in the kernel or in userspace programs where a script block runs as soon as the event fires. Probe types such as tracepoint for stable kernel events, kprobe for arbitrary kernel functions, and uprobe for userspace functions together cover practically every conceivable observation point.

# Install bpftrace (Debian/Ubuntu)
apt install bpftrace

# Simplest starting point: capture all open() calls system wide live
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

4. Practical example: measuring syscall latency

One of the most valuable questions in everyday hosting is how long certain system calls actually take, not on average across all processes, but as a distribution. bpftrace answers that with built-in histogram functions, storing the timestamp on syscall entry and recording the difference into a histogram on exit.

The script below measures the latency of all read calls per process name and renders it as a histogram. Unlike strace, no per-call overhead for a userspace context switch is incurred, aggregation happens directly inside the kernel, so this approach can run for hours on a production system without a problem.

# Latency of read() calls per process name as a histogram
bpftrace -e '
tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read /@start[tid]/ {
    @latency_ns[comm] = hist(nsecs - @start[tid]);
    delete(@start[tid]);
}'

5. Practical example: tracking file opens live

For the question of which process accesses which files, without burdening the entire process tree the way strace does, a probe on the openat tracepoint combined with a filter on the process name works well. That pays off especially when it is unclear which of several PHP-FPM workers is even touching the problematic file.

Another typical use case in a Magento context is spotting unexpected access to configuration files or backup directories, for instance to verify that a cron job really only touches the expected paths instead of accidentally reading production database dumps.

# Show all file opens by php-fpm processes live
bpftrace -e '
tracepoint:syscalls:sys_enter_openat
/comm == "php-fpm"/
{
    printf("%d %s\n", pid, str(args->filename));
}'

6. Ready-made tools instead of custom scripts: the BCC collection

For many standard questions, no custom bpftrace script needs to be written. The BCC project, a sibling project built on the same eBPF foundation, ships ready-made command line tools such as opensnoop for file opens, execsnoop for newly started processes, and biolatency for block I/O latency as a histogram.

These tools are available on many distributions via the bpfcc-tools package and already cover a large share of daily diagnostic needs without writing a custom bpftrace script. Writing your own script pays off mainly when a very specific combination of filter criteria is needed that no existing tool covers.

# Install BCC tools (Debian/Ubuntu)
apt install bpfcc-tools

# Show all newly opened files system wide live
opensnoop-bpfcc

# Block device I/O latency as a histogram over 10 seconds
biolatency-bpfcc 10 1

7. Requirements and limits of eBPF in practice

eBPF requires a reasonably current kernel, for stable tracepoints and full bpftrace functionality kernel 5.x or newer is recommended, along with CONFIG_DEBUG_INFO_BTF for type information based probes without a separate kernel headers package. On older enterprise distributions with long term maintained LTS kernels, availability of individual tracepoints should be checked first.

In containers, and especially in managed cloud environments, eBPF tracing is often restricted or disabled entirely, because it needs elevated capabilities such as CAP_BPF or CAP_SYS_ADMIN on the host, which are rarely passed through to individual containers for security reasons. On classic dedicated or KVM-virtualized hosting servers this is usually not an issue.

8. Comparison with strace and ftrace

strace observes a single process via ptrace, incurring an expensive context switch per syscall, whereas eBPF evaluates events directly inside the kernel and reports back only aggregated results, which is what makes continuous, system wide tracing practical in the first place. For a focused, deep-dive analysis of a single hanging process, strace often remains the faster reach, since no script needs to be written.

ftrace is the older, also kernel-built-in tracing mechanism that, among others, perf relies on, and was long the standard answer for function call tracing inside the kernel. eBPF conceptually builds on the same tracepoints ftrace uses, but offers far more programmability, since custom filter logic and aggregation can be expressed directly inside the kernel program instead of merely logging raw events.

9. Best practices for production use of bpftrace

A bpftrace script should always be verified on a test system with a similar kernel version first, before running on a production Magento hosting server, since tracepoint names and available fields can vary between kernel versions. A script that runs fine on one kernel can abort with a cryptic error on another if a referenced tracepoint simply does not exist there.

Aggregations such as @start[tid] should consistently be removed with delete() once evaluated, since bpftrace otherwise accumulates unbounded map entries over a long run and consumes memory itself. For one-liners on the command line that is often negligible, for scripts meant to run in the background for hours it is mandatory.

Anyone using bpftrace regularly for the same question, such as daily latency measurements for database volumes, should version the script and document which kernel version and which probe types it requires, so switching to a newer kernel does not silently produce wrong results.

# Check availability of a specific tracepoint before production use
bpftrace -l 'tracepoint:syscalls:sys_enter_openat'

# Store the script with a clearly documented kernel requirement
# scripts/bpftrace/read-latency.bt (kernel 5.10+, requires CONFIG_DEBUG_INFO_BTF)

10. Summary

11. FAQ: eBPF/bpftrace