Ask Your Kernel Anything: Building Native Observability Into a System You Compiled Yourself

There is a quiet irony in modern observability. Most administrators run half a dozen agents, exporters, and daemons to answer a simple question: what is my machine actually doing right now? Those tools work by inferring behavior through layers of abstraction — abstraction that exists because nobody who runs the system built it.

If you built your Linux From Scratch system by hand, that excuse is gone. You selected the kernel configuration. You compiled glibc. There is no vendor shim between you and the source of truth. That makes an LFS machine the finest observability laboratory you will ever own, and the tools to exploit it cost exactly nothing.

Start Where the Kernel Already Talks: /proc

Before you write a single line of code, spend an evening learning to read the kernel's own reporting interfaces. The proc filesystem is not documentation — it is a live, structured conversation with the scheduler, the memory manager, and the VM subsystem.

A few files repay close study:

• /proc/meminfo — the authoritative breakdown of memory state, more granular than any dashboard.
• /proc/pressure/{cpu,memory,io} — PSI (Pressure Stall Information), which tells you not how busy the system is, but how long tasks stalled waiting for resources. This is often the earliest meaningful signal of trouble.
• /proc/[pid]/stat and /proc/[pid]/status — per-process accounting you can parse with nothing more than awk.

A practical exercise: write a 40-line shell or Python script that samples PSI every five seconds and logs stalls above a threshold. You will catch memory pressure episodes that top never surfaces, because PSI measures waiting, not utilization.

Graduate to eBPF: Probes Without Payloads

Once you are comfortable with /proc, the next step is tracing — and on a modern kernel (5.x or later, which your LFS build almost certainly is), that means eBPF. The appeal for a hand-built system is structural: eBPF programs run inside the kernel's verifier sandbox, so you can attach probes to nearly any function without patching or reloading anything.

You have two realistic paths on an LFS base:

1. bcc and bpftrace. Porting these onto a from-source system is a genuine exercise in dependency management — LLVM, libbpf, and their prerequisites — but once present, bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }' gives you a live map of who is opening what.
2. Raw libbpf with CO-RE. For the committed builder, writing a small C program against libbpf, compiled against the same toolchain that built your kernel, is as close to the metal as userspace tracing gets.

One caveat worth planning for: enable CONFIG_BPF_SYSCALL, CONFIG_DEBUG_INFO_BTF, and kprobes in your kernel configuration now, before your next rebuild. Discovering their absence mid-investigation is a rite of passage, but an avoidable one.

Assemble the Daemon: Small, Honest, Yours

The final step is persistence. A monitoring daemon on an LFS system does not need to be general-purpose — it needs to answer your questions about your machine. A minimal design:

• A collection loop reading /proc and your eBPF maps on a fixed interval.
• A ring buffer or simple append-only log on disk — no database required at this scale.
• An optional HTTP endpoint exposing metrics in Prometheus text format, which takes about a hundred lines in C or Python and lets you plug into any existing dashboard later.
• An init script in your bootscripts directory, so the daemon starts under whatever init system you chose.

Because you control the toolchain, you can statically link the whole thing, strip it to a few hundred kilobytes, and know every syscall it makes — because you can trace it with your own eBPF probes. That closed loop, where your instrumentation can inspect your instrumentation, is unavailable on any stock distribution.

Why This Matters Beyond the Lab

The skills here transfer directly to professional work. Engineers who can read PSI output, write a bpftrace one-liner, and reason about where metrics originate are increasingly rare and increasingly valuable — the difference between watching a dashboard and understanding a system. Your LFS build is not just a hobby; it is the deepest possible rehearsal for debugging production infrastructure.

Start small: one script, one tracepoint, one log file. Within a week you will have a system that explains itself — in your own words, because you taught it.