Debugging #

Debugging a BPF program is kind of tricky for the following reasons:

• The return code of a tracepoint program is ignored.
• If your BPF program crashes or encounters an error, nothing will alert your userspace program.
• If something goes wrong, you'll mostly just see silent failures or no output.

There's basically two ways to debug your BPF program:

• "printf" debugging through TraceFS
• Writing diagnostics to a map and reading the map from userspace

"printf" debugging #

You can't write directly to userspace, but you can write to the kernel's trace log. The helper function bpf_trace_printk allows you to print to this trace log.

Messages written to the trace log can be read by running cat or tail on /sys/kernel/tracing/trace. I'm not sure if this is typical behavior, but if you cat this file while your program is running, you won't see the output in this "file".

You can also read /sys/kernel/tracing/trace_pipe, which streams output rather than buffering it, but there are caveats:

• The piped output is split between all readers of the pipe.
• The output read from the pipe isn't persisted anywhere, you consume it.

Writing to a map #

BPF responds to backpressure by dropping data, generally. For instance, it's possible to fail to allocate data in a map, and there's not really a good way to recover from that. As a debugging tool, it's our responsibility to let a user know when we've dropped data that they might be depending on.

If the event ringbuffer is full, we can't very well write diagnostic information to that ringbuffer. Instead, we keep a second ringbuffer dedicated to diagnostics. We write to this ringbuffer whenever we encounter an error so that userspace has some way of knowing.