Getting started with BCC (BPF Compiler Collection)

Posted on Edited on In , Views:

Intro to BCC

BCC is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples. It makes use of extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1 and above.

Install BCC from packages

In general, a Linux kernel version 4.1 or newer is required.

$ cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)

$ uname -r
5.7.12-1.el7.elrepo.x86_64

In addition, the kernel should have been compiled with the following flags set.

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# [optional, for tc filters]
CONFIG_NET_CLS_BPF=m
# [optional, for tc actions]
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
# [for Linux kernel versions 4.1 through 4.6]
CONFIG_HAVE_BPF_JIT=y
# [for Linux kernel versions 4.7 and later]
CONFIG_HAVE_EBPF_JIT=y
# [optional, for kprobes]
CONFIG_BPF_EVENTS=y
# Need kernel headers through /sys/kernel/kheaders.tar.xz
CONFIG_IKHEADERS=y

There are a few optional kernel flags needed for running bcc networking examples on vanilla kernel:

CONFIG_NET_SCH_SFQ=m
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_DUMMY=m
CONFIG_VXLAN=m

These kernel configuration might be set by default after the OS installation but you should double check as below.

$ cat /boot/config-$(uname -r)

To install the BCC tools from the official yum repository:

$ yum install bcc-tools
Installed:
  bcc-tools.x86_64 0:0.10.0-1.el7

Dependency Installed:
  bcc.x86_64 0:0.10.0-1.el7   python-bcc.x86_64 0:0.10.0-1.el7

The following BCC tools are pre-defined and available to use after installation.

$ cd /usr/share/bcc
$ ls
introspection  tools
$ cd /usr/share/bcc/tools
$ ls
argdist bpflist cobjnew dcstat ext4dist funclatency javagc llcstat nfsslower opensnoop phpstat pythonstat rubystat sofdsnoop syncsnoop tcpaccept tcpsubnet vfscount bashreadline btrfsdist cpudist deadlock ext4slower funcslower javaobjnew mdflush nodegc perlcalls  pidpersec reset-trace runqlat softirqs syscount tcpconnect tcptop vfsstat biolatency btrfsslower cpuunclaimed deadlock.c filelife gethostlatency  javastat memleak nodestat perlflow profile rubycalls runqlen solisten tclcalls tcpconnlat tcptracer wakeuptime biosnoop cachestat dbslower doc fileslower hardirqs javathreads mountsnoop offcputime perlstat pythoncalls rubyflow runqslower sslsniff tclflow tcpdrop tplist xfsdist biotop cachetop dbstat drsnoop filetop javacalls killsnoop mysqld_qslower offwaketime phpcalls pythonflow rubygc shmsnoop stackcount tclobjnew tcplife trace xfsslower bitesize capable dcsnoop execsnoop funccount javaflow lib nfsdist oomkill phpflow pythongc rubyobjnew slabratetop statsnoop tclstat tcpretrans ttysnoop

To add bcc directory to the $PATH:

$ vim .bash_profile
bcctools=/usr/share/bcc/tools
PATH=$PATH:$HOME/bin:$bcctools
export PATH
    
$ source ~/.bash_profile
$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin:/usr/share/bcc/tools

Install BCC from source

If you want to install a different version of BCC, you can refer to here. I tried this but it seems very tricky to install it successfully. I’ll not discuss it in this post.

Use the BCC tools

It’s not suprise if you see the following error for the first time run of BCC tools.

$ biolatency
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.7.12-1.el7.elrepo.x86_64/build/include/uapi/linux/ptrace.h:142:
In file included from /lib/modules/5.7.12-1.el7.elrepo.x86_64/build/arch/x86/include/asm/ptrace.h:5:
/lib/modules/5.7.12-1.el7.elrepo.x86_64/build/arch/x86/include/asm/segment.h:266:2: error: expected '(' after 'asm'
        alternative_io ("lsl %[seg],%[p]",
        ^
/lib/modules/5.7.12-1.el7.elrepo.x86_64/build/arch/x86/include/asm/alternative.h:240:2: note: expanded from macro 'alternative_io'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature)   \
        ^
/lib/modules/5.7.12-1.el7.elrepo.x86_64/build/include/linux/compiler_types.h:201:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:3:

This is because many BCC tools are broken with kernel 5.4+ and libbcc 0.10.

To fix this problem:

Modify as below for the BPF program definition.

Original code:

$ vim biolatency
<snippet>
# define BPF program
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <linux/blkdev.h>
<snippet>

Modified code:

$ vim biolatency
<snippet>
# define BPF program
bpf_text = """
#ifdef asm_inline
#undef asm_inline
#define asm_inline asm
#endif
#include <uapi/linux/ptrace.h>
#include <linux/blkdev.h>
<snippet>

Run the BCC tool again now:

$ biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
    usecs               : count     distribution
    0 -> 1          : 0        |                                        |
    2 -> 3          : 0        |                                        |
    4 -> 7          : 0        |                                        |
    8 -> 15         : 0        |                                        |
    16 -> 31        : 0        |                                        |
    32 -> 63        : 6        |********************                    |
    64 -> 127       : 12       |****************************************|
    128 -> 255      : 3        |**********                              |

Reference