Deploy systemtap on multiple systems

Posted on Edited on In , Views:

Normally, SystemTap scripts can only be run on systems where SystemTap is deployed together with the following required kernel packages.

  • kernel-devel-$(uname -r)
  • kernel-debuginfo-$(uname -r)
  • kernel-debuginfo-common-$(uname -m)-$(uname -r)

In the case that it’s neither feasible nor desired to install the denpendent packages on all the target systems. Using cross-instrumentation can work around. Cross-instrumentation is the process of generating SystemTap instrumentation modules from a SystemTap script on one system to be used on another system. This process offers the following benefits:

  • The kernel information packages for various machines can be installed on a single host machine.
  • Each target machine only needs one package to be installed to use the generated SystemTap instrumentation module: systemtap-runtime.

To build the instrumentation module, do the following on the host which has the dependent kernel packages installed:

[root@host1 ~]# uname -r
5.18.10-1.el7.elrepo.x86_64

[root@host1 ~]# rpm -qa | egrep "kernel-debug|kernel-devel|systemtap"
systemtap-client-4.0-13.el7.x86_64
systemtap-runtime-4.0-13.el7.x86_64
kernel-devel-3.10.0-1160.el7.x86_64
systemtap-devel-4.0-13.el7.x86_64
kernel-debuginfo-common-x86_64-3.10.0-1160.el7.x86_64
kernel-debuginfo-3.10.0-1160.el7.x86_64
systemtap-4.0-13.el7.x86_64

[root@host1 ~]# stap -r 3.10.0-1160.el7.x86_64 -e 'probe vfs.read {printf("read performed\n");} probe timer.s(5){exit()}' -m ported_stap -p4
ported_stap.ko

[root@host1 ~]# file ported_stap.ko
ported_stap.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1ad58adccafc9bf629f1edcbc285e9090e84dc35, not stripped

[root@host1 ~]# ls -la ported_stap.ko
-rw-r--r-- 1 root root 103608 Jul 20 22:33 ported_stap.ko

Once the instrumentation module is compiled, copy it to the target system and then run it using:

[root@host1 ~]# scp ported_stap.ko host2:/root

[root@host2 ~]# uname -r
3.10.0-1160.el7.x86_64

[root@host2 ~]# rpm -qa | egrep "kernel-debug|kernel-devel|systemtap"
systemtap-runtime-4.0-13.el7.x86_64

[root@host2 ~]# staprun ported_stap.ko
read performed
read performed
read performed
<...>

When users run a SystemTap script, a kernel module is built out of that script. SystemTap then loads the module into the kernel, allowing it to extract the specified data directly from the kernel. From the following command output, it’s clear that the SystemTap kernel module is dynamically loaded during the script run.

[root@host2 ~]# lsmod | grep ported_stap
ported_stap           139049  2
[root@host2 ~]# lsmod | grep ported_stap
[root@host2 ~]#

Reference