SystemTap

SystemTap provides free software (GPL) infrastructure to simplify the gathering of information about the running Linux system.

Installation

Simply install the systemtap package.

Standard kernel

You will need at least the linux-headers package installed.

Because Arch permanently strips debugging data from its distributed binaries (including the kernel), many normal/fancier systemtap capabilities are simply not available, so many examples at /usr/share/doc/systemtap/examples will not work. However, see stapprobes(3) for the NON-DWARF and AUTO-DWARF probe types for what should still work, for example:

kernel tracepoints: kernel.trace("*")
user-space probes: process("...").function("...") (for programs you build yourself with -g)
user-space markers: process("...").mark("...") (if they were configured with the <sys/sdt.h> markers)
perfctr-based probes: perf.*
non-dwarf kernel probes: kprobe.function("...") and nd_syscall.* tapset (if a /boot/System.map* file is available, see below).

Kernel rebuild

You may consider to build a linux-custom package to run SystemTap, but rebuilding the linux package is easy and efficient.

Prepare

First, follow the steps at Kernel/Arch build system#Getting the ingredients to get the original kernel build files. Then use makepkg --verifysource to get the additional files. By performing the verification, you can safely skip the steps on "Update checksum".

Modify config

Edit config.x86_64, turn on these options:

CONFIG_KPROBES=y
CONFIG_KPROBES_SANITY_TEST=n
CONFIG_KPROBE_EVENT=y
CONFIG_NET_DCCPPROBE=m
CONFIG_NET_SCTPPROBE=m
CONFIG_NET_TCPPROBE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=n
CONFIG_X86_DECODER_SELFTEST=n
CONFIG_DEBUG_INFO_VTA=y

By default only CONFIG_DEBUG_INFO and CONFIG_DEBUG_INFO_REDUCED are not set.

With the current linux kernel (tested with 3.15.2) you can simply append these lines into config.x86_64:

x86_64

echo '
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=n
' >> config.x86_64

Note If you want to put these lines into a self-maintained script, do not insert any space before CONFIG_* lines.

Update checksum

Tip You can safely skip this step if you verified the source files previously.

Run sha256sum config.x86_64 to get a new sha256sum.

In PKGBUILD file, the sha256sum=('sum-of-first' ... 'sum-of-last') has the same order with source=('first-source' ... 'last-source'), put your new sha256sum in the right place.

Build and install

Optional: It is recommended to set MAKEFLAGS="-j16" in /etc/makepkg.conf to speed up the compilation.

The rebuilt linux and linux-headers packages should be reinstalled, linux-docs does not matter.

Via this method, external modules (e.g. nvidia and virtualbox) do not need to be rebuilt.

Build custom kernel

Please reference this README

Troubleshooting

Pass 4 fails when launching

If you get the following error message, try into install systemtap-git^AUR

/usr/share/systemtap/runtime/stat.c:214:2: error: 'cpu_possible_map' undeclared (first use in this function)

System.map is missing

You can recover it where you build your linux kernel with DEBUG_INFO enabled

# cp src/linux-3.6/System.map /boot/System.map-3.6.7-1-ARCH

Alternatively,

# cp /proc/kallsyms /boot/System.map-$(uname -r)

Process return probes not available

If you are sure that your kernel configuration is correct, but on launching stap you get both of the following messages:

WARNING: Kernel function symbol table missing [man warning::symbols]
semantic error: process return probes not available [man error::inode-uprobes]

then SystemTap may have failed to verify support for this feature. You can fix this by following the steps in #System.map is missing.