Debugging/Getting traces

Related articles

• General troubleshooting
• Reporting bug guidelines
• Step-by-step debugging guide

This article aims to help in creating a debugging Arch package and using it to provide trace and debug information for reporting software bugs to developers.

Usually, executable files are stripped of human readable context to make them smaller. Not only that, enhanced debugging information is usually not added to the executable in the first place, which drastically reduces the quality of the trace. So, before getting traces with debug information, one has to rebuild the package without stripping and with debugging information.

Package names

The first thing to do is to obtain the names of the packages which require rebuilding.

When looking at debug messages, such as by using gdb on a core dump:

[...]
Backtrace was generated from '/usr/bin/epiphany'

(no debugging symbols found)
Using host libthread_db library "/lib/libthread_db.so.1".
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread -1241265952 (LWP 12630)]
(no debugging symbols found)
0xb7f25410 in __kernel_vsyscall ()
#0  0xb7f25410 in __kernel_vsyscall ()
#1  0xb741b45b in ?? () from /lib/libpthread.so.0
[...]

?? shows where debugging info is missing, as well as the name of library or executable which called the function. Similarly, when (no debugging symbols found) appears, you should look for the stated file names. For example, with pacman:

$ pacman -Qo /lib/libthread_db.so.1
/lib/libthread_db.so.1 is owned by glibc 2.5-8

The package is called glibc in version 2.5-8. Repeat this step for every package that needs debugging.

PKGBUILD

In order to build a package from source, the PKGBUILD file is required. See ABS for packages in the official repositories, and AUR#Acquire build files for packages in the AUR.

Compilation settings

At this stage, you can modify the global configuration file of makepkg if you will be using it only for debug purposes. In other cases, you should modify package's PKGBUILD file only for each package you would like to rebuild.

General

As of pacman 4.1, /etc/makepkg.conf has debug compilation flags in DEBUG_CFLAGS and DEBUG_CXXFLAGS. To use them, enable the debug makepkg option, and disable strip.

OPTIONS+=(debug !strip)

These settings will force compilation with debug symbols and will disable their stripping from executables. To apply this setting to a single package, modify the PKGBUILD:

options=(debug !strip)

Alternatively you can put the debug information in a separate package by enabling both debug and strip, debug symbols will then be stripped from the main package and placed, together with source files to aid in stepping through the debugger, in a separate foo-debug package.

Note: It is insufficient to simply install the newly compiled debug package, because the debugger will check that the file containing the debug symbols is from the same build as the associated library and executable. You must install both of the recompiled packages. In Arch, the debug symbol files are installed under /usr/lib/debug, and source files are installed under /usr/src/debug. See the GDB documentation for more information about debug packages.

Note that certain packages such as glibc are stripped regardless. Check the PKGBUILD for sections such as:

strip $STRIP_BINARIES usr/bin/{gencat,getconf,getent,iconv,iconvconfig} \
                      usr/bin/{ldconfig,locale,localedef,nscd,makedb} \
                      usr/bin/{pcprofiledump,pldd,rpcgen,sln,sprof} \
                      usr/lib/getconf/*

strip $STRIP_STATIC usr/lib/*.a

strip $STRIP_SHARED usr/lib/{libanl,libBrokenLocale,libcidn,libcrypt}-*.so \
                    usr/lib/libnss_{compat,db,dns,files,hesiod,nis,nisplus}-*.so \
                    usr/lib/{libdl,libm,libnsl,libresolv,librt,libutil}-*.so \
                    usr/lib/{libmemusage,libpcprofile,libSegFault}.so \
                    usr/lib/{audit,gconv}/*.so

And remove them where appropriate.

GTK3/GLib2

The eschwartz repository contains pre-built GTK3 and GLib2 packages with detached debug symbol packages.

Qt4

In addition to the previous general settings, pass -developer-build option to the configure script in the PKGBUILD. By default, -developer-build passes -Werror to the compiler, which may cause the compilation to fail. To avoid compilation errors, you may need pass -no-warnings-are-errors, too.

Qt5

The qt-debug repository contains pre-built Qt/PyQt packages with debug symbols. See also upstream instructions.

The eschwartz repository contains pre-built qt5-base and pyqt5 packages with detached debug symbol packages.

CMAKE (KDE) applications

KDE and related programs typically use cmake. To enable debug information and disable optimizations, change -DCMAKE_BUILD_TYPE to Debug, for KDE applications change it to debugfull. To enable debug information while keeping optimisations enabled, change -DCMAKE_BUILD_TYPE to RelWithDebInfo.

dhcpcd

Remember to pass the --debug flag to the configure script when building dhcpcd.

Building and installing the package

Build the package from source using makepkg while in the PKGBUILD's directory. This could take some time:

$ makepkg

Then install the built package:

# pacman -U glibc-2.26-1-x86_64.pkg.tar.gz

Getting the trace

The actual backtrace (or stack trace) can now be obtained via e.g. gdb, the GNU Debugger. Run it either via:

# gdb /path/to/file

or:

# gdb
(gdb) exec /path/to/file

The path is optional, if already set in the $PATH variable.

Then, within gdb, type run followed by any arguments you wish the program to start with, e.g.:

(gdb) run --no-daemon --verbose

to start execution of the file. Do whatever necessary to evoke the bug. For the actual log, type the lines:

(gdb) set logging file trace.log
(gdb) set logging on

and then:

(gdb) thread apply all bt full

to output the trace to trace.log into the directory gdb was started in. To exit, enter:

(gdb) set logging off
(gdb) quit

Tip: To debug an application written in python:

# gdb /usr/bin/python
(gdb) run <python application>

You can also debug an already running application, e.g.:

 # gdb --pid=$(pidof firefox)
 (gdb) continue

To debug an application that has already crashed, you will want to invoke gdb on its core dump.

Conclusion

Use the complete stack trace to inform developers of a bug you have discovered before. This will be highly appreciated by them and will help to improve your favorite program.

See also

• Debian Wiki - How To Get a Backtrace
• Gentoo Wiki - Backtraces with Gentoo
• Fedora - StackTraces
• GNOME - Getting Stack Traces
• gdb mini intro