Difference between revisions of "Step-by-step debugging guide"

From ArchWiki
Jump to: navigation, search
(Technique #1 - strace)
(update Pkg/AUR templates)
(Tag: wiki-scripts)
 
(43 intermediate revisions by 16 users not shown)
Line 1: Line 1:
[[Category:Development (English)]]
+
[[Category:Development]]
{{i18n|Step By Step Debugging Guide}}
+
[[Category:System administration]]
 
+
[[ja:ステップバイステップデバッグガイド]]
 +
{{Related articles start}}
 +
{{Related|General troubleshooting}}
 +
{{Related|Reporting bug guidelines}}
 +
{{Related|Debug - Getting Traces}}
 +
{{Related|Boot debugging}}
 +
{{Related articles end}}
 +
{{Move|Debugging|This article might as well be about debugging in general, so that other useful tools like {{Pkg|ltrace}} can be added here.}}
 
This page is mainly about how to gather more information in connection with bug reports. Even though the word "debug" is used, it's not intended as a guide for how to debug programs while developing.
 
This page is mainly about how to gather more information in connection with bug reports. Even though the word "debug" is used, it's not intended as a guide for how to debug programs while developing.
  
Line 8: Line 15:
 
=== Run it from the commandline ===
 
=== Run it from the commandline ===
  
If an application suddenly crashes, try running it from the commandline. If you're new to the commandline and use GNOME, press Alt+F2 and type "gnome-terminal", then type in the name of the application in lowercase letters. If you don't know the name of the executable, only the name of the package, the following command can be useful to find the name of the executable. Replace "packagename" with the name of the package:
+
{{Style|There should be no need to check that files in {{ic|/usr/bin/}} are executable, the below command could just be {{ic|pacman -Ql ''packagename'' {{!}} grep ' /usr/bin/.'}}.}}
  
for f in `pacman -Ql packagename | grep "/bin/" | cut -d" " -f2`; do file $f 2>/dev/null | grep -q executable && basename $f; done
+
If an application suddenly crashes, try running it from a [[terminal emulator]]. Type in the name of the application in lowercase letters. If you do not know the name of the executable, only the name of the package, the following command can find the name of the executable. Replace ''packagename'' with the name of the package:
  
=== Check if the application segfaults ===
+
for f in $(pacman -Ql ''packagename'' | grep /bin/ | cut -d' ' -f2); do file $f 2>/dev/null | grep -q executable && basename $f; done
  
If you see the word "segfault" or the phrase "segmentation fault", see if there is a file named "core" as well.
+
=== Check availability of a core dump ===
  
ls core
+
A core dump is a file containing a process's address space (memory) when the process terminates unexpectedly. If the application is compiled in a debug-friendly way, the "core" file can be used to find out where things went wrong.
  
If there is, the application has segfaulted. The "core" file can, if the application is compiled in a debug-friendly way, be used to find out where things went wrong. Sometimes the core file ends up in one of the directories the application has visited instead of the current directory.
+
The location of core dumps may vary depending on the operating system configuration. See [[core dump]] to find whether generation of core dump files is enabled on your system and where do they go.
  
== How to investigate a segfault ==
+
== Segmentation faults ==
  
There are a couple of techniques that can be used to figure out what went wrong. Put your detective hat on. 🔎
+
There are several techniques that can be used to figure out what went wrong. Put your detective hat on.
  
=== Technique #1 - gdb ===
+
=== Gdb ===
  
gdb is an ancient and well tested application for debugging applications. Try running the application (replace "appname" with the name of your executable) with gdb:
+
{{Pkg|gdb}} is an ancient and well tested application for debugging applications. Replace ''appname'' with the name of your executable:
  
  gdb appname
+
  $ gdb appname
 
  r
 
  r
 
  (wait for segfault)
 
  (wait for segfault)
 
  bt full
 
  bt full
  
Now post the output to one of the many pastebin sites on the web and include the URL if you file a bugreport.
+
Now post the output to a [[List_of_applications#Pastebin_clients|Pastebin client]] and include the URL in your bug report.
 +
 
 +
=== Improved gdb output ===
 +
 
 +
{{Merge|Debug - Getting Traces|Same subject}}
  
=== Technique #2 - even better gdb output ===
+
First [[Arch Build System|recompile]] the application in question with the '''-g''', '''-O0''' and '''-fbuiltin''' flags. Make sure "!strip" is in the options array in the PKGBUILD, then install the package and run it again with gdb, as above.
  
If you're able to, recompile the application in question with the '''-g''', '''-O0''' and '''-fbuiltin''' flags, make sure "!strip" is in the options array in the PKGBUILD, install and try running it again with gdb, like above.
+
This is what the options line can look like:
 +
 
 +
  options=('!strip')
  
 
One way of enabling '''-g''', '''-O0''' and '''-fbuiltin''' is to put these two lines at the very beginning of the build() function in the relevant PKGBUILD:
 
One way of enabling '''-g''', '''-O0''' and '''-fbuiltin''' is to put these two lines at the very beginning of the build() function in the relevant PKGBUILD:
Line 44: Line 57:
 
  export CXXFLAGS="$CXXFLAGS -O0 -fbuiltin -g"
 
  export CXXFLAGS="$CXXFLAGS -O0 -fbuiltin -g"
  
Note that '''-g''' enables debug symbols and '''-O0''' turns off optimizations. '''-O2''' is the normal choice, [http://funroll-loops.info/ '''-O3''' is usually overkill] and [http://stackoverflow.com/questions/1778538/how-many-gcc-optimization-levels-are-there '''-O4''' and above behaves exactly like '''-O3'''].
+
The meaning of the flags is the following: '''-g''' enables debug symbols and '''-O0''' turns off optimizations. ('''-O2''' is the most common optimization level. ([http://funroll-loops.info/ '''-O3''' is usually overkill] and [http://stackoverflow.com/questions/1778538/how-many-gcc-optimization-levels-are-there '''-O4''' and above behaves exactly like '''-O3''']).
  
If you have a "core" file, it can be used to get a backtrace:
+
If you have a "core" file, it can be used together with gdb to get a backtrace:
  
  gdb appname core
+
  $ gdb appname core
 
  bt full
 
  bt full
  
== How to investigate missing files or libraries ==
+
=== Valgrind ===
 +
 
 +
Assuming you have an unstripped binary without inlined functions, it is usually a good idea to also run that program through {{Pkg|valgrind}}. ''valgrind'' is a tool that emulates a CPU and usually shows where things go wrong or provide additional info in addition to gdb.
 +
 
 +
$ valgrind appname
 +
 
 +
it will provide a lot of helpful debug output if there is a crash. Consider {{ic|-v}} and {{ic|<nowiki>--leak-check=full</nowiki>}} to get even more info.
 +
 
 +
Alternatively, use:
 +
 
 +
$ valgrind --tool=callgrind appname
 +
 
 +
and run the output through {{Pkg|kcachegrind}} to graphically explore the functions the program uses. If a program hangs, this makes it easier to pinpoint the location of the error.
  
=== Technique #1 - strace ===
+
== Missing files or libraries ==
  
Strace is great for finding out, in detail, what an application is actually doing. If an application tries to open a file that just isn't there, it can be discovered by strace.
+
=== Strace ===
  
For finding which files a program named "appname" tries to open:
+
{{Pkg|strace}} finds out in detail what an application is actually doing. If an application tries to open a file that is not there, it can be discovered by strace.
 +
 
 +
For finding which files a program named ''appname'' tries to open:
  
 
  $ strace -eopen appname
 
  $ strace -eopen appname
  
Again, save the output, post it to a pastebin site and keep the URL in handy.
+
Save the output, post it to a [[List_of_applications#Pastebin_clients|Pastebin client]] and keep the URL in handy.
  
Tip: If you wish to grep the output from strace, you can use:
+
{{Tip|If you wish to grep the output from strace, you can try:
 +
{{ic|strace -o /dev/stdout appname <nowiki>|</nowiki> grep ''string''}}
 +
}}
  
$ strace -o /dev/stdout appname | grep something
+
=== LD_DEBUG ===
  
=== Technique #2 - LD_DEBUG ===
+
Setting {{ic|<nowiki>LD_DEBUG=files</nowiki>}} gives another overview of what files an application is looking for. For an application named ''appname'':
 
 
Setting LD_DEBUG to "files" is another way to get an overview of which files an application are looking for. For an application named "appname":
 
  
 
  LD_DEBUG=files appname > appname.log 2>&1
 
  LD_DEBUG=files appname > appname.log 2>&1
  
The output will end up in appname.log.
+
The output will end up in {{ic|appname.log}}.
  
For more information about this:
+
For more information, see {{man|8|ld-linux}}.
  
man ld-linux
+
=== Readelf ===
 
 
=== Technique #3 - no such file or directory ===
 
  
 
If you get "no such file or directory" when running an application, try the following command:
 
If you get "no such file or directory" when running an application, try the following command:
  
  readelf -a /usr/bin/appname | grep interp
+
  $ readelf -a /usr/bin/appname | grep interp
  
 
(replace /usr/bin/appname with the location of your executable)
 
(replace /usr/bin/appname with the location of your executable)
  
Make sure the interpreter in question (like /lib/ld-linux-x86-64.so.2) actually exists. Install '''ld-lsb''' from AUR if you have to.
+
Make sure the interpreter in question (like /lib/ld-linux-x86-64.so.2) actually exists. Install {{Pkg|ld-lsb}} from the [[AUR]] if need be.
 +
 
 +
== If it is not written in C or C++, but perhaps in Python ==
  
== If it's not written in C or C++, but perhaps in Python ==
+
Use {{Pkg|file}} on the executable to get more information (replace "appname" with your executable):
  
Use "file" on the executable to get more information (replace "appname" with your executable):
+
$ file /usr/bin/''appname''
  
file /usr/bin/appname
+
If it says "ELF" it is a binary executable and is usually written in C or C++. If it says "Python script" you know you are dealing with an application written in Python.
  
If it says "ELF" it's a binary exectuable and is usually written in C or C++. If it says "Python script" you know you're dealing with an application written in Python.
+
If it is a shell script, open up the shell script in a text editor and see (usually at the bottom of the file) if you can find the name of the real application (ELF file). You can then temporarily put "gdb" right in the shellscript, before the name of the executable, for debugging purposes. See the sections about gdb further up. Prefix the command with {{ic|gdb --args}} if the executable in question needs arguments as well.
  
If it's a shell script, open up the shell script in a text editor and see (usually at the bottom of the file) if you can find the name of the real application (ELF file). You can then temporarily put "gdb" right in the shellscript, before the name of the executable, for debugging purposes. See the sections about gdb further up.
+
For pure shell scripts, you can also use {{ic|bash -x ''script_name''}} or {{ic|bash -xv ''script_name''}}.
  
For Python applications, the output will often say which file and line number the crash occured at. If you're proficient with Python, you can try to fix this and include the fix in the bug report.
+
For Python applications, the output will often say which file and line number the crash occured at. If you are proficient with Python, you can try to fix this and include the fix in the bug report.
  
== Finally - report the bug ==
+
== Report the bug ==
  
 
Please report a bug at https://bugs.archlinux.org and possibly also directly to the developers of the application in question, then include a link in the Arch Linux bug report. This helps us all.
 
Please report a bug at https://bugs.archlinux.org and possibly also directly to the developers of the application in question, then include a link in the Arch Linux bug report. This helps us all.
  
However, if you think there's something wrong with the application itself, and not with how it is packaged, report the bug directly to upstream (which means the developers of the application). Normally, software streams from developers, through packagers/maintainers and down to users. Upstream means the other way, so for this case: directly to the developers of an application.
+
However, if you think there is something wrong with the application itself, and not with how it is packaged, report the bug directly to upstream (which means the developers of the application). Normally, software streams from developers, through packagers/maintainers and down to users. Upstream means the other way, so for this case: directly to the developers of an application.
  
 
== See also ==
 
== See also ==
  
*[[Reporting Bug Guidelines]] for detailed information about how to report bugs in a meaningful way.
+
*[https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces Gentoo guide for getting useful backtraces]
*[[Debug - Getting Traces]] for more information about collecting debug information and stack traces.
 
*[[General Troubleshooting]]
 
*[http://www.gentoo.org/proj/en/qa/backtraces.xml Gentoo guide for getting useful backtraces]
 

Latest revision as of 22:07, 9 November 2018

Tango-go-next.pngThis article or section is a candidate for moving to Debugging.Tango-go-next.png

Notes: This article might as well be about debugging in general, so that other useful tools like ltrace can be added here. (Discuss in Talk:Step-by-step debugging guide#)

This page is mainly about how to gather more information in connection with bug reports. Even though the word "debug" is used, it's not intended as a guide for how to debug programs while developing.

When an application fails

Run it from the commandline

Tango-edit-clear.pngThis article or section needs language, wiki syntax or style improvements. See Help:Style for reference.Tango-edit-clear.png

Reason: There should be no need to check that files in /usr/bin/ are executable, the below command could just be pacman -Ql packagename | grep ' /usr/bin/.'. (Discuss in Talk:Step-by-step debugging guide#)

If an application suddenly crashes, try running it from a terminal emulator. Type in the name of the application in lowercase letters. If you do not know the name of the executable, only the name of the package, the following command can find the name of the executable. Replace packagename with the name of the package:

for f in $(pacman -Ql packagename | grep /bin/ | cut -d' ' -f2); do file $f 2>/dev/null | grep -q executable && basename $f; done

Check availability of a core dump

A core dump is a file containing a process's address space (memory) when the process terminates unexpectedly. If the application is compiled in a debug-friendly way, the "core" file can be used to find out where things went wrong.

The location of core dumps may vary depending on the operating system configuration. See core dump to find whether generation of core dump files is enabled on your system and where do they go.

Segmentation faults

There are several techniques that can be used to figure out what went wrong. Put your detective hat on.

Gdb

gdb is an ancient and well tested application for debugging applications. Replace appname with the name of your executable:

$ gdb appname
r
(wait for segfault)
bt full

Now post the output to a Pastebin client and include the URL in your bug report.

Improved gdb output

Merge-arrows-2.pngThis article or section is a candidate for merging with Debug - Getting Traces.Merge-arrows-2.png

Notes: Same subject (Discuss in Talk:Step-by-step debugging guide#)

First recompile the application in question with the -g, -O0 and -fbuiltin flags. Make sure "!strip" is in the options array in the PKGBUILD, then install the package and run it again with gdb, as above.

This is what the options line can look like:

 options=('!strip')

One way of enabling -g, -O0 and -fbuiltin is to put these two lines at the very beginning of the build() function in the relevant PKGBUILD:

export CFLAGS="$CFLAGS -O0 -fbuiltin -g"
export CXXFLAGS="$CXXFLAGS -O0 -fbuiltin -g"

The meaning of the flags is the following: -g enables debug symbols and -O0 turns off optimizations. (-O2 is the most common optimization level. (-O3 is usually overkill and -O4 and above behaves exactly like -O3).

If you have a "core" file, it can be used together with gdb to get a backtrace:

$ gdb appname core
bt full

Valgrind

Assuming you have an unstripped binary without inlined functions, it is usually a good idea to also run that program through valgrind. valgrind is a tool that emulates a CPU and usually shows where things go wrong or provide additional info in addition to gdb.

$ valgrind appname

it will provide a lot of helpful debug output if there is a crash. Consider -v and --leak-check=full to get even more info.

Alternatively, use:

$ valgrind --tool=callgrind appname

and run the output through kcachegrind to graphically explore the functions the program uses. If a program hangs, this makes it easier to pinpoint the location of the error.

Missing files or libraries

Strace

strace finds out in detail what an application is actually doing. If an application tries to open a file that is not there, it can be discovered by strace.

For finding which files a program named appname tries to open:

$ strace -eopen appname

Save the output, post it to a Pastebin client and keep the URL in handy.

Tip: If you wish to grep the output from strace, you can try:

strace -o /dev/stdout appname | grep string

LD_DEBUG

Setting LD_DEBUG=files gives another overview of what files an application is looking for. For an application named appname:

LD_DEBUG=files appname > appname.log 2>&1

The output will end up in appname.log.

For more information, see ld-linux(8).

Readelf

If you get "no such file or directory" when running an application, try the following command:

$ readelf -a /usr/bin/appname | grep interp

(replace /usr/bin/appname with the location of your executable)

Make sure the interpreter in question (like /lib/ld-linux-x86-64.so.2) actually exists. Install ld-lsb from the AUR if need be.

If it is not written in C or C++, but perhaps in Python

Use file on the executable to get more information (replace "appname" with your executable):

$ file /usr/bin/appname

If it says "ELF" it is a binary executable and is usually written in C or C++. If it says "Python script" you know you are dealing with an application written in Python.

If it is a shell script, open up the shell script in a text editor and see (usually at the bottom of the file) if you can find the name of the real application (ELF file). You can then temporarily put "gdb" right in the shellscript, before the name of the executable, for debugging purposes. See the sections about gdb further up. Prefix the command with gdb --args if the executable in question needs arguments as well.

For pure shell scripts, you can also use bash -x script_name or bash -xv script_name.

For Python applications, the output will often say which file and line number the crash occured at. If you are proficient with Python, you can try to fix this and include the fix in the bug report.

Report the bug

Please report a bug at https://bugs.archlinux.org and possibly also directly to the developers of the application in question, then include a link in the Arch Linux bug report. This helps us all.

However, if you think there is something wrong with the application itself, and not with how it is packaged, report the bug directly to upstream (which means the developers of the application). Normally, software streams from developers, through packagers/maintainers and down to users. Upstream means the other way, so for this case: directly to the developers of an application.

See also