https://wiki.archlinux.org/api.php?action=feedcontributions&user=Mcnster&feedformat=atomArchWiki - User contributions [en]2024-03-29T13:58:27ZUser contributionsMediaWiki 1.41.0https://wiki.archlinux.org/index.php?title=LXD&diff=493497LXD2017-10-18T07:44:14Z<p>Mcnster: /* Required software */ Reworked this section for clarity, placing emphasis on unprivileged containers.</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:LXD]]<br />
'''[https://linuxcontainers.org/lxd/ LXD]''' is a container "hypervisor" and a new user experience for [[Linux Containers]].<br />
<br />
== Setup ==<br />
=== Required software ===<br />
Install [[LXC]] and the {{AUR|lxd}} package, then [[start]] {{ic|lxd.service}}.<br />
<br />
Verify that the running kernel is properly configured to run a container:<br />
$ lxc-checkconfig<br />
<br />
The safest type of container that LXD can create is ''unprivileged''. This is done by leveraging the Linux kernel's '''User Namespaces''' feature. However, due to more general security concerns, the default Arch kernel does ''not'' ship with User Namespaces enabled ({{ic|CONFIG_USER_NS}} is a kernel compile-time decision). You have three (3) options to use a kernel with {{ic|CONFIG_USER_NS}} and thereby create ''unprivileged'' containers:<br />
<br />
* Install the {{Pkg|linux-hardened}} kernel package along-side the default {{Pkg|linux}} kernel. When you wish to run unprivileged LXD containers, boot with {{Pkg|linux-hardened}} by selecting it in the bootloader. {{Pkg|linux-hardened}} is compiled with {{ic|CONFIG_USER_NS}}. Otherwise, run with {{Pkg|linux}} as normal.<br />
* Install the {{AUR|linux-userns}} or {{AUR|linux-lts-userns}} packages from the [[AUR]]. Both are compiled with {{ic|CONFIG_USER_NS}}, the latter being the Long-Term Support version. <br />
* Compile and install your own custom kernel with {{ic|CONFIG_USER_NS}} enabled.<br />
<br />
{{Note|If you decide to run a kernel '''without User Namespaces''', LXD containers will be ''privileged'' and that involves some risk (in the event a process escapes the container). See [[#Launching container without CONFIG_USER_NS]] below.}}<br />
<br />
=== Sub{u,g}id configuration ===<br />
<br />
You will need sub{u,g}ids for root, so that LXD can create the unprivileged containers:<br />
<br />
$ echo "root:1000000:65536" | sudo tee -a /etc/subuid /etc/subgid<br />
<br />
=== Accessing LXD as a unprivileged user ===<br />
<br />
By default the LXD daemon allows users in the {{ic|lxd}} group access, so add your user to the group:<br />
<br />
# usermod -a -G lxd <user><br />
<br />
=== LXD Networking ===<br />
<br />
LXD uses LXC's networking capabilities. By default it connects containers to the {{ic|lxcbr0}} network device. Refer to the LXC documentation on [[Linux Containers#Host network configuration|network configuration]] to set up a bridge for your containers.<br />
<br />
If you want to use a different interface than {{ic|lxcbr0}} edit the default using the lxc command line tool:<br />
<br />
$ lxc profile edit default<br />
<br />
An editor will open with a config file that by default contains:<br />
<br />
name: default<br />
config: {}<br />
devices:<br />
eth0:<br />
name: eth0<br />
nictype: bridged<br />
parent: lxcbr0<br />
type: nic<br />
<br />
You can set the {{ic|parent}} parameter to whichever bridge you want LXD to attach the containers to by default.<br />
<br />
==== Example network configuration ====<br />
<br />
Thanks to @jpic, the LXD package now provides some example networking configuration in {{ic|/usr/share/lxd/}}. To use this configuration run the following commands:<br />
<br />
$ ln -s /usr/share/lxd/dnsmasq-lxd.conf /etc/dnsmasq-lxd.conf<br />
$ ln -s /usr/share/lxd/systemd/system/dnsmasq@lxd.service /etc/systemd/system/dnsmasq@lxd.service <br />
$ ln -s /usr/share/lxd/netctl/lxd /etc/netctl/lxd<br />
$ ln -s /usr/share/lxd/dbus-1/system.d/dnsmasq-lxd.conf /etc/dbus-1/system.d/dnsmasq-lxd.conf<br />
<br />
If you use [[NetworkManager]], also symlink the following file:<br />
<br />
$ ln -s /usr/share/lxd/NetworkManager/dnsmasq.d/lxd.conf /etc/NetworkManager/dnsmasq.d/lxd.conf<br />
<br />
Change {{ic|parent: lxcbr0}} to {{ic|parent: lxd}}:<br />
<br />
$ lxc profile edit default<br />
<br />
Finally, [[enable]] and [[start]] {{ic|dnsmasq@lxd.service}} and {{ic|netctl@lxd.service}}.<br />
<br />
If you encounter issue with the provided example configuration, or have suggestions to improve it, please leave a comment on the {{AUR|lxd}} page.<br />
<br />
== Basic usage ==<br />
=== First steps ===<br />
<br />
LXD has two parts, the daemon (the lxd binary), and the client (the lxc binary). Now that the daemon is all configured and running, you can create a container:<br />
<br />
$ lxc launch ubuntu:14.04<br />
<br />
Alternatively, you can also use a remote LXD host as a source of images. One comes pre-configured in LXD, called "images" (images.linuxcontainers.org)<br />
<br />
$ lxc launch images:centos/7/amd64 centos<br />
<br />
== Advance usage ==<br />
=== Modify processes and files limit ===<br />
<br />
You may want to increase file descriptor limit or max user processes limit, since default file descriptor limit is 1024 on Archlinux<br />
<br />
$ sudo systemctl edit lxd<br />
<br />
And config as follow:<br />
<br />
[Service]<br />
LimitNOFILE=infinity<br />
LimitNPROC=infinity<br />
TasksMax=infinity<br />
<br />
Then restart lxd<br />
<br />
$ sudo systemctl restart lxd<br />
<br />
== Troubleshooting ==<br />
=== Launching container without CONFIG_USER_NS ===<br />
For launching images you must provide {{ic|1=security.privileged=true}} during image creation:<br />
<br />
$ lxc launch ubuntu:16.04 ubuntu -c security.privileged=true<br />
<br />
Or for already existed image you may edit config:<br />
<br />
$ lxc config edit ubuntu<br />
<br />
name: ubuntu<br />
profiles:<br />
- default<br />
config:<br />
...<br />
security.privileged: "true"<br />
...<br />
devices:<br />
root:<br />
path: /<br />
type: disk<br />
ephemeral: false<br />
<br />
Or to enable {{ic|1=security.privileged=true}} for new containers, edit the config for the default profile:<br />
<br />
$ lxc profile edit default<br />
<br />
== See also ==<br />
<br />
* [https://linuxcontainers.org/lxd/ The official LXD homepage]<br />
* [https://github.com/lxc/lxd The LXD GitHub page]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=LXD&diff=493482LXD2017-10-18T05:55:09Z<p>Mcnster: /* Required software */ Added reference to linux-hardened package having CONFIG_USER_NS set.</p>
<hr />
<div>[[Category:Virtualization]]<br />
[[ja:LXD]]<br />
'''[https://linuxcontainers.org/lxd/ LXD]''' is a container "hypervisor" and a new user experience for [[Linux Containers]].<br />
<br />
== Setup ==<br />
=== Required software ===<br />
Install [[LXC]] and the {{AUR|lxd}} package, then [[start]] {{ic|lxd.service}}.<br />
<br />
Verify that the running kernel is properly configured to run a container:<br />
$ lxc-checkconfig<br />
<br />
Due to security concerns, the default Arch kernel does '''not''' ship with the ability to run containers as an unprivileged user. LXD however needs this ability to run. You can either build a kernel yourself that has {{ic|CONFIG_USER_NS}} enabled, or use {{AUR|linux-userns}} or {{AUR|linux-lts-userns}} from the [[AUR]]. {{Pkg|linux-hardened}} in the Official Repositories also has {{ic|CONFIG_USER_NS}} enabled by default.<br />
<br />
{{Note|You still be able to run containers without CONFIG_USER_NS kernel feature. See: [[#Launching container without CONFIG_USER_NS]]}}<br />
<br />
=== Sub{u,g}id configuration ===<br />
<br />
You will need sub{u,g}ids for root, so that LXD can create the unprivileged containers:<br />
<br />
$ echo "root:1000000:65536" | sudo tee -a /etc/subuid /etc/subgid<br />
<br />
=== Accessing LXD as a unprivileged user ===<br />
<br />
By default the LXD daemon allows users in the {{ic|lxd}} group access, so add your user to the group:<br />
<br />
# usermod -a -G lxd <user><br />
<br />
=== LXD Networking ===<br />
<br />
LXD uses LXC's networking capabilities. By default it connects containers to the {{ic|lxcbr0}} network device. Refer to the LXC documentation on [[Linux Containers#Host network configuration|network configuration]] to set up a bridge for your containers.<br />
<br />
If you want to use a different interface than {{ic|lxcbr0}} edit the default using the lxc command line tool:<br />
<br />
$ lxc profile edit default<br />
<br />
An editor will open with a config file that by default contains:<br />
<br />
name: default<br />
config: {}<br />
devices:<br />
eth0:<br />
name: eth0<br />
nictype: bridged<br />
parent: lxcbr0<br />
type: nic<br />
<br />
You can set the {{ic|parent}} parameter to whichever bridge you want LXD to attach the containers to by default.<br />
<br />
==== Example network configuration ====<br />
<br />
Thanks to @jpic, the LXD package now provides some example networking configuration in {{ic|/usr/share/lxd/}}. To use this configuration run the following commands:<br />
<br />
$ ln -s /usr/share/lxd/dnsmasq-lxd.conf /etc/dnsmasq-lxd.conf<br />
$ ln -s /usr/share/lxd/systemd/system/dnsmasq@lxd.service /etc/systemd/system/dnsmasq@lxd.service <br />
$ ln -s /usr/share/lxd/netctl/lxd /etc/netctl/lxd<br />
$ ln -s /usr/share/lxd/dbus-1/system.d/dnsmasq-lxd.conf /etc/dbus-1/system.d/dnsmasq-lxd.conf<br />
<br />
If you use [[NetworkManager]], also symlink the following file:<br />
<br />
$ ln -s /usr/share/lxd/NetworkManager/dnsmasq.d/lxd.conf /etc/NetworkManager/dnsmasq.d/lxd.conf<br />
<br />
Change {{ic|parent: lxcbr0}} to {{ic|parent: lxd}}:<br />
<br />
$ lxc profile edit default<br />
<br />
Finally, [[enable]] and [[start]] {{ic|dnsmasq@lxd.service}} and {{ic|netctl@lxd.service}}.<br />
<br />
If you encounter issue with the provided example configuration, or have suggestions to improve it, please leave a comment on the {{AUR|lxd}} page.<br />
<br />
== Basic usage ==<br />
=== First steps ===<br />
<br />
LXD has two parts, the daemon (the lxd binary), and the client (the lxc binary). Now that the daemon is all configured and running, you can create a container:<br />
<br />
$ lxc launch ubuntu:14.04<br />
<br />
Alternatively, you can also use a remote LXD host as a source of images. One comes pre-configured in LXD, called "images" (images.linuxcontainers.org)<br />
<br />
$ lxc launch images:centos/7/amd64 centos<br />
<br />
== Advance usage ==<br />
=== Modify processes and files limit ===<br />
<br />
You may want to increase file descriptor limit or max user processes limit, since default file descriptor limit is 1024 on Archlinux<br />
<br />
$ sudo systemctl edit lxd<br />
<br />
And config as follow:<br />
<br />
[Service]<br />
LimitNOFILE=infinity<br />
LimitNPROC=infinity<br />
TasksMax=infinity<br />
<br />
Then restart lxd<br />
<br />
$ sudo systemctl restart lxd<br />
<br />
== Troubleshooting ==<br />
=== Launching container without CONFIG_USER_NS ===<br />
For launching images you must provide {{ic|1=security.privileged=true}} during image creation:<br />
<br />
$ lxc launch ubuntu:16.04 ubuntu -c security.privileged=true<br />
<br />
Or for already existed image you may edit config:<br />
<br />
$ lxc config edit ubuntu<br />
<br />
name: ubuntu<br />
profiles:<br />
- default<br />
config:<br />
...<br />
security.privileged: "true"<br />
...<br />
devices:<br />
root:<br />
path: /<br />
type: disk<br />
ephemeral: false<br />
<br />
Or to enable {{ic|1=security.privileged=true}} for new containers, edit the config for the default profile:<br />
<br />
$ lxc profile edit default<br />
<br />
== See also ==<br />
<br />
* [https://linuxcontainers.org/lxd/ The official LXD homepage]<br />
* [https://github.com/lxc/lxd The LXD GitHub page]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493476General troubleshooting2017-10-17T23:54:55Z<p>Mcnster: /* See also */ Added urls that were in the "See also" 3rd level subsection above.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Approach ===<br />
<br />
Rather than approaching an issue by stating,<br />
<br />
''Application X does not work.''<br />
<br />
you will find it more helpful to formulate your issue in the context of the system as a whole, as:<br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== Message: "file: could not find any magic files!" ==<br />
<br />
If you see this message, it likely indicates that a package update has corrupted the dynamic linker run-time bindings file and your system is now essentially crippled. You will not be able to recompile or reinstall the package responsible or rebuild the [[initramfs]] until you fix it.<br />
<br />
=== Problem ===<br />
<br />
A package update likely added an invalid {{ic|''filename''.conf}} to the directory {{ic|/etc/ld.so.conf.d}} or edited {{ic|/etc/ld.so.conf}} incorrectly. The result is that the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} is being re-generated with invalid data. This can potentially cause all programs on the system that depend on shared libraries to fail (ie. almost all of them).<br />
<br />
=== Solution ===<br />
<br />
# Boot with the '''''Arch Linux Installation CD'''''.<br />
# Mount your root {{ic|/}} filesystem on {{ic|/mnt}} and your {{ic|/boot}} filesystem on {{ic|/mnt/boot}} and chroot into the broken system by executing {{ic|# arch-chroot /mnt}}.<br />
# Examine the file {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
# Examine the files located in the directory {{ic|/etc/ld.so.conf.d/}} and remove any invalid files.<br />
# Rebuild the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} by executing {{ic|# ldconfig}}.<br />
# Rebuild the [[Initramfs]] by executing {{ic|# mkinitcpio -p linux}}.<br />
# Exit the chroot, unmount filesystems, and reboot back into your installed system.<br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Memtest-like tools to add to grub.cfg on UltimateBootCD.com<br />
* [[Wikipedia:BIOS Boot partition|BIOS Boot partition]] on Wikipedia<br />
* [https://fedoraproject.org/wiki/QA/Sysrq\x20QA/Sysrq Using sysrq] on Fedoraproject.org<br />
* [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console] on Freedesktop.org<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues] on Archive.org</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493475General troubleshooting2017-10-17T23:54:03Z<p>Mcnster: /* See also */ Removed this 3rd level section and placed the "See also"'s at the end of the article for consistency.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Approach ===<br />
<br />
Rather than approaching an issue by stating,<br />
<br />
''Application X does not work.''<br />
<br />
you will find it more helpful to formulate your issue in the context of the system as a whole, as:<br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== Message: "file: could not find any magic files!" ==<br />
<br />
If you see this message, it likely indicates that a package update has corrupted the dynamic linker run-time bindings file and your system is now essentially crippled. You will not be able to recompile or reinstall the package responsible or rebuild the [[initramfs]] until you fix it.<br />
<br />
=== Problem ===<br />
<br />
A package update likely added an invalid {{ic|''filename''.conf}} to the directory {{ic|/etc/ld.so.conf.d}} or edited {{ic|/etc/ld.so.conf}} incorrectly. The result is that the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} is being re-generated with invalid data. This can potentially cause all programs on the system that depend on shared libraries to fail (ie. almost all of them).<br />
<br />
=== Solution ===<br />
<br />
# Boot with the '''''Arch Linux Installation CD'''''.<br />
# Mount your root {{ic|/}} filesystem on {{ic|/mnt}} and your {{ic|/boot}} filesystem on {{ic|/mnt/boot}} and chroot into the broken system by executing {{ic|# arch-chroot /mnt}}.<br />
# Examine the file {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
# Examine the files located in the directory {{ic|/etc/ld.so.conf.d/}} and remove any invalid files.<br />
# Rebuild the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} by executing {{ic|# ldconfig}}.<br />
# Rebuild the [[Initramfs]] by executing {{ic|# mkinitcpio -p linux}}.<br />
# Exit the chroot, unmount filesystems, and reboot back into your installed system.<br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493473General troubleshooting2017-10-17T23:48:47Z<p>Mcnster: /* See also */ Fixed link.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Approach ===<br />
<br />
Rather than approaching an issue by stating,<br />
<br />
''Application X does not work.''<br />
<br />
you will find it more helpful to formulate your issue in the context of the system as a whole, as:<br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Memtest-like tools to add to grub.cfg on UltimateBootCD.com<br />
* [[Wikipedia:BIOS Boot partition|BIOS Boot partition]] on Wikipedia<br />
* [https://fedoraproject.org/wiki/QA/Sysrq\x20QA/Sysrq Using sysrq] on Fedoraproject.org<br />
* [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console] on Freedesktop.org<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues] on Archive.org<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== Message: "file: could not find any magic files!" ==<br />
<br />
If you see this message, it likely indicates that a package update has corrupted the dynamic linker run-time bindings file and your system is now essentially crippled. You will not be able to recompile or reinstall the package responsible or rebuild the [[initramfs]] until you fix it.<br />
<br />
=== Problem ===<br />
<br />
A package update likely added an invalid {{ic|''filename''.conf}} to the directory {{ic|/etc/ld.so.conf.d}} or edited {{ic|/etc/ld.so.conf}} incorrectly. The result is that the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} is being re-generated with invalid data. This can potentially cause all programs on the system that depend on shared libraries to fail (ie. almost all of them).<br />
<br />
=== Solution ===<br />
<br />
# Boot with the '''''Arch Linux Installation CD'''''.<br />
# Mount your root {{ic|/}} filesystem on {{ic|/mnt}} and your {{ic|/boot}} filesystem on {{ic|/mnt/boot}} and chroot into the broken system by executing {{ic|# arch-chroot /mnt}}.<br />
# Examine the file {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
# Examine the files located in the directory {{ic|/etc/ld.so.conf.d/}} and remove any invalid files.<br />
# Rebuild the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} by executing {{ic|# ldconfig}}.<br />
# Rebuild the [[Initramfs]] by executing {{ic|# mkinitcpio -p linux}}.<br />
# Exit the chroot, unmount filesystems, and reboot back into your installed system.<br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493472General troubleshooting2017-10-17T23:46:06Z<p>Mcnster: /* See also */ Made URLS and comments appear consistent.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Approach ===<br />
<br />
Rather than approaching an issue by stating,<br />
<br />
''Application X does not work.''<br />
<br />
you will find it more helpful to formulate your issue in the context of the system as a whole, as:<br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Memtest-like tools to add to grub.cfg on UltimateBootCD.com<br />
* [[Wikipedia:BIOS Boot partition|BIOS Boot partition]] on Wikipedia<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq Using sysrq] on Fedoraproject.org<br />
* [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console] on Freedesktop.org<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues] on Archive.org<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== Message: "file: could not find any magic files!" ==<br />
<br />
If you see this message, it likely indicates that a package update has corrupted the dynamic linker run-time bindings file and your system is now essentially crippled. You will not be able to recompile or reinstall the package responsible or rebuild the [[initramfs]] until you fix it.<br />
<br />
=== Problem ===<br />
<br />
A package update likely added an invalid {{ic|''filename''.conf}} to the directory {{ic|/etc/ld.so.conf.d}} or edited {{ic|/etc/ld.so.conf}} incorrectly. The result is that the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} is being re-generated with invalid data. This can potentially cause all programs on the system that depend on shared libraries to fail (ie. almost all of them).<br />
<br />
=== Solution ===<br />
<br />
# Boot with the '''''Arch Linux Installation CD'''''.<br />
# Mount your root {{ic|/}} filesystem on {{ic|/mnt}} and your {{ic|/boot}} filesystem on {{ic|/mnt/boot}} and chroot into the broken system by executing {{ic|# arch-chroot /mnt}}.<br />
# Examine the file {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
# Examine the files located in the directory {{ic|/etc/ld.so.conf.d/}} and remove any invalid files.<br />
# Rebuild the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} by executing {{ic|# ldconfig}}.<br />
# Rebuild the [[Initramfs]] by executing {{ic|# mkinitcpio -p linux}}.<br />
# Exit the chroot, unmount filesystems, and reboot back into your installed system.<br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493471General troubleshooting2017-10-17T23:36:58Z<p>Mcnster: /* Be more specific */ "Let calm heads prevail". :)</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Approach ===<br />
<br />
Rather than approaching an issue by stating,<br />
<br />
''Application X does not work.''<br />
<br />
you will find it more helpful to formulate your issue in the context of the system as a whole, as:<br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== Message: "file: could not find any magic files!" ==<br />
<br />
If you see this message, it likely indicates that a package update has corrupted the dynamic linker run-time bindings file and your system is now essentially crippled. You will not be able to recompile or reinstall the package responsible or rebuild the [[initramfs]] until you fix it.<br />
<br />
=== Problem ===<br />
<br />
A package update likely added an invalid {{ic|''filename''.conf}} to the directory {{ic|/etc/ld.so.conf.d}} or edited {{ic|/etc/ld.so.conf}} incorrectly. The result is that the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} is being re-generated with invalid data. This can potentially cause all programs on the system that depend on shared libraries to fail (ie. almost all of them).<br />
<br />
=== Solution ===<br />
<br />
# Boot with the '''''Arch Linux Installation CD'''''.<br />
# Mount your root {{ic|/}} filesystem on {{ic|/mnt}} and your {{ic|/boot}} filesystem on {{ic|/mnt/boot}} and chroot into the broken system by executing {{ic|# arch-chroot /mnt}}.<br />
# Examine the file {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
# Examine the files located in the directory {{ic|/etc/ld.so.conf.d/}} and remove any invalid files.<br />
# Rebuild the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} by executing {{ic|# ldconfig}}.<br />
# Rebuild the [[Initramfs]] by executing {{ic|# mkinitcpio -p linux}}.<br />
# Exit the chroot, unmount filesystems, and reboot back into your installed system.<br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493470General troubleshooting2017-10-17T23:29:32Z<p>Mcnster: /* Solution */ Clarity.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Be more specific ===<br />
<br />
When attempting to resolve an issue, '''never''' approach it as: <br />
<br />
''Application X does not work.''<br />
<br />
Instead, look at it in its entirety: <br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== Message: "file: could not find any magic files!" ==<br />
<br />
If you see this message, it likely indicates that a package update has corrupted the dynamic linker run-time bindings file and your system is now essentially crippled. You will not be able to recompile or reinstall the package responsible or rebuild the [[initramfs]] until you fix it.<br />
<br />
=== Problem ===<br />
<br />
A package update likely added an invalid {{ic|''filename''.conf}} to the directory {{ic|/etc/ld.so.conf.d}} or edited {{ic|/etc/ld.so.conf}} incorrectly. The result is that the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} is being re-generated with invalid data. This can potentially cause all programs on the system that depend on shared libraries to fail (ie. almost all of them).<br />
<br />
=== Solution ===<br />
<br />
# Boot with the '''''Arch Linux Installation CD'''''.<br />
# Mount your root {{ic|/}} filesystem on {{ic|/mnt}} and your {{ic|/boot}} filesystem on {{ic|/mnt/boot}} and chroot into the broken system by executing {{ic|# arch-chroot /mnt}}.<br />
# Examine the file {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
# Examine the files located in the directory {{ic|/etc/ld.so.conf.d/}} and remove any invalid files.<br />
# Rebuild the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} by executing {{ic|# ldconfig}}.<br />
# Rebuild the [[Initramfs]] by executing {{ic|# mkinitcpio -p linux}}.<br />
# Exit the chroot, unmount filesystems, and reboot back into your installed system.<br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493469General troubleshooting2017-10-17T23:28:55Z<p>Mcnster: /* Solution */ Word change for clarity.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Be more specific ===<br />
<br />
When attempting to resolve an issue, '''never''' approach it as: <br />
<br />
''Application X does not work.''<br />
<br />
Instead, look at it in its entirety: <br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== Message: "file: could not find any magic files!" ==<br />
<br />
If you see this message, it likely indicates that a package update has corrupted the dynamic linker run-time bindings file and your system is now essentially crippled. You will not be able to recompile or reinstall the package responsible or rebuild the [[initramfs]] until you fix it.<br />
<br />
=== Problem ===<br />
<br />
A package update likely added an invalid {{ic|''filename''.conf}} to the directory {{ic|/etc/ld.so.conf.d}} or edited {{ic|/etc/ld.so.conf}} incorrectly. The result is that the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} is being re-generated with invalid data. This can potentially cause all programs on the system that depend on shared libraries to fail (ie. almost all of them).<br />
<br />
=== Solution ===<br />
<br />
# Boot with the '''''Arch Linux Installation CD'''''.<br />
# Mount your root {{ic|/}} filesystem on {{ic|/mnt}} and your {{ic|/boot}} filesystem on {{ic|/mnt/boot}} and chroot into it by executing {{ic|# arch-chroot /mnt}}.<br />
# Examine the file {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
# Examine the files located in the directory {{ic|/etc/ld.so.conf.d/}} and remove any invalid files.<br />
# Rebuild the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} by executing {{ic|# ldconfig}}.<br />
# Rebuild the [[Initramfs]] by executing {{ic|# mkinitcpio -p linux}}.<br />
# Exit the chroot, unmount filesystems, and reboot back into your installed system.<br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493468General troubleshooting2017-10-17T23:26:54Z<p>Mcnster: /* Message: "file: could not find any magic files!" */ Cleaned up, clarified, and wiki-fied.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Be more specific ===<br />
<br />
When attempting to resolve an issue, '''never''' approach it as: <br />
<br />
''Application X does not work.''<br />
<br />
Instead, look at it in its entirety: <br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== Message: "file: could not find any magic files!" ==<br />
<br />
If you see this message, it likely indicates that a package update has corrupted the dynamic linker run-time bindings file and your system is now essentially crippled. You will not be able to recompile or reinstall the package responsible or rebuild the [[initramfs]] until you fix it.<br />
<br />
=== Problem ===<br />
<br />
A package update likely added an invalid {{ic|''filename''.conf}} to the directory {{ic|/etc/ld.so.conf.d}} or edited {{ic|/etc/ld.so.conf}} incorrectly. The result is that the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} is being re-generated with invalid data. This can potentially cause all programs on the system that depend on shared libraries to fail (ie. almost all of them).<br />
<br />
=== Solution ===<br />
<br />
# Boot with the '''''Arch Linux Installation CD'''''.<br />
# Mount your root {{ic|/}} filesystem on {{ic|/mnt}} and your {{ic|/boot}} filesystem on {{ic|/mnt/boot}} and chroot into it by executing {{ic|# arch-chroot /mnt}}.<br />
# Examine the file {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
# Examine the files located in the directory {{ic|/etc/ld.so.conf.d/}} and remove any invalid files.<br />
# Rebuild the dynamic linker run-time bindings file {{ic|/etc/ld.so.cache}} by executing {{ic|# ldconfig}}.<br />
# Rebuild the [[Initramfs]] by executing {{ic|# mkinitcpio -p linux}}.<br />
# Reboot back to your installed system.<br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493467General troubleshooting2017-10-17T22:57:43Z<p>Mcnster: /* file: could not find any magic files! */ Added "Message: " to clarify that this is a diagnostic message.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Be more specific ===<br />
<br />
When attempting to resolve an issue, '''never''' approach it as: <br />
<br />
''Application X does not work.''<br />
<br />
Instead, look at it in its entirety: <br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== Message: "file: could not find any magic files!" ==<br />
<br />
{{Style|See [[Help:Style]] and related articles.}}<br />
<br />
''Example:'' After an every-day routine update or following the installation of a package you are given the following error:<br />
# file: could not find any magic files!<br />
This will most likely leave your system crippled. And, any attempts made to recompile/reinstall the package(s) responsible for the breakage will fail. Also, any attempts made to try to rebuild the [[initramfs]] will result in the following:<br />
# mkinitcpio -p linux<br />
==> Building image from preset: 'default'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img<br />
file: could not find any magic files!<br />
==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
==> Building image from preset: 'fallback'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect<br />
file: could not find any magic files!<br />
@==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
<br />
Typically a previously installed application had placed a configuration file within {{ic|/etc/ld.so.conf.d/}} or it had made changes to {{ic|/etc/ld.so.conf}} which are now invalid.<br />
#Boot into the Arch Linux Live CD / Installation Media.<br />
#Mount your root ({{ic|'''/'''}}) partition to {{ic|/mnt}} and chroot into it by executing {{ic|# arch-chroot /mnt}}.<br />
{{Note|{{ic|arch-chroot}} does not automatically mount the {{ic|/boot}} filesystem. If you need it mounted, mount it prior to executing the {{ic|arch-chroot}}.}}<br />
#Examine {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
#Examine the files located inside the directory {{ic|/etc/ld.so.conf.d/}} and remove all invalid files.<br />
#Rebuild the [[initramfs]].<br />
# mkinitcpio -p linux<br />
#Reboot back to your installed system.<br />
#Once booted, reinstall the package that was responsible for leaving your system inoperable using:<br />
# pacman -S <package><br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493466General troubleshooting2017-10-17T22:56:59Z<p>Mcnster: /* error while loading shared libraries */ Added "Message:" prefix to section heading to be clear.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Be more specific ===<br />
<br />
When attempting to resolve an issue, '''never''' approach it as: <br />
<br />
''Application X does not work.''<br />
<br />
Instead, look at it in its entirety: <br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== Message: "error while loading shared libraries" ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== file: could not find any magic files! ==<br />
<br />
{{Style|See [[Help:Style]] and related articles.}}<br />
<br />
''Example:'' After an every-day routine update or following the installation of a package you are given the following error:<br />
# file: could not find any magic files!<br />
This will most likely leave your system crippled. And, any attempts made to recompile/reinstall the package(s) responsible for the breakage will fail. Also, any attempts made to try to rebuild the [[initramfs]] will result in the following:<br />
# mkinitcpio -p linux<br />
==> Building image from preset: 'default'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img<br />
file: could not find any magic files!<br />
==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
==> Building image from preset: 'fallback'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect<br />
file: could not find any magic files!<br />
@==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
<br />
Typically a previously installed application had placed a configuration file within {{ic|/etc/ld.so.conf.d/}} or it had made changes to {{ic|/etc/ld.so.conf}} which are now invalid.<br />
#Boot into the Arch Linux Live CD / Installation Media.<br />
#Mount your root ({{ic|'''/'''}}) partition to {{ic|/mnt}} and chroot into it by executing {{ic|# arch-chroot /mnt}}.<br />
{{Note|{{ic|arch-chroot}} does not automatically mount the {{ic|/boot}} filesystem. If you need it mounted, mount it prior to executing the {{ic|arch-chroot}}.}}<br />
#Examine {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
#Examine the files located inside the directory {{ic|/etc/ld.so.conf.d/}} and remove all invalid files.<br />
#Rebuild the [[initramfs]].<br />
# mkinitcpio -p linux<br />
#Reboot back to your installed system.<br />
#Once booted, reinstall the package that was responsible for leaving your system inoperable using:<br />
# pacman -S <package><br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493465General troubleshooting2017-10-17T22:54:22Z<p>Mcnster: /* file: could not find any magic files! */ Fixed broken link.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Be more specific ===<br />
<br />
When attempting to resolve an issue, '''never''' approach it as: <br />
<br />
''Application X does not work.''<br />
<br />
Instead, look at it in its entirety: <br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== error while loading shared libraries ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== file: could not find any magic files! ==<br />
<br />
{{Style|See [[Help:Style]] and related articles.}}<br />
<br />
''Example:'' After an every-day routine update or following the installation of a package you are given the following error:<br />
# file: could not find any magic files!<br />
This will most likely leave your system crippled. And, any attempts made to recompile/reinstall the package(s) responsible for the breakage will fail. Also, any attempts made to try to rebuild the [[initramfs]] will result in the following:<br />
# mkinitcpio -p linux<br />
==> Building image from preset: 'default'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img<br />
file: could not find any magic files!<br />
==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
==> Building image from preset: 'fallback'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect<br />
file: could not find any magic files!<br />
@==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
<br />
Typically a previously installed application had placed a configuration file within {{ic|/etc/ld.so.conf.d/}} or it had made changes to {{ic|/etc/ld.so.conf}} which are now invalid.<br />
#Boot into the Arch Linux Live CD / Installation Media.<br />
#Mount your root ({{ic|'''/'''}}) partition to {{ic|/mnt}} and chroot into it by executing {{ic|# arch-chroot /mnt}}.<br />
{{Note|{{ic|arch-chroot}} does not automatically mount the {{ic|/boot}} filesystem. If you need it mounted, mount it prior to executing the {{ic|arch-chroot}}.}}<br />
#Examine {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
#Examine the files located inside the directory {{ic|/etc/ld.so.conf.d/}} and remove all invalid files.<br />
#Rebuild the [[initramfs]].<br />
# mkinitcpio -p linux<br />
#Reboot back to your installed system.<br />
#Once booted, reinstall the package that was responsible for leaving your system inoperable using:<br />
# pacman -S <package><br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493464General troubleshooting2017-10-17T22:51:09Z<p>Mcnster: /* file: could not find any magic files! */ Fixed broken link.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Be more specific ===<br />
<br />
When attempting to resolve an issue, '''never''' approach it as: <br />
<br />
''Application X does not work.''<br />
<br />
Instead, look at it in its entirety: <br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== error while loading shared libraries ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== file: could not find any magic files! ==<br />
<br />
{{Style|See [[Help:Style]] and related articles.}}<br />
<br />
''Example:'' After an every-day routine update or following the installation of a package you are given the following error:<br />
# file: could not find any magic files!<br />
This will most likely leave your system crippled. And, any attempts made to recompile/reinstall the package(s) responsible for the breakage will fail. Also, any attempts made to try to rebuild the [[initramfs]] will result in the following:<br />
# mkinitcpio -p linux<br />
==> Building image from preset: 'default'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img<br />
file: could not find any magic files!<br />
==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
==> Building image from preset: 'fallback'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect<br />
file: could not find any magic files!<br />
@==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
<br />
Typically a previously installed application had placed a configuration file within {{ic|/etc/ld.so.conf.d/}} or it had made changes to {{ic|/etc/ld.so.conf}} which are now invalid.<br />
#Boot into the Arch Linux Live CD / Installation Media.<br />
#Mount your root ({{ic|'''/'''}}) partition to {{ic|/mnt}} and chroot into it by executing {{ic|# arch-chroot /mnt}}.<br />
{{Note|[[Change root#Change_root|arch-chroot]]{{Broken section link}} leaves mounting the {{ic|/boot}} partition up to the user.}}<br />
#Examine {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
#Examine the files located inside the directory {{ic|/etc/ld.so.conf.d/}} and remove all invalid files.<br />
#Rebuild the [[initramfs]].<br />
# mkinitcpio -p linux<br />
#Reboot back to your installed system.<br />
#Once booted, reinstall the package that was responsible for leaving your system inoperable using:<br />
# pacman -S <package><br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493463General troubleshooting2017-10-17T22:37:09Z<p>Mcnster: /* Reboot into root shell and fix problem */ Word change for clarity.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Be more specific ===<br />
<br />
When attempting to resolve an issue, '''never''' approach it as: <br />
<br />
''Application X does not work.''<br />
<br />
Instead, look at it in its entirety: <br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all parameters.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== error while loading shared libraries ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== file: could not find any magic files! ==<br />
<br />
{{Style|See [[Help:Style]] and related articles.}}<br />
<br />
''Example:'' After an every-day routine update or following the installation of a package you are given the following error:<br />
# file: could not find any magic files!<br />
This will most likely leave your system crippled. And, any attempts made to recompile/reinstall the package(s) responsible for the breakage will fail. Also, any attempts made to try to rebuild the [[initramfs]] will result in the following:<br />
# mkinitcpio -p linux<br />
==> Building image from preset: 'default'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img<br />
file: could not find any magic files!<br />
==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
==> Building image from preset: 'fallback'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect<br />
file: could not find any magic files!<br />
@==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
<br />
Typically a previously installed application had placed a configuration file within {{ic|/etc/ld.so.conf.d/}} or it had made changes to {{ic|/etc/ld.so.conf}} which are now invalid.<br />
#Boot into the Arch Linux Live CD / Installation Media.<br />
#Mount your root ({{ic|'''/'''}}) partition to {{ic|/mnt}} and using [[Change root#Change_root|arch-chroot]]{{Broken section link}}, [[chroot]] into your system.<br />
{{Note|[[Change root#Change_root|arch-chroot]]{{Broken section link}} leaves mounting the {{ic|/boot}} partition up to the user.}}<br />
#Examine {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
#Examine the files located inside the directory {{ic|/etc/ld.so.conf.d/}} and remove all invalid files.<br />
#Rebuild the [[initramfs]].<br />
# mkinitcpio -p linux<br />
#Reboot back to your installed system.<br />
#Once booted, reinstall the package that was responsible for leaving your system inoperable using:<br />
# pacman -S <package><br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=493462Kernel Panics2017-10-17T22:35:15Z<p>Mcnster: /* Deleted Article */ After merge into "General troubleshooting".</p>
<hr />
<div></div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Talk:General_troubleshooting&diff=493461Talk:General troubleshooting2017-10-17T22:33:48Z<p>Mcnster: /* Merged article "Kernel panics" into this article. */ new section</p>
<hr />
<div>== Blacklisting radeon module ==<br />
<br />
:''Moved from [[Talk:Beginners' guide]]. -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 14:04, 15 August 2015 (UTC)''<br />
<br />
Installed Arch on my laptop, during pacstrap the screen went blank, pressing SPACE, CTL+C ... didn't helped only modprobe.blacklist=radeon enabled me to go through the whole installation process.<br />
My graphic card is ATI M96 aka Mobility Radeon HD 4650.<br />
I believe this info and similar problems should be added to the beginner's guide on a Installation's Issues Troubleshooting section. I believe this is important enough to dual post and separate it from the Removing "Kernel modules" talk. p.s. I may add that this is my first desktop Linux experience--[[User:Dhead|Dhead]] ([[User talk:Dhead|talk]]) 06:20, 4 March 2013 (UTC)<br />
<br />
:<s>The beginners' guide should not contain hardware-specific info, if the issue is common enough links can be added to [[Beginners%27_guide#Troubleshooting_boot_problems]].</s> -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 14:40, 27 December 2014 (UTC)<br />
<br />
==Expansion needed==<br />
<br />
I didn't realize the only page linking here was the Beginners' Guide - thanks for catching my blunder Kynikos. I got rid of that link because I figured this page could only be frustrating for beginners looking for general troubleshooting help. I'm having trouble pinning down the scope - can we have a brainstorm for what kind of content could go in here? | [[User:Emiralle|Emiralle]] 14:03, 11 October 2011 (EDT)<br />
:I have a bunch of random stuff that might fit in here or in [[General Recommendations]] or in the respective articles e.g. about xorg-server. Would e.g. [https://bbs.archlinux.org/viewtopic.php?pid=1001550#p1001550] (seems what you mean in your first bullet-point) or [https://bbs.archlinux.org/viewtopic.php?pid=988292#p988292] fit here? -- [[User:Karol|Karol]] 14:41, 11 October 2011 (EDT)<br />
::I think that finding troubleshooting tips that are really "general" can be quite difficult, usually the purpose of troubleshooting itself is trying to narrow the field of possible causes of the problem in order to find the specific cause and then solve it. That's why it's much more likely that a troubleshooting procedure can belong much more easily to an article treating a specific piece of software rather than here. -- [[User:Kynikos|Kynikos]] 15:05, 13 October 2011 (EDT)<br />
:::I think some of these are close to the mark, but I'm not sure I believe in this article anymore. General troubleshooting is great in a paper encyclopedia where you don't have a powerful search function, but I have a hard time not stepping on other articles' toes here. I say move [[General Troubleshooting#Finding startup files]] to [[FAQ]], and move [[General Troubleshooting#Single user mode]] either to FAQ or merge with [[Runlevels]]. Maybe this page could redirect to FAQ since it's a similar idea and might be helpful. | [[User:Emiralle|Emiralle]] 16:15, 13 October 2011 (EDT)<br />
To start things off: <br />
*How about something regarding programs unable to source files or otherwise not working when they don't have the right permissions? It's not particular to a given application, though it's quite related to [[Users and groups]]. | [[User:Emiralle|Emiralle]] 14:03, 11 October 2011 (EDT)<br />
*Something general about missing or conflicting kernel modules/drivers/firmware and how to go about diagnosing such issues. | [[User:Emiralle|Emiralle]] 14:03, 11 October 2011 (EDT)<br />
::Um... also this one would probably be more related to [[Kernel modules]]. -- [[User:Kynikos|Kynikos]] 15:05, 13 October 2011 (EDT)<br />
*Bring here something from the [[FAQ]]? -- [[User:Kynikos|Kynikos]] 15:05, 13 October 2011 (EDT)<br />
*FAQ-ize the two sections of this article, move them to [[FAQ]] and delete this article? (You requested brainstorming, after all :P ) -- [[User:Kynikos|Kynikos]] 15:05, 13 October 2011 (EDT)<br />
*Maybe this page could be a FAQ-like list of links to help requests in the forum which have found a solution? -- [[User:Kynikos|Kynikos]] 15:05, 13 October 2011 (EDT)<br />
*Focus on the concept of general Problem Solving Procedures instead of general Troubleshooting? For example list suggestions like "If a GUI application crashes, try to look at its output in tty1 (or the virtual console where you started your wm)". -- [[User:Kynikos|Kynikos]] 15:05, 13 October 2011 (EDT)<br />
:: I vote for this idea. We can make this page as "General" as it can. Do not add solution of a particular application. And consider "non-programmer" as the target reader. What request a programming skill or knowledge should go to [[Step By Step Debugging Guide]]. We can add info about <br />
:: * Search from google<br />
:: * Recall what changed<br />
:: * Downgrade package<br />
:: * Search mailing list<br />
:: * Create a new user for a clean environment<br />
:: And so on.<br />
:: We can add step by step introduction on ''how to narrow the field of possible causes of the problem'' . For reference this artical :[http://www.maximumpc.com/article/features/linux_troubleshooting_guide_fix_most_common_problems?page=0,2] -- [[User:Fengchao|Fengchao]] 23:52, 1 February 2012 (EST)<br />
:::These are certainly good ideas, however I'd like also to point you to [[ArchWiki:Reports#System_recovery_category]] where we briefly discussed about merging this article with [[Step By Step Debugging Guide]], as it could be the easiest solution. -- [[User:Kynikos|Kynikos]] 09:05, 4 February 2012 (EST)<br />
*More ideas...<br />
<br />
=== Related article names ===<br />
<br />
Currently there's several "debugging" articles:<br />
<br />
* [[Step_By_Step_Debugging_Guide]] in [[:Category:Development]]<br />
* [[Debug_-_Getting_Traces]] in [[:Category:Package development]]<br />
* [[Boot_debugging]] in [[:Category:Boot process]] and [[:Category:System recovery]]<br />
<br />
[[General troubleshooting]] is in [[:Category:System administration]] and [[:Category:System recovery]].<br />
<br />
Suggestions:<br />
<br />
* one category, existing such as [[:Category:System recovery]], or create a new one such as [[:Category:Troubleshooting]];<br />
* sub-articles such as <s>[[Troubleshooting/Traces]]</s>, [[Troubleshooting/Step by step]], [[Troubleshooting/Boot]], <br />
<s>[[Troubleshooting/General]]</s>.<br />
<br />
-- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 09:44, 10 August 2014 (UTC)<br />
<br />
:What would be the scope of the main [[Troubleshooting]] page? Would [[General troubleshooting]] be moved to use that title?<br />
:[[Step_By_Step_Debugging_Guide]] and [[Debug_-_Getting_Traces]] should be merged into one article, the introductions mention the same purpose. Alternatively, the info for C{,++} programs could be split from info for "others"; this might had been the intention of [[Debug - Getting Traces]].<br />
:-- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 09:59, 10 August 2014 (UTC)<br />
<br />
::No details are implied in the title, so adding "General" to it is redundant IMO. However, only the four first sections (''Attention to detail'' ... ''Additional support'') of [[General troubleshooting]] are "general" - see the discussion above for more suggestions on that regard. The others handle specific issues, which I'm unsure belong there.<br />
::Regarding [[Step_By_Step_Debugging_Guide]], as the "others" section is limited in scope, I would agree for merging it with [[Debug - Getting Traces]]. -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 22:03, 12 August 2014 (UTC)<br />
<br />
:::To me, simple "Troubleshooting" seems too vague, I don't mind including some word in the title specifying that the content is generic. Naming the page "Troubleshooting" might encourage addition of too specific sections similar to [[General_troubleshooting#Session_permissions]]. IMO "General troubleshooting" or "Debugging guide" are good titles. -- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 21:04, 18 August 2014 (UTC)<br />
<br />
=== Introduction ===<br />
<br />
:''Moved from [[Talk:Beginners' guide]].'' -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 12:41, 25 June 2015 (UTC)<br />
<br />
Apart from that there should be a section at the very beginning explaining how to troubleshoot your own problems. Learning dmesg -HLkd and journalctl -b etc were helpful tools for me. I also appreciated learning lsblk, lsmod, ls etc from the various articles, but a quick over view of these helpful commands on this page would help newbies like myself. [[User:Victoroux|Victoroux]] ([[User talk:Victoroux|talk]]) 14:01, 7 June 2013 (UTC)<br />
<br />
: Sorry! Just reading over again, and realizing that I could've saved a tonne of time if I knew the problem of "a bunch of white letters clustered on my screen" was an error I could check. It usually happened when the firmware didn't support something (in my case) but telling the user what he can do when this happens helps ease the wiki hopping. I finally, finally figured out how to debug most of my own problems and I think that is the number one thing this guide should do. No offense, but it would also lessen the load on the "newbie corner" on the forums (not that I know it's loaded or not, but less is better, right?). That way no matter what's written in the guide, if it's incorrect or leads to a bad result, the user can figure out why and what to do.. [[User:Victoroux|Victoroux]] ([[User talk:Victoroux|talk]]) 14:05, 7 June 2013 (UTC)<br />
<br />
:: Another [[Keyboard_Shortcuts|useful article]] that could be mentioned (rebooting from black screens, yay!) [[User:Victoroux|Victoroux]] ([[User talk:Victoroux|talk]]) 00:27, 8 June 2013 (UTC)<br />
<br />
:::Regarding your second point, [[General troubleshooting]] is in Related articles - it could be expanded there. --[[User:Alad|Alad]] ([[User talk:Alad|talk]]) 14:25, 2 July 2014 (UTC)<br />
<br />
== "Single user mode" section is outdated ==<br />
<br />
The section suggests changing the runlevel.<br />
<br />
It could be replaced with a simple reference to man systemd.target<br />
<br />
[[User:Head on a Stick|Head on a Stick]] ([[User talk:Head on a Stick|talk]]) 22:43, 16 November 2014 (UTC)<br />
<br />
== user buses ==<br />
<br />
How is [https://wiki.archlinux.org/index.php?title=General_troubleshooting&curid=11976&diff=414390&oldid=411991] related to session permissions? -- [[User:Alad|Alad]] ([[User talk:Alad|talk]]) 12:43, 11 January 2016 (UTC)<br />
<br />
== Microcode updates ==<br />
<br />
Re: [https://redd.it/6wt802 this thread on Reddit] and [https://wiki.archlinux.org/index.php?title=General_troubleshooting&type=revision&diff=488006&oldid=488005 this edit], microcode updates have broken at least one attempt to run Arch. While the recommendation is to make sure they're installed and applied, in this specific instance things exploded, so it might be worthwhile to at least note that microcode updates may break things once in a while. Are you sure this should be left out? (Pinging [[User:Lahwaacz|Lahwaacz]]) [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 22:38, 29 August 2017 (UTC)<br />
<br />
:The reddit thread, which was the source for your edit, does not discuss any details regarding the system which was supposed to be broken by microcode updates, so it's impossible to tell if the concern is valid or not. I see it as just one isolated case out of many. What percentage of these i3-7100U CPUs have this problem? What's Intel's position on this? -- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 23:30, 29 August 2017 (UTC)<br />
<br />
::Alright, I can't find much about this issue anywhere, so this is fine as it is now. -- [[User:DragoonAethis|DragoonAethis]] ([[User talk:DragoonAethis|talk]]) 11:55, 5 September 2017 (UTC)<br />
<br />
== Merged article "Kernel panics" into this article. ==<br />
<br />
I have merged the article "Kernel panics" into this "General troubleshooting" article as directed by the Merge template on the former. [[User:Mcnster|Mcnster]] ([[User talk:Mcnster|talk]]) 22:33, 17 October 2017 (UTC)</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=General_troubleshooting&diff=493460General troubleshooting2017-10-17T22:31:28Z<p>Mcnster: /* Boot problems */ Merged article "Kernel panics" into 2nd level section following "Boot problems" as suggested by Merge template.</p>
<hr />
<div>[[Category:System administration]]<br />
[[Category:System recovery]]<br />
[[Category:Getting and installing Arch]]<br />
[[es:General troubleshooting]]<br />
[[ja:一般的なトラブルシューティング]]<br />
[[ru:General troubleshooting]]<br />
{{Related articles start}}<br />
{{Related|Reporting bug guidelines}}<br />
{{Related|Step-by-step debugging guide}}<br />
{{Related|Debug - Getting Traces}}<br />
{{Related|IRC Collaborative Debugging}}<br />
{{Related articles end}}<br />
<br />
This article explains some methods for general troubleshooting. For application specific issues, please reference the particular wiki page for that program.<br />
<br />
== General procedures ==<br />
<br />
=== Attention to detail ===<br />
<br />
In order to resolve an issue that you are having, it is ''absolutely crucial'' to have a firm basic understanding of how that specific subsystem functions. How does it work, and what does it need to run without error? If you cannot comfortably answer these question then you would best review the [[Table of contents|Archwiki]] article for the subsystem that you are having trouble with. Once you feel like you've understood it, it will be easier for you to pinpoint the cause of the problem.<br />
<br />
=== Questions/checklist ===<br />
<br />
The following gives a number of questions for you whenever dealing with a malfunctioning system. Under each question there are notes explaining how you should be answering each question, followed by some light examples on how to easily gather data output and what tools can be used to review logs and the journal.<br />
<br />
# What is the issue(s)?<br />
#: Be ''as precise as possible''. This will help you not get confused and/or side-tracked when looking up specific information.<br />
# Are there error messages? (if any)<br />
#: Copy and paste ''full outputs'' that contain '''error messages''' related to your issue into a separate file, such as {{ic|$HOME/issue.log}}. For example, to forward the output of the following [[mkinitcpio]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ mkinitcpio -p linux >> $HOME/issue.log''}}<br />
# Can you reproduce the issue?<br />
#: If so, give ''exact'' '''step-by-step''' instructions/commands needed to do so.<br />
# When did you first encounter these issues and what was changed between then and when the system was operating without error?<br />
#:If it occurred right after an update then, list '''all packages that were updated'''. Include ''version numbers'', also, paste the entire update from [[pacman]].log ({{ic|/var/log/pacman.log}}). Also take note of the statuses of ''any'' service(s) needed to support the malfunctioning application(s) using [[systemd]]'s systemctl tools. For example, to forward the output of the following [[Systemd#Basic_systemctl_usage|systemd]] command to {{ic|$HOME/issue.log}}:<br />
#: {{bc|$ systemctl status dhcpcd@eth0.service >> $HOME/issue.log}}<br />
#: {{Note|Using {{ic|'''>>'''}} will ensure any previous text in {{ic|$HOME/issue.log}} will not be overwritten.}}<br />
<br />
=== Be more specific ===<br />
<br />
When attempting to resolve an issue, '''never''' approach it as: <br />
<br />
''Application X does not work.''<br />
<br />
Instead, look at it in its entirety: <br />
<br />
''Application X produces Y error(s) when performing Z tasks under conditions A and B.<br />
<br />
=== Additional support ===<br />
<br />
With all the information in front of you you should have a good idea as to what is going on with the system and you can now start working on a proper fix.<br />
<br />
If you require any additional support, it can be found on [https://bbs.archlinux.org the forums] or IRC at irc.freenode.net #archlinux See [[IRC channels]] for other options.<br />
<br />
When asking for support post the '''complete''' output/logs, not just what you think are the significant sections. Sources of information include:<br />
<br />
* Full output of any command involved - don't just select what you think is relevant.<br />
* Output from systemd's {{ic|journalctl}}. For more extensive output, use the {{ic|1=systemd.log_level=debug}} boot parameter.<br />
* Log files (have a look in {{ic|/var/log}})<br />
* Relevant configuration files<br />
* Drivers involved<br />
* Versions of packages involved<br />
* Kernel: {{ic|dmesg}}. For a boot problem, at least the last 10 lines displayed, preferably more<br />
* Networking: Exact output of commands involved, and any configuration files<br />
* Xorg: {{ic|/var/log/Xorg.0.log}}, and prior logs if you have overwritten the problematic one<br />
* Pacman: If a recent upgrade broke something, look in {{ic|/var/log/pacman.log}}<br />
<br />
One of the better ways to post this information is to use an online pastebin. You can [[install]] the {{pkg|pbpst}} or {{pkg|gist}} package to automatically upload information. For example, to upload the content of your systemd journal from this boot you would do:<br />
<br />
# journalctl -xb | pbpst -S<br />
<br />
A link will then be output that you can paste to the forum or IRC.<br />
<br />
Additionally, before posting your question, you may wish to review [http://www.catb.org/esr/faqs/smart-questions.html how to ask smart questions]. See also [[Code of conduct]].<br />
<br />
== Boot problems ==<br />
<br />
Diagnosing errors during the [[boot process]] involves changing the [[kernel parameters]], and rebooting the system.<br />
<br />
If booting the system is not possible, boot from a [https://www.archlinux.org/download/ live image] and [[change root]] to the existing system.<br />
<br />
=== Console messages ===<br />
<br />
After the boot process, the screen is cleared and the login prompt appears, leaving users unable to read init output and error messages. This default behavior may be modified using methods outlined in the sections below.<br />
<br />
Note that regardless of the chosen option, kernel messages can be displayed for inspection after booting by using {{ic|dmesg}} or all logs from the current boot with {{ic|journalctl -b}}.<br />
<br />
==== Flow control ====<br />
<br />
This is basic management that applies to most terminal emulators, including virtual consoles (vc):<br />
<br />
* Press {{ic|Ctrl+S}} to pause the output<br />
* And {{ic|Ctrl+Q}} to resume it<br />
<br />
This pauses not only the output, but also programs which try to print to the terminal, as they will block on the {{ic|write()}} calls for as long as the output is paused. If your ''init'' appears frozen, make sure the system console is not paused.<br />
<br />
To see error messages which are already displayed, see [[Getty#Have boot messages stay on tty1]].<br />
<br />
==== Scrollback ====<br />
<br />
Scrollback allows the user to go back and view text which has scrolled off the screen of a text console. This is made possible by a buffer created between the video adapter and the display device called the scrollback buffer. By default, the key combinations of {{ic|Shift+PageUp}} and {{ic|Shift+PageDown}} scroll the buffer up and down.<br />
<br />
If scrolling up all the way does not show you enough information, you need to expand your scrollback buffer to hold more output. This is done by tweaking the kernel's framebuffer console (fbcon) with the [[kernel parameter]] {{ic|1=fbcon=scrollback:Nk}} where {{ic|N}} is the desired buffer size is kilobytes. The default size is 32k.<br />
<br />
If this does not work, your framebuffer console may not be properly enabled. Check the [https://www.kernel.org/doc/Documentation/fb/fbcon.txt Framebuffer Console documentation] for other parameters, e.g. for changing the framebuffer driver.<br />
<br />
==== Debug output ====<br />
<br />
Most kernel messages are hidden during boot. You can see more of these messages by adding different kernel parameters. The simplest ones are:<br />
<br />
* {{ic|debug}} enables debug messages for both the kernel and [[systemd]]<br />
* {{ic|ignore_loglevel}} forces ''all'' kernel messages to be printed<br />
<br />
Other parameters you can add that might be useful in certain situations are:<br />
* {{ic|1=earlyprintk=vga,keep}} prints kernel messages very early in the boot process, in case the kernel would crash before output is shown. You must change {{ic|vga}} to {{ic|efi}} for [[EFI]] systems<br />
* {{ic|1=log_buf_len=16M}} allocates a larger (16MB) kernel message buffer, to ensure that debug output is not overwritten<br />
<br />
There are also a number of separate debug parameters for enabling debugging in specific subsystems e.g. {{ic|bootmem_debug}}, {{ic|sched_debug}}. Check the [https://www.kernel.org/doc/Documentation/kernel-parameters.txt kernel parameter documentation] for specific information.<br />
<br />
{{Note|If you cannot scroll back far enough to view the desired boot output, you should increase the size of the [[#Scrollback|scrollback buffer]].}}<br />
<br />
=== Recovery shells ===<br />
<br />
Getting an interactive shell at some stage in the boot process can help you pinpoint exactly where and why something is failing. There are several kernel parameters for doing so, but they all launch a normal shell which you can {{ic|exit}} to let the kernel resume what it was doing:<br />
* {{ic|rescue}} launches a shell shortly after the root filesystem is remounted read/write<br />
* {{ic|emergency}} launches a shell even earlier, before most filesystems are mounted<br />
* {{ic|1=init=/bin/sh}} (as a last resort) changes the init program to a root shell. {{ic|rescue}} and {{ic|emergency}} both rely on [[systemd]], but this should work even if ''systemd'' is broken<br />
<br />
Another option is systemd's debug-shell which adds a root shell on {{ic|tty9}} (accessible with Ctrl+Alt+F9). It can be enabled by either adding {{ic|systemd.debug-shell}} to the [[kernel parameters]], or by [[enabling]] {{ic|debug-shell.service}}. Take care to disable the service when done to avoid the security risk of leaving a root shell open on every boot.<br />
<br />
=== Blank screen with Intel video ===<br />
<br />
This is most likely due to a problem with [[kernel mode setting]]. Try [[Kernel mode setting#Disabling modesetting|disabling modesetting]] or changing the [[Intel#KMS Issue: console is limited to small area|video port]].<br />
<br />
=== Stuck while loading the kernel ===<br />
<br />
Try disabling ACPI by adding the {{ic|1=acpi=off}} kernel parameter.<br />
<br />
=== Debugging kernel modules ===<br />
<br />
See [[Kernel modules#Obtaining information]].<br />
<br />
=== Debugging hardware ===<br />
<br />
* You can display extra debugging information about your hardware by following [[udev#Debug output]].<br />
* Ensure that [[Microcode]] updates are applied on your system.<br />
* Test your device's RAM with [http://www.memtest.org/ Memtest86+]. Unstable RAM may lead to some extremely odd issues, ranging from random crashes to data corruption.<br />
<br />
=== See also ===<br />
<br />
* [http://wiki.ultimatebootcd.com/index.php?title=Tools List of Tools for UBCD] - Can be added to custom menu.lst like memtest<br />
* Wikipedia's page on [[Wikipedia:BIOS Boot partition|BIOS Boot partition]]<br />
* [https://fedoraproject.org/wiki/QA/Sysrq QA/Sysrq] - Using sysrq<br />
* systemd documentation: [http://freedesktop.org/wiki/Software/systemd/Debugging#Debug_Logging_to_a_Serial_Console Debug Logging to a Serial Console]<br />
* [https://web.archive.org/web/20120217124742/http://www.lesswatts.org/projects/acpi/debug.php How to Isolate Linux ACPI Issues]<br />
<br />
== Kernel panics ==<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
=== Examine panic message ===<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
==== Example scenario: bad module ====<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
=== Reboot into root shell and fix problem ===<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.<br />
<br />
== Package management ==<br />
<br />
See [[Pacman#Troubleshooting]] for general topics, and [[pacman/Package signing#Troubleshooting]] for issues with PGP keys.<br />
<br />
== fuser ==<br />
<br />
{{Expansion|Write an example how to use it.}}<br />
<br />
''fuser'' is a command-line utility for identifying processes using resources such as files, filesystems and TCP/UDP ports.<br />
<br />
''fuser'' is provided by the {{Pkg|psmisc}} package, which should be already installed as part of the {{Grp|base}} group.<br />
<br />
== Session permissions ==<br />
<br />
{{Note|You must be using [[systemd]] as your init system for local sessions to work.[https://www.archlinux.org/news/d-bus-now-launches-user-buses/] It is required for polkit permissions and ACLs for various devices (see {{ic|/usr/lib/udev/rules.d/70-uaccess.rules}} and [http://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/])}}<br />
<br />
First, make sure you have a valid local session within X:<br />
<br />
$ loginctl show-session $XDG_SESSION_ID<br />
<br />
This should contain {{ic|1=Remote=no}} and {{ic|1=Active=yes}} in the output. If it does not, make sure that X runs on the same tty where the login occurred. This is required in order to preserve the logind session.<br />
<br />
A D-Bus session should also be started along with X. See [[D-Bus#Starting the user session]] for more information on this.<br />
<br />
Basic [[polkit]] actions do not require further set-up. Some polkit actions require further authentication, even with a local session. A polkit authentication agent needs to be running for this to work. See [[polkit#Authentication agents]] for more information on this.<br />
<br />
== error while loading shared libraries ==<br />
<br />
{{Accuracy|Or the program needs to be rebuilt after a [[System_maintenance#Partial_upgrades_are_unsupported|soname bump]].}}<br />
<br />
If, while using a program, you get an error similar to: <br />
<br />
error while loading shared libraries: libusb-0.1.so.4: cannot open shared object file: No such file or directory<br />
<br />
Use [[pacman]] or [[pkgfile]] to search for the package that owns the missing library:<br />
<br />
{{hc|$ pacman -Fs libusb-0.1.so.4|<br />
extra/libusb-compat 0.1.5-1<br />
usr/lib/libusb-0.1.so.4<br />
}}<br />
<br />
In this case, the {{Pkg|libusb-compat}} package needs to be [[installed]].<br />
<br />
The error could also mean that the package that you used to install your program does not list the library as a dependency in its [[PKGBUILD]]: if it is an official package, [[report a bug]]; if it is an [[AUR]] package, report it to the maintainer using its page in the AUR website.<br />
<br />
== file: could not find any magic files! ==<br />
<br />
{{Style|See [[Help:Style]] and related articles.}}<br />
<br />
''Example:'' After an every-day routine update or following the installation of a package you are given the following error:<br />
# file: could not find any magic files!<br />
This will most likely leave your system crippled. And, any attempts made to recompile/reinstall the package(s) responsible for the breakage will fail. Also, any attempts made to try to rebuild the [[initramfs]] will result in the following:<br />
# mkinitcpio -p linux<br />
==> Building image from preset: 'default'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img<br />
file: could not find any magic files!<br />
==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
==> Building image from preset: 'fallback'<br />
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect<br />
file: could not find any magic files!<br />
@==> ERROR: invalid kernel specifier: `/boot/vmlinuz-linux'<br />
<br />
Typically a previously installed application had placed a configuration file within {{ic|/etc/ld.so.conf.d/}} or it had made changes to {{ic|/etc/ld.so.conf}} which are now invalid.<br />
#Boot into the Arch Linux Live CD / Installation Media.<br />
#Mount your root ({{ic|'''/'''}}) partition to {{ic|/mnt}} and using [[Change root#Change_root|arch-chroot]]{{Broken section link}}, [[chroot]] into your system.<br />
{{Note|[[Change root#Change_root|arch-chroot]]{{Broken section link}} leaves mounting the {{ic|/boot}} partition up to the user.}}<br />
#Examine {{ic|/etc/ld.so.conf}} and remove any invalid lines found.<br />
#Examine the files located inside the directory {{ic|/etc/ld.so.conf.d/}} and remove all invalid files.<br />
#Rebuild the [[initramfs]].<br />
# mkinitcpio -p linux<br />
#Reboot back to your installed system.<br />
#Once booted, reinstall the package that was responsible for leaving your system inoperable using:<br />
# pacman -S <package><br />
<br />
== See also ==<br />
<br />
* [http://www.tuxradar.com/content/how-fix-most-common-linux-problems Fix the Most Common Problems]<br />
* [https://www.reddit.com/r/archlinux/comments/tjjwr/archlinux_a_howto_in_troubleshooting_for_newcomers/ A how-to in troubleshooting for newcomers]</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Talk:Kernel_Panics&diff=493459Talk:Kernel Panics2017-10-17T22:27:37Z<p>Mcnster: /* Intent to update Kernel Panics article */ Moved this article to its own section in "General troubleshooting".</p>
<hr />
<div>How do I make computer automaticaly restart after kernel panic? [[User:Obsrv|Obsrv]] 22:53, 30 September 2008 (EDT)<br />
: http://www.google.com/search?hl=en&q=linux+restart+%22kernel+panic%22 --[[User:Byte|byte]] 20:28, 2 October 2008 (EDT)<br />
<br />
== Login Wrong ==<br />
<br />
If one uses arch to login, the response to the mount command is "mount: only root can do that". You might want to try root instead. - [[User:KitchM|KitchM]] 00:14, 22 October 2009 (EDT)<br />
<br />
== Kernel ID ==<br />
<br />
Need to explain how to identify the last kernel used. - [[User:KitchM|KitchM]] 00:38, 22 October 2009 (EDT)<br />
<br />
== "Recovery Console" evolved ==<br />
<br />
It would be nice to add that as a sort of countermeasure there is the possibility of add init=/bin/sh to the grub's kernel line in order to quickly access the battlefield and solve some things without a rescue cd.<br />
<br />
== Chroot Instructions ==<br />
<br />
I'm having some troubles with a possible kernel panic. I found this page disagrees with the actual [[Chroot]] page. So while I procrastinate on my own problem and attempt to take a breather before all my hair falls out, I thought I would make this suggestion. Either one of these pages should be updated to agree with the other. The [[Chroot]] article has a newer edit date so I tend to think it should be the standard.<br />
;[[Chroot]] article reads:<br />
<pre><br />
mount -t proc proc proc/<br />
mount -t sysfs sys sys/<br />
mount -o bind /dev dev/<br />
</pre>(user has already "cd /mnt/arch" hence no leading /mnt/)<br />
;[[Kernel Panics]] article reads:<br />
<pre><br />
# mount -t proc none /mnt/proc<br />
# mount -t sysfs none /mnt/sys<br />
# mount --bind /dev /mnt/dev"<br />
</pre><br />
<br />
Perhaps the end result is equal or insignificant but it is a bit confusing for someone who relies on these articles, e.g. me.<br />
<br />
== Blacklisting modules ==<br />
Link to [[Kernel_modules#Using_kernel_command_line]]? -- [[User:Kynikos|Kynikos]] 19:31, 9 June 2011 (EDT)<br />
<br />
== Intent to update Kernel Panics article ==<br />
<br />
Hi. I would like to take a stab at updating this article with consideration given to the above comments. My approach would be to write the article here under [[Kernel Panics]] with the final goal of integrating it into [[General troubleshooting]] once it passes muster.<br />
21:01, 3 October 2017 (UTC)<br />
<br />
:Feel free to proceed, though since this will be a full rewrite I'd like to remind that [[ArchWiki:Contributing#The_3_fundamental_rules]] should be followed. #3 already accomplished ;) -- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 21:13, 3 October 2017 (UTC)<br />
<br />
::Got it: use edit summary, and make many small edits rather than few large ones so its easier to check. Okay. 21:40, 3 October 2017 (UTC)<br />
<br />
::I have completed my revison of this article and will post it shortly. I have change the focus somewhat, from general procedures like re-installing a kernel and rolling back faulty software to info specific to kernel panics--what they are, what causes them, how to read an oops crash dump, and how to get back "into" the machine after a panic happens. This seems to make sense in the context of the proposal to merge this article into [[General troubleshooting]]. I think it would fit well as a 2nd level section immediately following [[General troubleshooting#Boot problems]]. [[User:Mcnster|Mcnster]] ([[User talk:Mcnster|talk]]) 20:26, 8 October 2017 (UTC)<br />
<br />
::Its been 9 days since I posted my edits. I'll assume silence equals a consensus of "That's fine. Proceed." So I am moving this article into a 2nd level section immediately following [[General troubleshooting#Boot problems]], as directed in the Merge template. 22:26, 17 October 2017 (UTC)</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster&diff=492749User:Mcnster2017-10-08T20:44:46Z<p>Mcnster: /* Pages I've worked on */ Fixed formatting.</p>
<hr />
<div>"Have keyboard, will edit."<br />
<br />
== Pages I'm working on ==<br />
<br />
== Pages I've worked on ==<br />
<br />
* [[User:Mcnster/Kernel Panics]] 20:43, 8 October 2017 (UTC)<br />
* [[User:Mcnster/EVE Online]] 16:29, 20 September 2017 (UTC)</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster&diff=492747User:Mcnster2017-10-08T20:43:58Z<p>Mcnster: /* Pages I've worked on */ Finished kernel panics.</p>
<hr />
<div>"Have keyboard, will edit."<br />
<br />
== Pages I'm working on ==<br />
<br />
== Pages I've worked on ==<br />
<br />
[[User:Mcnster/Kernel Panics]] [[User:Mcnster|Mcnster]] ([[User talk:Mcnster|talk]]) 20:43, 8 October 2017 (UTC)<br />
[[User:Mcnster/EVE Online]] 16:29, 20 September 2017 (UTC)</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster&diff=492746User:Mcnster2017-10-08T20:43:18Z<p>Mcnster: /* Pages I'm working on */ Moved Kernel Panics to done.</p>
<hr />
<div>"Have keyboard, will edit."<br />
<br />
== Pages I'm working on ==<br />
<br />
== Pages I've worked on ==<br />
[[User:Mcnster/EVE Online]] 16:29, 20 September 2017 (UTC)</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492744Kernel Panics2017-10-08T20:41:35Z<p>Mcnster: Removed 'out of date' flag in header.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492743Kernel Panics2017-10-08T20:40:33Z<p>Mcnster: Added new 2nd level section "Reboot into root shell and fix problem". That's all the changes. Done for the day.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492742Kernel Panics2017-10-08T20:38:58Z<p>Mcnster: Added new 3rd level section "Example scenario: bad module".</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492740Kernel Panics2017-10-08T20:37:14Z<p>Mcnster: Added 2nd level section 'Examine panic message'.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492739Kernel Panics2017-10-08T20:35:48Z<p>Mcnster: /* Reboot */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492738Kernel Panics2017-10-08T20:35:29Z<p>Mcnster: /* Option 2: Reinstall kernel */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492736Kernel Panics2017-10-08T20:35:05Z<p>Mcnster: /* Roll back to previous kernel version */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492735Kernel Panics2017-10-08T20:34:49Z<p>Mcnster: /* Chroot to your normal root */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
===Roll back to previous kernel version===<br />
If you keep your downloaded pacman packages, you now can easily roll back. If you did not keep them, you have to [[Downgrading packages#Finding_your_older_version|find a way]]{{Broken section link}} to get a previous kernel version on your system now.<br />
<br />
Let us suppose you kept the previous versions. We will now install the last working one.<br />
<br />
First, you need to get the kernel details:<br />
# find /var/cache/pacman/pkg -name 'linux-4*'<br />
<br />
Now, use the kernel details in the command below.<br />
<br />
# pacman -U /var/cache/pacman/pkg/linux-4.''xx-x''.pkg.tar.xz<br />
(Of course, make sure that you adapt this line to your own kernel version. You can find the ones you still have in your cache by examining the directory above.)<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492734Kernel Panics2017-10-08T20:34:32Z<p>Mcnster: /* Gather your files for later troubleshooting */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
===Chroot to your normal root===<br />
Now, you will have to chroot to the partition mounted in {{ic|/mnt}}. Newer kernels use an initial ramdisk to set up the kernel environment: when you reinstall a kernel, that initial ramdisk will be regenerated with [[mkinitcpio]]. One of mkinitcpio's features is that it does automatic detection to find out what kernel modules are required for starting up your computer. For this autodetection to work, {{ic|/dev}}, {{ic|/sys}}, and {{ic|/proc}} need to mounted in your chroot; make sure to read [[Change root]].<br />
<br />
To chroot to your normal root mounted at {{ic|/mnt}}, run this command:<br />
# arch-chroot /mnt /bin/bash<br />
<br />
If you do not want to use the [[Bash]] shell, remove {{ic|/bin/bash}} from the {{ic|arch-chroot}} command.<br />
<br />
{{Note|You need the {{Pkg|arch-install-scripts}} package in order to use {{ic|arch-chroot}}.}}<br />
<br />
===Roll back to previous kernel version===<br />
If you keep your downloaded pacman packages, you now can easily roll back. If you did not keep them, you have to [[Downgrading packages#Finding_your_older_version|find a way]]{{Broken section link}} to get a previous kernel version on your system now.<br />
<br />
Let us suppose you kept the previous versions. We will now install the last working one.<br />
<br />
First, you need to get the kernel details:<br />
# find /var/cache/pacman/pkg -name 'linux-4*'<br />
<br />
Now, use the kernel details in the command below.<br />
<br />
# pacman -U /var/cache/pacman/pkg/linux-4.''xx-x''.pkg.tar.xz<br />
(Of course, make sure that you adapt this line to your own kernel version. You can find the ones you still have in your cache by examining the directory above.)<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492733Kernel Panics2017-10-08T20:34:13Z<p>Mcnster: /* Mount your partitions */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
===Gather your files for later troubleshooting===<br />
This is a good point to stop and gather your information onto another drive or partition so that it can be analyzed and/or emailed for outside viewing before the files change again. Simply create a separate directory on your main partition or mount a USB drive to contain the files. Then you may copy any files you will need to keep unchanged during the next boot with your new kernel.<br />
<br />
===Chroot to your normal root===<br />
Now, you will have to chroot to the partition mounted in {{ic|/mnt}}. Newer kernels use an initial ramdisk to set up the kernel environment: when you reinstall a kernel, that initial ramdisk will be regenerated with [[mkinitcpio]]. One of mkinitcpio's features is that it does automatic detection to find out what kernel modules are required for starting up your computer. For this autodetection to work, {{ic|/dev}}, {{ic|/sys}}, and {{ic|/proc}} need to mounted in your chroot; make sure to read [[Change root]].<br />
<br />
To chroot to your normal root mounted at {{ic|/mnt}}, run this command:<br />
# arch-chroot /mnt /bin/bash<br />
<br />
If you do not want to use the [[Bash]] shell, remove {{ic|/bin/bash}} from the {{ic|arch-chroot}} command.<br />
<br />
{{Note|You need the {{Pkg|arch-install-scripts}} package in order to use {{ic|arch-chroot}}.}}<br />
<br />
===Roll back to previous kernel version===<br />
If you keep your downloaded pacman packages, you now can easily roll back. If you did not keep them, you have to [[Downgrading packages#Finding_your_older_version|find a way]]{{Broken section link}} to get a previous kernel version on your system now.<br />
<br />
Let us suppose you kept the previous versions. We will now install the last working one.<br />
<br />
First, you need to get the kernel details:<br />
# find /var/cache/pacman/pkg -name 'linux-4*'<br />
<br />
Now, use the kernel details in the command below.<br />
<br />
# pacman -U /var/cache/pacman/pkg/linux-4.''xx-x''.pkg.tar.xz<br />
(Of course, make sure that you adapt this line to your own kernel version. You can find the ones you still have in your cache by examining the directory above.)<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492732Kernel Panics2017-10-08T20:33:53Z<p>Mcnster: /* Start from the installation CD */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
===Mount your partitions===<br />
When booted, you are in a minimal but functional live GNU/Linux environment with some basic tools.<br />
Now, you have to mount your normal root disk (or partition) to {{ic|/mnt}}.<br />
# mount /dev/sdXY /mnt<br />
<br />
If you are using legacy IDE drives, then use the command:<br />
# mount /dev/hdXY /mnt<br />
<br />
If you use a separate boot partition, do not forget to mount it with:<br />
# mount /dev/sdXZ /mnt/boot<br />
<br />
===Gather your files for later troubleshooting===<br />
This is a good point to stop and gather your information onto another drive or partition so that it can be analyzed and/or emailed for outside viewing before the files change again. Simply create a separate directory on your main partition or mount a USB drive to contain the files. Then you may copy any files you will need to keep unchanged during the next boot with your new kernel.<br />
<br />
===Chroot to your normal root===<br />
Now, you will have to chroot to the partition mounted in {{ic|/mnt}}. Newer kernels use an initial ramdisk to set up the kernel environment: when you reinstall a kernel, that initial ramdisk will be regenerated with [[mkinitcpio]]. One of mkinitcpio's features is that it does automatic detection to find out what kernel modules are required for starting up your computer. For this autodetection to work, {{ic|/dev}}, {{ic|/sys}}, and {{ic|/proc}} need to mounted in your chroot; make sure to read [[Change root]].<br />
<br />
To chroot to your normal root mounted at {{ic|/mnt}}, run this command:<br />
# arch-chroot /mnt /bin/bash<br />
<br />
If you do not want to use the [[Bash]] shell, remove {{ic|/bin/bash}} from the {{ic|arch-chroot}} command.<br />
<br />
{{Note|You need the {{Pkg|arch-install-scripts}} package in order to use {{ic|arch-chroot}}.}}<br />
<br />
===Roll back to previous kernel version===<br />
If you keep your downloaded pacman packages, you now can easily roll back. If you did not keep them, you have to [[Downgrading packages#Finding_your_older_version|find a way]]{{Broken section link}} to get a previous kernel version on your system now.<br />
<br />
Let us suppose you kept the previous versions. We will now install the last working one.<br />
<br />
First, you need to get the kernel details:<br />
# find /var/cache/pacman/pkg -name 'linux-4*'<br />
<br />
Now, use the kernel details in the command below.<br />
<br />
# pacman -U /var/cache/pacman/pkg/linux-4.''xx-x''.pkg.tar.xz<br />
(Of course, make sure that you adapt this line to your own kernel version. You can find the ones you still have in your cache by examining the directory above.)<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492731Kernel Panics2017-10-08T20:33:24Z<p>Mcnster: /* Option 1: Check bootloader configuration */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
===Start from the installation CD===<br />
The first step is booting the installation CD. Once booted, you are presented with an automatically logged-in virtual console as the ''root'' user.<br />
<br />
===Mount your partitions===<br />
When booted, you are in a minimal but functional live GNU/Linux environment with some basic tools.<br />
Now, you have to mount your normal root disk (or partition) to {{ic|/mnt}}.<br />
# mount /dev/sdXY /mnt<br />
<br />
If you are using legacy IDE drives, then use the command:<br />
# mount /dev/hdXY /mnt<br />
<br />
If you use a separate boot partition, do not forget to mount it with:<br />
# mount /dev/sdXZ /mnt/boot<br />
<br />
===Gather your files for later troubleshooting===<br />
This is a good point to stop and gather your information onto another drive or partition so that it can be analyzed and/or emailed for outside viewing before the files change again. Simply create a separate directory on your main partition or mount a USB drive to contain the files. Then you may copy any files you will need to keep unchanged during the next boot with your new kernel.<br />
<br />
===Chroot to your normal root===<br />
Now, you will have to chroot to the partition mounted in {{ic|/mnt}}. Newer kernels use an initial ramdisk to set up the kernel environment: when you reinstall a kernel, that initial ramdisk will be regenerated with [[mkinitcpio]]. One of mkinitcpio's features is that it does automatic detection to find out what kernel modules are required for starting up your computer. For this autodetection to work, {{ic|/dev}}, {{ic|/sys}}, and {{ic|/proc}} need to mounted in your chroot; make sure to read [[Change root]].<br />
<br />
To chroot to your normal root mounted at {{ic|/mnt}}, run this command:<br />
# arch-chroot /mnt /bin/bash<br />
<br />
If you do not want to use the [[Bash]] shell, remove {{ic|/bin/bash}} from the {{ic|arch-chroot}} command.<br />
<br />
{{Note|You need the {{Pkg|arch-install-scripts}} package in order to use {{ic|arch-chroot}}.}}<br />
<br />
===Roll back to previous kernel version===<br />
If you keep your downloaded pacman packages, you now can easily roll back. If you did not keep them, you have to [[Downgrading packages#Finding_your_older_version|find a way]]{{Broken section link}} to get a previous kernel version on your system now.<br />
<br />
Let us suppose you kept the previous versions. We will now install the last working one.<br />
<br />
First, you need to get the kernel details:<br />
# find /var/cache/pacman/pkg -name 'linux-4*'<br />
<br />
Now, use the kernel details in the command below.<br />
<br />
# pacman -U /var/cache/pacman/pkg/linux-4.''xx-x''.pkg.tar.xz<br />
(Of course, make sure that you adapt this line to your own kernel version. You can find the ones you still have in your cache by examining the directory above.)<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492730Kernel Panics2017-10-08T20:33:00Z<p>Mcnster: /* Troubleshooting */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Option 1: Check bootloader configuration==<br />
Another possibility is an error in the [[Boot Loader#Configuration files|bootloader's configuration]]{{Broken section link}}. For example, repartitioning hard drives can change partitions' order. GRUB users may recall whether repartitioning has occurred recently and make sure the ''root'' and ''kernel'' lines match up with the new partitioning scheme. And examine the file for typos and extraneous characters. An extra space, or a character in the wrong place will cause a kernel panic.<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
===Start from the installation CD===<br />
The first step is booting the installation CD. Once booted, you are presented with an automatically logged-in virtual console as the ''root'' user.<br />
<br />
===Mount your partitions===<br />
When booted, you are in a minimal but functional live GNU/Linux environment with some basic tools.<br />
Now, you have to mount your normal root disk (or partition) to {{ic|/mnt}}.<br />
# mount /dev/sdXY /mnt<br />
<br />
If you are using legacy IDE drives, then use the command:<br />
# mount /dev/hdXY /mnt<br />
<br />
If you use a separate boot partition, do not forget to mount it with:<br />
# mount /dev/sdXZ /mnt/boot<br />
<br />
===Gather your files for later troubleshooting===<br />
This is a good point to stop and gather your information onto another drive or partition so that it can be analyzed and/or emailed for outside viewing before the files change again. Simply create a separate directory on your main partition or mount a USB drive to contain the files. Then you may copy any files you will need to keep unchanged during the next boot with your new kernel.<br />
<br />
===Chroot to your normal root===<br />
Now, you will have to chroot to the partition mounted in {{ic|/mnt}}. Newer kernels use an initial ramdisk to set up the kernel environment: when you reinstall a kernel, that initial ramdisk will be regenerated with [[mkinitcpio]]. One of mkinitcpio's features is that it does automatic detection to find out what kernel modules are required for starting up your computer. For this autodetection to work, {{ic|/dev}}, {{ic|/sys}}, and {{ic|/proc}} need to mounted in your chroot; make sure to read [[Change root]].<br />
<br />
To chroot to your normal root mounted at {{ic|/mnt}}, run this command:<br />
# arch-chroot /mnt /bin/bash<br />
<br />
If you do not want to use the [[Bash]] shell, remove {{ic|/bin/bash}} from the {{ic|arch-chroot}} command.<br />
<br />
{{Note|You need the {{Pkg|arch-install-scripts}} package in order to use {{ic|arch-chroot}}.}}<br />
<br />
===Roll back to previous kernel version===<br />
If you keep your downloaded pacman packages, you now can easily roll back. If you did not keep them, you have to [[Downgrading packages#Finding_your_older_version|find a way]]{{Broken section link}} to get a previous kernel version on your system now.<br />
<br />
Let us suppose you kept the previous versions. We will now install the last working one.<br />
<br />
First, you need to get the kernel details:<br />
# find /var/cache/pacman/pkg -name 'linux-4*'<br />
<br />
Now, use the kernel details in the command below.<br />
<br />
# pacman -U /var/cache/pacman/pkg/linux-4.''xx-x''.pkg.tar.xz<br />
(Of course, make sure that you adapt this line to your own kernel version. You can find the ones you still have in your cache by examining the directory above.)<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492729Kernel Panics2017-10-08T20:32:35Z<p>Mcnster: /* What to do */ Deleted section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Troubleshooting==<br />
To make troubleshooting easier, ensure that the kernel is not in quiet mode. Remove 'quiet' from the kernel line in GRUB, if it is found there. Upon boot, check the output immediately before the panic, and decide whether there is any useful information. There are probably too many causes for a kernel panic to keep well-documented in this wiki. Make sure that your system's configuration in /boot is correct, and that none of the computer's hardware is faulty - it is good idea to run memtest from the Arch install/rescue CD or another utility (red entries are bad). If you believe the configuration in /boot may be erroneous, try Option 1 to repair your bootloader setup. If you believe the kernel panic is the fault of the kernel itself, follow Option 2 in order to reinstall the existing version or an earlier kernel.<br />
<br />
==Option 1: Check bootloader configuration==<br />
Another possibility is an error in the [[Boot Loader#Configuration files|bootloader's configuration]]{{Broken section link}}. For example, repartitioning hard drives can change partitions' order. GRUB users may recall whether repartitioning has occurred recently and make sure the ''root'' and ''kernel'' lines match up with the new partitioning scheme. And examine the file for typos and extraneous characters. An extra space, or a character in the wrong place will cause a kernel panic.<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
===Start from the installation CD===<br />
The first step is booting the installation CD. Once booted, you are presented with an automatically logged-in virtual console as the ''root'' user.<br />
<br />
===Mount your partitions===<br />
When booted, you are in a minimal but functional live GNU/Linux environment with some basic tools.<br />
Now, you have to mount your normal root disk (or partition) to {{ic|/mnt}}.<br />
# mount /dev/sdXY /mnt<br />
<br />
If you are using legacy IDE drives, then use the command:<br />
# mount /dev/hdXY /mnt<br />
<br />
If you use a separate boot partition, do not forget to mount it with:<br />
# mount /dev/sdXZ /mnt/boot<br />
<br />
===Gather your files for later troubleshooting===<br />
This is a good point to stop and gather your information onto another drive or partition so that it can be analyzed and/or emailed for outside viewing before the files change again. Simply create a separate directory on your main partition or mount a USB drive to contain the files. Then you may copy any files you will need to keep unchanged during the next boot with your new kernel.<br />
<br />
===Chroot to your normal root===<br />
Now, you will have to chroot to the partition mounted in {{ic|/mnt}}. Newer kernels use an initial ramdisk to set up the kernel environment: when you reinstall a kernel, that initial ramdisk will be regenerated with [[mkinitcpio]]. One of mkinitcpio's features is that it does automatic detection to find out what kernel modules are required for starting up your computer. For this autodetection to work, {{ic|/dev}}, {{ic|/sys}}, and {{ic|/proc}} need to mounted in your chroot; make sure to read [[Change root]].<br />
<br />
To chroot to your normal root mounted at {{ic|/mnt}}, run this command:<br />
# arch-chroot /mnt /bin/bash<br />
<br />
If you do not want to use the [[Bash]] shell, remove {{ic|/bin/bash}} from the {{ic|arch-chroot}} command.<br />
<br />
{{Note|You need the {{Pkg|arch-install-scripts}} package in order to use {{ic|arch-chroot}}.}}<br />
<br />
===Roll back to previous kernel version===<br />
If you keep your downloaded pacman packages, you now can easily roll back. If you did not keep them, you have to [[Downgrading packages#Finding_your_older_version|find a way]]{{Broken section link}} to get a previous kernel version on your system now.<br />
<br />
Let us suppose you kept the previous versions. We will now install the last working one.<br />
<br />
First, you need to get the kernel details:<br />
# find /var/cache/pacman/pkg -name 'linux-4*'<br />
<br />
Now, use the kernel details in the command below.<br />
<br />
# pacman -U /var/cache/pacman/pkg/linux-4.''xx-x''.pkg.tar.xz<br />
(Of course, make sure that you adapt this line to your own kernel version. You can find the ones you still have in your cache by examining the directory above.)<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492728Kernel Panics2017-10-08T20:32:15Z<p>Mcnster: /* Definition */ Deleted section (really).</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==What to do==<br />
Basically, the problem is that the operating system doesn't start correctly. Various behavior may be expressed, such as that one may get the computer to freeze, or the operating system may give an error message of some sort or one may not go to the place they were expecting (Command prompt, Desktop or whathaveyou). This will require some basic troubleshooting from the command line, if you can boot to it, or from a boot disk if it will get you a command prompt or your favorite interface.<br />
<br />
==Troubleshooting==<br />
To make troubleshooting easier, ensure that the kernel is not in quiet mode. Remove 'quiet' from the kernel line in GRUB, if it is found there. Upon boot, check the output immediately before the panic, and decide whether there is any useful information. There are probably too many causes for a kernel panic to keep well-documented in this wiki. Make sure that your system's configuration in /boot is correct, and that none of the computer's hardware is faulty - it is good idea to run memtest from the Arch install/rescue CD or another utility (red entries are bad). If you believe the configuration in /boot may be erroneous, try Option 1 to repair your bootloader setup. If you believe the kernel panic is the fault of the kernel itself, follow Option 2 in order to reinstall the existing version or an earlier kernel.<br />
<br />
==Option 1: Check bootloader configuration==<br />
Another possibility is an error in the [[Boot Loader#Configuration files|bootloader's configuration]]{{Broken section link}}. For example, repartitioning hard drives can change partitions' order. GRUB users may recall whether repartitioning has occurred recently and make sure the ''root'' and ''kernel'' lines match up with the new partitioning scheme. And examine the file for typos and extraneous characters. An extra space, or a character in the wrong place will cause a kernel panic.<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
===Start from the installation CD===<br />
The first step is booting the installation CD. Once booted, you are presented with an automatically logged-in virtual console as the ''root'' user.<br />
<br />
===Mount your partitions===<br />
When booted, you are in a minimal but functional live GNU/Linux environment with some basic tools.<br />
Now, you have to mount your normal root disk (or partition) to {{ic|/mnt}}.<br />
# mount /dev/sdXY /mnt<br />
<br />
If you are using legacy IDE drives, then use the command:<br />
# mount /dev/hdXY /mnt<br />
<br />
If you use a separate boot partition, do not forget to mount it with:<br />
# mount /dev/sdXZ /mnt/boot<br />
<br />
===Gather your files for later troubleshooting===<br />
This is a good point to stop and gather your information onto another drive or partition so that it can be analyzed and/or emailed for outside viewing before the files change again. Simply create a separate directory on your main partition or mount a USB drive to contain the files. Then you may copy any files you will need to keep unchanged during the next boot with your new kernel.<br />
<br />
===Chroot to your normal root===<br />
Now, you will have to chroot to the partition mounted in {{ic|/mnt}}. Newer kernels use an initial ramdisk to set up the kernel environment: when you reinstall a kernel, that initial ramdisk will be regenerated with [[mkinitcpio]]. One of mkinitcpio's features is that it does automatic detection to find out what kernel modules are required for starting up your computer. For this autodetection to work, {{ic|/dev}}, {{ic|/sys}}, and {{ic|/proc}} need to mounted in your chroot; make sure to read [[Change root]].<br />
<br />
To chroot to your normal root mounted at {{ic|/mnt}}, run this command:<br />
# arch-chroot /mnt /bin/bash<br />
<br />
If you do not want to use the [[Bash]] shell, remove {{ic|/bin/bash}} from the {{ic|arch-chroot}} command.<br />
<br />
{{Note|You need the {{Pkg|arch-install-scripts}} package in order to use {{ic|arch-chroot}}.}}<br />
<br />
===Roll back to previous kernel version===<br />
If you keep your downloaded pacman packages, you now can easily roll back. If you did not keep them, you have to [[Downgrading packages#Finding_your_older_version|find a way]]{{Broken section link}} to get a previous kernel version on your system now.<br />
<br />
Let us suppose you kept the previous versions. We will now install the last working one.<br />
<br />
First, you need to get the kernel details:<br />
# find /var/cache/pacman/pkg -name 'linux-4*'<br />
<br />
Now, use the kernel details in the command below.<br />
<br />
# pacman -U /var/cache/pacman/pkg/linux-4.''xx-x''.pkg.tar.xz<br />
(Of course, make sure that you adapt this line to your own kernel version. You can find the ones you still have in your cache by examining the directory above.)<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Kernel_Panics&diff=492727Kernel Panics2017-10-08T20:31:30Z<p>Mcnster: New introductory section.</p>
<hr />
<div>[[Category:System recovery]]<br />
[[Category:Kernel]]<br />
[[cs:Kernel Panics]]<br />
[[el:Kernel Panics]]<br />
[[es:Kernel Panics]]<br />
[[fr:Kernel Panics]]<br />
[[it:Kernel Panics]]<br />
[[ja:カーネルパニック]]<br />
[[ro:Kernel panics]]<br />
[[tr:Çekirdek Sorunları]]<br />
[[zh-hans:Kernel Panics]]<br />
{{out of date|Last '''major''' update to this page was November 2009.}}<br />
{{Merge|General troubleshooting|If you remove all the excess verbosity and duplicate instructions, a few paragraphs remain which can go to [[General troubleshooting]]}}<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
==Definition==<br />
A decent definition of Kernel Panic comes to us from Wikipedia, which states in part; "A kernel panic is an action taken by an operating system upon detecting an internal fatal error from which it cannot safely recover; the term is largely specific to Unix and Unix-like systems. The equivalent in Microsoft Windows operating systems is the Blue Screen of Death."<br />
<br />
See also [[Wikipedia:Kernel panic]].<br />
<br />
==What to do==<br />
Basically, the problem is that the operating system doesn't start correctly. Various behavior may be expressed, such as that one may get the computer to freeze, or the operating system may give an error message of some sort or one may not go to the place they were expecting (Command prompt, Desktop or whathaveyou). This will require some basic troubleshooting from the command line, if you can boot to it, or from a boot disk if it will get you a command prompt or your favorite interface.<br />
<br />
==Troubleshooting==<br />
To make troubleshooting easier, ensure that the kernel is not in quiet mode. Remove 'quiet' from the kernel line in GRUB, if it is found there. Upon boot, check the output immediately before the panic, and decide whether there is any useful information. There are probably too many causes for a kernel panic to keep well-documented in this wiki. Make sure that your system's configuration in /boot is correct, and that none of the computer's hardware is faulty - it is good idea to run memtest from the Arch install/rescue CD or another utility (red entries are bad). If you believe the configuration in /boot may be erroneous, try Option 1 to repair your bootloader setup. If you believe the kernel panic is the fault of the kernel itself, follow Option 2 in order to reinstall the existing version or an earlier kernel.<br />
<br />
==Option 1: Check bootloader configuration==<br />
Another possibility is an error in the [[Boot Loader#Configuration files|bootloader's configuration]]{{Broken section link}}. For example, repartitioning hard drives can change partitions' order. GRUB users may recall whether repartitioning has occurred recently and make sure the ''root'' and ''kernel'' lines match up with the new partitioning scheme. And examine the file for typos and extraneous characters. An extra space, or a character in the wrong place will cause a kernel panic.<br />
<br />
==Option 2: Reinstall kernel==<br />
Reinstalling the kernel is probably the best bet when no other major system modifications have taken place recently.<br />
<br />
===Start from the installation CD===<br />
The first step is booting the installation CD. Once booted, you are presented with an automatically logged-in virtual console as the ''root'' user.<br />
<br />
===Mount your partitions===<br />
When booted, you are in a minimal but functional live GNU/Linux environment with some basic tools.<br />
Now, you have to mount your normal root disk (or partition) to {{ic|/mnt}}.<br />
# mount /dev/sdXY /mnt<br />
<br />
If you are using legacy IDE drives, then use the command:<br />
# mount /dev/hdXY /mnt<br />
<br />
If you use a separate boot partition, do not forget to mount it with:<br />
# mount /dev/sdXZ /mnt/boot<br />
<br />
===Gather your files for later troubleshooting===<br />
This is a good point to stop and gather your information onto another drive or partition so that it can be analyzed and/or emailed for outside viewing before the files change again. Simply create a separate directory on your main partition or mount a USB drive to contain the files. Then you may copy any files you will need to keep unchanged during the next boot with your new kernel.<br />
<br />
===Chroot to your normal root===<br />
Now, you will have to chroot to the partition mounted in {{ic|/mnt}}. Newer kernels use an initial ramdisk to set up the kernel environment: when you reinstall a kernel, that initial ramdisk will be regenerated with [[mkinitcpio]]. One of mkinitcpio's features is that it does automatic detection to find out what kernel modules are required for starting up your computer. For this autodetection to work, {{ic|/dev}}, {{ic|/sys}}, and {{ic|/proc}} need to mounted in your chroot; make sure to read [[Change root]].<br />
<br />
To chroot to your normal root mounted at {{ic|/mnt}}, run this command:<br />
# arch-chroot /mnt /bin/bash<br />
<br />
If you do not want to use the [[Bash]] shell, remove {{ic|/bin/bash}} from the {{ic|arch-chroot}} command.<br />
<br />
{{Note|You need the {{Pkg|arch-install-scripts}} package in order to use {{ic|arch-chroot}}.}}<br />
<br />
===Roll back to previous kernel version===<br />
If you keep your downloaded pacman packages, you now can easily roll back. If you did not keep them, you have to [[Downgrading packages#Finding_your_older_version|find a way]]{{Broken section link}} to get a previous kernel version on your system now.<br />
<br />
Let us suppose you kept the previous versions. We will now install the last working one.<br />
<br />
First, you need to get the kernel details:<br />
# find /var/cache/pacman/pkg -name 'linux-4*'<br />
<br />
Now, use the kernel details in the command below.<br />
<br />
# pacman -U /var/cache/pacman/pkg/linux-4.''xx-x''.pkg.tar.xz<br />
(Of course, make sure that you adapt this line to your own kernel version. You can find the ones you still have in your cache by examining the directory above.)<br />
<br />
==Reboot==<br />
{{Note|If you choose to do anything else before you reboot, remember that you are still in the chroot environment and will likely have to exit and login again.}}<br />
Now is the time to reboot and see if the system modifications have stopped the panic.<br />
If reverting to an older kernel works, do not forget to [https://www.archlinux.org/news/ check the arch-newspage] to check what went wrong with the kernel build. If there is no mention of the problem there, then go to the bug reporting area and search for it there. If you still do not find it, open a new bug report and attach those files you saved during the troubleshooting step above.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=Talk:Kernel_Panics&diff=492726Talk:Kernel Panics2017-10-08T20:26:53Z<p>Mcnster: /* Intent to update Kernel Panics article */ Announcing completed revisions to article.</p>
<hr />
<div>How do I make computer automaticaly restart after kernel panic? [[User:Obsrv|Obsrv]] 22:53, 30 September 2008 (EDT)<br />
: http://www.google.com/search?hl=en&q=linux+restart+%22kernel+panic%22 --[[User:Byte|byte]] 20:28, 2 October 2008 (EDT)<br />
<br />
== Login Wrong ==<br />
<br />
If one uses arch to login, the response to the mount command is "mount: only root can do that". You might want to try root instead. - [[User:KitchM|KitchM]] 00:14, 22 October 2009 (EDT)<br />
<br />
== Kernel ID ==<br />
<br />
Need to explain how to identify the last kernel used. - [[User:KitchM|KitchM]] 00:38, 22 October 2009 (EDT)<br />
<br />
== "Recovery Console" evolved ==<br />
<br />
It would be nice to add that as a sort of countermeasure there is the possibility of add init=/bin/sh to the grub's kernel line in order to quickly access the battlefield and solve some things without a rescue cd.<br />
<br />
== Chroot Instructions ==<br />
<br />
I'm having some troubles with a possible kernel panic. I found this page disagrees with the actual [[Chroot]] page. So while I procrastinate on my own problem and attempt to take a breather before all my hair falls out, I thought I would make this suggestion. Either one of these pages should be updated to agree with the other. The [[Chroot]] article has a newer edit date so I tend to think it should be the standard.<br />
;[[Chroot]] article reads:<br />
<pre><br />
mount -t proc proc proc/<br />
mount -t sysfs sys sys/<br />
mount -o bind /dev dev/<br />
</pre>(user has already "cd /mnt/arch" hence no leading /mnt/)<br />
;[[Kernel Panics]] article reads:<br />
<pre><br />
# mount -t proc none /mnt/proc<br />
# mount -t sysfs none /mnt/sys<br />
# mount --bind /dev /mnt/dev"<br />
</pre><br />
<br />
Perhaps the end result is equal or insignificant but it is a bit confusing for someone who relies on these articles, e.g. me.<br />
<br />
== Blacklisting modules ==<br />
Link to [[Kernel_modules#Using_kernel_command_line]]? -- [[User:Kynikos|Kynikos]] 19:31, 9 June 2011 (EDT)<br />
<br />
== Intent to update Kernel Panics article ==<br />
<br />
Hi. I would like to take a stab at updating this article with consideration given to the above comments. My approach would be to write the article here under [[Kernel Panics]] with the final goal of integrating it into [[General troubleshooting]] once it passes muster.<br />
21:01, 3 October 2017 (UTC)<br />
<br />
:Feel free to proceed, though since this will be a full rewrite I'd like to remind that [[ArchWiki:Contributing#The_3_fundamental_rules]] should be followed. #3 already accomplished ;) -- [[User:Lahwaacz|Lahwaacz]] ([[User talk:Lahwaacz|talk]]) 21:13, 3 October 2017 (UTC)<br />
<br />
::Got it: use edit summary, and make many small edits rather than few large ones so its easier to check. Okay. 21:40, 3 October 2017 (UTC)<br />
<br />
::I have completed my revison of this article and will post it shortly. I have change the focus somewhat, from general procedures like re-installing a kernel and rolling back faulty software to info specific to kernel panics--what they are, what causes them, how to read an oops crash dump, and how to get back "into" the machine after a panic happens. This seems to make sense in the context of the proposal to merge this article into [[General troubleshooting]]. I think it would fit well as a 2nd level section immediately following [[General troubleshooting#Boot problems]]. [[User:Mcnster|Mcnster]] ([[User talk:Mcnster|talk]]) 20:26, 8 October 2017 (UTC)</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492722User:Mcnster/Kernel Panics2017-10-08T20:10:33Z<p>Mcnster: /* Reboot into root shell and fix problem */ Removed word.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492721User:Mcnster/Kernel Panics2017-10-08T20:08:29Z<p>Mcnster: /* Example scenario: bad module */ Case change.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492720User:Mcnster/Kernel Panics2017-10-08T20:07:14Z<p>Mcnster: /* Examine panic message */ Clarified.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic on tty1:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''Oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492719User:Mcnster/Kernel Panics2017-10-08T20:05:19Z<p>Mcnster: /* Kernel Panics */ Added 2 words.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both panics and oops occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''Oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492718User:Mcnster/Kernel Panics2017-10-08T20:00:43Z<p>Mcnster: /* Reboot into root shell and fix problem */ Expanded.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''Oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic, roll-back a faulty update, or fix a configuration problem.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492717User:Mcnster/Kernel Panics2017-10-08T19:56:11Z<p>Mcnster: /* Reboot into root shell and fix problem */ Rollback.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''Oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic or roll-back a faulty update.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492716User:Mcnster/Kernel Panics2017-10-08T19:54:26Z<p>Mcnster: /* Reboot into root shell and fix problem */ Grammar.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''Oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell before the machine deadlocks:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492715User:Mcnster/Kernel Panics2017-10-08T19:52:55Z<p>Mcnster: /* Reboot into root shell and fix problem */ Added sentence re intents.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''Oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
You'll need a root shell to make changes to the system so the panic no longer occurs. If the panic occurs on boot, there are several strategies to obtain a root shell prior to the panic occurring:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492714User:Mcnster/Kernel Panics2017-10-08T19:50:18Z<p>Mcnster: /* Examine panic message */ Grammar.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message scrolls away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''Oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
If the panic occurs on boot, there are several strategies to obtain a root shell prior to the panic occurring:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492713User:Mcnster/Kernel Panics2017-10-08T19:49:14Z<p>Mcnster: /* Examine panic message */ Word change.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message output by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message is scrolling away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''Oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
If the panic occurs on boot, there are several strategies to obtain a root shell prior to the panic occurring:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic.</div>Mcnsterhttps://wiki.archlinux.org/index.php?title=User:Mcnster/Kernel_Panics&diff=492712User:Mcnster/Kernel Panics2017-10-08T19:45:42Z<p>Mcnster: Deleted.</p>
<hr />
<div>= Kernel Panics =<br />
<br />
A ''kernel panic'' occurs when the Linux kernel enters an unrecoverable failure state. The state typically originates from buggy hardware drivers resulting in the machine being deadlocked, non-responsive, and requiring a reboot. Just prior to deadlock, a diagnostic message is generated, consisting of: the ''machine state'' when the failure ocurred, a ''call trace'' leading to the kernel function that recognized the failure, and a listing of currently loaded modules. Thankfully, kernel panics don't happen very often using ''mainline'' versions of the kernel--such as those supplied by the official repositories--but when they do happen, you need to know how to deal with them.<br />
<br />
{{Note|Kernel panics are sometimes referred to as ''oops'' or ''kernel oops''. While both occur as the result of a failure state, an ''oops'' is more general in that it does not ''necessarily'' result in a deadlocked machine--sometimes the kernel can recover from an oops by killing the offending task and carrying on.}}<br />
<br />
{{Tip|Pass the kernel parameter {{ic|1=oops=panic}} at boot or write {{ic|1}} to {{ic|/proc/sys/kernel/panic_on_oops}} to force a recoverable oops to issue a panic instead. This is advisable is you are concerned about the small chance of system instability resulting from an oops recovery which may make future errors difficult to diagnose.}}<br />
<br />
== Examine panic message ==<br />
<br />
If a kernel panic occurs very early in the boot process, you may see a message on the console containing "Kernel panic - not syncing:", but once [[Systemd]] is running, kernel messages will typically be captured and written to the system log. However, when a panic occurs, the diagnostic message written by the kernel is ''almost never'' written to the log file on disk because the machine deadlocks before {{ic|system-journald}} gets the chance. Therefore, the only way to examine the panic message is to view it on the console as it happens (without resorting to setting up a ''kdump crashkernel''). You can do this by booting with the following kernel parameters and attempting to reproduce the panic:<br />
<br />
{{bc|1=systemd.journald.forward_to_console=1 console=tty1}}<br />
<br />
{{Tip|In the event that the panic message is scrolling away too quickly to examine, try passing the kernel parameter {{ic|1=pause_on_oops=''seconds''}} at boot.}}<br />
<br />
=== Example scenario: bad module ===<br />
<br />
It is possible to make a best guess as to what subsystem or module is causing the panic using the information in the diagnostic message. In this scenario, we have a panic on some imaginary machine during boot. Pay attention to the lines highlighted in '''bold''':<br />
<br />
{{bc|'''kernel: BUG: unable to handle kernel NULL pointer dereference at (null)''' [1]<br />
'''kernel: IP: fw_core_init+0x18/0x1000 [firewire_core]''' [2]<br />
kernel: PGD 718d00067 <br />
kernel: P4D 718d00067 <br />
kernel: PUD 7b3611067 <br />
kernel: PMD 0 <br />
kernel: <br />
kernel: Oops: 0002 [#1] PREEMPT SMP<br />
'''kernel: Modules linked in: firewire_core(+) crc_itu_t cfg80211 rfkill ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 ...''' [3] <br />
kernel: CPU: 6 PID: 1438 Comm: modprobe Tainted: P O 4.13.3-1-ARCH #1<br />
kernel: Hardware name: Gigabyte Technology Co., Ltd. H97-D3H/H97-D3H-CF, BIOS F5 06/26/2014<br />
kernel: task: ffff9c667abd9e00 task.stack: ffffb53b8db34000<br />
kernel: RIP: 0010:fw_core_init+0x18/0x1000 [firewire_core]<br />
kernel: RSP: 0018:ffffb53b8db37c68 EFLAGS: 00010246<br />
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000<br />
kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffffc16d3af4<br />
kernel: RBP: ffffb53b8db37c70 R08: 0000000000000000 R09: ffffffffae113e95<br />
kernel: R10: ffffe93edfdb9680 R11: 0000000000000000 R12: ffffffffc16d9000<br />
kernel: R13: ffff9c6729bf8f60 R14: ffffffffc16d5710 R15: ffff9c6736e55840<br />
kernel: FS: 00007f301fc80b80(0000) GS:ffff9c675dd80000(0000) knlGS:0000000000000000<br />
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />
kernel: CR2: 0000000000000000 CR3: 00000007c6456000 CR4: 00000000001406e0<br />
kernel: Call Trace:<br />
'''kernel: do_one_initcall+0x50/0x190''' [4]<br />
kernel: ? do_init_module+0x27/0x1f2<br />
kernel: do_init_module+0x5f/0x1f2<br />
kernel: load_module+0x23f3/0x2be0<br />
kernel: SYSC_init_module+0x16b/0x1a0<br />
kernel: ? SYSC_init_module+0x16b/0x1a0<br />
kernel: SyS_init_module+0xe/0x10<br />
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5<br />
kernel: RIP: 0033:0x7f301f3a2a0a<br />
kernel: RSP: 002b:00007ffcabbd1998 EFLAGS: 00000246 ORIG_RAX: 00000000000000af<br />
kernel: RAX: ffffffffffffffda RBX: 0000000000c85a48 RCX: 00007f301f3a2a0a<br />
kernel: RDX: 000000000041aada RSI: 000000000001a738 RDI: 00007f301e7eb010<br />
kernel: RBP: 0000000000c8a520 R08: 0000000000000001 R09: 0000000000000085<br />
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000c79208<br />
kernel: R13: 0000000000c8b4d8 R14: 00007f301e7fffff R15: 0000000000000030<br />
kernel: Code: <c7> 04 25 00 00 00 00 01 00 00 00 bb f4 ff ff ff e8 73 43 9c ec 48 <br />
kernel: RIP: fw_core_init+0x18/0x1000 [firewire_core] RSP: ffffb53b8db37c68<br />
kernel: CR2: 0000000000000000<br />
kernel: ---[ end trace 71f4306ea1238f17 ]---<br />
'''kernel: Kernel panic - not syncing: Fatal exception''' [5]<br />
kernel: Kernel Offset: 0x80000000 from 0xffffffff810000000 (relocation range: 0xffffffff800000000-0xfffffffffbffffffff<br />
kernel: ---[ end Kernel panic - not syncing: Fatal exception}}<br />
<br />
* [1] Indicates the type of error that caused the panic. In this case it was a programmer bug.<br />
* [2] Indicates that the panic happened in a function called ''fw_core_init'' in module ''firewire_core''.<br />
* [3] Indicates that ''firewire_core'' was the latest module to be loaded.<br />
* [4] Indicates that the function that called function ''fw_core_init'' was ''do_one_initcall''.<br />
* [5] Indicates that this ''Oops'' message is, in fact, a kernel panic and the system is now deadlocked.<br />
<br />
We can surmise then, that the panic occurred during the initialization routine of module ''firewire_core'' as it was loaded. (We might assume then, that the machine's firewire hardware is incompatible with this version of the firewire driver module due to a programmer error, and will have to wait for a new release.) In the meantime, the easiest way to get the machine running again is to prevent the module from being loaded. We can do this in one of two ways:<br />
<br />
* If the module is being loaded during the execution of the ''initramfs'', reboot with the kernel parameter {{ic|1=rd.blacklist=firewire_core}}.<br />
* Otherwise reboot with the kernel parameter {{ic|1=module_blacklist=firewire_core}}.<br />
<br />
== Reboot into root shell and fix problem ==<br />
<br />
If the panic occurs on boot, there are several strategies to obtain a root shell prior to the panic occurring:<br />
<br />
* Reboot with the kernel parameter {{ic|emergency}}, {{ic|rd.emergency}}, or {{ic|-b}} to receive a prompt to login just after the root filesystem is mounted and {{ic|systemd}} is started.<br />
: {{Note|At this point, the root filesystem will be mounted '''read-only'''. Execute {{ic|# mount -o remount,rw /}} to make changes.}}<br />
* Reboot with the kernel parameter {{ic|rescue}}, {{ic|rd.rescue}}, {{ic|single}}, {{ic|s}}, {{ic|S}}, or {{ic|1}} to receive a prompt to login just after local filesystems are mounted.<br />
* Reboot with the kernel parameter {{ic|1=systemd.debug-shell=1}} to obtain a very early root shell on tty9. Switch to it with by pressing {{ic|Ctrl-Alt-F9}}.<br />
* Experiment by rebooting with different sets of kernel parameters to possibly disable the kernel feature that is causing the panic. Try the "old standbys" {{ic|1=acpi=off}} and {{ic|nolapic}}.<br />
: {{Tip|See {{ic|Documentation/admin-guide/kernel-parameters.txt}} in the Linux kernel source tree for all possible options.}}<br />
* As a last resort, boot with the '''Arch Linux Installation CD''' and mount the root filesystem on {{ic|/mnt}} then execute {{ic|# arch-chroot /mnt}}.<br />
<br />
Disable the service or program that is causing the panic.</div>Mcnster