Difference between revisions of "Systemd/Journal"

From ArchWiki
Jump to navigation Jump to search
(update interlanguage links)
Tag: wiki-scripts
m (add ES interlanguage link)
 
(14 intermediate revisions by 7 users not shown)
Line 2: Line 2:
 
[[Category:Daemons]]
 
[[Category:Daemons]]
 
[[Category:Logging]]
 
[[Category:Logging]]
[[pt:Systemd/Journal]]
+
[[es:Systemd (Español)/Journal]]
 +
[[ja:Systemd/ジャーナル]]
 +
[[pt:Systemd (Português)/Journal]]
 +
[[ru:Systemd (Русский)/Journal]]
 
[[zh-hans:Systemd/Journal]]
 
[[zh-hans:Systemd/Journal]]
See [[systemd]] for the main article.
 
 
 
''systemd'' has its own logging system called the journal; therefore, running a {{ic|syslog}} daemon is no longer required. To read the log, use:
 
''systemd'' has its own logging system called the journal; therefore, running a {{ic|syslog}} daemon is no longer required. To read the log, use:
  
 
  # journalctl
 
  # journalctl
  
In Arch Linux, the directory {{ic|/var/log/journal/}} is a part of the {{Pkg|systemd}} package, and the journal (when {{ic|1=Storage=}} is set to {{ic|auto}} in {{ic|/etc/systemd/journald.conf}}) will write to {{ic|/var/log/journal/}}.  If you or some program delete that directory, ''systemd'' will '''not''' recreate it automatically and instead will write its logs to {{ic|/run/systemd/journal}} in a nonpersistent way. However, the folder will be recreated when you set {{ic|1=Storage=persistent}} and [[restart]] {{ic|systemd-journald.service}} (or reboot).
+
In Arch Linux, the directory {{ic|/var/log/journal/}} is a part of the {{Pkg|systemd}} package, and the journal (when {{ic|1=Storage=}} is set to {{ic|auto}} in {{ic|/etc/systemd/journald.conf}}) will write to {{ic|/var/log/journal/}}.  If that directory is deleted, ''systemd'' will '''not''' recreate it automatically and instead will write its logs to {{ic|/run/systemd/journal}} in a nonpersistent way. However, the folder will be recreated if {{ic|1=Storage=persistent}} is added to {{ic|journald.conf}} and {{ic|systemd-journald.service}} is [[restart]]ed (or the system is rebooted).
  
 
Systemd journal classifies messages by [[#Priority level|Priority level]] and [[#Facility|Facility]]. Logging classification corresponds to classic [[wikipedia:Syslog|Syslog]] protocol ([https://tools.ietf.org/html/rfc5424 RFC 5424]).
 
Systemd journal classifies messages by [[#Priority level|Priority level]] and [[#Facility|Facility]]. Logging classification corresponds to classic [[wikipedia:Syslog|Syslog]] protocol ([https://tools.ietf.org/html/rfc5424 RFC 5424]).
  
==Priority level==
+
== Priority level ==
  
 
A syslog severity code (in systemd called priority) is used to mark the importance of a message [https://tools.ietf.org/html/rfc5424#section-6.2.1 RFC 5424 Section 6.2.1].
 
A syslog severity code (in systemd called priority) is used to mark the importance of a message [https://tools.ietf.org/html/rfc5424#section-6.2.1 RFC 5424 Section 6.2.1].
Line 22: Line 23:
 
! Value !! Severity !! Keyword  !! Description || Examples
 
! Value !! Severity !! Keyword  !! Description || Examples
 
|-
 
|-
| 0 || Emergency || emerg || System is unusable || Severe Kernel BUG, systemd dumped core.<br>This level should not be used by applications.
+
| 0 || Emergency || emerg || System is unusable || Severe Kernel BUG, [[Systemd-coredump|systemd dumped core]].<br>This level should not be used by applications.
 
|-
 
|-
 
| 1 || Alert || alert || Should be corrected immediately || Vital subsystem goes out of work. Data loss. <br>{{ic|kernel: BUG: unable to handle kernel paging request at ffffc90403238ffc|}}.
 
| 1 || Alert || alert || Should be corrected immediately || Vital subsystem goes out of work. Data loss. <br>{{ic|kernel: BUG: unable to handle kernel paging request at ffffc90403238ffc|}}.
Line 28: Line 29:
 
| 2 || Critical || crit || Critical conditions || Crashes, coredumps. Like familiar flash:<br>{{ic|systemd-coredump[25319]: Process 25310 (plugin-containe) of user 1000 dumped core}}<br>Failure in the system primary application, like X11.  
 
| 2 || Critical || crit || Critical conditions || Crashes, coredumps. Like familiar flash:<br>{{ic|systemd-coredump[25319]: Process 25310 (plugin-containe) of user 1000 dumped core}}<br>Failure in the system primary application, like X11.  
 
|-
 
|-
| 3 || Error || err || Error conditions || Not severe error reported:<br>{{ic|kernel: usb 1-3: 3:1: cannot get freq at ep 0x84}},<br>{{ic|systemd[1]: Failed unmounting /var.}},<br>{{ic|libvirtd[1720]: internal error: Failed to initialize a valid firewall backend}}).
+
| 3 || Error || err || Error conditions || Not severe error reported:<br>{{ic|kernel: usb 1-3: 3:1: cannot get freq at ep 0x84}},<br>{{ic|systemd[1]: Failed unmounting /var.}},<br>{{ic|libvirtd[1720]: internal error: Failed to initialize a valid firewall backend}}
 
|-
 
|-
 
| 4 || Warning || warning || May indicate that an error will occur if action is not taken. ||  A non-root file system has only 1GB free.<br>{{ic|org.freedesktop. Notifications[1860]: (process:5999): Gtk-WARNING **: Locale not supported by C library. Using the fallback 'C' locale}}.
 
| 4 || Warning || warning || May indicate that an error will occur if action is not taken. ||  A non-root file system has only 1GB free.<br>{{ic|org.freedesktop. Notifications[1860]: (process:5999): Gtk-WARNING **: Locale not supported by C library. Using the fallback 'C' locale}}.
 
|-
 
|-
| 5 || Notice || notice || Events that are unusual, but not error conditions. || {{ic|systemd[1]: var.mount: Directory /var to mount over is not empty, mounting anyway}}. {{ic|gcr-prompter[4997]: Gtk: GtkDialog mapped without a transient parent. This is discouraged}}.
+
| 5 || Notice || notice || Events that are unusual, but not error conditions. || {{ic|systemd[1]: var.mount: Directory /var to mount over is not empty, mounting anyway}},<br>{{ic|gcr-prompter[4997]: Gtk: GtkDialog mapped without a transient parent. This is discouraged}}
 
|-
 
|-
| 6 || Informational || info ||  Normal operational messages that require no action. || {{ic|lvm[585]:  7 logical volume(s) in volume group "archvg" now active}}.
+
| 6 || Informational || info ||  Normal operational messages that require no action. || {{ic|lvm[585]:  7 logical volume(s) in volume group "archvg" now active}}
 
|-
 
|-
| 7 || Debug || debug || Information useful to developers for debugging the application. || {{ic|kdeinit5[1900]: powerdevil: Scheduling inhibition from ":1.14" "firefox" with cookie 13 and reason "screen"}}.
+
| 7 || Debug || debug || Information useful to developers for debugging the application. || {{ic|kdeinit5[1900]: powerdevil: Scheduling inhibition from ":1.14" "firefox" with cookie 13 and reason "screen"}}
 
|}
 
|}
  
If you cannot find a message on the expected priority level, also search a couple of levels above and below: these rules are recommendations, and the developer of the affected application may have a different perception of the issue's importance from yours.
+
These rules are recommendations, and the priority level of a given error is at the application developer's discretion. It is always possible that the error will be at a higher or lower level than expected.
  
==Facility==
+
== Facility ==
  
 
A syslog facility code is used to specify the type of program that is logging the message [https://tools.ietf.org/html/rfc5424#section-6.2.1 RFC 5424 Section 6.2.1].
 
A syslog facility code is used to specify the type of program that is logging the message [https://tools.ietf.org/html/rfc5424#section-6.2.1 RFC 5424 Section 6.2.1].
Line 48: Line 49:
 
! Facility code !! Keyword !! Description !! Info
 
! Facility code !! Keyword !! Description !! Info
 
|-
 
|-
| 0 || kern || kernel messages
+
| 0 || kern || Kernel messages
 
|-
 
|-
| 1 || user || user-level messages
+
| 1 || user || User-level messages
 
|-
 
|-
| 2 || mail || mail system || Archaic POSIX still supported and sometimes used (for more {{man|1|mail}})
+
| 2 || mail || Mail system || Archaic POSIX still supported and sometimes used (for more {{man|1|mail}})
 
|-
 
|-
| 3 || daemon || system daemons || All daemons, including systemd and its subsystems
+
| 3 || daemon || System daemons || All daemons, including systemd and its subsystems
 
|-
 
|-
| 4 || auth || security/authorization messages || Also watch for different facility 10
+
| 4 || auth || Security/authorization messages || Also watch for different facility 10
 
|-
 
|-
| 5 || syslog || messages generated internally by syslogd || For syslogd implementations (not used by systemd, see facility 3)
+
| 5 || syslog || Messages generated internally by syslogd || For syslogd implementations (not used by systemd, see facility 3)
 
|-
 
|-
| 6 || lpr || line printer subsystem (archaic subsystem)
+
| 6 || lpr || Line printer subsystem (archaic subsystem)
 
|-
 
|-
| 7 || news || network news subsystem (archaic subsystem)
+
| 7 || news || Network news subsystem (archaic subsystem)
 
|-
 
|-
 
| 8 || uucp || UUCP subsystem (archaic subsystem)
 
| 8 || uucp || UUCP subsystem (archaic subsystem)
 
|-
 
|-
| 9 || || clock daemon || systemd-timesyncd
+
| 9 || || Clock daemon || systemd-timesyncd
 
|-
 
|-
| 10 || authpriv || security/authorization messages || Also watch for different facility 4
+
| 10 || authpriv || Security/authorization messages || Also watch for different facility 4
 
|-
 
|-
 
| 11 || ftp || FTP daemon
 
| 11 || ftp || FTP daemon
Line 74: Line 75:
 
| 12 || - || NTP subsystem
 
| 12 || - || NTP subsystem
 
|-
 
|-
| 13 || - || log audit
+
| 13 || - || Log audit
 
|-
 
|-
| 14 || - || log alert
+
| 14 || - || Log alert
 
|-
 
|-
| 15 || cron || scheduling daemon
+
| 15 || cron || Scheduling daemon
 
|-
 
|-
| 16 || local0 || local use 0  (local0)
+
| 16 || local0 || Local use 0  (local0)
 
|-
 
|-
| 17 || local1 || local use 1  (local1)
+
| 17 || local1 || Local use 1  (local1)
 
|-
 
|-
| 18 || local2 || local use 2  (local2)
+
| 18 || local2 || Local use 2  (local2)
 
|-
 
|-
| 19 || local3 || local use 3  (local3)
+
| 19 || local3 || Local use 3  (local3)
 
|-
 
|-
| 20 || local4 || local use 4  (local4)
+
| 20 || local4 || Local use 4  (local4)
 
|-
 
|-
| 21 || local5 || local use 5  (local5)
+
| 21 || local5 || Local use 5  (local5)
 
|-
 
|-
| 22 || local6 || local use 6  (local6)
+
| 22 || local6 || Local use 6  (local6)
 
|-
 
|-
| 23 || local7 || local use 7  (local7)
+
| 23 || local7 || Local use 7  (local7)
 
|}
 
|}
  
So, useful facilities to watch: 0,1,3,4,9,10,15.
+
Useful facilities to watch: 0, 1, 3, 4, 9, 10, 15.
  
 
== Filtering output ==
 
== Filtering output ==
  
''journalctl'' allows you to filter the output by specific fields. Be aware that if there are many messages to display or filtering of large time span has to be done, the output of this command can be delayed for quite some time.
+
''journalctl'' allows for the filtering of the output by specific fields. If there are many messages to display or filtering of large time span has to be done, the output of this command can be extensively delayed.
 
 
{{Tip|While the journal is stored in a binary format, the content of stored messages is not modified. This means it is viewable with ''strings'', for example for recovery in an environment which does not have ''systemd'' installed. Example command:
 
{{bc|$ strings /mnt/arch/var/log/journal/af4967d77fba44c6b093d0e9862f6ddd/system.journal <nowiki>| grep -i</nowiki> ''message''}}
 
}}
 
  
 
Examples:
 
Examples:
  
* Show all messages from this boot: {{bc|# journalctl -b}} However, often one is interested in messages not from the current, but from the previous boot (e.g. if an unrecoverable system crash happened). This is possible through optional offset parameter of the {{ic|-b}} flag: {{ic|journalctl -b -0}} shows messages from the current boot, {{ic|journalctl -b -1}} from the previous boot, {{ic|journalctl -b -2}} from the second previous and so on – you can see the list of boots with their numbers by using {{ic|journalctl --list-boots}}. See {{man|1|journalctl}} for full description, the semantics is much more powerful.
+
* Show all messages from this boot: {{bc|# journalctl -b}} However, often one is interested in messages not from the current, but from the previous boot (e.g. if an unrecoverable system crash happened). This is possible through optional offset parameter of the {{ic|-b}} flag: {{ic|journalctl -b -0}} shows messages from the current boot, {{ic|journalctl -b -1}} from the previous boot, {{ic|journalctl -b -2}} from the second previous and so on – you can see the list of boots with their numbers by using {{ic|journalctl --list-boots}}. See {{man|1|journalctl}} for a full description; the semantics are more powerful than indicated here.
 
* Show all messages from date (and optional time): {{bc|1=# journalctl --since="2012-10-30 18:17:16"}}
 
* Show all messages from date (and optional time): {{bc|1=# journalctl --since="2012-10-30 18:17:16"}}
 
* Show all messages since 20 minutes ago: {{bc|1=# journalctl --since "20 min ago"}}
 
* Show all messages since 20 minutes ago: {{bc|1=# journalctl --since "20 min ago"}}
Line 117: Line 114:
 
* Show all messages by a specific unit: {{bc|# journalctl -u man-db.service}}
 
* Show all messages by a specific unit: {{bc|# journalctl -u man-db.service}}
 
* Show kernel ring buffer: {{bc|1=# journalctl -k}}
 
* Show kernel ring buffer: {{bc|1=# journalctl -k}}
* Show only error, critical, and alert priority messages {{bc|# journalctl -p err..alert}} Numbers also can be used, {{ic|journalctl -p 3..1}}. If single number/keyword used, {{ic|journalctl -p 3}} - all higher priority levels also included.
+
* Show only error, critical and alert priority messages: {{bc|# journalctl -p err..alert}} You can use numeric log level too, like {{ic|journalctl -p 3..1}}. If single number/log level is used, {{ic|journalctl -p 3}}, then all higher priority log levels are also included (i.e. 0 to 3 in this case).
 
* Show auth.log equivalent by filtering on syslog facility: {{bc|1=# journalctl SYSLOG_FACILITY=10}}
 
* Show auth.log equivalent by filtering on syslog facility: {{bc|1=# journalctl SYSLOG_FACILITY=10}}
* If your journal directory (by default located under {{ic|/var/log/journal}}) contains huge amount of log data then {{ic|journalctl}} can take several minutes in filtering output. You can speed it up significantly by using {{ic|--file}} option to force {{ic|journalctl}} to look only into most recent journal: {{bc|# journalctl --file /var/log/journal/*/system.journal -f}}
+
* If the journal directory (by default located under {{ic|/var/log/journal}}) contains a large amount of log data then {{ic|journalctl}} can take several minutes to filter output. It can be sped up significantly by using {{ic|--file}} option to force {{ic|journalctl}} to look only into most recent journal: {{bc|# journalctl --file /var/log/journal/*/system.journal -f}}
  
See {{man|1|journalctl}}, {{man|7|systemd.journal-fields}}, or Lennart's [http://0pointer.de/blog/projects/journalctl.html blog post] for details.
+
See {{man|1|journalctl}}, {{man|7|systemd.journal-fields}}, or [http://0pointer.de/blog/projects/journalctl.html Lennart Poettering's blog post] for details.
  
 
{{Tip|1=
 
{{Tip|1=
Line 130: Line 127:
 
  $ SYSTEMD_LESS=FRXMK journalctl
 
  $ SYSTEMD_LESS=FRXMK journalctl
  
If you would like to set this behaviour as default, [[Environment variables#Per_user|export]] the variable from {{ic|~/.bashrc}} or {{ic|~/.zshrc}}.
+
To set this behaviour as default, [[Environment variables#Per_user|export]] the variable from {{ic|~/.bashrc}} or {{ic|~/.zshrc}}.
 +
}}
 +
 
 +
{{Tip|While the journal is stored in a binary format, the content of stored messages is not modified. This means it is viewable with ''strings'', for example for recovery in an environment which does not have ''systemd'' installed, e.g.:
 +
 
 +
{{bc|$ strings /mnt/arch/var/log/journal/af4967d77fba44c6b093d0e9862f6ddd/system.journal {{!}} grep -i ''message''}}
 +
 
 
}}
 
}}
  
 
== Journal size limit ==
 
== Journal size limit ==
  
If the journal is persistent (non-volatile), its size limit is set to a default value of 10% of the size of the underlying file system but capped to 4 GiB. For example, with {{ic|/var/log/journal/}} located on a 20 GiB partition, journal data may take up to 2 GiB. On a 50 GiB partition, it would max at 4 GiB.
+
If the journal is persistent (non-volatile), its size limit is set to a default value of 10% of the size of the underlying file system but capped at 4 GiB. For example, with {{ic|/var/log/journal/}} located on a 20 GiB partition, journal data may take up to 2 GiB. On a 50 GiB partition, it would max at 4 GiB.
  
 
The maximum size of the persistent journal can be controlled by uncommenting and changing the following:
 
The maximum size of the persistent journal can be controlled by uncommenting and changing the following:
Line 143: Line 146:
 
}}
 
}}
  
It is also possible to use the drop-in snippets configuration override mechanism rather than editing the global configuration file. In this case do not forget to place the overrides under the {{ic|[Journal]}} header:
+
It is also possible to use the drop-in snippets configuration override mechanism rather than editing the global configuration file. In this case, place the overrides under the {{ic|[Journal]}} header:
  
 
{{hc|/etc/systemd/journald.conf.d/00-journal-size.conf|2=
 
{{hc|/etc/systemd/journald.conf.d/00-journal-size.conf|2=
Line 150: Line 153:
 
}}
 
}}
  
[[Restart]] the {{ic|systemd-journald.service}} after changing this setting to immediately apply the new limit.
+
[[Restart]] the {{ic|systemd-journald.service}} after changing this setting to apply the new limit.
  
 
See {{man|5|journald.conf}} for more info.
 
See {{man|5|journald.conf}} for more info.
Line 156: Line 159:
 
== Clean journal files manually ==
 
== Clean journal files manually ==
  
Journal files can be globally removed from {{ic|/var/log/journal/}} using ''e.g.'' {{ic|rm}}, or can be trimmed according to various criteria using {{ic|journalctl}}. Examples:
+
Journal files can be globally removed from {{ic|/var/log/journal/}} using ''e.g.'' {{ic|rm}}, or can be trimmed according to various criteria using {{ic|journalctl}}. For example:
  
 
* Remove archived journal files until the disk space they use falls below 100M: {{bc|1=# journalctl --vacuum-size=100M}}
 
* Remove archived journal files until the disk space they use falls below 100M: {{bc|1=# journalctl --vacuum-size=100M}}
Line 185: Line 188:
  
 
== Specify a different journal to view ==
 
== Specify a different journal to view ==
 +
 
There may be a need to check the logs of another system that is dead in the water, like booting from a live system to recover a production system. In such case, one can mount the disk in e.g. {{ic|/mnt}}, and specify the journal path via {{ic|-D}}/{{ic|--directory}}, like so:
 
There may be a need to check the logs of another system that is dead in the water, like booting from a live system to recover a production system. In such case, one can mount the disk in e.g. {{ic|/mnt}}, and specify the journal path via {{ic|-D}}/{{ic|--directory}}, like so:
  
 
  $ journalctl -D ''/mnt''/var/log/journal -xe
 
  $ journalctl -D ''/mnt''/var/log/journal -xe

Latest revision as of 20:27, 20 September 2019

systemd has its own logging system called the journal; therefore, running a syslog daemon is no longer required. To read the log, use:

# journalctl

In Arch Linux, the directory /var/log/journal/ is a part of the systemd package, and the journal (when Storage= is set to auto in /etc/systemd/journald.conf) will write to /var/log/journal/. If that directory is deleted, systemd will not recreate it automatically and instead will write its logs to /run/systemd/journal in a nonpersistent way. However, the folder will be recreated if Storage=persistent is added to journald.conf and systemd-journald.service is restarted (or the system is rebooted).

Systemd journal classifies messages by Priority level and Facility. Logging classification corresponds to classic Syslog protocol (RFC 5424).

Priority level

A syslog severity code (in systemd called priority) is used to mark the importance of a message RFC 5424 Section 6.2.1.

Value Severity Keyword Description Examples
0 Emergency emerg System is unusable Severe Kernel BUG, systemd dumped core.
This level should not be used by applications.
1 Alert alert Should be corrected immediately Vital subsystem goes out of work. Data loss.
kernel: BUG: unable to handle kernel paging request at ffffc90403238ffc.
2 Critical crit Critical conditions Crashes, coredumps. Like familiar flash:
systemd-coredump[25319]: Process 25310 (plugin-containe) of user 1000 dumped core
Failure in the system primary application, like X11.
3 Error err Error conditions Not severe error reported:
kernel: usb 1-3: 3:1: cannot get freq at ep 0x84,
systemd[1]: Failed unmounting /var.,
libvirtd[1720]: internal error: Failed to initialize a valid firewall backend
4 Warning warning May indicate that an error will occur if action is not taken. A non-root file system has only 1GB free.
org.freedesktop. Notifications[1860]: (process:5999): Gtk-WARNING **: Locale not supported by C library. Using the fallback 'C' locale.
5 Notice notice Events that are unusual, but not error conditions. systemd[1]: var.mount: Directory /var to mount over is not empty, mounting anyway,
gcr-prompter[4997]: Gtk: GtkDialog mapped without a transient parent. This is discouraged
6 Informational info Normal operational messages that require no action. lvm[585]: 7 logical volume(s) in volume group "archvg" now active
7 Debug debug Information useful to developers for debugging the application. kdeinit5[1900]: powerdevil: Scheduling inhibition from ":1.14" "firefox" with cookie 13 and reason "screen"

These rules are recommendations, and the priority level of a given error is at the application developer's discretion. It is always possible that the error will be at a higher or lower level than expected.

Facility

A syslog facility code is used to specify the type of program that is logging the message RFC 5424 Section 6.2.1.

Facility code Keyword Description Info
0 kern Kernel messages
1 user User-level messages
2 mail Mail system Archaic POSIX still supported and sometimes used (for more mail(1))
3 daemon System daemons All daemons, including systemd and its subsystems
4 auth Security/authorization messages Also watch for different facility 10
5 syslog Messages generated internally by syslogd For syslogd implementations (not used by systemd, see facility 3)
6 lpr Line printer subsystem (archaic subsystem)
7 news Network news subsystem (archaic subsystem)
8 uucp UUCP subsystem (archaic subsystem)
9 Clock daemon systemd-timesyncd
10 authpriv Security/authorization messages Also watch for different facility 4
11 ftp FTP daemon
12 - NTP subsystem
13 - Log audit
14 - Log alert
15 cron Scheduling daemon
16 local0 Local use 0 (local0)
17 local1 Local use 1 (local1)
18 local2 Local use 2 (local2)
19 local3 Local use 3 (local3)
20 local4 Local use 4 (local4)
21 local5 Local use 5 (local5)
22 local6 Local use 6 (local6)
23 local7 Local use 7 (local7)

Useful facilities to watch: 0, 1, 3, 4, 9, 10, 15.

Filtering output

journalctl allows for the filtering of the output by specific fields. If there are many messages to display or filtering of large time span has to be done, the output of this command can be extensively delayed.

Examples:

  • Show all messages from this boot:
    # journalctl -b
    However, often one is interested in messages not from the current, but from the previous boot (e.g. if an unrecoverable system crash happened). This is possible through optional offset parameter of the -b flag: journalctl -b -0 shows messages from the current boot, journalctl -b -1 from the previous boot, journalctl -b -2 from the second previous and so on – you can see the list of boots with their numbers by using journalctl --list-boots. See journalctl(1) for a full description; the semantics are more powerful than indicated here.
  • Show all messages from date (and optional time):
    # journalctl --since="2012-10-30 18:17:16"
  • Show all messages since 20 minutes ago:
    # journalctl --since "20 min ago"
  • Follow new messages:
    # journalctl -f
  • Show all messages by a specific executable:
    # journalctl /usr/lib/systemd/systemd
  • Show all messages by a specific process:
    # journalctl _PID=1
  • Show all messages by a specific unit:
    # journalctl -u man-db.service
  • Show kernel ring buffer:
    # journalctl -k
  • Show only error, critical and alert priority messages:
    # journalctl -p err..alert
    You can use numeric log level too, like journalctl -p 3..1. If single number/log level is used, journalctl -p 3, then all higher priority log levels are also included (i.e. 0 to 3 in this case).
  • Show auth.log equivalent by filtering on syslog facility:
    # journalctl SYSLOG_FACILITY=10
  • If the journal directory (by default located under /var/log/journal) contains a large amount of log data then journalctl can take several minutes to filter output. It can be sped up significantly by using --file option to force journalctl to look only into most recent journal:
    # journalctl --file /var/log/journal/*/system.journal -f

See journalctl(1), systemd.journal-fields(7), or Lennart Poettering's blog post for details.

Tip: By default, journalctl truncates lines longer than screen width, but in some cases, it may be better to enable wrapping instead of truncating. This can be controlled by the SYSTEMD_LESS environment variable, which contains options passed to less (the default pager) and defaults to FRSXMK (see less(1) and journalctl(1) for details).

By omitting the S option, the output will be wrapped instead of truncated. For example, start journalctl as follows:

$ SYSTEMD_LESS=FRXMK journalctl
To set this behaviour as default, export the variable from ~/.bashrc or ~/.zshrc.
Tip: While the journal is stored in a binary format, the content of stored messages is not modified. This means it is viewable with strings, for example for recovery in an environment which does not have systemd installed, e.g.:
$ strings /mnt/arch/var/log/journal/af4967d77fba44c6b093d0e9862f6ddd/system.journal | grep -i message

Journal size limit

If the journal is persistent (non-volatile), its size limit is set to a default value of 10% of the size of the underlying file system but capped at 4 GiB. For example, with /var/log/journal/ located on a 20 GiB partition, journal data may take up to 2 GiB. On a 50 GiB partition, it would max at 4 GiB.

The maximum size of the persistent journal can be controlled by uncommenting and changing the following:

/etc/systemd/journald.conf
SystemMaxUse=50M

It is also possible to use the drop-in snippets configuration override mechanism rather than editing the global configuration file. In this case, place the overrides under the [Journal] header:

/etc/systemd/journald.conf.d/00-journal-size.conf
[Journal]
SystemMaxUse=50M

Restart the systemd-journald.service after changing this setting to apply the new limit.

See journald.conf(5) for more info.

Clean journal files manually

Journal files can be globally removed from /var/log/journal/ using e.g. rm, or can be trimmed according to various criteria using journalctl. For example:

  • Remove archived journal files until the disk space they use falls below 100M:
    # journalctl --vacuum-size=100M
  • Make all journal files contain no data older than 2 weeks.
    # journalctl --vacuum-time=2weeks

See journalctl(1) for more info.

Journald in conjunction with syslog

Compatibility with a classic, non-journald aware syslog implementation can be provided by letting systemd forward all messages via the socket /run/systemd/journal/syslog. To make the syslog daemon work with the journal, it has to bind to this socket instead of /dev/log (official announcement).

The default journald.conf for forwarding to the socket is ForwardToSyslog=no to avoid system overhead, because rsyslog or syslog-ng pull the messages from the journal by itself.

See Syslog-ng#Overview and Syslog-ng#syslog-ng and systemd journal, or rsyslog respectively, for details on configuration.

Forward journald to /dev/tty12

Create a drop-in directory /etc/systemd/journald.conf.d and create a fw-tty12.conf file in it:

/etc/systemd/journald.conf.d/fw-tty12.conf
[Journal]
ForwardToConsole=yes
TTYPath=/dev/tty12
MaxLevelConsole=info

Then restart systemd-journald.service.

Specify a different journal to view

There may be a need to check the logs of another system that is dead in the water, like booting from a live system to recover a production system. In such case, one can mount the disk in e.g. /mnt, and specify the journal path via -D/--directory, like so:

$ journalctl -D /mnt/var/log/journal -xe