https://wiki.archlinux.org/api.php?action=feedcontributions&user=Nuclearsandwich&feedformat=atomArchWiki - User contributions [en]2024-03-28T15:00:28ZUser contributionsMediaWiki 1.41.0https://wiki.archlinux.org/index.php?title=Syslog-ng&diff=132225Syslog-ng2011-02-26T23:00:35Z<p>Nuclearsandwich: /* syslog-ng.conf */</p>
<hr />
<div>[[Category:Daemons and system services (English)]]<br />
==Quick Start==<br />
Syslog-ng is a great logging replacement/enhancement for syslog. I used to use rsyslog, now I only use syslog-ng. The power of syslog-ng lies in the configuration file syslog-ng.conf.<br />
<br />
For a quick start, here there is a classic configuration file slightly modified from the one in the <br />
[http://www.gentoo.org/doc/en/security/security-handbook.xml?part=1&chap=3#doc_chap4 Gentoo Security Guide], the default syslog-ng.conf provided with the source distribution, and my own personal preferences. [[User:AskApache|AskApache]] 22:10, 14 September 2010 (EDT)<br />
<br />
== syslog-ng.conf ==<br />
<pre><br />
@version: 3.0<br />
# For a description of syslog-ng configuration file directives, please read<br />
# the syslog-ng Administrator's guide at:<br />
#<br />
# http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html<br />
#<br />
<br />
##########################################################<br />
# OPTIONS<br />
#<br />
options {<br />
create_dirs(yes);<br />
# use_dns(no);<br />
use_dns(persist_only);<br />
dns_cache_hosts(/etc/hosts);<br />
dns_cache_expire(87600);<br />
<br />
# disable the chained hostname format in logs (default is enabled)<br />
chain_hostnames(0);<br />
<br />
# the number of lines fitting in the output queue<br />
log_fifo_size(512);<br />
<br />
# enable or disable directory creation for destination files<br />
create_dirs(yes);<br />
<br />
# default owner, group, and permissions for log files (defaults are 0, 0, 0600)<br />
owner(root);<br />
group(log);<br />
perm(0640);<br />
<br />
# default owner, group, and permissions for created directories (defaults are 0, 0, 0700)<br />
dir_owner(root);<br />
dir_group(root);<br />
dir_perm(0740); <br />
<br />
# the time to wait before a died connection is re-established (default is 60)<br />
time_reopen(10);<br />
<br />
# the time to wait before an idle destination file is closed (default is 60)<br />
time_reap(360);<br />
<br />
# default no<br />
use_fqdn(no);<br />
<br />
keep_hostname(yes);<br />
<br />
# disable stats<br />
stats_freq(0);<br />
}; <br />
<br />
<br />
##########################################################<br />
# SOURCES<br />
#<br />
source local_src {<br />
# message generated by Syslog-NG<br />
internal();<br />
<br />
# standard Linux log source (this is the default place for the syslog() function to send logs to)<br />
unix-stream("/dev/log");<br />
<br />
# from a chrooted bind install<br />
unix-stream("/var/named/chroot/dev/log");<br />
<br />
# messages from the kernel<br />
file("/proc/kmsg" program_override("kernel: "));<br />
};<br />
<br />
# source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); };<br />
# source s_pipe { pipe("/dev/pipe" pad_size(2048)); };<br />
<br />
<br />
<br />
##########################################################<br />
# DESTINATIONS<br />
#<br />
destination d_file { file("/var/log/$YEAR.$MONTH.$DAY/everything.log" template("$HOUR:$MIN:$SEC [$LEVEL] [$FACILITY] [$PROGRAM] $MSG\n") template_escape(no)); };<br />
<br />
destination d_askapacheloghost {<br />
tcp("askapacheloghost.dyndns.org" port(65514));<br />
udp("askapacheloghost.dyndns.org" port(65514));<br />
udp("askapacheloghost.dyndns.org" port(514));<br />
};<br />
<br />
destination d_authlog { file("/var/log/auth.log"); };<br />
destination d_cron { file("/var/log/cron.log"); };<br />
destination d_daemon { file("/var/log/daemon.log"); };<br />
destination d_kern { file("/var/log/kern.log"); };<br />
destination d_lpr { file("/var/log/lpr.log"); };<br />
destination d_user { file("/var/log/user.log"); };<br />
destination d_uucp { file("/var/log/uucp.log"); };<br />
destination d_ppp { file("/var/log/ppp.log"); };<br />
<br />
destination d_mail { file("/var/log/mail.log"); };<br />
destination d_mailinfo { file("/var/log/mail.info"); };<br />
destination d_mailwarn { file("/var/log/mail.warn"); };<br />
destination d_mailerr { file("/var/log/mail.err"); };<br />
<br />
destination d_newscrit { file("/var/log/news/news.crit"); };<br />
destination d_newserr { file("/var/log/news/news.err"); };<br />
destination d_newsnotice { file("/var/log/news/news.notice"); };<br />
<br />
destination d_debug { file("/var/log/debug"); };<br />
destination d_messages { file("/var/log/messages"); };<br />
<br />
destination d_everything { file("/var/log/everything"); };<br />
destination d_console { usertty("root"); };<br />
destination d_console_all { file("/dev/tty12"); };<br />
destination d_loghost { udp("loghost" port(999)); };<br />
destination d_xconsole { pipe("/dev/xconsole"); };<br />
<br />
<br />
<br />
##########################################################<br />
# FILTERS<br />
#<br />
filter f_auth { facility(auth); };<br />
filter f_authpriv { facility(auth, authpriv); }; <br />
filter f_syslog { program(syslog-ng); };<br />
filter f_cron { facility(cron); };<br />
filter f_daemon { facility(daemon); };<br />
filter f_kernel { facility(kern) and not filter(f_iptables); };<br />
filter f_lpr { facility(lpr); };<br />
filter f_mail { facility(mail); };<br />
filter f_news { facility(news); };<br />
filter f_user { facility(user); };<br />
filter f_uucp { facility(cron); };<br />
filter f_news { facility(news); };<br />
filter f_ppp { facility(local2); };<br />
filter f_debug { not facility(auth, authpriv, news, mail); };<br />
filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news, cron) and not program(syslog-ng) and not filter(f_iptables); };<br />
filter f_everything { level(debug..emerg); };<br />
filter f_emergency { level(emerg); };<br />
filter f_info { level(info); };<br />
filter f_notice { level(notice); };<br />
filter f_warn { level(warn); };<br />
filter f_crit { level(crit); };<br />
filter f_err { level(err); };<br />
filter f_iptables { match("IN=" value("MESSAGE")) and match("OUT=" value("MESSAGE")); };<br />
filter f_acpid { program("acpid"); };<br />
filter f_failed { match("regex" value("failed")); };<br />
filter f_denied { match("regex" value("denied")); };<br />
filter f_noshorewall { not match("regex" value("Shorewall")); }; # Filter everything except regex keyword Shorewall<br />
filter f_shorewall { match("regex" value("Shorewall")); }; # Filter regex keyword Shorewall<br />
<br />
<br />
<br />
<br />
##########################################################<br />
# LOG<br />
#<br />
log { source(local_src); destination(d_askapacheloghost); };<br />
log { source(local_src); destination(d_file); };<br />
<br />
log { source(local_src); filter(f_authpriv); destination(d_authlog); };<br />
log { source(local_src); filter(f_user); destination(d_user); };<br />
<br />
log { source(local_src); filter(f_cron); destination(d_cron); };<br />
log { source(local_src); filter(f_daemon); destination(d_daemon); };<br />
log { source(local_src); filter(f_kern); destination(d_kern); };<br />
log { source(local_src); filter(f_lpr); destination(d_lpr); };<br />
log { source(local_src); filter(f_mail); destination(d_mail); };<br />
log { source(local_src); filter(f_uucp); destination(d_uucp); };<br />
log { source(local_src); filter(f_mail); filter(f_info); destination(d_mailinfo); };<br />
log { source(local_src); filter(f_mail); filter(f_warn); destination(d_mailwarn); };<br />
log { source(local_src); filter(f_mail); filter(f_err); destination(d_mailerr); };<br />
log { source(local_src); filter(f_news); filter(f_crit); destination(d_newscrit); };<br />
log { source(local_src); filter(f_news); filter(f_err); destination(d_newserr); };<br />
log { source(local_src); filter(f_news); filter(f_notice); destination(d_newsnotice); };<br />
log { source(local_src); filter(f_debug); destination(d_debug); };<br />
log { source(local_src); filter(f_messages); destination(d_messages); };<br />
log { source(local_src); filter(f_ppp); destination(d_ppp); };<br />
log { source(local_src); destination(d_messages); };<br />
<br />
#default log<br />
log { source(local_src); destination(console_all); };<br />
</pre><br />
<br />
== Sources ==<br />
Syslog-ng receives log messages from a source. To define a source you should follow the following syntax:<br />
<br />
source <identifier> { source-driver(params); source-driver(params); ... };<br />
<br />
<br />
You can look at the identifiers and source-drivers in the [http://www.balabit.com/support/documentation/ official manuals]. <br />
This will follow the manual to explain the configuration file above. The unix-stream() source-driver opens the given AF_UNIX<br />
[http://en.wikipedia.org/wiki/Berkeley_sockets socket] and starts listening on it for messages. <br />
The internal() source-driver gets messages generated by syslog-ng.<br />
<br />
Therefore, the following means: src gets messages from /dev/log socket and syslog-ng.<br />
<br />
source src { unix-stream("/dev/log"); internal(); };<br />
<br />
<br />
The kernel sends log messages to /proc/kmsg and the file() driver reads log messages from files. Therefore, the following means:<br />
kernsrc gets messages from file /proc/kmsg<br />
<br />
source kernsrc { file("/proc/kmsg"); };<br />
<br />
<br />
In the default configuration file after emerging syslog-ng, the source is defined as:<br />
<br />
source src { unix-stream("/dev/log"); internal(); pipe("/proc/kmsg"); };<br />
<br />
Reading messages by pipe("/proc/kmsg") gives a better performance but because it opens its argument in read-write mode can be a security<br />
hazard as the [http://www.balabit.com/dl/white_papers/syslog_admin_guide_en.pdf syslog-ng admin guide] states in section 7.1.6:<br />
<br />
"Pipe is very similar to the file() driver, but there are a few differences, for example pipe() opens its argument in read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg." (You can follow this discussion in [http://forums.gentoo.org/viewtopic-t-558161.html this post].)<br />
<br />
<br />
== Destinations ==<br />
In syslog-ng log messages are sent to files. The syntax is very similar to sources:<br />
<br />
destination <identifier> {destination-driver(params); destination-driver(params); ... };<br />
<br />
<br />
You will be normally logging to a file, but you could log to a different destination-driver: pipe, unix socket, TCP-UDP ports,<br />
terminals or to specific programs. Therefore, this means sending authlog messages to /var/log/auth.log:<br />
<br />
destination authlog { file("/var/log/auth.log"); };<br />
<br />
<br />
If the user is logged in, usertty() sends messages to the terminal of the specified user. If you want to send console messages<br />
to root's terminal if it is logged in:<br />
<br />
destination console { usertty("root"); };<br />
<br />
<br />
Messages can be sent to a pipe with pipe(). The following sends xconsole messages to the pipe /dev/xconsole. <br />
This needs some more configuration, so you could look at the sub-section xconsole below.<br />
<br />
destination xconsole { pipe("/dev/xconsole"); };<br />
<br />
<br />
To send messages on the network, use udp(). The following will send your log data out to another server.<br />
<br />
destination remote_server { udp("10.0.0.2" port(514)); };<br />
<br />
<br />
<br />
<br />
== Creating Filters for Messages ==<br />
The syntax for the filter statement is:<br />
<br />
filter <identifier> { expression; };<br />
<br />
<br />
Functions can be used in the expression, such as the fuction facility() which selects messages based on the facility codes. <br />
The linux kernel has a few facilities you can use for logging. Each facility has a log-level; where debug is the most verbose,<br />
and panic only shows serious errors. You can find the facilities, log levels and priority names in /usr/include/sys/syslog.h.<br />
To filter those messages coming from authorisation, like <br />
''<nowiki>May 11 23:42:31 mimosinnet su(pam_unix)[18569]: session opened for user root by (uid=1000)</nowiki>'', use the following:<br />
<br />
filter f_auth { facility(auth); };<br />
<br />
<br />
The facility expression can use the boolean operators ''and'', ''or'', and ''not'', so the following filter<br />
selects those messages not coming from authorisation, network news or mail:<br />
<br />
filter f_debug { not facility(auth, authpriv, news, mail); };<br />
<br />
<br />
The funciont level() selects messages based on its priority level, so if you want to select informational levels:<br />
<br />
filter f_info { level(info); };<br />
<br />
<br />
Functions and boolean operators can be combined in more complex expressions. The following line filters messages with a priority level from<br />
informational to warning not coming from atuh, authpriv, mail and news facilities:<br />
<br />
filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); };<br />
<br />
<br />
Messages can also be selected by matching a regular expression in the message with the function match("regex" value("keyword")). For example:<br />
<br />
filter f_failed { match("regex" value("failed")); };<br />
<br />
== Log Paths ==<br />
Syslog-ng connects sources, filters and destinations with log statements. The syntax is:<br />
<pre>log {source(s1); source(s2); ...<br />
filter(f1); filter(f2); ...<br />
destination(d1); destination(d2); ...<br />
flags(flag1[, flag2...]); };</pre><br />
<br />
<br />
The following for example sends messages from 'src' source to 'mailinfo' destination filtered by 'f_info' filter.<br />
<br />
log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); };<br />
<br />
<br />
== Tips and Tricks ==<br />
After understanding the logic behind syslog-ng, many possible and complex configuration are possible. Here there are some examples.<br />
<br />
=== Failover Logging to Remote Host ===<br />
This setup shows how to send the default unencrypted syslog packets across both tcp and udp protocols, using the standard port (514) and an alternate port. This is sending the same output to the same machine 4 different ways to try and make sure packets make it. Mostly useful if you are debugging a remote server that fails to reboot. The different ports and protocols are to make it past any firewall filters or other network problems. Also useful for port-forwarding and using tunnels. Something like this setup is ideal to tunnel across an ssh connection that the prone-to-failover host initiates through a reverse connection.<br />
<br />
<pre><br />
#sending to a remote syslog server on tcp and udp ports (not encrypted)<br />
destination askapache_failover_loghost {<br />
tcp("208.86.158.195" port(25214));<br />
udp("208.86.158.195" port(25214));<br />
udp("mysyslog1.dyndns.org" port(514));<br />
};<br />
log { <br />
source(src); <br />
destination(askapache_failover_loghost);<br />
};<br />
</pre><br />
<br />
<br />
And then on the loghost receiving these logs:<br />
<pre><br />
#a usb redirected console for flexible viewing<br />
destination debugging_console {<br />
file("/dev/ttyU1");<br />
};<br />
<br />
# listens on ips and ports, sets the incoming settings<br />
source prone_to_failover_host {<br />
tcp(ip(208.86.158.195),port(25214));<br />
udp(ip(208.86.158.195) port(25214));<br />
<br />
udp(default-facility(syslog) default-priority(emerg));<br />
tcp(default-facility(syslog) default-priority(emerg));<br />
}<br />
<br />
# log it<br />
log {<br />
source(prone_to_failover_host); <br />
destination(debugging_console);<br />
};<br />
</pre><br />
<br />
=== Log directly to MySQL ===<br />
[[Syslog-ng directly to MySQL]]<br />
<br />
=== Move log to another file ===<br />
In order to move some log from /var/log/messages to another file:<br />
<br />
<pre><br />
#sshd configuration<br />
destination ssh { file("/var/log/ssh.log"); };<br />
filter f_ssh { program("sshd"); };<br />
log { source(src); filter(f_ssh); destination(ssh); };<br />
</pre><br />
<br />
<br />
=== Configuring as a loghost ===<br />
Configuring your system to be a loghost is quite simple. Drop the following into your configuration, and create the needed directory.<br />
With this simple configuration, log filenames will be based on the [http://en.wikipedia.org/wiki/FQDN FQDN] of the remote host,<br />
and located in /var/log/remote/. After creating the remote directory, reload your syslog-ng configuration.<br />
<br />
<br />
<pre><br />
source net { udp(); };<br />
destination remote { file("/var/log/remote/$FULLHOST"); };<br />
log { source(net); destination(remote); };<br />
</pre><br />
<br />
<br />
=== Improve Performance ===<br />
Syslog-ng's performance can be improved in different ways:<br />
<br />
==== Avoid redundant processing and disk space ====<br />
A single log message can be sent to different log files several times. For example, in the initial configuration file, we have the following definitions:<br />
<br />
<pre><br />
destination cron { file("/var/log/cron.log"); };<br />
destination messages { file("/var/log/messages"); };<br />
filter f_cron { facility(cron); };<br />
filter f_messages { level(info..warn) <br />
and not facility(auth, authpriv, mail, news); };<br />
log { source(src); filter(f_cron); destination(cron); };<br />
log { source(src); filter(f_messages); destination(messages); };<br />
</pre><br />
<br />
<br />
The same message from the 'cron' facility will end up in both the cron.log and messages file. To change this behavior we can use the final flag, <br />
ending up further processing with the message. Therefore, in this example, if we want messages from the 'cron' facility not ending up in the<br />
messages file, we should change the cron's log sentence by:<br />
<br />
log { source(src); filter(f_cron); destination(cron); flags(final); };<br />
<br />
another way is to exclude the cron facility from f_messages filter:<br />
filter f_messages { level(info..warn) and not facility(cron, auth, authpriv, mail, news); };<br />
<br />
=== Postgresql Destination ===<br />
This section will use two roles: ''syslog'' and ''logwriter''. ''syslog'' will be the administrator of the database ''syslog'' and ''logwriter'' will only be able to add records to the ''logs'' table.<br />
<br />
No longer needed to create table for logs. Syslog-ng will create automatically.<br />
<br />
psql -U postgres<br />
<br />
postgres=# CREATE ROLE syslog WITH LOGIN;<br />
postgres=# \password syslog # Using the \password function is secure because<br />
postgres=# \password logwriter # the password isn't saved in history.<br />
postgres=# CREATE DATABASE syslog OWNER syslog;<br />
postgres=# \q # You're done here for the moment<br />
<br />
Edit pg_hba.conf to allow ''syslog'' and ''logwriter'' to establish a connection to PostgreSQL.<br />
<br />
/var/lib/postgresql/8.4/data/pg_hba.conf<br />
<pre><br />
# TYPE DATABASE USER CIDR-ADDRESS METHOD<br />
<br />
host syslog logwriter 192.168.0.1/24 md5<br />
host syslog syslog 192.168.0.10/32 md5<br />
</pre><br />
<br />
<br />
Tell PostgreSQL to reload the configuration files:<br />
/etc/rc.d/postgresql-8.4 reload<br />
<br />
<br />
Edit /etc/syslog-ng.conf so that it knows where and how to write to PostgreSQL. Syslog-ng will utilize the ''logwriter'' role.<br />
<br />
<pre><br />
...<br />
#<br />
# SQL logging support<br />
#<br />
<br />
destination d_pgsql {<br />
sql(type(pgsql)<br />
host("127.0.0.1") username("logwriter") password("password")<br />
database("syslog")<br />
table("logs_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") #or whatever you want, example ${HOST}" for hosts, ${LEVEL}" for levels.. etc<br />
columns("datetime varchar(16)", "host varchar(32)", "program varchar(8)", "message varchar(200)")<br />
values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSG")<br />
indexes("datetime", "host", "program", "pid", "message"));<br />
};<br />
<br />
<br />
log { source(src); destination(d_pgsql); };<br />
</pre><br />
<br />
<br />
Finally, restart Syslog-ng.<br />
/etc/rc.d/syslog-ng restart<br />
<br />
<br />
And check to see if things are being logged.<br />
psql -U logwriter -d syslog<br />
syslog=> SELECT * FROM <your table name> ORDER BY datetime DESC LIMIT 10;<br />
<br />
=== ISO 8601 timestamps ===<br />
'''Before''' :<br />
#logger These timestamps are not optimal.<br />
#tail -n 1 /var/log/messages.log<br />
Feb 18 14:25:01 hostname logger: These timestamps are not optimal.<br />
#<br />
<br />
Add <tt>ts_format(iso);</tt><br />
to ''/etc/syslog-ng.conf'' in the options section. Example:<br />
options {<br />
stats_freq (0);<br />
flush_lines (0);<br />
time_reopen (10);<br />
log_fifo_size (1000);<br />
long_hostnames(off); <br />
use_dns (no);<br />
use_fqdn (no);<br />
create_dirs (no);<br />
keep_hostname (yes);<br />
perm(0640);<br />
group("log");<br />
ts_format(iso); #make ISO8601 timestamps<br />
};<br />
<br />
Then :<br />
# killall -HUP syslog-ng<br />
<br />
'''After''' :<br />
#logger Now THAT is a timestamp!<br />
#tail -n 2 /var/log/messages.log<br />
Feb 18 14:25:01 hostname logger: These timestamps are not optimal.<br />
2010-02-18T20:23:58-05:00 electron logger: Now THAT is a timestamp!<br />
#<br />
<br />
=== RFC 3339 timestamps ===<br />
same as above, except use ''rfc3339'' instead of ''iso'' for <tt>ts_format</tt><br />
<br />
<br />
== External Links ==<br />
* [http://en.gentoo-wiki.com/wiki/Syslog-ng Syslog-ng Gentoo wiki]<br />
* [http://en.wikipedia.org/wiki/ISO_8601 ISO_8601] Wikipedia page for ISO 8601<br />
* [http://tools.ietf.org/html/rfc3339 RFC3339] Text of RFC 3339<br />
* [http://www.syslog.org/syslog-ng/v2/#reference_destinationdrivers syslog-ng_manual] syslog-ng v2.0 reference manual<br />
* [http://freshmeat.net/projects/syslog-ng/ Syslog-ng Project Page on Freshmeat]<br />
* [http://www.balabit.com/support/documentation/ Portal to Syslog-ng Documentation]<br />
* [http://www.gentoo.org/doc/en/security/security-handbook.xml?part=1&chap=3 Gentoo's Security Handbook on Logging]<br />
* [http://www.kdough.net/docs/syslog_postgresql/ Syslog Logging with PostgreSQL HOWTO]</div>Nuclearsandwichhttps://wiki.archlinux.org/index.php?title=Talk:Syslog-ng&diff=132223Talk:Syslog-ng2011-02-26T22:59:33Z<p>Nuclearsandwich: /* Typo in Shorewall config? */</p>
<hr />
<div>after the example syslog-ng.conf, and aside from the timestamps and remote loghost tips, most of this article has been adapted from the gentoo wiki page for syslog-ng.conf .. FYI<br />
<br />
So yes it needs updating for arch please <br />
<br />
[[User:AskApache|AskApache]] 22:19, 14 September 2010 (EDT)<br />
<br />
== Is match() example right? ==<br />
<br />
The example:<br />
filter f_failed { match("regex" value("failed")); };<br />
is in my opinion bad.<br />
<br />
List of supported values in value() should be: "HOST", "HOST_FROM", "MESSAGE", "PROGRAM", "PID", "MSGID" and "SOURCE".<br />
<br />
More info: https://lists.balabit.hu/pipermail/syslog-ng/2009-April/012789.html<br />
<br />
Better example could be:<br />
<br />
filter f_grsecurity { match("^grsec" value("MESSAGE")); };<br />
''This is real/working example from my syslog-ng config.''<br />
<br />
[[User:Tojaj|Tojaj]] 16:39, 8 February 2011 (EST)<br />
<br />
== Reversal typo in Shorewall examples ==<br />
<br />
The example:<br />
filter f_shorewall { not match("regex" value("Shorewall")); }; # Filter everything except regex keyword Shorewall<br />
filter f_noshorewall { match("regex" value("Shorewall")); }; # Filter regex keyword Shorewall<br />
I believe the identifiers are switched. I have switched them to what I think they are intended to be.<br />
[[User:nuclearsandwich|nuclearsandwich]] 14:58, 26 February 2011 (PST)</div>Nuclearsandwich