balabit / syslog-ng-incubator Goto Github PK

View Code? Open in Web Editor NEW

52.0 52.0 28.0 600 KB

Experimental modules for syslog-ng 3.5+

License: Other

C 85.39% Lua 1.70% Perl 0.15% Shell 0.13% Makefile 5.61% M4 6.45% C++ 0.57%

syslog-ng-incubator's People

Contributors

Stargazers

Watchers

syslog-ng-incubator's Issues

lua: A way to set global variables from the syslog-ng config

I'd like to build some SCL stuff over a destination implemented in Lua, and for that, I'd need to tweak some configuration options. It would be tremendously useful if I could do that without having to edit the Lua script itself, because I want to install the lua script via Debian packages, so it won't be a config file you can tweak and upgrade and all that.

If I could also use template type hints too, that would be grand.

(I will have a stab at this tomorrow, if all goes well, but will likely fail, because I know nothing about Lua)

trouble installing lua destination

... compiled syslog-ng 3.5 from source (it works great):

syslog-ng -V
syslog-ng 3.5.4.1
Installer-Version: 3.5.4.1
Revision: ssh+git://[email protected]/var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.5#master#4090ee62163780ae68a0c83cfdc23998c904fe97
Compile-Date: Apr  9 2014 14:56:33
Available-Modules: afprog,cryptofuncs,affile,linux-kmsg-format,afsocket,afamqp,json-plugin,syslogformat,system-source,confgen,afsocket-notls,afuser,basicfuncs,afstomp,redis,afsocket-tls,csvparser,tfgeoip,afmongodb,dbparser
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: off
Enable-TCP-Wrapper: off
Enable-Linux-Caps: off
Enable-Pcre: on

... compiled lua from source:

lua -v ... Lua 5.2.3  Copyright (C) 1994-2013 Lua.org, PUC-Rio

... I see:

/usr/lib/x86_64-linux-gnu/liblua5.1.so.0
/usr/lib/x86_64-linux-gnu/liblua5.1-c++.so.0.0.0
/usr/lib/x86_64-linux-gnu/liblua5.1-c++.so.0
/usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0
/usr/local/lib/liblua.a
/usr/share/doc/liblua5.1-0

cd syslog-ng-incubator
autoreconf -i
./configure
I see:

checking for LUA... no

syslog-ng Incubator 0.2.1 configured
------------------------------------
 Modules:
  basicfuncs-plus      yes
  graphite             yes
  logmongource:        no
  lua:                 no  <<--- ???
  monitor-source:      no 
  riemann:             no
  rss                  yes
  trigger-source       yes
  tfgetent             yes

what am I missing ?
I was hoping to try the logs to elasticsearch via lua example, but I get this:

Error parsing destination, destination plugin lua not found in /usr/local/etc/syslog-ng.conf at line 9, column 3:

  lua(
  ^^^

thanks

lua destination formatting issues

The source of the lua destination has a couple of stylistic issues:

The * being at the wrong place when declaring pointers:
- lua_dd_load_file
- lua_dd_call_init_func
- lua_dd_check_existence_of_queue_func
- lua_dd_set_template
- lua_dd_set_filename
- lua_dd_get_template_options
- lua_dd_set_init_func
- lua_dd_set_queue_func
- lua_dd_set_mode
- lua_dd_new
- LuaDestDriver
- Various function declarations in lua-dest.h
- lua_message_new
- lua_message_metatable__new_index
- lua_message_metatable__index
- lua_message_metatable__gc
- lua_message_to_logmsg (in lua-msg.h)
- lua_get_pointer_from_userdata
- lua_check_userdata
- lua_check_and_convert_userdata
Indentation is wrong at:
- lua_keywords
Variable declared somewhere else than the function header:
- lua_get_pointer_from_userdata

@talien, can you fix these and submit a PR?

perl: module integration

Curently the perl destination tries to load the script given using the script() directive.
IMHO it would be nice to be able to use an existing module in @INC instead:

perl(
  module("My::SyslogNG::Module")
  value-pairs(
    deinit-func("bye")
    scope(nv-pairs)
  )
);

This would for instance implicitly set init-func("My::SyslogNG::Module::init") and semi-implicitly deinit-func("My::SyslogNG::Module::bye")

python is not found on FreeBSD unless a hardcoded version number is passed to configure

When running configure on FreeBSD, python is not found, unless a hardcoded version number is passed to it:

python: no (python)

And config.log-ban shows:

configure:16589: checking for PYTHON
configure:16596: $PKG_CONFIG --exists --print-errors "$with_python >= 2.6"
Package python was not found in the pkg-config search path.
Perhaps you should add the directory containing python.pc' to the PKG_CONFIG_PATH environment variable Package 'python', required by 'world', not found configure:16599: $? = 1 configure:16613: $PKG_CONFIG --exists --print-errors "$with_python >= 2.6" Package python was not found in the pkg-config search path. Perhaps you should add the directory containingpython.pc'
to the PKG_CONFIG_PATH environment variable
Package 'python', required by 'world', not found
configure:16616: $? = 1
configure:16630: result: no
Package python was not found in the pkg-config search path.
Perhaps you should add the directory containing `python.pc'
to the PKG_CONFIG_PATH environment variable
Package 'python', required by 'world', not found

One can use --with-python=python-2.7 or similar, but it's a problem, as six different python versions are supported in FreeBSD ports and one can't be sure which one is installed.

lua: segfault when using macros in lua destination params

The following generates a segfault:

destination d_elasticsearch {
  elasticsearch(
    index("netflow-$YEAR.$MONTH.$DAY")
  );
};

stack trace:

#0  0x00007f22200d1192 in log_macro_expand (result=0x82b8f8, id=22, escape=0, opts=0x70cc58, tz=0, seq_num=0, context_id=0x0, msg=0x0)
    at lib/template/templates.c:532
        buf =
    "\370\032\006 \"\177\000\000\034\250\006 \"\177\000\000\070;v \"\177", '\000' <repeats 18 times>"\310, \324t \"\177\000\000\367\247\006 \"\177\000\000\260\316S\037\"\177\000"
        length = 0
        stamp = 0x20
        sstamp = {tv_sec = 32, tv_usec = 1, zone_offset = 0}
        tmp_hour = 0
        tm = 0x70cc58
        tm_storage = {tm_sec = 537266304, tm_min = 32546, tm_hour = 0, tm_mday = 1, tm_mon = 595, tm_year = 1, tm_wday = 0, tm_yday = 0, tm_isdst = 48,
          tm_gmtoff = 139784531536593, tm_zone = 0x7f2220063238 "tC"}
        t = 0
        zone_ofs = 8567032
        __PRETTY_FUNCTION__ = "log_macro_expand"
#1  0x00007f22200d3227 in log_template_append_format_with_context (self=0x8225a0, messages=0x7fffac1a4c70, num_messages=1, opts=0x70cc58, tz=0, seq_num=0,
    context_id=0x0, result=0x82b8f8) at lib/template/templates.c:1352
        len = 8
        msg_ndx = 0
        p = 0x81c8a0 = {0x822650, 0x822690, 0x8226d0}
        e = 0x822650
#2  0x00007f22200d359c in log_template_append_format (self=0x8225a0, lm=0x0, opts=0x0, tz=0, seq_num=0, context_id=0x0, result=0x82b8f8)
    at lib/template/templates.c:1412
No locals.
#3  0x00007f22200ad1b5 in vp_pairs_foreach (data=0x822310, user_data=0x7fffac1a4da0) at lib/value-pairs.c:203
        vp = 0x817a20
        msg = 0x0
        seq_num = 0
        scope_set = 0x7f5f00
        template_options = 0x0
        sb = 0x82b8f0
        vpc = 0x822310
#4  0x00007f221f54767b in g_ptr_array_foreach (array=0x81c800, func=0x7f22200ad0f9 <vp_pairs_foreach>, user_data=0x7fffac1a4da0) at garray.c:1306
        i = <value optimized out>
        __PRETTY_FUNCTION__ = "g_ptr_array_foreach"
#5  0x00007f22200ad8c5 in value_pairs_foreach_sorted (vp=0x817a20, func=0x7f221a86e594 <lua_dd_inject_global_variable>, compare_func=
    0x7f221e664580 <__strcmp_sse42>, msg=0x0, seq_num=0, template_options=0x0, user_data=0x8193f0) at lib/value-pairs.c:365
        args = {0x817a20, 0x7f221a86e594, 0x0, 0x0, 0x8193f0, 0x7f5f00, 0x0}
        result = 1
        helper_args = {0x7f221a86e594, 0x8193f0, 0x7fffac1a4d9c}
        scope_set = 0x7f5f00
#6  0x00007f22200ad94e in value_pairs_foreach (vp=0x817a20, func=0x7f221a86e594 <lua_dd_inject_global_variable>, msg=0x0, seq_num=0, template_options=0x0,
    user_data=0x8193f0) at lib/value-pairs.c:381
No locals.
#7  0x00007f221a86e716 in lua_dd_inject_all_global_variables (state=0x8193f0, globals=0x817a20) at modules/lua/lua-dest.c:193
No locals.
#8  0x00007f221a86e7ed in lua_dd_init (s=0x8192b0) at modules/lua/lua-dest.c:218
---Type <return> to continue, or q <return> to quit---
        self = 0x8192b0
        cfg = 0x70cb50
#9  0x00007f2220085b84 in log_pipe_init (s=0x8192b0, cfg=0x70cb50) at lib/logpipe.h:253
No locals.
#10 0x00007f22200878f6 in cfg_tree_start (self=0x70ccc0) at lib/cfg-tree.c:1072
        i = 25
#11 0x00007f2220081695 in cfg_init (cfg=0x70cb50) at lib/cfg.c:218
        regerr = 0
#12 0x00007f222009df0a in main_loop_initialize_state (cfg=0x70cb50, persist_filename=0x7f22200d8000 "/var/lib/syslog-ng/syslog-ng.persist")
    at lib/mainloop.c:527
        success = 0
#13 0x00007f222009e4d3 in main_loop_init () at lib/mainloop.c:739
No locals.
#14 0x0000000000401845 in main (argc=1, argv=0x7fffac1a51b8) at syslog-ng/main.c:246
        rc = 0
        ctx = 0x7032e0
        error = 0x0

Ideas for a Riak destination

I've been researching Riak, during which it occurred to me, it would make an excellent log store. What I envision, is a way to push log into buckets, into keys that use the Set Data Type. Values can just be JSON.

Something like this:

riak(host("localhost") port(8089)
       bucket("log_${HOST}")
       key("logs-${YEAR}-${MONTH}-${DAY}")
       value("$(format-json --scope rfc5424)"));

And that's about it. Riak can be talked to over Protocol Buffers, you don't need much more than connect + serialize + update. The harder part is making sure that the bucket has the appropriate data type property set, that may need some head scratching. Alternatively, if we do not allow templates in bucket(), and assume that the bucket already exists and is set up properly, that works too.

CEF parser request

One of my client wants to parse messages in CEF format, a solution would be neccassary for him. :)
Thanks!
Jagresz

java destination can't be disabled

While trying to package 0.4.0, without JDK in the build chroot, configure enables Java support. Compilation obviously fails:

[ 13s] GEN modules/java/java-grammar.y
[ 13s] YACC modules/java/java-grammar.c
[ 13s] d ./modules/java ./modules/java/SyslogNgClassLoader.java
[ 13s] make: d: Command not found
[ 13s] make: [modules/java/org/syslog_ng/SyslogNgDestination.class] Error 127 (ignored)
[ 13s] d ./modules/java ./modules/java/SyslogNg.java
[ 13s] make: d: Command not found
[ 13s] make: [modules/java/org/syslog_ng/SyslogNgDestination.class] Error 127 (ignored)
[ 13s] d ./modules/java -cp ./modules/java ./modules/java/SyslogNgDestination.java
[ 13s] make: d: Command not found
[ 13s] make: [modules/java/org/syslog_ng/SyslogNgDestination.class] Error 127 (ignored)
[ 13s] jar -cvf ./modules/java/SyslogNg.jar -C ./modules/java/ org
[ 13s] make: jar: Command not found
[ 13s] make: *** [modules/java/org/syslog_ng/SyslogNgDestination.class] Error 127

On --disable-java:

[ 5s] GEN modules/java/java-grammar.y
[ 5s] YACC modules/java/java-grammar.c
[ 5s] make: *** No rule to make target modules/java/SyslogNg.h', needed byall'. Stop.
[ 5s] error: Bad exit status from /var/tmp/rpm-tmp.7jm9h9 (%build)

Test suite

The Incubator could really use a test suite.

Question: cron callback?

I'm using the perl destination to stage events to Elasticsearch using Search::Elasticsearch::Bulk.
This works very well and I'm using the auto-flushing mechanism to batch packets of messages.
The obvious problem with that approach is latency for low traffic sources, and I was thinking about adding a timeout to force the flush. Do you think adding a periodic callback would be a sane solution, e.g. periodic-func("periodic" timeout(60s))? Or should I rather use the trigger source?

systemd

0.4.0 compiles fine on openSUSE 13.1, but the same source package fails on openSUSE 13.2 with the following error:

[ 43s] CC modules/lua/modules_lua_libluautil_la-lua-template.lo
[ 44s] CCLD modules/lua/libluautil.la
[ 44s] CCLD modules/monitor-source/libmonitor-source.la
[ 44s] /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lsystemd
[ 44s] collect2: error: ld returned 1 exit status
[ 44s] Makefile:1497: recipe for target 'modules/monitor-source/libmonitor-source.la' failed
[ 44s] make[1]: *** [modules/monitor-source/libmonitor-source.la] Error 1
[ 44s] make[1]: Leaving directory '/home/abuild/rpmbuild/BUILD/syslog-ng-incubator-0.4.0'
[ 44s] Makefile:1239: recipe for target 'all' failed
[ 44s] make: *** [all] Error 2
[ 44s] error: Bad exit status from /var/tmp/rpm-tmp.l9j5Re (%build)

grok compile warning

After compiling incubator with grok support on openSUSE, I get the following error message from post build checks (which fails the package building process):

[   23s] E: syslog-ng-incubator 64bit-portability-issue modules/grok/grok-parser-grammar.y:400

The offending messages from the build logs are:

[   19s]   CC       modules/grok/modules_grok_libgrok_parser_la-grok-parser-grammar.lo
[   19s] modules/grok/grok-parser-grammar.y: In function 'grok_parse':
[   19s] modules/grok/grok-parser-grammar.y:400:5: warning: implicit declaration of function 'grok_instance_new' [-Wimplicit-function-declaration]
[   19s]          : KW_GROK_MATCH { last_grok_instance = grok_instance_new(); }  '(' string grok_instance_options ')' { grok_instance_set_pattern(last_grok_instance, $4); free($4); }
[   19s]      ^
[   19s] modules/grok/grok-parser-grammar.y:400:26: warning: assignment makes pointer from integer without a cast [enabled by default]
[   19s]          : KW_GROK_MATCH { last_grok_instance = grok_instance_new(); }  '(' string grok_instance_options ')' { grok_instance_set_pattern(last_grok_instance, $4); free($4); }
[   19s]                           ^
[   19s]   CC       modules/grok/modules_grok_libgrok_parser_la-grok-parser.lo

grok parser

test issue for trello

Enhancement: date parser

One of my users needs to manipulate dates. I think date parsing is currently one of the missing pieces and very handy for instance in logstash. Something in the likes of

$(format-date --in "%Y%m%d-%H:%M:%S" --out "ISO8601" <date>)

tfgetent does not compile on FreeBSD

tfgetent is found by configure:

tfgetent yes

But compilation fails:

/bin/sh ./libtool --tag=CC --mode=compile cc -DHAVE_CONFIG_H -I. -I/usr/local/include/syslog-ng -I/usr/local/include/glib-2.0 -I/usr/local/include -I/usr/local/include/eventlog -I./modules/getent -O2 -pipe -fno-strict-aliasing -MT modules/getent/modules_getent_libtfgetent_la-tfgetent.lo -MD -MP -MF modules/getent/.deps/modules_getent_libtfgetent_la-tfgetent.Tpo -c -o modules/getent/modules_getent_libtfgetent_la-tfgetent.lo test -f 'modules/getent/tfgetent.c' || echo './'modules/getent/tfgetent.c
libtool: compile: cc -DHAVE_CONFIG_H -I. -I/usr/local/include/syslog-ng -I/usr/local/include/glib-2.0 -I/usr/local/include -I/usr/local/include/eventlog -I./modules/getent -O2 -pipe -fno-strict-aliasing -MT modules/getent/modules_getent_libtfgetent_la-tfgetent.lo -MD -MP -MF modules/getent/.deps/modules_getent_libtfgetent_la-tfgetent.Tpo -c modules/getent/tfgetent.c -fPIC -DPIC -o modules/getent/.libs/modules_getent_libtfgetent_la-tfgetent.o
modules/getent/tfgetent.c:36:10: fatal error: 'features.h' file not found

include <features.h>

1 error generated.
gmake[2]: *** [modules/getent/modules_getent_libtfgetent_la-tfgetent.lo] Error 1
gmake[2]: Leaving directory /root/syslog-ng-incubator/work/syslog-ng-incubator-0.3.1' gmake[1]: *** [all] Error 2 gmake[1]: Leaving directory/root/syslog-ng-incubator/work/syslog-ng-incubator-0.3.1'
*** Error code 1

casting errors for floats

The messages still make it to riemann though

syslog-ng config snippet:

destination d_riemann {
  riemann(
    server("...")
    port(...)
    metric("${appacct.metric}")
    ...
  )
}

error examples in log:

Casting error; value='0.498863', type-hint='float'
Casting error; value='1017.625511', type-hint='float'
Casting error; value='0.489130', type-hint='float'

there are traces of graphite in 0.4.0 sources

0.4.0 should not have anything related to graphite in it, as this feature is already merged into syslog-ng 3.6

linux-3f8h:~/home:czanik:syslog-ng36/syslog-ng-incubator # xzcat syslog-ng-incubator-0.4.0.tar.xz | tar tvf - | grep -i graph
drwxr-xr-x 0/0 0 2014-12-10 15:49 syslog-ng-incubator-0.4.0/scl/graphite/
-rw-rw-r-- 1000/1000 1235 2014-11-04 09:56 syslog-ng-incubator-0.4.0/scl/graphite/plugin.conf
-rw-rw-r-- 1000/1000 137 2014-11-04 09:56 syslog-ng-incubator-0.4.0/scl/graphite/Makefile.am
-rw-rw-r-- 1000/1000 288 2014-11-04 09:56 syslog-ng-incubator-0.4.0/scl/graphite/graphite-example.conf

perl: segfault

I'm getting a segfault when trying to push data to Elasticsearch using the perl plugin.

elasticsearch.pm

#!/usr/bin/perl

use warnings;
use strict;
use Search::Elasticsearch;
use Data::Dumper;

my $es;
my $bulk;

sub init {
        $es = Search::Elasticsearch -> new();

        $bulk = $es -> bulk_helper(
                index => 'perl',
                type => 'perl',
        );
}

sub queue {
        my ($data) = @_;
        print Dumper $data;
        $bulk -> create_docs(
                $data
        );
}

sub deinit {
        $bulk -> flush
}

syslog-ng -Fedv -f /etc/syslog-ng/syslog-ng.conf.es

Starting to read include file; filename='/etc/syslog-ng/scl.conf', depth='1'
Global value changed; define='scl-root', value='/usr/share/syslog-ng/include/scl'
Global value changed; define='include-path', value='/etc/syslog-ng:/usr/share/syslog-ng/include'
Starting to read include file; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2'
Module loaded and initialized successfully; module='system-source'
Finishing include; filename='/usr/share/syslog-ng/include/scl/system/plugin.conf', depth='2'
Starting to read include file; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2'
Reading path for candidate modules; path='/lib64/syslog-ng'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='afstomp.so', module='afstomp'
Registering candidate plugin; module='afstomp', context='destination', name='stomp', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='graphite.so', module='graphite'
Registering candidate plugin; module='graphite', context='template-func', name='graphite_output', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='mod-perl.so', module='mod-perl'
Registering candidate plugin; module='mod-perl', context='destination', name='perl', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='syslog-ng-crypto.so', module='syslog-ng-crypto'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='dbparser.so', module='dbparser'
Registering candidate plugin; module='dbparser', context='parser', name='db-parser', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='basicfuncs.so', module='basicfuncs'
Registering candidate plugin; module='basicfuncs', context='template-func', name='grep', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='if', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='echo', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='length', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='substr', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='strip', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='sanitize', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='lowercase', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='uppercase', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='replace-delimiter', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='+', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='-', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='*', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='/', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='%', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='ipv4-to-int', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='indent-multi-line', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='context-length', preference='0'
Registering candidate plugin; module='basicfuncs', context='template-func', name='env', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='afsocket-tls.so', module='afsocket-tls'
Registering candidate plugin; module='afsocket-tls', context='source', name='unix-stream', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-stream', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='unix-dgram', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='unix-dgram', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='tcp', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='tcp6', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='tcp6', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='udp', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='udp', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='udp6', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='udp6', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='syslog', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='syslog', preference='100'
Registering candidate plugin; module='afsocket-tls', context='source', name='network', preference='100'
Registering candidate plugin; module='afsocket-tls', context='destination', name='network', preference='100'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='lua.so', module='lua'
Registering candidate plugin; module='lua', context='destination', name='lua', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='afsocket.so', module='afsocket'
Registering candidate plugin; module='afsocket', context='source', name='unix-stream', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='unix-stream', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='unix-dgram', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='unix-dgram', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='tcp', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='tcp', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='tcp6', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='tcp6', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='udp', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='udp', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='udp6', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='udp6', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='syslog', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='syslog', preference='100'
Registering candidate plugin; module='afsocket', context='source', name='network', preference='100'
Registering candidate plugin; module='afsocket', context='destination', name='network', preference='100'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='affile.so', module='affile'
Registering candidate plugin; module='affile', context='source', name='file', preference='0'
Registering candidate plugin; module='affile', context='source', name='pipe', preference='0'
Registering candidate plugin; module='affile', context='destination', name='file', preference='0'
Registering candidate plugin; module='affile', context='destination', name='pipe', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='linux-kmsg-format.so', module='linux-kmsg-format'
Registering candidate plugin; module='linux-kmsg-format', context='format', name='linux-kmsg', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='basicfuncs-plus.so', module='basicfuncs-plus'
Registering candidate plugin; module='basicfuncs-plus', context='template-func', name='//', preference='0'
Registering candidate plugin; module='basicfuncs-plus', context='template-func', name='or', preference='0'
Registering candidate plugin; module='basicfuncs-plus', context='template-func', name='padding', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='mod-python.so', module='mod-python'
Registering candidate plugin; module='mod-python', context='destination', name='python', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='rss.so', module='rss'
Registering candidate plugin; module='rss', context='destination', name='rss', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='afuser.so', module='afuser'
Registering candidate plugin; module='afuser', context='destination', name='usertty', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='trigger-source.so', module='trigger-source'
Registering candidate plugin; module='trigger-source', context='source', name='trigger', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='csvparser.so', module='csvparser'
Registering candidate plugin; module='csvparser', context='parser', name='csv-parser', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='monitor-source.so', module='monitor-source'
Registering candidate plugin; module='monitor-source', context='source', name='monitor', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='system-source.so', module='system-source'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='tfgeoip.so', module='tfgeoip'
Registering candidate plugin; module='tfgeoip', context='template-func', name='geoip', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='afsocket-notls.so', module='afsocket-notls'
Registering candidate plugin; module='afsocket-notls', context='source', name='unix-stream', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-stream', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='unix-dgram', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='unix-dgram', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='tcp', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='tcp6', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='tcp6', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='udp', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='udp', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='udp6', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='udp6', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='syslog', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='syslog', preference='0'
Registering candidate plugin; module='afsocket-notls', context='source', name='network', preference='0'
Registering candidate plugin; module='afsocket-notls', context='destination', name='network', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='tfgetent.so', module='tfgetent'
Registering candidate plugin; module='tfgetent', context='template-func', name='getent', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='pacctformat.so', module='pacctformat'
Registering candidate plugin; module='pacctformat', context='format', name='pacct', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='afmongodb.so', module='afmongodb'
Registering candidate plugin; module='afmongodb', context='destination', name='mongodb', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='json-plugin.so', module='json-plugin'
Registering candidate plugin; module='json-plugin', context='parser', name='json-parser', preference='0'
Registering candidate plugin; module='json-plugin', context='template-func', name='format_json', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='confgen.so', module='confgen'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='afamqp.so', module='afamqp'
Registering candidate plugin; module='afamqp', context='destination', name='amqp', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='afprog.so', module='afprog'
Registering candidate plugin; module='afprog', context='source', name='program', preference='0'
Registering candidate plugin; module='afprog', context='destination', name='program', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='cryptofuncs.so', module='cryptofuncs'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='uuid', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='hash', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha1', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha256', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='sha512', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='md4', preference='0'
Registering candidate plugin; module='cryptofuncs', context='template-func', name='md5', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='riemann.so', module='riemann'
Registering candidate plugin; module='riemann', context='destination', name='riemann', preference='0'
Reading shared object for a candidate module; path='/lib64/syslog-ng', fname='syslogformat.so', module='syslogformat'
Registering candidate plugin; module='syslogformat', context='format', name='syslog', preference='0'
Registering candidate plugin; module='syslogformat', context='parser', name='syslog-parser', preference='0'
Finishing include; filename='/usr/share/syslog-ng/include/scl/pacct/plugin.conf', depth='2'
Starting to read include file; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2'
Module loaded and initialized successfully; module='confgen'
Finishing include; filename='/usr/share/syslog-ng/include/scl/syslogconf/plugin.conf', depth='2'
Starting to read include file; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2'
Finishing include; filename='/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2'
Finishing include; filename='/etc/syslog-ng/scl.conf', depth='1'
Module loaded and initialized successfully; module='dbparser'
Module loaded and initialized successfully; module='csvparser'
Module loaded and initialized successfully; module='json-plugin'
Module loaded and initialized successfully; module='afsocket-tls'
Module loaded and initialized successfully; module='affile'
Module loaded and initialized successfully; module='tfgetent'
Module loaded and initialized successfully; module='tfgeoip'
Module loaded and initialized successfully; module='mod-perl'
Finishing include; content='destination block elasticsearch_perl', depth='1'
Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf.e]
  Compiling s_internal reference [source] at [/etc/syslog-ng/syslog-ng.conf.e]
    Compiling s_internal sequence [source] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf.e]
        Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf.e]
  Compiling d_rsyslog reference [destination] at [/etc/syslog-ng/syslog-ng.conf.e]
    Compiling d_rsyslog sequence [destination] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf.e]
        Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf.e]
Compiling #unnamed sequence [log] at [/etc/syslog-ng/syslog-ng.conf.e]
  Compiling s_netflow_labo1 reference [source] at [/etc/syslog-ng/syslog-ng.conf.e]
    Compiling s_netflow_labo1 sequence [source] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf.e]
        Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf.e]
  Compiling p_csv reference [parser] at [/etc/syslog-ng/syslog-ng.conf.e]
    Compiling p_csv sequence [parser] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf.e]
  Compiling r_enrich reference [rewrite] at [/etc/syslog-ng/syslog-ng.conf.e]
    Compiling r_enrich sequence [rewrite] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf.e]
  Compiling d_all_fifo reference [destination] at [/etc/syslog-ng/syslog-ng.conf.e]
    Compiling d_all_fifo sequence [destination] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf.e]
        Compiling #unnamed single [log] at [/etc/syslog-ng/syslog-ng.conf.e]
  Compiling d_elasticsearch reference [destination] at [/etc/syslog-ng/syslog-ng.conf.e]
    Compiling d_elasticsearch sequence [destination] at [/etc/syslog-ng/syslog-ng.conf.e]
      Compiling #unnamed junction [log] at [/etc/syslog-ng/syslog-ng.conf.e]
        Compiling #unnamed single [log] at [#buffer:2:3]
Syslog connection established; fd='7', server='AF_INET(127.0.0.1:514)', local='AF_INET(0.0.0.0:0)'
Module loaded and initialized successfully; module='syslogformat'
Initializing Perl destination; driver='d_elasticsearch#0', script='/var/tmp/elasticsearch.pm'
Running application hooks; hook='1'
Running application hooks; hook='3'
syslog-ng starting up; version='3.5.4.1'
Worker thread started; driver='d_elasticsearch#0'
Incoming log entry; line='cycle proto   ipLoc   dir     ipExt   ptLoc   ptExt   tcpFlg  incTraf outTraf incPkts outPkts FstTime LstTime'
Message parsing complete; result='1', rule='p_csv', location='/etc/syslog-ng/syslog-ng.conf.es:35:3'
Rewrite expression evaluation result; value='flowevt.svcLoc', new_value='', rule='r_enrich', location='/etc/syslog-ng/syslog-ng.conf.es:59:3'
Rewrite expression evaluation result; value='flowevt.svcExt', new_value='', rule='r_enrich', location='/etc/syslog-ng/syslog-ng.conf.es:60:3'
Rewrite expression evaluation result; value='flowevt.geoipLoc', new_value='', rule='r_enrich', location='/etc/syslog-ng/syslog-ng.conf.es:61:3'
Rewrite expression evaluation result; value='flowevt.geoipExt', new_value='', rule='r_enrich', location='/etc/syslog-ng/syslog-ng.conf.es:62:3'
Initializing destination file writer; template='/var/tmp/syslog-ng.fifo', filename='/var/tmp/syslog-ng.fifo'
Incoming log entry; line='00:15:00      6       134.158.106.105 <       130.246.219.148 1095    53426   24      60632   1445844 807    815      23:59:58        00:00:00'
Message parsing complete; result='1', rule='p_csv', location='/etc/syslog-ng/syslog-ng.conf.es:35:3'
$VAR1 = {
          'HOST' => 'cctest38',
          'PRIORITY' => 'notice',
          'DATE' => 'Jun 10 11:16:05',
          'FACILITY' => 'user',
          'MESSAGE' => 'cycle   proto   ipLoc   dir     ipExt   ptLoc   ptExt   tcpFlg  incTraf outTraf incPkts outPkts FstTime LstTime'
        };
Rewrite expression evaluation result; value='flowevt.svcLoc', new_value='', rule='r_enrich', location='/etc/syslog-ng/syslog-ng.conf.es:59:3'
Rewrite expression evaluation result; value='flowevt.svcExt', new_value='', rule='r_enrich', location='/etc/syslog-ng/syslog-ng.conf.es:60:3'
Rewrite expression evaluation result; value='flowevt.geoipLoc', new_value='FR', rule='r_enrich', location='/etc/syslog-ng/syslog-ng.conf.es:61:3'
Rewrite expression evaluation result; value='flowevt.geoipExt', new_value='GB', rule='r_enrich', location='/etc/syslog-ng/syslog-ng.conf.es:62:3'
Incoming log entry; line='00:15:00      6       193.48.99.122   >       130.199.149.37  1094    41082   27      149212  22692888       2869     7676    23:59:55        00:00:02'
Message parsing complete; result='1', rule='p_csv', location='/etc/syslog-ng/syslog-ng.conf.es:35:3'
Rewrite expression evaluation result; value='flowevt.svcLoc', new_value='', rule='r_enrich', location='/etc/syslog-ng/syslog-ng.conf.es:59:3'
Segmentation fault (core dumped)

Core was generated by `syslog-ng -Fedv -f /etc/syslog-ng/syslog-ng.conf.es'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007ffa2f2f0bdf in XS_Scalar__Util_weaken (my_perl=<value optimized out>, cv=0xc7a730) at ListUtil.c:659
659         dVAR; dXSARGS;
Missing separate debuginfos, use: debuginfo-install GeoIP-1.4.8-1.el6.x86_64 eventlog-0.2.13-1.el6.x86_64 ivykis-0.36.2-1.el6.x86_64 json-c-0.10-2.el6.x86_64 libnet-1.1.6-7.el6.x86_64 perl-Package-Stash-XS-0.25-1.el6.x86_64 perl-Sub-Name-0.05-6.el6.x86_64 perl-Variable-Magic-0.45-0.el6.x86_64
(gdb) bt full
#0  0x00007ffa2f2f0bdf in XS_Scalar__Util_weaken (my_perl=<value optimized out>, cv=0xc7a730) at ListUtil.c:659
        sp = <value optimized out>
        ax = <value optimized out>
        mark = <value optimized out>
#1  0x00007ffa304ef815 in Perl_pp_entersub (my_perl=0xa45000) at pp_hot.c:2888
        markix = 1
        sp = <value optimized out>
        sv = 0xe8c360
        gv = 0xa45000
        cv = 0xc7a730
        cx = <value optimized out>
        gimme = 128
        hasargs = <value optimized out>
#2  0x00007ffa304edb06 in Perl_runops_standard (my_perl=0xa45000) at run.c:40
No locals.
#3  0x00007ffa304955df in Perl_call_sv (my_perl=0xa45000, sv=0xe504f0, flags=4) at perl.c:2721
        sp = <value optimized out>
        myop = {op_next = 0x0, op_sibling = 0x0, op_ppaddr = 0, op_targ = 0, op_type = 0, op_opt = 0, op_latefree = 0, op_latefreed =
    0, op_attached = 0, op_spare = 0, op_flags = 66 'B', op_private = 0 '

zmq compile warnings

When packaging 0.4.0 on openSUSE and enable zmq support, the compilation succeeds with many warnings. On the other hand post build scripts fail with the following error:

[ 22s] E: syslog-ng-incubator 64bit-portability-issue modules/zmq/zmq-grammar.y:379

The related compilation log is:

[ 18s] CC modules/zmq/modules_zmq_libzmq_la-zmq-grammar.lo
[ 18s] modules/zmq/zmq-grammar.y: In function 'zmq_parse':
[ 18s] modules/zmq/zmq-grammar.y:379:13: warning: implicit declaration of function 'zmq_sd_new' [-Wimplicit-function-declaration]
[ 18s] last_driver = *instance = zmq_sd_new(configuration);
[ 18s] ^
[ 18s] modules/zmq/zmq-grammar.y:379:37: warning: assignment makes pointer from integer without a cast [enabled by default]
[ 18s] last_driver = *instance = zmq_sd_new(configuration);
[ 18s] ^
[ 18s] modules/zmq/zmq-grammar.y:420:13: warning: implicit declaration of function 'zmq_sd_set_address' [-Wimplicit-function-declaration]
[ 18s] zmq_sd_set_address(last_driver, $3);
[ 18s] ^
[ 18s] modules/zmq/zmq-grammar.y:426:13: warning: implicit declaration of function 'zmq_sd_set_port' [-Wimplicit-function-declaration]
[ 18s] zmq_sd_set_port(last_driver, $3);
[ 18s] ^
[ 18s] CC modules/zmq/modules_zmq_libzmq_la-zmq-plugin.lo
[ 19s] CC modules/zmq/modules_zmq_libzmq_la-zmq-destination.lo
[ 19s] modules/zmq/zmq-destination.c: In function 'zmq_worker_insert':
[ 19s] modules/zmq/zmq-destination.c:151:12: warning: unused variable 'success' [-Wunused-variable]
[ 19s] gboolean success = TRUE;
[ 19s] ^
[ 19s] CC modules/zmq/modules_zmq_libzmq_la-zmq-source.lo
[ 19s] CC modules/zmq/modules_zmq_libzmq_la-zmq-parser.lo
[ 19s] CC modules/zmq/modules_zmq_libzmq_la-zmq-transport.lo
[ 19s] modules/zmq/zmq-transport.c: In function 'log_transport_zmq_new':
[ 19s] modules/zmq/zmq-transport.c:64:20: warning: assignment from incompatible pointer type [enabled by default]
[ 19s] self->super.read = log_transport_zmq_read_method;
[ 19s] ^
[ 19s] CCLD modules/zmq/libzmq.la

perl segfault if script not found

syslog-ng segfaults when it can't find the script for the perl module.
The ERRNO is hidden as it gets spit on STDERR.

See syslog-ng/syslog-ng/issues/221

graphite: Can we have an SCL?

It would be really neat to have a graphite() SCL, that embeds a graphite-output template function in a by-default well argumented network() destination.

rss destination requires glibc-2.28

When running syslog-ng, we get:

Error opening plugin module; module='rss', error='/usr/lib/syslog-ng/librss.so: undefined symbol: g_list_free_full'

This seems to be due to the fact that g_list_free_full() was introduced into glibc 2.28
and my test server only has 2.26.

riemann module segfaults upon startup (syslog-ng -s suffices

syslog-ng -s
[...]
Core was generated by `/usr/sbin/syslog-ng -s -p /var/run/syslog-ng.pid --no-caps'. Program terminated with signal 6, Aborted.
#0  0x00007f9d48681925 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);

I can submit core file of course

Enhancement: NS template function

I have a use case where I need to resolve UID and GID in a log message.
It would be awesome if it were possible to resolve name-service entries.

Proposed syntax:

$(getent passwd ${usracct.uid})
$(getent hosts ${flowevt.src_ip})
$(getent services ${appacct.svc_id})

java destination on RHEL6 (CentOS6)

When I try to compile 0.4.1 on CentOS6 with OpenJDK 1.7, I get the following error during compilation:

  CC       modules/java/modules_java_libmod_java_la-java-grammar.lo
  CC       modules/java/modules_java_libmod_java_la-java-plugin.lo
  CC       modules/java/modules_java_libmod_java_la-java-destination.lo
  CC       modules/java/modules_java_libmod_java_la-java-destination-proxy.lo
modules/java/java-destination-proxy.c:59: error: redefinition of typedef 'JavaDestinationProxy'
modules/java/java-destination-proxy.h:30: note: previous declaration of 'JavaDestinationProxy' was here
make[1]: Leaving directory `/builddir/build/BUILD/syslog-ng-incubator-0.4.1'
make[1]: *** [modules/java/modules_java_libmod_java_la-java-destination-proxy.lo] Error 1

Question: java destination LD_LIBRARY_PATH

is it possible to make the driver load the library automatically without having to set LD_LIBRARY_PATH?

basicfuncs-plus requires glib 2.32+

Due to GRWLock, basicfuncs-plus requires glib 2.32+. The configure script should check for that, and disable the module if an older glib is found.

(Via Peter Czanik)

perl: unclean shutdown

When shutting down syslog-ng, it seems the perl plugin is not shutting down cleanly:

# syslog-ng -Fv
^C
Scalars leaked: -8
Attempt to free non-existent shared string '_array_iterator', Perl interpreter: 0x7f18bc0008c0 during global destruction.
Attempt to free non-existent shared string '_natatime_iterator', Perl interpreter: 0x7f18bc0008c0 during global destruction.
Attempt to free non-existent shared string '_XScompiled', Perl interpreter: 0x7f18bc0008c0 during global destruction.
Attempt to free non-existent shared string '/usr/lib64/perl5/List/MoreUtils.pm', Perl interpreter: 0x7f18bc0008c0 during global destruction.
Scalars leaked: -142
Attempt to free unreferenced scalar: SV 0x7f18bc10ffe8, Perl interpreter: 0x7f18b40008c0 during global destruction.
Unbalanced string table refcount: (1) for "_XScompiled" during global destruction.
Unbalanced string table refcount: (1) for "_natatime_iterator" during global destruction.
Unbalanced string table refcount: (1) for "_array_iterator" during global destruction.
Unbalanced string table refcount: (1) for "/usr/lib64/perl5/List/MoreUtils.pm" during global destruction.
Scalars leaked: 150

perl/lua/python destinations: handle errors

It would be awesome if we could have some way to handle errors.
For instance for perl, we could have new callbacks on_error which would fire up in case of a non-true return value, with the callback function as argument. By default this callback could be to throw an error. Then syslog-ng could restart the process after a configurable time. Of course if flow-control is on it would be nice not to lose messages during that time.

Enhancement: perl integration

My company developed many perl modules to integrate with its configuration management and monitoring. Given the lua destination module, I would love to see a similar perl integration!

BTW: congratulations for the enlightening incubator extremely useful modules like riemann and lua

multiple java destinations java.lang.UnsatisfiedLinkError

It is apparently not possible to use multiple java destinations using the same class:

Exception in thread "main" java.lang.UnsatisfiedLinkError: Native Library /usr/lib64/syslog-ng/libmod-java.so already loaded in another classloader
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1895)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1847)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1119)
        at org.syslog_ng.SyslogNg.<clinit>(SyslogNg.java:28)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:344)
        at org.syslog_ng.SyslogNgClassLoader.loadClass(SyslogNgClassLoader.java:57)

getent: the module is slow

How could the performance be improved?
When disabling the function, I get a 100x increase in performance

perl segfault only when multiple destinations

Replacing old ill-defined issue.
I'm hitting a segfault when using multiple perl destinations, namely the same perl script multiple times.
To reproduce: have two identical perl destination blocks with different names, and have a log path pushing to both. Afer some time there's a segfault

segfault at 7ff4b8f222c0 ip 00007ff4b7cb8442 sp 00007ff4b8f222c0 error 6 in libevtlog.so.0.0.0[7ff4b7cb6000+4000]

The download badge show pre-releases too

The download badge should show the last release that is NOT a pre-release. The link should point towards the same release, too.

--enable / --disable configure switches

From the packager point of view, --enable-xxx and --disable-xxx configure switches are very useful, as features are not accidentally changed in a package. Please use these for incubator features!

Moving modules to syslog-ng 3.6?

I think a few modules could graduate from the Incubator to syslog-ng 3.6. Namely riemann, trigger-source, graphite, and perhaps the $(or), $(//) and $(padding) template functions.

What do you think?

Built grammar files are not portable

The grammar files built by the Incubator are not portable, therefore they should not be included in the make dist tarball: having grammar files built with 3.5 will make the compile fail with 3.6 and vice versa.

Instead, these files should always be rebuilt, and documentation updated to have bison and flex installed too.

(Via Peter Czanik)

getent: handle errors

I'm stumbling on the following error message for large groups:

syslog-ng[9522]: $(getent group) failed; key='124', errno='Numerical result out of range (34)'

Found some references:

Enhancement: printf template function

I have a client who wants to concatenate two macros $prefix and $suffix while padding the second to a fixed number of digits, e.g. foo and 52 should yield foo0052, while bar and 0 should yield bar0000.

It seems natural to me to solve this more generally by implementing a printf-like template function, e.g.:

${prefix}$(printf "%d" "${suffix}")

lua: trouble getting ES destination to work

I tried the shipped example to no avail.
Basically when activating the log statement, I get a defunct syslog-ng process. A kill -HUP after commenting the destination gets the server up and running again.
Here's my relevant information:

lua:

# rpm -qa lua\*
lua-5.1.4-4.1.el6.x86_64
lua-socket-2.0.2-4.el6.x86_64

/usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf:

block destination elasticsearch(
  host("localhost") port(9200)
  index("syslog-ng") type("message")
  body("$(format_json --scope nv_pairs --key PROGRAM --pair @timestamp=\"${R_ISODATE}\" --pair @message=\"${MSG}\")")
){
  lua(
    script("/usr/share/syslog-ng/include/scl/elasticsearch.lua")
    template("`body`\n")
    init-func("elastic_init")
    queue-func("elastic_queue")
    globals(
      es_batch_size(int("100"))
      es_host("`host`")
      es_port("`port`")
      es_index("`index`")
      es_type("`type`")
    )
  );
};

/etc/syslog-ng/syslog-ng.conf:

destination d_elasticsearch {
  elasticsearch(index("test"));
};

Error parsing kafka, Error compiling template (Unknown template function "format-json")

Hello, I have installed librdkafka library and syslog-ng-incubator with the kafka module:

git clone git://github.com/balabit/syslog-ng-incubator.git
cd syslog-ng-incubator
autoreconf -i
./configure --with-librdkafka=/usr/local/include/
make && make install

But when I configure my syslog-ng.conf like this :

source s_system {
  system();
};
destination d_kafka {
  kafka(properties(metadata.broker.list("localhost:9092"))
        topic("syslogng")
        payload("$(format-json --scope all-nv-pairs --scope core)")
        partition("$PROGRAM")
  );
};
log {
  source(s_system);
  destination(d_kafka);
};

It failed:

[root@syslog-ng librdkafka-0.8.5]# /etc/init.d/syslog-ng restart
Stopping syslog-ng:                                        [FAILED]
Error parsing kafka, Error compiling template (Unknown template function "format-json") in /etc/syslog-ng/syslog-ng.conf at line 76, column 17:

        payload("$(format-json --scope all-nv-pairs --scope core)")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng

I don't know what to do at this point and I can't find somebody having the same problem.
Any help, or some direction would be much appreciated.

Enhancement: perl: allow to pass parameters to callbacks

It would feel more natural (IMHO) to pass parameters to callbacks instead of going through value-pairs, maybe something like:

perl(
  script("elasticsearch.pm")
  init-func("init", pair("es_bulk_size", 100))
  queue-func("queue", pair("es_index", "syslog-$YEAR.$MONTH.$DAY"), pair("es_type", "syslog"))
  deinit-func("deinit")
);

or alternatively

perl(
  script(...)
  queue-func("init")
  queue-param(pair(...))
);

typo in the example config of python destination

In the example config (python-example.conf) there is:

script("python_example")

but the name of the python script is: "python-example.py"

python: crash on message containing WINDOWS-1252 character 0x92

It looks like the crash happens here:

syslog-ng-incubator/modules/python/python-dest.c

Line 249 in 54179c5

PyDict_SetItemString(dict, name, PyUnicode_FromString(value));

(although syslog-ng provides no output, and simply shuts down unexpectedly)

I haven't dug into the code deeply enough to know whether syslog-ng has already converted utf-8 to ascii on the input-side, but if it attempted to, it left some non-ascii characters in the message which blow up here.

I think there are 2 bugs worth addressing here:

syslog-ng should probably sanitize all non-ascii input characters, even if they are not valid utf-8. [1]
syslog-ng-mod-python should probably have much more error handling code since thrown exceptions currently kill syslog-ng hard.

[1] yes, users can choose proper character sets for their input and avoid this problem, but syslog-ng should probably protect users to some degree

java destination on FreeBSD

Tested on FreeBSD 10.1 with openjdk 1.7 and 1.8. The results are the same:

Already the output of configure looks strange:

checking for JAVA_VERSION... test: deprecated
"1.7.0_71"
1: bad number
test: deprecated
"1.7.0_71"
1: bad number
deprecated
"1.7.0_71"
1.7

And compilation fails:

GEN modules/java/java-grammar.y
YACC modules/java/java-grammar.c
/usr/local/bin/javavm -d ./modules/java ./modules/java/SyslogNgClassLoader.java
javavm: warning: The use of 'javavm' as a synonym for 'java' is deprecated
Unrecognized option: -d
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Makefile:2919: recipe for target
'modules/java/org/syslog_ng/SyslogNgDestination.class' failed
gmake[1]: *** [modules/java/org/syslog_ng/SyslogNgDestination.class] Error 1
gmake[1]: Leaving directory
'/root/syslog-ng-incubator/work/syslog-ng-incubator-0.4.0'
*** Error code 1

configure really needs an --enable-debug option.

Incubator configure does not have --enable-debug, we should create it like in syslog-ng.

lua: memleak

I've been running the elasticsearch.lua script since yesterday, and syslog-ng has been linearily increasing its RSS value ever since:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6140 root      20   0 17.2g  16g 5624 S  3.6 54.2  36:34.14 syslog-ng
11947 elastics  20   0 40.5g 8.2g 671m S  2.3 26.2 520:33.26 java

Please advise on the procedure to follow to track that down

graphite: Would be nice to make the timestamp configurable

The $(graphite-output) template function currently uses the hard-coded R_UNIXTIME macro to insert a time stamp. It would be nice if it could use something else, but compatible. As far as I see, that would require parsing the options twice, to pick out the graphite-specific options, and leave only the rest for value_pairs_new_from_cmdline().

balabit / syslog-ng-incubator Goto Github PK

syslog-ng-incubator's People

Contributors

Stargazers

Watchers

Forkers

syslog-ng-incubator's Issues

include <features.h>

Recommend Projects

Recommend Topics

Recommend Org