Odd syslog-ng problem

Sun May 11 07:36:06 UTC 2014

Peter, it's a bit difficult to tell what's going on without seeing the
rest of the syslog-ng configuration and your script's source code.

However, a couple possibilities come to mind:
- Your script is only reading one line at a time. syslog-ng starts a
program() output persistently and expects that it can send multiple
messages into its pipe to your script's stdin.
- Messages are being buffered inside of syslog-ng. Check out the
flush_lines() and flush_timeout() flags to syslog-ng's program()
output. Find the right page for your version, but here's v3.3.:
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html/reference_destination_program.html
- Messages are being buffered in your shell or script. Maybe try some
non-blocking IO with a smallish buffer to see data as it comes in
before a whole line or block fills and flushes in.

To Anurag's question about open source log management with a WebUI, I
agree with Blake: logstash ingesting syslog and inputting it into
elasticsearch makes for a great backend for Kibana.
The logstash grok filter is great for pulling apart and indexing weird
vendor-specific logging formats:
http://logstash.net/docs/1.4.1/filters/grok

Cheers,
jof

On Sat, May 10, 2014 at 2:24 AM, Peter Persson <webbax at webbax.se> wrote:
> Hey,
>
> I got a weird problem with my syslog-ng setup, im logging from alot of
> cisco machines and that works great.
> The problem is that when i "pass" this further to a shell program, some
> lines disapere.
>
> My destination looks like this
> destination hosts {
>    file("/var/log/ciscorouters/$HOST.log"
>    owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes));
>    program("/scripts/irc/syslog_wrapper_new.sh" template(t_irctempl));
> };
> The "/var/log/ciscorouters/$HOST.log" writes correct, but the data thats
> putted trough to "/scripts/irc/syslog_wrapper_new.sh" only get the first
> line, if it gets flooded (like 5 rows per second).
>
> Do anyone of you have any idea of what might be the problem?
>
> Regards,
> Peter