improving signal to noise ratio from centralized network syslogs

Scott Weeks surfer at mauigateway.com
Sat Feb 3 21:49:57 UTC 2018


--- jmaimon at jmaimon.com wrote:
Centralized logging is a good thing. However, 
what happens is that every repetitive, annoying 
but not (usually) important thing fills up the 
log with reams of what you are not looking for.
---------------------------------------

Apologies, I'm late to the party.  But I just 
want to add one thing for the archives.  It's 
along with what Rich Kulawiec said, "it forces 
you to look at your own data, which is really 
helpful.  You'll be surprised at what you find 
if you've never done it before."  This is 
accurate.  It's fun to see what your network 
is putting out.

This is all from memory (I've done it so many 
times it's in there permanently... :-) as I 
don't have a unix server or a router in front 
of me to use, so don't hold me to exact 
details...

And it's mainly for the newbies.

Have all the routers send to one syslog file, 
switches to another and other devices to a 
third on a *nix box: For example, send the 
router messages to /var/log/router.log and 
the switch messages to /var/log/switch.log  
This is done with the 'logging facility' 
command on the devices: 

After defining your syslog server's IP 
address and the level of messaging you want
(I set it to debug because I want to see 
everything):

on the routers: logging facility local0
on the switches:  logging facility local1

on the logging server in:  /etc/rsyslog.conf
local0.* /var/log/router.log
local1.* /var/log/switch.log

Use logrotate to manage the log files as they 
can get quite large.

Then, you can watch your network in real time 
like so (below is all one line):

tail -f /var/log/router.log /var/log/switch.log 
| egrep -vi 'term1|term2|termN'

'egrep -v' takes out all the lines you don't 
want to see while the syslog messages scroll
across the screen.

Say there is a battery condition on router1 
and a duplex mismatch on a switch I don't want 
to see:

tail -f /var/log/router.log /var/log/switch.log 
| egrep -vi 'router1.*battery|switch1.*duplex.*mismatch'

For me, N can get to 40-50 sometimes, so I put 
it into a shell script like so:

vi log.sh

---------------------------
#! /bin/sh

tail -f /var/log/router.log /var/log/switch.log 
| egrep -v 'term1|term2|termN'
---------------------------

then, run it like so: ./log.sh

It's all netgeek fun-n-games from there on. :)

scott



More information about the NANOG mailing list