improving signal to noise ratio from centralized network syslogs
Scott Weeks
surfer at mauigateway.com
Sat Feb 3 21:49:57 UTC 2018
--- jmaimon at jmaimon.com wrote:
Centralized logging is a good thing. However,
what happens is that every repetitive, annoying
but not (usually) important thing fills up the
log with reams of what you are not looking for.
---------------------------------------
Apologies, I'm late to the party. But I just
want to add one thing for the archives. It's
along with what Rich Kulawiec said, "it forces
you to look at your own data, which is really
helpful. You'll be surprised at what you find
if you've never done it before." This is
accurate. It's fun to see what your network
is putting out.
This is all from memory (I've done it so many
times it's in there permanently... :-) as I
don't have a unix server or a router in front
of me to use, so don't hold me to exact
details...
And it's mainly for the newbies.
Have all the routers send to one syslog file,
switches to another and other devices to a
third on a *nix box: For example, send the
router messages to /var/log/router.log and
the switch messages to /var/log/switch.log
This is done with the 'logging facility'
command on the devices:
After defining your syslog server's IP
address and the level of messaging you want
(I set it to debug because I want to see
everything):
on the routers: logging facility local0
on the switches: logging facility local1
on the logging server in: /etc/rsyslog.conf
local0.* /var/log/router.log
local1.* /var/log/switch.log
Use logrotate to manage the log files as they
can get quite large.
Then, you can watch your network in real time
like so (below is all one line):
tail -f /var/log/router.log /var/log/switch.log
| egrep -vi 'term1|term2|termN'
'egrep -v' takes out all the lines you don't
want to see while the syslog messages scroll
across the screen.
Say there is a battery condition on router1
and a duplex mismatch on a switch I don't want
to see:
tail -f /var/log/router.log /var/log/switch.log
| egrep -vi 'router1.*battery|switch1.*duplex.*mismatch'
For me, N can get to 40-50 sometimes, so I put
it into a shell script like so:
vi log.sh
---------------------------
#! /bin/sh
tail -f /var/log/router.log /var/log/switch.log
| egrep -v 'term1|term2|termN'
---------------------------
then, run it like so: ./log.sh
It's all netgeek fun-n-games from there on. :)
scott
More information about the NANOG
mailing list