improving signal to noise ratio from centralized network syslogs

Mon Feb 5 02:25:36 UTC 2018

In addition to that, you can use some fancy awk colour coding, so you can make it highlight certain lines based on content.. I use this for my e-mail logs, but I’m sure it could be adapted:

tail -n 1000 -f /var/log/mail-submission.log | grep smtp.*relay | awk '
    /sent/ {print "\033[32m" $0 "\033[39m"}
    /bounced/ {print "\033[31m" $0 "\033[39m"}
    /deferred/ {print "\033[33m" $0 "\033[39m"}
'

> On 4 Feb 2018, at 5:49 am, Scott Weeks <surfer at mauigateway.com> wrote:
> 
> 
> --- jmaimon at jmaimon.com wrote:
> Centralized logging is a good thing. However, 
> what happens is that every repetitive, annoying 
> but not (usually) important thing fills up the 
> log with reams of what you are not looking for.
> ---------------------------------------
> 
> Apologies, I'm late to the party.  But I just 
> want to add one thing for the archives.  It's 
> along with what Rich Kulawiec said, "it forces 
> you to look at your own data, which is really 
> helpful.  You'll be surprised at what you find 
> if you've never done it before."  This is 
> accurate.  It's fun to see what your network 
> is putting out.
> 
> This is all from memory (I've done it so many 
> times it's in there permanently... :-) as I 
> don't have a unix server or a router in front 
> of me to use, so don't hold me to exact 
> details...
> 
> And it's mainly for the newbies.
> 
> Have all the routers send to one syslog file, 
> switches to another and other devices to a 
> third on a *nix box: For example, send the 
> router messages to /var/log/router.log and 
> the switch messages to /var/log/switch.log  
> This is done with the 'logging facility' 
> command on the devices: 
> 
> After defining your syslog server's IP 
> address and the level of messaging you want
> (I set it to debug because I want to see 
> everything):
> 
> on the routers: logging facility local0
> on the switches:  logging facility local1
> 
> on the logging server in:  /etc/rsyslog.conf
> local0.* /var/log/router.log
> local1.* /var/log/switch.log
> 
> Use logrotate to manage the log files as they 
> can get quite large.
> 
> Then, you can watch your network in real time 
> like so (below is all one line):
> 
> tail -f /var/log/router.log /var/log/switch.log 
> | egrep -vi 'term1|term2|termN'
> 
> 'egrep -v' takes out all the lines you don't 
> want to see while the syslog messages scroll
> across the screen.
> 
> Say there is a battery condition on router1 
> and a duplex mismatch on a switch I don't want 
> to see:
> 
> tail -f /var/log/router.log /var/log/switch.log 
> | egrep -vi 'router1.*battery|switch1.*duplex.*mismatch'
> 
> For me, N can get to 40-50 sometimes, so I put 
> it into a shell script like so:
> 
> vi log.sh
> 
> ---------------------------
> #! /bin/sh
> 
> tail -f /var/log/router.log /var/log/switch.log 
> | egrep -v 'term1|term2|termN'
> ---------------------------
> 
> then, run it like so: ./log.sh
> 
> It's all netgeek fun-n-games from there on. :)
> 
> scott