improving signal to noise ratio from centralized network syslogs
rsk at gsp.org
Wed Jan 31 15:17:46 UTC 2018
On Thu, Jan 25, 2018 at 11:10:02PM -0500, Joe Maimon wrote:
> What I am interested in is an automated zoom-in zoom-out tool to mask the
> repetition of "normal" events and allow the unusual to stand out.
This is an approach outlined by Marcus Ranum years ago; he called it
"artificial stupidity", and it works. (Of course, an inverse check
that makes sure routine boring things are still happening is also
a good idea.)
You could use any number of elaborate (and sometimes expensive) tools
to do this, but I recommend rolling your own with Perl or similar.
This is goodness for two reasons: first, it forces you to look at your
own data, which is really helpful. You'll be surprised at what you
find if you've never done it before. Second, it lets you customize for
your environment at every step.
I have written dozens of these, some as trivial as a few lines of code,
some quite extensive. None of them "solve" the problem per se, they just
all take bites out of it. But this admittedly-simplistic (and deliberately
so) approach has flagged a lot of issues, and because it's simple,
it's easy to connect to other monitoring/alerting plumbing.
More information about the NANOG