improving signal to noise ratio from centralized network syslogs

Fri Jan 26 06:01:11 UTC 2018

On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon at jmaimon.com> wrote:

> Hey All,
>
> Centralized logging is a good thing. However, what happens is that every
> repetitive, annoying but not (usually) important thing fills up the log
> with reams of what you are not looking for.
>
> Networks are a noisy place and silencing every logged condition is
> impractical and sometimes undesirable.
>
> What I am interested in is an automated zoom-in zoom-out tool to mask
> the repetition of "normal" events and allow the unusual to stand out.
>
> Add to that an ability to identify gaps in the background noise. (The
> dog that didnt bark)
>
> What I am not interested in are solutions based upon preconfigured
> filters and definitions and built in analysis for supported
> (prepopulated definitions) platforms, this is all about pattern
> mining/masking and should be self discoverable. Ideally a command tool
> to generate static versions of the analysis coupled with a web platform
> (with zoom +- buttons)  for realtime.
>
> I made a crude run of it with SLCT, using its generated patterns to grep
> -v, and that in and of itself was useful, but needs a bit of work. Also,
> its not quite real time.
>
> Any ideas would be greatly appreciated.

Not cheap, but Splunk comes to mind.

>
>
> Joe
>
-- 

"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler