improving signal to noise ratio from centralized network syslogs

Fri Jan 26 11:30:46 UTC 2018

Splunk is the obvious solution that most organizations with a mature
security group will likely already have in their portfolio.

Going a step further, and with an abundance of skill, ability, and
forethought: either ELK (or any derivative there of such as: Elasticache,
Fluentd, Kibana), or rsyslog|syslog-ng + database + loganalzyer.

Grep-fu will pay dividends in any of the three options (do nothing, go
proprietary, go open).

~Steven

On Fri, Jan 26, 2018 at 1:01 AM, Michael Loftis <mloftis at wgops.com> wrote:

> On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon at jmaimon.com> wrote:
>
> > Hey All,
> >
> > Centralized logging is a good thing. However, what happens is that every
> > repetitive, annoying but not (usually) important thing fills up the log
> > with reams of what you are not looking for.
> >
> > Networks are a noisy place and silencing every logged condition is
> > impractical and sometimes undesirable.
> >
> > What I am interested in is an automated zoom-in zoom-out tool to mask
> > the repetition of "normal" events and allow the unusual to stand out.
> >
> > Add to that an ability to identify gaps in the background noise. (The
> > dog that didnt bark)
> >
> > What I am not interested in are solutions based upon preconfigured
> > filters and definitions and built in analysis for supported
> > (prepopulated definitions) platforms, this is all about pattern
> > mining/masking and should be self discoverable. Ideally a command tool
> > to generate static versions of the analysis coupled with a web platform
> > (with zoom +- buttons)  for realtime.
> >
> > I made a crude run of it with SLCT, using its generated patterns to grep
> > -v, and that in and of itself was useful, but needs a bit of work. Also,
> > its not quite real time.
> >
> > Any ideas would be greatly appreciated.
>
>
> Not cheap, but Splunk comes to mind.
>
> >
> >
> > Joe
> >
> --
>
> "Genius might be described as a supreme capacity for getting its possessors
> into trouble of all kinds."
> -- Samuel Butler
>

-- 
Steven M. Miano
http://stevenmiano.com