improving signal to noise ratio from centralized network syslogs

Fri Jan 26 16:41:51 UTC 2018

     ELK stack.

     Java RAM devoring monster but Kibana makes indexing easy.

-----
Alain Hebert                                ahebert at pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770     Beaconsfield, Quebec     H9W 6G7
Tel: 514-990-5911  http://www.pubnix.net    Fax: 514-990-9443

On 01/26/18 01:01, Michael Loftis wrote:
> On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon at jmaimon.com> wrote:
>
>> Hey All,
>>
>> Centralized logging is a good thing. However, what happens is that every
>> repetitive, annoying but not (usually) important thing fills up the log
>> with reams of what you are not looking for.
>>
>> Networks are a noisy place and silencing every logged condition is
>> impractical and sometimes undesirable.
>>
>> What I am interested in is an automated zoom-in zoom-out tool to mask
>> the repetition of "normal" events and allow the unusual to stand out.
>>
>> Add to that an ability to identify gaps in the background noise. (The
>> dog that didnt bark)
>>
>> What I am not interested in are solutions based upon preconfigured
>> filters and definitions and built in analysis for supported
>> (prepopulated definitions) platforms, this is all about pattern
>> mining/masking and should be self discoverable. Ideally a command tool
>> to generate static versions of the analysis coupled with a web platform
>> (with zoom +- buttons)  for realtime.
>>
>> I made a crude run of it with SLCT, using its generated patterns to grep
>> -v, and that in and of itself was useful, but needs a bit of work. Also,
>> its not quite real time.
>>
>> Any ideas would be greatly appreciated.
>
> Not cheap, but Splunk comes to mind.
>
>>
>> Joe
>>