improving signal to noise ratio from centralized network syslogs

Fri Jan 26 17:17:31 UTC 2018

     +1 for Graylog, you can pour ALL your syslog data into it, and then
configure what are called streams.  Streams are a way to whittle down the
incoming log flows and see something LESS than everything.  You can create
a stream that only shows these 6 devices, or one that only shows log info
from the RPD daemon on your Juniper routers.

     In your case, you could use the stream rules to create a stream that
filters out the background noise with regex expressions.  You're not losing
anything, you still have the full log data captured, and you can see it in
the portal, but if you click on one of your streams, you see filtered data
based, on your rulesets.  We've been using it for about 2 years now I
think.

     It's open source, easy to set up, supports LDAP, multiple input types
(beyond just udp syslog), and the community is pretty solid.

Sincerely,
Casey Russell
Network Engineer
[image: KanREN] <http://www.kanren.net>
[image: phone]785-856-9809
2029 Becker Drive, Suite 282
Lawrence, Kansas 66047
[image: linkedin]
<https://www.linkedin.com/company/92399?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A92399%2Cidx%3A1-1-1%2CtarId%3A1440002635645%2Ctas%3AKanREN>
[image:
twitter] <https://twitter.com/TheKanREN> [image: twitter]
<http://www.kanren.net/feed/> need support? <support at kanren.net>

On Fri, Jan 26, 2018 at 10:41 AM, Alain Hebert <ahebert at pubnix.net> wrote:

>     ELK stack.
>
>     Java RAM devoring monster but Kibana makes indexing easy.
>
> -----
> Alain Hebert                                ahebert at pubnix.net
> PubNIX Inc.
> 50 boul. St-Charles
> <https://maps.google.com/?q=50+boul.+St-Charles&entry=gmail&source=g>
> P.O. Box 26770     Beaconsfield, Quebec     H9W 6G7
> Tel: 514-990-5911  http://www.pubnix.net    Fax: 514-990-9443
>
>
> On 01/26/18 01:01, Michael Loftis wrote:
>
>> On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon at jmaimon.com> wrote:
>>
>> Hey All,
>>>
>>> Centralized logging is a good thing. However, what happens is that every
>>> repetitive, annoying but not (usually) important thing fills up the log
>>> with reams of what you are not looking for.
>>>
>>> Networks are a noisy place and silencing every logged condition is
>>> impractical and sometimes undesirable.
>>>
>>> What I am interested in is an automated zoom-in zoom-out tool to mask
>>> the repetition of "normal" events and allow the unusual to stand out.
>>>
>>> Add to that an ability to identify gaps in the background noise. (The
>>> dog that didnt bark)
>>>
>>> What I am not interested in are solutions based upon preconfigured
>>> filters and definitions and built in analysis for supported
>>> (prepopulated definitions) platforms, this is all about pattern
>>> mining/masking and should be self discoverable. Ideally a command tool
>>> to generate static versions of the analysis coupled with a web platform
>>> (with zoom +- buttons)  for realtime.
>>>
>>> I made a crude run of it with SLCT, using its generated patterns to grep
>>> -v, and that in and of itself was useful, but needs a bit of work. Also,
>>> its not quite real time.
>>>
>>> Any ideas would be greatly appreciated.
>>>
>>
>> Not cheap, but Splunk comes to mind.
>>
>>
>>> Joe
>>>
>>>
>