Service Provider NetFlow Collectors

Wed Jan 2 12:08:10 UTC 2019

I would advise against InfluxDB in this case - flow data has a very high (and open) tag cardinality which is not suited to Influx (although their recently new index format has improved this).

I’m currently pushing sFlow through Pmacct —> Kafka —> Clickhouse (columnar store) with a summing merge tree database engine.
Clickhouse is very fast for queries across columns as well as aggregating down them (e.g. summing number of bytes).

For example this is the results of a query of nearly a year’s worth of MAC-to-MAC flows (7-tuple) queried for the last 7 days between two given sets of MACs:

2016 rows in set. Elapsed: 0.208 sec. Processed 17.56 million rows, 1.03 GB (84.51 million rows/s., 4.97 GB/s.)

There is also a Grafana datasource plugin for Clickhouse :)

- Tim

> On 2 Jan 2019, at 7:50 pm, H I Baysal <hibaysal at gmail.com> wrote:
> 
> PMACCT (Works Awesome)
> push to influxdb ( Works awesome)
> 
> With some custom scripts to add/match interface descriptions. And you can query whatever you want in grafana :D
> And grafana has a nice API for rendering a dashboardgraph to a PNG and you can send this png to whatever chat/bot or mail you want.
> 
> And all for free with 99% of accuracy.
> 
> (Mucho gracias to Paulo :D )
> 
> 
> On 01-01-19 05:56, Avi Freedman wrote:
>> We do have a minimum for commercial service that's more like $1500/mo but we are coming out with a free tier in Q1 with lower retention (among other deltas, but including fully slice and dice flow analytics +BGP that it sounded like Erik might be looking for).
>> 
>> Feel free to ping me if anyone would like to help us test the free tier in January.
>> 
>> Thanks,
>> 
>> Avi Freedman
>> CEO, Kentik
>> 
>>> Doesn't Kentik cost like $2000 a month minimum?
>>> 
>>> 
>>> On Mon, Dec 31, 2018 at 11:57 AM Matthew Crocker <matthew at corp.crocker.com>
>>> wrote:
>>> 
>>>>  +1 Kentik as well,  DDoS, RTBH, Netflow.  Cloud based so I don't have to
>>>> worry about it.
>>>> 
>>>> On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" <
>>>> nanog-bounces at nanog.org on behalf of bryan at shout.net> wrote:
>>>> 
>>>>     +1 Kentik ...
>>>> 
>>>>     We've been using their DDoS/RTBH mitigation with good success.
>>>> 
>>>> 
>>>>     On 12/31/18 3:52 AM, Eric Lindsjö wrote:
>>>>     > Hi,
>>>>     >
>>>>     > We use kentik and we're very happy. Works great, tons of new
>>>> features
>>>>     > coming along all the time. Going to start looking into ddos
>>>> detection
>>>>     > and mitigation soon.
>>>>     >
>>>>     > Would recommend.
>>>>     >
>>>>     > Kind regards,
>>>>     > Eric Lindsjö
>>>>     >
>>>>     >
>>>>     > On 12/31/2018 04:29 AM, Erik Sundberg wrote:
>>>>     >>
>>>>     >> Hi Nanog….
>>>>     >>
>>>>     >> We are looking at replacing our Netflow collector. I am wonder what
>>>>     >> other service providers are using to collect netflow data off their
>>>>     >> Core and Edge Routers. Pros/Cons… What to watch out for any info
>>>> would
>>>>     >> help.
>>>>     >>
>>>>     >> We are mainly looking to analyze the netflow data. Bonus if it does
>>>>     >> ddos detection and mitigation.
>>>>     >>
>>>>     >> We are looking at
>>>>     >>
>>>>     >> ManageEngine Netflow Analyzer
>>>>     >>
>>>>     >> PRTG
>>>>     >>
>>>>     >> Plixer – Scrutinizer
>>>>     >>
>>>>     >> PeakFlow
>>>>     >>
>>>>     >> Kentik
>>>>     >>
>>>>     >> Solarwinds NTA
>>>>     >>
>>>>     >> Thanks in advance…
>>>>     >>
>>>>     >> Erik
>>>>     >>
>>>>     >>
>>>>     >>
>>>> ------------------------------------------------------------------------
>>>>     >>
>>>>     >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
>>>> documents,
>>>>     >> files or previous e-mail messages attached to it may contain
>>>>     >> confidential information that is legally privileged. If you are not
>>>>     >> the intended recipient, or a person responsible for delivering it
>>>> to
>>>>     >> the intended recipient, you are hereby notified that any
>>>> disclosure,
>>>>     >> copying, distribution or use of any of the information contained in
>>>> or
>>>>     >> attached to this transmission is STRICTLY PROHIBITED. If you have
>>>>     >> received this transmission in error please notify the sender
>>>>     >> immediately by replying to this e-mail. You must destroy the
>>>> original
>>>>     >> transmission and its attachments without reading or saving in any
>>>>     >> manner. Thank you.
>>>>     >
>>>> 
>>>> 
>>>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20190102/7abeb483/attachment.html>