Service Provider NetFlow Collectors

Wed Jan 2 12:37:29 UTC 2019

Hi Tim,

That absolutely depends on the amount of TAGs you use, and how you 
aggregate, etc.
I am collecting DSTAS, SRCAS, en DST AS per IP. And influx is not even 
sweating a single drop....

We have a 4 Tbps of traffic during peak, and as well as pmacct and 
influxdb or running very very smooth.

(With the mentioned aggregations I can see what a single customer costs 
with Transit, Peering and IX (per IP even if needed) )
And dst AS per port/description/ethernet name

 From your mail i derive that you just pushed everything to influx from 
flows, you have to be a bit smarter with the layout, aggregations and 
continuous queries.
(collect what you need)

On 02-01-19 13:08, Tim Raphael wrote:
> I would advise against InfluxDB in this case - flow data has a very 
> high (and open) tag cardinality which is not suited to Influx 
> (although their recently new index format has improved this).
>
> I’m currently pushing sFlow through Pmacct —> Kafka —> Clickhouse 
> (columnar store) with a summing merge tree database engine.
> Clickhouse is very fast for queries across columns as well as 
> aggregating down them (e.g. summing number of bytes).
>
> For example this is the results of a query of nearly a year’s worth of 
> MAC-to-MAC flows (7-tuple) queried for the last 7 days between two 
> given sets of MACs:
> /
> /
> /2016 rows in set. Elapsed: 0.208 sec. Processed 17.56 million rows, 
> 1.03 GB (84.51 million rows/s., 4.97 GB/s.)/
> /
> /
> There is also a Grafana datasource plugin for Clickhouse :)
> /
> /
> /- /Tim
>
>
>> On 2 Jan 2019, at 7:50 pm, H I Baysal <hibaysal at gmail.com 
>> <mailto:hibaysal at gmail.com>> wrote:
>>
>> PMACCT (Works Awesome)
>> push to influxdb ( Works awesome)
>>
>> With some custom scripts to add/match interface descriptions. And you 
>> can query whatever you want in grafana :D
>> And grafana has a nice API for rendering a dashboardgraph to a PNG 
>> and you can send this png to whatever chat/bot or mail you want.
>>
>> And all for free with 99% of accuracy.
>>
>> (Mucho gracias to Paulo :D )
>>
>>
>> On 01-01-19 05:56, Avi Freedman wrote:
>>> We do have a minimum for commercial service that's more like 
>>> $1500/mo but we are coming out with a free tier in Q1 with lower 
>>> retention (among other deltas, but including fully slice and dice 
>>> flow analytics +BGP that it sounded like Erik might be looking for).
>>>
>>> Feel free to ping me if anyone would like to help us test the free 
>>> tier in January.
>>>
>>> Thanks,
>>>
>>> Avi Freedman
>>> CEO, Kentik
>>>
>>>> Doesn't Kentik cost like $2000 a month minimum?
>>>>
>>>>
>>>> On Mon, Dec 31, 2018 at 11:57 AM Matthew Crocker 
>>>> <matthew at corp.crocker.com <mailto:matthew at corp.crocker.com>>
>>>> wrote:
>>>>
>>>>>  +1 Kentik as well,  DDoS, RTBH, Netflow.  Cloud based so I don't 
>>>>> have to
>>>>> worry about it.
>>>>>
>>>>> On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" <
>>>>> nanog-bounces at nanog.org <mailto:nanog-bounces at nanog.org> on behalf 
>>>>> of bryan at shout.net <mailto:bryan at shout.net>> wrote:
>>>>>
>>>>>     +1 Kentik ...
>>>>>
>>>>>     We've been using their DDoS/RTBH mitigation with good success.
>>>>>
>>>>>
>>>>>     On 12/31/18 3:52 AM, Eric Lindsjö wrote:
>>>>>     > Hi,
>>>>>     >
>>>>>     > We use kentik and we're very happy. Works great, tons of new
>>>>> features
>>>>>     > coming along all the time. Going to start looking into ddos
>>>>> detection
>>>>>     > and mitigation soon.
>>>>>     >
>>>>>     > Would recommend.
>>>>>     >
>>>>>     > Kind regards,
>>>>>     > Eric Lindsjö
>>>>>     >
>>>>>     >
>>>>>     > On 12/31/2018 04:29 AM, Erik Sundberg wrote:
>>>>>     >>
>>>>>     >> Hi Nanog….
>>>>>     >>
>>>>>     >> We are looking at replacing our Netflow collector. I am 
>>>>> wonder what
>>>>>     >> other service providers are using to collect netflow data 
>>>>> off their
>>>>>     >> Core and Edge Routers. Pros/Cons… What to watch out for any 
>>>>> info
>>>>> would
>>>>>     >> help.
>>>>>     >>
>>>>>     >> We are mainly looking to analyze the netflow data. Bonus if 
>>>>> it does
>>>>>     >> ddos detection and mitigation.
>>>>>     >>
>>>>>     >> We are looking at
>>>>>     >>
>>>>>     >> ManageEngine Netflow Analyzer
>>>>>     >>
>>>>>     >> PRTG
>>>>>     >>
>>>>>     >> Plixer – Scrutinizer
>>>>>     >>
>>>>>     >> PeakFlow
>>>>>     >>
>>>>>     >> Kentik
>>>>>     >>
>>>>>     >> Solarwinds NTA
>>>>>     >>
>>>>>     >> Thanks in advance…
>>>>>     >>
>>>>>     >> Erik
>>>>>     >>
>>>>>     >>
>>>>>     >>
>>>>> ------------------------------------------------------------------------
>>>>>     >>
>>>>>     >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
>>>>> documents,
>>>>>     >> files or previous e-mail messages attached to it may contain
>>>>>     >> confidential information that is legally privileged. If you 
>>>>> are not
>>>>>     >> the intended recipient, or a person responsible for 
>>>>> delivering it
>>>>> to
>>>>>     >> the intended recipient, you are hereby notified that any
>>>>> disclosure,
>>>>>     >> copying, distribution or use of any of the information 
>>>>> contained in
>>>>> or
>>>>>     >> attached to this transmission is STRICTLY PROHIBITED. If 
>>>>> you have
>>>>>     >> received this transmission in error please notify the sender
>>>>>     >> immediately by replying to this e-mail. You must destroy the
>>>>> original
>>>>>     >> transmission and its attachments without reading or saving 
>>>>> in any
>>>>>     >> manner. Thank you.
>>>>>     >
>>>>>
>>>>>
>>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20190102/64f222ef/attachment.html>