Open source Netflow analysis for monitoring AS-to-AS traffic

Wed Mar 27 23:38:04 UTC 2024

In the same vein, if you can get your devices exporting sFlow, or for
others reading that do have sFlow capable devices: the sFlow-RT team has
built ready to deploy, all in one docker containers using Grafana and
Prometheus that you can stand up within minutes to start visualizing and
easily querying/processing sFlow data from your routers, with no prior
experience with the underlying software needed.

https://blog.sflow.com/2023/07/deploy-real-time-network-dashboards.html
https://github.com/sflow-rt/prometheus-grafana

On Wed, Mar 27, 2024 at 12:00 PM Peter Phaal <peter.phaal at gmail.com> wrote:

> Brian, you may want to see if your routers support sFlow (vendors have
> added the feature over the last few years).
>
> In particular, see if it includes support for the sFlow extended_gateway
> structure:
>
> /* Extended Gateway Data */
> /* opaque = flow_data; enterprise = 0; format = 1003 */
>
> struct extended_gateway {
>    next_hop nexthop;           /* Address of the border router that should
>                                   be used for the destination network */
>    unsigned int as;            /* Autonomous system number of router */
>    unsigned int src_as;        /* Autonomous system number of source */
>    unsigned int src_peer_as;   /* Autonomous system number of source peer
> */
>    as_path_type dst_as_path<>; /* Autonomous system path to the
> destination */
>    unsigned int communities<>; /* Communities associated with this route */
>    unsigned int localpref;     /* LocalPref associated with this route */
> }
>
> The dst_as_path field is particularly valuable since it allows you to see
> who your customers are peering with.
>
> While not a complete solution, you might want to take a look at sflowtool,
> https://github.com/sflow/sflowtool, to decode the sFlow records and
> convert them to JSON. It's not hard to write a Python script to calculate
> BGP peering metrics and push the results into a time series database
> (Prometheus, InfluxDB, etc) and build dashboards in Grafana. The following
> article gives a few examples:
>
> https://blog.sflow.com/2018/12/sflow-to-json.html
>
> On Tue, Mar 26, 2024 at 5:06 PM Brian Knight via NANOG <nanog at nanog.org>
> wrote:
>
>> What's presently the most commonly used open source toolset for
>> monitoring AS-to-AS traffic?
>>
>> I want to see with which ASes I am exchanging the most traffic across my
>> transits and IX links. I want to look for opportunities to peer so I can
>> better sell expansion of peering to upper management.
>>
>> Our routers are mostly $VENDOR_C_XR so Netflow support is key.
>>
>> In the past, I've used AS-Stats
>> <https://github.com/manuelkasper/AS-Stats> for this purpose. However, it
>> is particularly CPU and disk IO intensive. Also, it has not been actively
>> maintained since 2017.
>>
>> InfluxDB wants to sell me
>> <https://www.influxdata.com/what-are-netflow-and-sflow/> on Telegraf +
>> InfluxDB + Chronograf + Kapacitor, but I can't find any clear guide on what
>> hardware I would need for that, never mind how to set up the software. It
>> does appear to have an open source option, however.
>>
>> pmacct seems to be good at gathering Netflow, but doesn't seem to analyze
>> data. I don't see any concise howto guides for setting this up for my
>> purpose, however.
>>
>> I'm aware Kentik does this very well, but I have no budget at the moment,
>> my testing window is longer than the 30 day trial, and we are not prepared
>> to share our Netflow data with a third party.
>>
>> Elastiflow <https://www.elastiflow.com/> appears to have been open source
>> <https://github.com/robcowart/elastiflow?tab=readme-ov-file> at one time
>> in the past, but no longer. Since it too appears to be hosted, I have the
>> same objections as I do with Kentik above.
>>
>> On-list and off-list replies are welcome.
>>
>> Thanks,
>>
>> -Brian
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20240327/168edf09/attachment.html>