NANOG36-NOTES 2006.02.14 talk 2 Netflow Visualization Tools

Wed Feb 15 06:39:13 UTC 2006

Roland Dobbins - that's me asking about the time intervals for the  
bins and the TCP flags stuff.

;>

Note that 5-minute bins may not always be optimal for opsec - 5  
minutes minimum to see something happening and then 5 minutes to see  
if your mitigation action was effective is a long time.  With NetFlow- 
based anomaly-detection systems, the active flow timeout value is  
generally turned down to one minute; the operator may -choose- to  
suppress certain types of alarms for a set period, or configure  
threshold-transition delays, but being stuck at a practical minimum  
of 10 minutes between detection and confirmation of mitigation due to  
data-conversion overhead (the collected flow telemetry must be  
converted into another format prior to analysis) may be an issue, in  
some circumstances.

On Feb 14, 2006, at 8:13 PM, Vicky Røde wrote:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> thanks for taking notes.
>
> comments in-line:
>
> Matthew Petach wrote:
>> 2006.02.14 talk 2 Netflow tools
>>
>> Bill Yurcik
>> byurcik at ncsa.uiuc.edu
>>
>> NVisionIP and VisFlowConnect-IP
>>
>> probably a dozen tools out there, this is just
>> two of them.  Concenses is there's something to
>> this.
>>
>> They're an edge network, comes into ISP domain,
>> their tools are used by entities with many
>> subnet blocks.
>>
>> Overview
>> Project Motifivation
>> Netflows for Security
>> Two visualization tools
>>  NVisionIP
>>  VisFlowConnect-IP
>> Summary
>>
>> Internet Security:
>> N-Dimensional Work Space
>>
>> large--already lots of data to process
>> complex--combinatorics explode quickly
>> time dynamics--things can change quickly!
>> Visualizations can help!
>>  in near-realtime
>>  overview-browse-details on demand
>>
>> People are wired to do near-realtime processing
>> of visual information, so that's a good way to
>> present information for humans.
>> HCI says use overview-browse-details paradigm.
>>
>> Netflows for security
>> can identify connection-oriented stats to see
>> things like attacks, DoS, DDoS, etc.
>> Most people don't use the data portion of the
>> flow field, the first 64 bytes, they just look
>> at header info or aggregated flow records.
>>
>> Can spot how many users are on your system at
>> a given time, to schedule upgrades.
>>
>> Who are your top talkers?
>>
>> How long do my users surf?  What are people using
>> the network for?
>>
>> Where do users go?   Where did they come from?
>>
>> Are users following the security policy?
>>
>> What are the top N destination ports?
>> Is there traffic to vulnerable hosts?
>>
>> Can you identify and block scanners/bad guys?
>>
>> This doesn't replace other systems like syslog, etc.;
>> it integrates and works alongside them.
>>
>> architecture slide for NCSA.
>>
>> Can't really do sampled view for security, so probably
>> need distributed flow collector farm to get all the
>> raw data safely.
>>
>> Two visualization tools:
>> NVisionIP, VisFlowConnect-IP
>>
>> focus on quick overview of tools
>> security.ncsa.uiuc.edu/
>>
>> 3 level hierarchical tool;
>> galaxy view (small multiple view) ((machine view))
>>
>> Galaxy is overview of the whole network.
>> color and shape of dots is each host in a network.
>> settable parameters for each dot.
>>
>> Animated toolbar and clock show changes over time
>> in the galaxy.
>> Lets you get high-level content quickly and easily.
>>
>> Domain view lets you drill in a bit more; small
>> multiple view looks at the traffic within the
>> block.
>> upper histogram is lower, well known ports; lower
>> histogram is ports over 1024
>>
>> You can click on a given multiple view entry to
>> delve into one machine.
>> Many graphs for each machine in the most detailed
>> view.
>>
>> well known ports first, then rest of ports (sorted)
>> then source and destination traffic broken out.
>>
>> Designed for class Bs.
>>
>> http://security.ncsa.uiuc.edu/distribution/ 
>> VisFlowConnectDownload.html
>>
>> 3 vertical lines, comes from edge network perspective;
>> middle line is edge network to manage.  You set range
>> of networks you care about.  Outside lines are people
>> sourcing or sinking traffic to you, from outside
>> domains.
>>
>> There's a time axis, traffic only shown for the slice
>> of time currently under consideration.
>> Uses VCR-like controls to move time forward/backward
>>
>> Lets you see traffic/interactivity, drill into that
>> domain, see host level connectivity flows.
>>
>> Shows MS Blaster virus traffic as an example.
>>
>> Example 2, a scan example.  Just because it looks
>> like one IP hitting many others doesn't mean it's
>> really a security incident, though; could be a
>> cluster getting traffic.
>>
>> web crawlers hitting NCSA web servers make for
>> a very charateristic pattern over time.
>>
>> Summary
>> Netflows analysis is non-trivial,
>>
>> NVisionIP
>> VisFlowConnect-IP
>>
>> lots of references listed in very fine blue font.
>>
>> http://security.ncsa.uiuc.edu/distribution/NVisionIPDownload
>>
>> Avi Freedman, Akamai, Argus was mentioned a lot; it
>> lets you grab symmetric netflows, but also does TCP
>> analysis, shows some performance data as well.  not
>> sure if people are studying the impact of correlating
>> argus data with flow data.
>>
>> Roland Douta? of Cisco; many people are using netflow
>> to track security issues.  They now have ingress and
>> egress flow data on many of their platforms.
>> In reading paper describing it, there's data conversion
>> that needs to happen into an internal format that
>> nVision can understand.  It reads log files at the
>> moment, takes about 5 minutes to process files.  Lets
>> them take different file data sources, make the tool
>> for visualization independent of the input format.
>> They can read large files, but there is a performance
>> hit when doing it.
>> Are they planning on doing further work on the tool
>> to collect TCP flags, for frags, drop traffic, etc?
>> They've looked at it, but they leave it to IDS tools
>> for flag activity.  Might be of interest to consider
>> for future versions of the tools.
>>
>> Last question came up, echoed about argus.
>> Question about interactivity, they are working on
>> feedback through tools.  Question about alarming
>> on patterns; but once you start alarming or putting
>> up visual indicators, it distracts from rest of
>> the overall pattern, you tend to miss other information.
> - ----------------
> the last part was me, virendra rode from riverdomain. my question was
> mostly related to a possibility of setting priority bit(s) in order to
> control (rate-limit, if you will) session(s) that could lead to  
> congestion.
>
> since argus is already integrated and performs traffic auditing (i
> think) setting priority bit(s) would be a nice feature to integrate  
> down
> the path. then again, i understand this is a performance   
> monitoring tool.
>
>
> that's all.
>
>
>
> regards,
> /virendra
>
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.5 (MingW32)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
> iD8DBQFD8qpUpbZvCIJx1bcRAnzaAKCsI29SetdMSJaLr3LR01MGp87CmACgnCEf
> 7RDnyaGsad++GevXjt2MIQY=
> =/55T
> -----END PGP SIGNATURE-----

----------------------------------------------------------------------
Roland Dobbins <rdobbins at cisco.com> // 408.527.6376 voice

      Everything has been said.  But nobody listens.

                    -- Roger Shattuck