Network Storage

Sun Apr 15 13:38:28 UTC 2012

In a message written on Thu, Apr 12, 2012 at 05:16:27PM -0400, Maverick wrote:
> 1) My goal is to store the traffic may be fore ever, and analyze it in
> the future for security related incidents detected by ids/ips.

Let's just assume you have enough disk space that you can write out
every packet, or even just packet header.  That's a hard problem,
but you've received plenty of suggestions on how to go down that
path.

Once you have that data, how are you going to process it?

Yes, disk reads are faster than disk writes, but not by that much.
If it takes you 24 hours to write a day of data to disk, it might
take you 12 hours just to read it all back off and process it.
Processing a weeks worth of back data could take days.  I'm also
not even starting to count the CPU and memory necessary to build
state tables and statistical analysis tables to generate useful
data.

There's a reason why most network traffic tools summarize early,
as early as on the network device when using Netflow type collection.
It's not just to save storage space on disk, but it's to make the
processing of the data fast enough that it can be done in a short
enough time that the data is still relevant when the processing is
complete.

-- 
       Leo Bicknell - bicknell at ufp.org - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 826 bytes
Desc: not available
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20120415/e7b9bd33/attachment.sig>