iam at st-andrews.ac.uk
Thu Apr 12 16:18:05 CDT 2012
You'll need to build an array that'll random read/write upwards of 200MB/s if you want to get a semi-reliable capture to disk. That means SSD if you're very rich, or many spindles (preferably 15k's) in a stripe/ raid10 if you're building from your scrap pile. Bear in mind that write cache won't help you, as the io isn't going to be bursty, rather a continuous stream.
Another great help is scoping what you're looking for and pre-processing before writing out only the 'interesting' bits, thus reducing the io requirement. It does depend what you're trying to do, as headers can be adequate for many applications.
Aligning your partitions with the physical disk geometry can produce surprising speedups, as can stripe block size changes, but that's generally empirical, and depends on your workload.
Sent: 12/04/2012, 21:27
To: nanog at nanog.org
Subject: Network Storage
Can you please comment on what is best solution for storing network
traffic. We have been graciously granted access by our network
administrator to capture traffic but the one Tera byte disk space is
no match with the data that we are seeing, so it fills up quickly. We
can't get additional space on the server itself so I am looking for
some external solutions. Can you please suggest something that would
be best for Gbps speeds .
More information about the NANOG