from the academic side of the house
JP Velders
jpv at veldersjes.net
Sun Apr 29 12:21:24 UTC 2007
On Thu, 26 Apr 2007, Simon Leinen wrote:
> Date: Thu, 26 Apr 2007 19:11:37 +0200
> From: Simon Leinen <simon at limmat.switch.ch>
> Subject: Re: from the academic side of the house
> [ ... ]
> Another host issue would be interrupts and CPU load for checksum, but
> most modern 10GE (and also GigE!) adapters offload checksum
> segmentation and reassembly, as well as checksum computation and
> validation to the adapter if the OS/driver supports it.
Correct, though sometimes the performance can vary wildly depending on
driver and firmware. I had to turn off all the checksumming with
certain driver versions for the Myrinet 10GE cards, later drivers
performed OK.
Also, having multi-cpu and multi-core machines helps enormously,
binding a NIC/driver to a certain core, and having the receiving
process on the neighbouring core can dramatically increase throughput.
But per adapter/driver differences in which offloading actually
increases performance sometimes still is a black art, necessitating
painstaking testing...
> The adapters used in this record (Chelsio S310E) contain a full TOE
> (TCP Offload Engine) that can run the entire TCP state machine on the
> adapter, although I'm more sure whether they made use of that.
I believe that they did not use it. In the past we also ran into
problems and performance dips when using the accellerated Chelsio
cards. Without TOE accelleration things worked much smoother.
Kei's team did modify iperf such that they used mmap's to gain more of
a zero-copy behaviour, depending on drivers this can also give much
better performance, akin to say Myricom's claims for Myrinet...
Kind regards,
JP Velders
More information about the NANOG
mailing list