More on PC Routers from BGPD

Sat Apr 30 03:00:11 UTC 1994

>My comment was incorrect on T1/E1.  We were trying to do full DES
>encryption of a T1 stream and couldn't saturate the link using a
>modified BSDI kernel and a 486/66.  Full T1/E1 is possible using
>existing cards if you don't try to encrypt.

   Additionally, some early versions of BSDI had problems with doing the
processing of the protocol queues via netisr in a timely manner and this
limited the maximum packet rate. I haven't tested more recent versions of
BSDI for this problem.

>Wrt ethernet - I was under the impression that the fastest EISA card
>was the 3C579 and the fastest ISA the TN-1500.

   I've never benchmarked a 3c579 (or seen one for that matter), but if it's
anything like other recent boards from 3COM...I would be surprised if it
actually worked very well. perhaps the important point that I should make here
is that bus master DMA does not necessarily mean fast or low overhead.

>  I'd be surprised to
>hear that a WD/SMC EISA card can saturate 2-3 ethernets at small
>packet sizes (typical traffic seen by a router).  This is more a
>reflection of my opinion of PC card technology, not FreeBSD or BSDI.

   Actually, I think it could saturate 2 ethers with an SMC ISA (not EISA)
board. Maybe I should actually try this. :-) The driver I wrote for FreeBSD is
actually quite well done (I'm patting myself on the back :-)) and uses a
multi buffering technique I invented that has to my knowledge never been
previously implemented. A typical "ttcp" run looks like this:

(transmitter)
[corbin:davidg]> ttcp -t implode
ttcp-t: buflen=8192, nbuf=2048, align=16384/+0, port=5001  tcp  -> implode
ttcp-t: socket
ttcp-t: connect
ttcp-t: 16777216 bytes in 14.95 real seconds = 1096.08 KB/sec +++
ttcp-t: 2048 I/O calls, msec/call = 7.47, calls/sec = 137.01
ttcp-t: 0.0user 3.6sys 0:14real 24% 32i+188d 110maxrss 0+2pf 5602+9csw

(receiver)
[implode:davidg]> ttcp -r
ttcp-r: buflen=8192, nbuf=2048, align=16384/+0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 198.145.90.2
ttcp-r: 16777216 bytes in 14.96 real seconds = 1095.00 KB/sec +++
ttcp-r: 11372 I/O calls, msec/call = 1.35, calls/sec = 760.03
ttcp-r: 0.0user 2.4sys 0:14real 16% 32i+188d 152maxrss 0+2pf 11265+33csw

   I think the most important thing above is the (low) percent CPU time. The
actual consumed CPU time is somewhat higher (not all of the system time is
reflected in the percentage) - kernel profiling shows it to be about 30%.

>If anyone has measured troughput under small packet load, I'd be
>interested in hearing the result.

   Do you have a suggestion on how I might do such a test? It's a problem
because TCP will coalesce socket writes until a full frame is gathered unless
the TCP_NODELAY socket option is used. I believe setting this option will have
other bad side effects (like setting the PUSH bit, and thus might solicit an
immediate acknowledge). I'd be tempted to do a UDP test, but due to bugs in
the UDP layer of all Net/2 derived code I've seen (including BSDI, NetBSD, and
FreeBSD), the UDP performance is rather low - and additionally is difficult to
test because of the lack of connection flow control. In fact, BSDI and NetBSD
have a bug that causes a pathological condition whereby partial UDP datagrams
will be fragmented/sent and resent repeatedly should a client application
attempt to write packets too quickly to the socket.

>Wrt DS3 - I didn't think a HSSI card for EISA existed.  Does one
>exist?

   I don't know; I did say that "this assertion is unproven". I based the
opinion on extensive kernel profiling analysis of the socket and TCP
performance.

>  Also, keep in mind that routers see an average packet size on
>the order of 200 bytes.  Do you really think an EISA bus can handle
>the small packet throughput?  Big packets are easy, too bad real
>traffic isn't all big packets.

   All the analysis I've done of bottlenecks in FreeBSD has shown that the
actual overhead of packet handling was fairly low compared to the time to
get/put data out to ISA shared memory. Unlike BSDI, FreeBSD's interrupt
overhead is extremely low and would not be significantly affected by the
increased interrupt rate. This isn't the only factor of course, and the
large number of variables makes it difficult to guess how well it might work
without actually testing specific hardware.

>BTW - I've corrected my own untruths by way of Cc.  Thanks for the
>note and the good work that you are doing on FreeBSD.

   Thanks. I hope this rather lengthy reply isn't totally boring other folks on
this mailing list. BTW, is this a list I should be subscribed to?

-DG