Thousands of hosts on a gigabit LAN, maybe not

Fri May 8 19:53:28 UTC 2015

John Levine wrote:
> Some people I know (yes really) are building a system that will have
> several thousand little computers in some racks.  Each of the
> computers runs Linux and has a gigabit ethernet interface.  It occurs
> to me that it is unlikely that I can buy an ethernet switch with
> thousands of ports, and even if I could, would I want a Linux system
> to have 10,000 entries or more in its ARP table.
>
> Most of the traffic will be from one node to another, with
> considerably less to the outside.  Physical distance shouldn't be a
> problem since everything's in the same room, maybe the same rack.
>
> What's the rule of thumb for number of hosts per switch, cascaded
> switches vs. routers, and whatever else one needs to design a dense
> network like this?  TIA
>
>

It's become fairly commonplace to build supercomputers out of clusters 
of 100s, or 1000s of commodity PCs, see, for example:
www.rocksclusters.org
http://www.rocksclusters.org/presentations/tutorial/tutorial-1.pdf
or
http://www.dodlive.mil/files/2010/12/CondorSupercomputerbrochure_101117_kb-3.pdf 
(a cluster of 1760 playstations at AFRL Rome Labs)

Interestingly, all the documentation I can find is heavy on the software 
layers used to cluster resources - but there's little about hardware 
configuration other than pretty pictures of racks with lots of CPUs and 
lots of wires.

If the people you know are trying to do something similar - it might be 
worth some nosing around the Rocks community, or some phone calls.  I 
expect that interconnect architecture and latency might be a bit of an 
issue for this sort of application.

Miles Fidelman

-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra