Thousands of hosts on a gigabit LAN, maybe not

Miles Fidelman mfidelman at
Fri May 8 20:02:13 UTC 2015

Forgot to mention - you might also want to check out Beowulf clusters - 
there's an email list at - probably some useful 
info in the list archives, maybe a good place to post your query.


Miles Fidelman wrote:
> John Levine wrote:
>> Some people I know (yes really) are building a system that will have
>> several thousand little computers in some racks.  Each of the
>> computers runs Linux and has a gigabit ethernet interface.  It occurs
>> to me that it is unlikely that I can buy an ethernet switch with
>> thousands of ports, and even if I could, would I want a Linux system
>> to have 10,000 entries or more in its ARP table.
>> Most of the traffic will be from one node to another, with
>> considerably less to the outside.  Physical distance shouldn't be a
>> problem since everything's in the same room, maybe the same rack.
>> What's the rule of thumb for number of hosts per switch, cascaded
>> switches vs. routers, and whatever else one needs to design a dense
>> network like this?  TIA
> It's become fairly commonplace to build supercomputers out of clusters 
> of 100s, or 1000s of commodity PCs, see, for example:
> or
> (a cluster of 1760 playstations at AFRL Rome Labs)
> Interestingly, all the documentation I can find is heavy on the 
> software layers used to cluster resources - but there's little about 
> hardware configuration other than pretty pictures of racks with lots 
> of CPUs and lots of wires.
> If the people you know are trying to do something similar - it might 
> be worth some nosing around the Rocks community, or some phone calls.  
> I expect that interconnect architecture and latency might be a bit of 
> an issue for this sort of application.
> Miles Fidelman

In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

More information about the NANOG mailing list