Thousands of hosts on a gigabit LAN, maybe not

Fri May 8 19:26:54 UTC 2015

- The more switches a packet has to go through, the higher the latency, so
your response times may deteriorate if you cascade too many switches.
Legend says up to 4 is a good number, any further you risk creating a big
mess.

- The more switches you add, the higher your bandwidth utilized by
broadcasts in the same subnet.
http://en.wikipedia.org/wiki/Broadcast_radiation

- If you have only one connection between each switch, each switch is going
to be limited to that rate (1gbps in this case), possibly creating a
bottleneck depending on your application and how exactly it behaves.
Consider aggregating uplinks.

- Bundling too many Ethernet cables will cause interference (cross-talk),
so keep that in mind. I'd purchase F/S/FTP cables and the like.

Here I am going off on a tangent: if your friends want to build a "super
computer" then there's a way to calculate the most "efficient" number of
nodes given your constraints (e.g. linear optimization). This could save
you time, money and headaches. An example: maximize the number of TFLOPS
while minimizing number of nodes (i.e. number of switch ports). Just a
quick thought.

On Fri, May 8, 2015 at 1:53 PM, John Levine <johnl at iecc.com> wrote:

> Some people I know (yes really) are building a system that will have
> several thousand little computers in some racks.  Each of the
> computers runs Linux and has a gigabit ethernet interface.  It occurs
> to me that it is unlikely that I can buy an ethernet switch with
> thousands of ports, and even if I could, would I want a Linux system
> to have 10,000 entries or more in its ARP table.
>
> Most of the traffic will be from one node to another, with
> considerably less to the outside.  Physical distance shouldn't be a
> problem since everything's in the same room, maybe the same rack.
>
> What's the rule of thumb for number of hosts per switch, cascaded
> switches vs. routers, and whatever else one needs to design a dense
> network like this?  TIA
>
> R's,
> John
>