Thousands of hosts on a gigabit LAN, maybe not

Fri May 8 23:20:51 UTC 2015

The real answer to this is being able to cram them into a single chassis which can multiplex the network through a backplane.  Something like the HP Moonshot ARM system or the way others like Google build high density compute with integrated Ethernet switching. 

Phil

-----Original Message-----
From: "John Levine" <johnl at iecc.com>
Sent: ‎5/‎8/‎2015 2:59 PM
To: "nanog at nanog.org" <nanog at nanog.org>
Subject: Thousands of hosts on a gigabit LAN, maybe not

Some people I know (yes really) are building a system that will have
several thousand little computers in some racks.  Each of the
computers runs Linux and has a gigabit ethernet interface.  It occurs
to me that it is unlikely that I can buy an ethernet switch with
thousands of ports, and even if I could, would I want a Linux system
to have 10,000 entries or more in its ARP table.

Most of the traffic will be from one node to another, with
considerably less to the outside.  Physical distance shouldn't be a
problem since everything's in the same room, maybe the same rack.

What's the rule of thumb for number of hosts per switch, cascaded
switches vs. routers, and whatever else one needs to design a dense
network like this?  TIA

R's,
John