Thousands of hosts on a gigabit LAN, maybe not

Dave Taht dave.taht at
Fri May 8 19:26:16 UTC 2015

On Fri, May 8, 2015 at 11:53 AM, John Levine <johnl at> wrote:
> Some people I know (yes really) are building a system that will have
> several thousand little computers in some racks.

Very cool-ly crazy.

> Each of the
> computers runs Linux and has a gigabit ethernet interface.  It occurs
> to me that it is unlikely that I can buy an ethernet switch with
> thousands of ports, and even if I could, would I want a Linux system
> to have 10,000 entries or more in its ARP table.

Agreed. :) You don't really want 10,000 entries in a routing FIB
table either, but I was seriously encouraged by the work going
on in linux 4.0 and 4.1 to improve those lookups.

I'd love to know the actual scalability of some modern
routing protocols (isis, babel, ospfv3, olsrv2, rpl) with that
many nodes too....

> Most of the traffic will be from one node to another, with
> considerably less to the outside.  Physical distance shouldn't be a
> problem since everything's in the same room, maybe the same rack.

That is an awful lot of ports to fit in a rack (48 ports, 36 2U slots
in the rack (and is that too high?) = 1728
ports) A thought is you could make it meshier using multiple
interfaces per tiny linux box? Put, say
3-6 interfaces and have a very few switches interconnecting given
clusters (and multiple paths
to each switch). That would reduce your arp table (and fib table) by a
lot at the cost of adding

> What's the rule of thumb for number of hosts per switch, cascaded
> switches vs. routers, and whatever else one needs to design a dense
> network like this?  TIA

max per vlan 4096. Still a lot.

Another approach might be max density on a switch (48?) per cluster,
routed (not switched) 10GigE
to another 10GigE+ switch.

I'd love to know the rule of thumbs here also, I imagine some rules
must exist for those in the VM
or VXLAN worlds.

> R's,
> John

Dave Täht
Open Networking needs **Open Source Hardware**

More information about the NANOG mailing list