Thousands of hosts on a gigabit LAN, maybe not

Sun May 10 08:15:05 UTC 2015

If you need that kind of density, I recommend a Clos fabric. Arista, Juniper, Brocade, Big Switch BCF and Cisco all have solutions that would allow you to build a high-density leaf/spine. You can build the Cisco solution with NXOS or ACI, depending which models you choose. The prices on these solutions are all somewhat in the same ballpark based on list pricing I've seen... even Cisco (the Nexus 9k is surprisingly in the same range as branded whitebox). There is also Pluribus which offers a fabric, but their niche is having server procs on board the switches and it seems like your project involves physical rather than virtual servers. Still, the Pluribus could be used without taking advantage of the on board server compute I suppose.
I also recommend looking into a solution that supports VXLAN (or GENEVE, or whatever overlay works for your needs) simply because MAC is carried in Layer-3 so you won't have to deal with spanning tree or monstrous mac tables. But you don't need to do an overlay if you just segment with traditional VLANs.
I'm guessing you don't need HA (A/B uplinks utilizing LACP) for these servers?
Also, do you need line rate forwarding? Having 1,000 devices with 1Gb uplinks doesn't necessarily mean that full throughput is required... the clustering and the applications may be sporadic and bursty? I have seen load-testing clusters, hadoop and data warehousing pushing high volumes but the individual NICs in the clusters never actually hit capacity... If you need line-rate, then you need to do a deep dive with several of the vendors because there are significant differences in buffers on some models.
And... what support do you need? Just one spare on the shelf or full vendor support on every switch? That will impact which vendor you choose.
I'd like to hear more about this effort once you get it going. Which vendor you went with, how you tuned it, and why you selected who you did. Also, how it works.
LFoD
> Date: Sun, 10 May 2015 01:17:07 +0000
> From: johnl at iecc.com
> To: nanog at nanog.org
> Subject: Re: Thousands of hosts on a gigabit LAN, maybe not
> 
> In article <CAHf3uWyPQn1NS_umjZ-zNuk3i5uFcZBu9L39b-crovG6yUm2qA at mail.gmail.com> you write:
> >Juniper OCX1100 have 72 ports in 1U.
> 
> Yeah, too bad it costs $32,000.  Other than that it'd be perfect.
> 
> R's,
> John