[TECH] Pica8 & Cumulus Networks

Mon Nov 2 21:52:49 UTC 2015

Yoann THOMAS writes:
> Under a Cloud project I ask myself to use equipment based on the Pica8
> or Cumulus Networks.

Ah, quite different beasts.

Cumulus Networks tries to really make the switch look like a Linux
system with hardware-accelerated forwarding, so you can use stock
programs that manipulate routing, e.g. Quagga, and all forwarding
between the high-speed ports is done "in hardware".

Most other systems including Pica8 treat the high-speed interfaces as
different; you need special software to manipulate the configuration of
the forwarding ASIC.  I think in the case of Pica8 it's OpenFlow/Open
vSwitch, for other systems it will be some sort of a ASIC-specific SDK.

A colleague has built a proof-of-concept L3 leaf/spine network (using
OSPFv2/OSPFv3 according to local tradition) with six 32x40GE Quanta
switches running Cumulus Linux.  So far it has been quite pleasant.
There have been a few glitches, but those usually get fixed pretty
quickly.  We configure the switches very much like GNU/Linux servers, in
our case using Puppet (Ansible or Chef would work just as well).

> All in order to mount a Spine & Leaf architecture

> - Spine 40Gbps
> - Leaf in 10Gbps

One interesting option is to get (e.g. 1RU 32x) 40G switches for both
spine and leaf, and connect the servers using 1:4 break-out cables.
Fewer SKUs, better port density at the cost of funny cabling.  Also
gives you a bit more flexibility with respect to uplinks (can have more
than 6*40GE per leaf if needed) and downlinks (easy to connect some
servers at 40GE).

The new 32*100GE switches also look interesting, but they might still be
prohibitively expensive (although you can save on spine count and
cabling) unless you NEED the bandwidth or want to build something
future-proof.  They are even more flexible in that you can drive the
ports as 4*10GE, 4*25GE (could be an attractive high-speed option once
25GE server adapters become common), 40GE, 2*50GE, 100GE.

We have looked at Edge-Core and Quanta and they both look pretty solid.
I think they are also both used by some of the Web "hypergiants".
Others may be just as good - basically it's always the same Broadcom
switching silicon (Trident II/II+ in the 40GE, Tomahawk in the 100GE
switches) with a bit of glue; there may be subtle differences between
vendors in quality, box design, airflow etc.

It's a bit unhealthy that Broadcom is so dominant in this market - but
probably not undeserved.  There are a few alternative switching
chipsets, e.g. Mellanox, Cavium XPliant that look competitive (at least
on paper) and that may be more "open" than Broadcom's.  I think both the
software vendors (e.g. Cumulus Networks) and the ODMs (Edge-Core, Quanta
etc.) are interested in these.
-- 
Simon.