100G - Whitebox

Nick Hilliard nick at foobar.org
Sun Aug 20 22:12:12 UTC 2017

Fredrik Korsbäck wrote:
> The only viable merchant silicon chip that would be useful for a IXP
> is from the StrataDNX-family which house the
> jericho/qumran/petra/arad chips from broadcom. No packetbuffer in the
> exhangepoint will shred performance significantly, especially when
> one of your bursty 100G customers starts sending data into 1/10G
> customers.
> To the best of my knowledge the only one that offers DNX in
> whitebox-fashion is Agema and Edgecore. But why whitebox? Except on a
> very few occasions whitebox is just "i like paying hardware and
> software on different invoices = whitebox" the TCO is just the same
> but. As an exchangepoint i also see that it can hard to reap the
> benefits of all the hipstershit going on in these NOS-startups, you
> want spanning-tree, port-security, something to loadbalance over
> links and perhaps a overlaying-technology if the IXP becomes to big
> and distributed, like vxlan. This is to easy almost.

so yeah, hmm.

spanning-tree: urgh.
port-security: no thank you.  Static ACLs only.
core ecmp/lag load-balance: don't use vpls.

Buffering is hard and gets down to having a good understanding of cost /
benefit analysis of your core network and your stakeholders'
requirements.  The main problem category which IXPs will find it
difficult to handle is the set of situations where two participant
networks are exchanging individual traffic streams at a rate which is
comparable to the egress port speed of the receiving network.

This could be 100G-connected devices sending 50G-80G traffic streams to
other 100G-connected devices, but the other main situation where this
would occur would be high speed CDNs sending traffic to access networks
where the individual ISP->customer links would be provisioned in roughly
the same speed category as the IXP-ISP link.  For example, a small
provider doing high speed gpon or docsis, with a small IXP port, e.g.
because it's only for a couple of small peers, or maybe it's a backup
port or something.

In that situation, tcp streams will flood the IXP port at rates which
are comparable to the ISP-access layer.  If you're not buffering in that
situation, the ISP will end up in trouble because they'll drop packets
like crazy and the IXP can end up in trouble because shared buffers will
be exhausted and may be unavailable for other IXP participants.

Mostly you can engineer around this, but it's not as simple as saying
that small-buffer switches aren't suitable for an IXP.  They can be, but
it depends on the network engineering requirements of the ixp
participants, and how the ixp is designed.  No simple answers here,
sorry :-(

The flip side of this is that individual StrataDNX asics don't offer the
raw performance grunt of the StrataXGS family, and there is a serious
difference in costs and performance between a 1U box with a single
tomahawk and a 2U box with a 4-way jericho config, make no mistake about it.

Otherwise white boxes can reduce costs if you know what you're doing and
you're spot on that TCO should be one of the main determinants if the
performance characteristics are otherwise similar.  TCO is a strange
beast though.  Everything from kit capex to FLS to depreciation term
counts forwards TCO, so it's important to understand your cost base and
organisational costing model thoroughly before making any decisions.


More information about the NANOG mailing list