The Making of a Router

Sat Dec 28 06:39:55 UTC 2013

There has been a lot of conversation lately regarding 10Gbps+ routing
without higher cost devices such as the junipers. I have been looking into
a few options myself, below are my opinions so far. What are your
recommendations, real life experiences and ideas?

-    Mikrotik Cloud Core Router
The Mikrotik CCR might have 2 SFP+ ports but with any ACLs, etc fast path
is disabled, this already limits the functionality a lot. The BGP
calculations only happen on a single core which provides very slow
performance for full routing tables. RouterOS is very unstable and had a
large number of bugs even with version 6. I have had issues using them even
on some small test environments, would not recommend this hardware for
nearly any setup.
-    Linux Based Software Routing
Quagga is great for BGP with the correct CPUs and configurations. Vyatta or
VyOS provides a stable and simple configuration method for Quagga. The
issues with all of the options currently available is forwarding plane
performance, you are only looking at 1Gbps+ at line rate. Most providers
will have to deal with DDOS attacks at one point or another and would not
recommend taking the chance. If you are only looking at 1Gbps or less worth
of traffic this is a great option.
DDOS attacks information from just the Arbor Networks hardware.
http://www.digitalattackmap.com/
Userspace processing of the forwarding plane will help a lot to overcome
this issue. There are a few different solutions out there but the most
common is Intel DPDK. Some of you would know about the Intel DPDK from the
upcoming brocade vRouter 5600 which supports 10Gbps line rate per core. I
can see Intel DPDK being used for other solutions such as DDOS filtering as
currently you require specialised hardware such as Arbor Networks or
NSFOCUS. It would be much cheaper if you could do some filtering from x86
hardware at line rate.
http://blog.lukego.com/blog/2013/01/04/kernel-bypass-networking/
Brocade vRouter 5600 might be an option when it is released depending on
price. As you still need to get all the hardware required and make sure you
do your research regarding the chipsets, etc. Most Intel SFP+ NIC will
handle around 9MPPS but has great support for drivers. Solarflare have some
nice NICs that can handle 16MPPS but I can see a lot more reviews for
different manufacturers coming out after the vRouter release. Hopefully
VyOS or some other open source project can integrate Intel DPDK.
-    OpenFlow
OpenFlow is a great method for really high PPS but the major limiting
factor is the flow entries and flow mods. I personally like this
architecture as it allows the control plane to run on X86 and the Data
Plane to run on specialised hardware. For providers with 1 IP transit
provider and a few peering IX most OpenFlow hardware will support enough
flow entries. The issue is supporting providers with a reasonable number of
full routing tables; I think summarization will help a decent amount to
lower the flow entries required. NoviSwitch 1248 supports 1 million flow
entries which is a reasonable number for smaller providers. I have only
started to get my hand dirty with OpenFlow and would like to know if anyone
is using it in production for routing? What OpenFlow controller are you
using? E.g. RouteFlow
https://sites.google.com/site/routeflow/
-    Brocade CER
The older model CER devices had a lot of issues/bugs but the newer models
such as BR-CER-2024C-4X-RT-AC seem to be a lot more stable. There are
reviews on webhostingtalk with people pushing more than 30Gbps on the newer
models without issue. Based on other people’s comments such as Jon Sands
the units should be around 10K each new which makes the units cost
affective for a lot of implementations. If you are lucky enough to find one
second hand you would only be looking around $5-6K. The 2024C-4X-RT has 4
SFP+ ports which is alright but would really like to see some larger
options. Currently a lot of people just create a port channel with all 4
ports to a SFP+ switch which allow them to connect more ports up but need
to be careful about overprovisioning.
-    Layer 3 SFP+ Switch
Great for providers with only one uplink as they just use a default route
but most providers require more than one uplink. There are lot of cheap
options out there even the junipers are not that costly.

Regards,
Steven.

Date: Fri, 27 Dec 2013 21:34:00 -0500 (EST)
From: "Justin M. Streiner" <streiner at cluebyfour.org>
To: William Waites <wwaites at tardis.ed.ac.uk>
Cc: nanog at nanog.org
Subject: Re: The Making of a Router
Message-ID: <Pine.LNX.4.64.1312272133090.22688 at whammy.cluebyfour.org>
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed

On Sat, 28 Dec 2013, William Waites wrote:

> On Fri, 27 Dec 2013 07:23:36 -0500 (EST), "Justin M. Streiner" <
streiner at cluebyfour.org> said:
>
>    > You end up combining some of the downsides of a hardware-based
>    > router with some of the downsides of a server (new attack
>    > vectors, another device that needs to be backed up, patched, and
>    > monitored...
>
> Might be a good idea to back up, patch and monitor your routers
> too... Just sayin'

Yes, a given.

jms