1GE L3 aggregation

Mon Jun 20 06:22:56 UTC 2016

On 18/Jun/16 17:37, James Jun wrote:

> One issue with pushing IP transit (L3-wise) with small boxes down to the
> metro is that if a particular customer comes under attack, any DDoS in
> excess of 10-30 Gbps is going to totally destroy the remote site down to
> the floor and then some, until NOC intervenes to restore service.

A DoS/DDoS attack on a central IP gateway is no safer from such a
scenario. In fact, given the level of aggregation on a centralized
router, the level of impact is likely to be higher. Moreover, the attack
would still spread from the centralized router to the end customer until
the NOC intervenes. So you aren't really gaining much by centralizing
the router.

>
> A Big Expensive Router at head-end site fed with big pipes to your IP core
> just needs a subscriber line rate policer configured on the customer EVC
> off the NNI facing your metro transport network, largely protecting your
> metro POP during an attack.

So what do you do when an attack is coming from one of your Metro-E
customers to another Metro-E customer? In such a case, you've just
unnecessarily involved your centralized router in something it should
not be aware of.

>
> There are also issues with control-plane policing (or limited options
> there of) with some of these low-end platforms.

On the ASR920's we use, CoPP is reasonable. But as with everything else,
you have to make some compromises if you want to keep your costs down
without sacrificing too much in operation. Given the spread you'd get
with IP/MPLS in the Access, the problem is broken down into smaller,
more manageable chunks, which appeals more to me.

> BGP Selective Download has its own drawbacks too--in that, it's largely
> meant to be used on single-tailed environment with FIB only having single
> point of egress.
>
> Consider a topology where an ASR920 in the metro is dual-homed to two 
> peering sites using variably distant dark fiber (say 30km to Site A,
> 90km to Site B), with IGP costs configured to conform to fiber distances.

For us, that is not a Metro-E Access ring. It's too wide and would
normally be broken up by some kind of PE Aggregation. If you're building
Metro-E Access rings that wide, you're going to end up in trouble sooner
rather than later.

We build Metro-E Access rings within 1ms, and while 1ms will give you a
100km cable radius, we'd never build a Metro-E Access ring that wide. So
we're typically running the rings at between 1km - 10km. And since we do
latency-based IGP cost, there is no difference whether a ring is 1km or
10km wide (or even in your example, 30km vs. 90km).

>
> How will you guarantee that the best-path that ASR920 chooses for your
> customer taking full table to be actually congruent with the box's
> real forwarding path?  Your 920 may choose site A as best-path, only
> to see FIB-programmed default route to force it out on site B.  If you're
> doing active-standby on your fiber uplinks, then it would not be an issue;
> or may be in metro environment where latency differences are minimal (<1ms),
> you probably don't care in practice to be bothered with that.

Not sure I get your scenario.

We use BGP-SD where our Metro-E devices have iBGP sessions to our RR's.
We download 0/0 + ::/0 + some other routes into FIB, and keep the rest
in control plane. We do not see any discrepancy in NEXT_HOP data between
the control or data plane when we run BGP-SD.

Have you run into the issues you describe when you've turned on BGP-SD?

>
> Yes, there are some operational complexities and issues with L2vpn'ing
> customers to head-end router -- such as, link-state propagation needs to
> be properly validated; and now you're burning two ports instead of one,
> one at the terminus, one at the access, doubling SPOF and maintenance
> liabilities.

The only use-case we've seen for centralizing routing is in a BNG scenario.

I once attempted a distributed BNG design, but the issue was load on the
control plane of each router (DHCP, session management, e.t.c.). One
could deploy a dedicated edge router for BNG terminations alongside an
edge router for Business/Wholesale customers, but that gets pricey, quickly.

But even with centralized BNG's, we are now looking at ways to
distribute them further with current virtualization technologies, into
smaller islands that each can handle a decent amount of aggregate bandwidth.

>
> At the end of the day, it's lack of full-featured ports at reasonable cost
> that drive centralization to head-ends.  Spamming Small Expensive Routers
> (ASR9001/MX104) in every small metro site doesn't scale (btdt myself), but
> neither is hacking up BGP to work on platforms that aren't really meant to
> function as heavy L3 routers (e.g. ASR920, ME3600), IMHO.

I disagree.

Adding high-end BGP support to the ME3600X/3800X might have been an
afterthought, but we are lucky that it had sufficient resources to
support it.

On the ASR920, that was designed in from Day One, and we've been happy
running full BGP there as well (in addition to doing it on the ME3600X
as well).

Been doing this 2010, and it's going well. The level of simplicity we
enjoy, and how quickly we can turn up a customer service, the
decoupling/independence of devices and the ability to run maintenance
activities in a controlled way that minimize aggregate impact are
benefits I'd never trade for a centralized router approach.

Mark.