AW: AW: AW: /27 the new /24

Mon Oct 5 04:45:27 UTC 2015

On Sat, Oct 03, 2015 at 08:10:36AM -0500, Mike Hammett wrote:
> 
> People keep thinking I want Level 3 to replace a loaded 6500 with a CCR and that's simply not what I'm saying at all. The point of rattling off the newer\smaller hardware was to say that if the site doesn't require 40G\100G, doesn't have the revenue to support an MX480, etc. you should put in a smaller\cheaper box. 
> Cost is a non-issue at that point because the smaller gear that's all you need will have far less operational cost. Someone thought a particular POP was going to be a big hit... and wasn't. 

In an SP environment, there is an escalating operating cost and network complexity to having small full-featured routers (ie. MX80, ASR9001, CER2k, etc) at every data center, POP or anywhere you need to terminate customers.  The reality is that small routers (even if you were to use ghetto routers) have poor economics in port density.  It's feasible for a startup ISP to spam MX80 or equivalent anytime they need more ports, but there comes a point where plopping a big chassis is the way to go.

At my place, we started with MX80s to cheap out on router ports anytime we had to light a data center.  That only got us far and we ended up having to migrate to ASR 9010s and start phasing out small routers.  The increasing complexity of having dozens of small routers and managing LSP mesh to remainder of the network is ugly.  Moreover, full-table BGP routers are also the places where you exercise edge policy with complex routing policies.  Even with automation, managing dozens of those in a region that could have been served by only 2 routers is annoying.  It's easier to haul IP customers to fewer, but more reliable large-chassis platforms and use packet-optical network to get to the customer premise.

Between the above and the lack of control-plane redundancy on most small routers, there are operational complexities & economic realities to keep in mind; it's not strictly about whether a site requires 40G/100G.  

> On the flip side, if there are 200 ports of customers chances are you need the big interfaces that aren't on the old boxes. You have the bigger revenue. Heck, the new big boxes probably still use less power than the old big boxes anyway. 

The idea has its merits, however in practice, it isn't feasible.  People don't put in line cards into their router with expectation that they need to be replaced 2 years down the road because FIB TCAM ran out.  Even if you have the revenue to justify new line cards, constant migration of customer interfaces means disruptive maintenance for that customer.  We'd prefer IP network to be as reliable as dial-tone, if possible.

The global routing table is approaching 600k today.  Lot of line cards in installed base today only handle ~1.0/1.3 to ~1.8 million IPv4.  When you start replacing those line cards (and mind you, a 24x10GE line card has a list price running into $300k range), the next logical level is 4 million IPv4.  With all the deaggs with /24s, just how long of time are we going to have with /27 explosion before 4 million FIB runs out of space?

I can see /25 being contemplated, but the cost of moving to /27 just isn't worth it at the moment.

> 
> What I learned from this thread: Once you mention MT\UBNT routers, people assume you're using a MT\UBNT hammer everywhere. 

I'm not aware of any carrier-grade network that operates on these things.

Best,
james