AW: AW: /27 the new /24

Jürgen Jaritsch jj at anexia.at
Fri Oct 2 23:11:47 UTC 2015


Hi Mike,

sorry, this was probably sent to quick ... let me please explain my POV of your statement:

I want to concentrate my detailed answer only to the backbone situation which is often handled by the 6500/7600 - I guess all of us know that the 6500/7600 has a ton of additional features ...


6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone capacities built with platforms that only had n*100M interfaces another 3-5 years before. Their only invest in these 3-5 years was to add the Gig line cards, install some software updates and add new fibre optics (GBICs). Chassis, cabling, management interfaces etc could be remain in the cabinet - they only had to replace ONE line card (let's say for a few thousand bucks) and with this invest they were able to scale up their capacities. Of course: at some point they also had to replace the SUPs, PSUs, FANs, etc. But the invest in the surrounding stuff is nothing compared with completely new machines.

So what all these companies did was buying a machine with an basic configuration and since 10(!) years they are able to expand this machines with (more or less) small and cheap upgrades. 

In backbone situations the 6500/7600 are definitely at the end of the resources the platform can provide. Most of the carriers (and of course also the bigger ISPs) had a real chance to evaluate a new model/vendor to ran future networks (with possibly also a very good scale-up path and scaling- and upgrade-options). Most of the before mentioned are already in an migration process (let's take a look at Seabone ... they are migration from Cisco to a mix of Juniper and Huawei).

Summary: there are strict limitations within the Cisco 6500/7600 platform and these limitations forces the big players to move this boxes out (or move them into other parts of their network). The limitation with 1Mio routes is not a secret and the admins of these boxes decide what they want to use (e.g. 768k routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If the global routing table reaches the 768k mark (I guess this will happen in the next 12-18months) most of the boxes will crash again (as it happened in Aug 2014). 


Regarding the words "I have a small router which handles multiple full tables ...": push and pull a few full tables at the same time and you'll see what's happening: the CCRs are SLOW. And why? Because the software is not as good as it could be: the BGP daemon uses only one core of a 36(?) core CPU. Same problem in the past with the EoIP daemon (not sure if they fixed it on the CCRs - they fixed it on x86).

Routerboards are nice and cool and to be honest: I'm a big fan of this stuff (also Ubiquiti). But with this boxes you're not able to ran a stable enterprise class carrier network with >99,5% uptime. And that’s thei MAIN reason why "the old shit" is still online :).

Hopefully my words explained my hard "you know nothing" blabla ?

Best regards


Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: JJaritsch at anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-----Ursprüngliche Nachricht-----
Von: NANOG [mailto:nanog-bounces at nanog.org] Im Auftrag von Mike Hammett
Gesendet: Freitag, 02. Oktober 2015 21:33
Cc: NANOG <nanog at nanog.org>
Betreff: Re: AW: /27 the new /24

Hrm. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


----- Original Message -----

From: "Jürgen Jaritsch" <jj at anexia.at> 
To: "Mike Hammett" <nanog at ics-il.net>, "NANOG" <nanog at nanog.org> 
Sent: Friday, October 2, 2015 2:25:10 PM 
Subject: AW: /27 the new /24 

> Stop using old shit. 

Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about carrier grade networks. 




Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: JJaritsch at anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-----Ursprüngliche Nachricht----- 
Von: NANOG [mailto:nanog-bounces at nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 20:38 
An: NANOG <nanog at nanog.org> 
Betreff: Re: /27 the new /24 

Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router. 

I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient. 


I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse. 

Stop using old shit. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


----- Original Message ----- 

From: "William Herrin" <bill at herrin.us> 
To: "Mike Hammett" <nanog at ics-il.net> 
Cc: "NANOG" <nanog at nanog.org> 
Sent: Friday, October 2, 2015 1:09:16 PM 
Subject: Re: /27 the new /24 

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog at ics-il.net> wrote: 
> How many routers out there have this limitation? A $100 router 
> I bought ten years ago could manage many full tables. If 
> someone's network can't match that today, should I really have 
> any pity for them? 

Hi Mike, 

The technology doesn't work the way you think it does. Or more 
precisely, it only works the way you think it does on small (cheap) 
end-user routers. Those routers do everything in software on a 
general-purpose CPU using radix tries for the forwarding table (FIB). 
They don't have to (and can't) handle both high data rates and large 
routing tables at the same time. 

For a better understanding how the big iron works, check out 
https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see 
folks here talk about TCAM. This stands for Ternary Content 
Addressable Memory. It's a special circuit, different from DRAM and 
SRAM, used by most (but not all) big iron routers. The TCAM permits an 
O(1) route lookup instead of an O(log n) lookup. The architectural 
differences which balloon from there move the router cost from your 
$100 router into the hundreds of thousands of dollars. 

Your BGP advertisement doesn't just have to be carried on your $100 
router. It also has to be carried on the half-million-dollar routers. 
That makes it expensive. 

Though out of date, this paper should help you better understand the 
systemic cost of a BGP route advertisement: 
http://bill.herrin.us/network/bgpcost.html 

Regards, 
Bill Herrin 




-- 
William Herrin ................ herrin at dirtside.com bill at herrin.us 
Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/> 




More information about the NANOG mailing list