AW: AW: /27 the new /24

Sat Oct 3 00:52:17 UTC 2015

I don't expect carriers to be running UBNT\Mikrotik, but the boxes that have been there for 10 years have more than paid for themselves (unless they're a shitty business). It's time to rip and replace with whatever is appropriate for that site. No, I obviously don't think I'm going to change anyone's opinion on the matter (at least not anyone that matters in one of these networks). What I was saying is that my little business with meager means (and revenues) can afford a box to do it. They can too. 

I don't doubt their situation sucks... but either you fix it or you don't. Time and the rest of the Internet won't wait for them. 

If their business hasn't boomed, maybe it's time to replace that old 6500 with a 4500x or a QFX-5100 or an x670 or whatever. Your decreased power bill alone will pay it off. If it has boomed, then ten years of revenues should get you whatever the bigger Ciscos are or an MX or whatever the bigger Extremes are. 

Don't whine about my choices in gear I mentioned. I was just throwing things out there. Old big, new small if no money or old big new big if money. 

BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle full table imports in a significantly reduced time. Oh, and I'm not sure that you couldn't do at least three nines with MT\UBNT. Well, no experience with the EdgeRouters yet. 

----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest Internet Exchange 
http://www.midwest-ix.com 

----- Original Message -----

From: "Jürgen Jaritsch" <jj at anexia.at> 
To: "Mike Hammett" <nanog at ics-il.net> 
Cc: "NANOG" <nanog at nanog.org> 
Sent: Friday, October 2, 2015 6:11:47 PM 
Subject: AW: AW: /27 the new /24 

Hi Mike, 

sorry, this was probably sent to quick ... let me please explain my POV of your statement: 

I want to concentrate my detailed answer only to the backbone situation which is often handled by the 6500/7600 - I guess all of us know that the 6500/7600 has a ton of additional features ... 

6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone capacities built with platforms that only had n*100M interfaces another 3-5 years before. Their only invest in these 3-5 years was to add the Gig line cards, install some software updates and add new fibre optics (GBICs). Chassis, cabling, management interfaces etc could be remain in the cabinet - they only had to replace ONE line card (let's say for a few thousand bucks) and with this invest they were able to scale up their capacities. Of course: at some point they also had to replace the SUPs, PSUs, FANs, etc. But the invest in the surrounding stuff is nothing compared with completely new machines. 

So what all these companies did was buying a machine with an basic configuration and since 10(!) years they are able to expand this machines with (more or less) small and cheap upgrades. 

In backbone situations the 6500/7600 are definitely at the end of the resources the platform can provide. Most of the carriers (and of course also the bigger ISPs) had a real chance to evaluate a new model/vendor to ran future networks (with possibly also a very good scale-up path and scaling- and upgrade-options). Most of the before mentioned are already in an migration process (let's take a look at Seabone ... they are migration from Cisco to a mix of Juniper and Huawei). 

Summary: there are strict limitations within the Cisco 6500/7600 platform and these limitations forces the big players to move this boxes out (or move them into other parts of their network). The limitation with 1Mio routes is not a secret and the admins of these boxes decide what they want to use (e.g. 768k routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If the global routing table reaches the 768k mark (I guess this will happen in the next 12-18months) most of the boxes will crash again (as it happened in Aug 2014). 

Regarding the words "I have a small router which handles multiple full tables ...": push and pull a few full tables at the same time and you'll see what's happening: the CCRs are SLOW. And why? Because the software is not as good as it could be: the BGP daemon uses only one core of a 36(?) core CPU. Same problem in the past with the EoIP daemon (not sure if they fixed it on the CCRs - they fixed it on x86). 

Routerboards are nice and cool and to be honest: I'm a big fan of this stuff (also Ubiquiti). But with this boxes you're not able to ran a stable enterprise class carrier network with >99,5% uptime. And that’s thei MAIN reason why "the old shit" is still online :). 

Hopefully my words explained my hard "you know nothing" blabla ? 

Best regards 

Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: JJaritsch at anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-----Ursprüngliche Nachricht----- 
Von: NANOG [mailto:nanog-bounces at nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 21:33 
Cc: NANOG <nanog at nanog.org> 
Betreff: Re: AW: /27 the new /24 

Hrm. 

----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest Internet Exchange 
http://www.midwest-ix.com 

----- Original Message ----- 

From: "Jürgen Jaritsch" <jj at anexia.at> 
To: "Mike Hammett" <nanog at ics-il.net>, "NANOG" <nanog at nanog.org> 
Sent: Friday, October 2, 2015 2:25:10 PM 
Subject: AW: /27 the new /24 

> Stop using old shit. 

Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about carrier grade networks. 

Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: JJaritsch at anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-----Ursprüngliche Nachricht----- 
Von: NANOG [mailto:nanog-bounces at nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 20:38 
An: NANOG <nanog at nanog.org> 
Betreff: Re: /27 the new /24 

Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router. 

I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient. 

I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse. 

Stop using old shit. 

----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest Internet Exchange 
http://www.midwest-ix.com 

----- Original Message ----- 

From: "William Herrin" <bill at herrin.us> 
To: "Mike Hammett" <nanog at ics-il.net> 
Cc: "NANOG" <nanog at nanog.org> 
Sent: Friday, October 2, 2015 1:09:16 PM 
Subject: Re: /27 the new /24 

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog at ics-il.net> wrote: 
> How many routers out there have this limitation? A $100 router 
> I bought ten years ago could manage many full tables. If 
> someone's network can't match that today, should I really have 
> any pity for them? 

Hi Mike, 

The technology doesn't work the way you think it does. Or more 
precisely, it only works the way you think it does on small (cheap) 
end-user routers. Those routers do everything in software on a 
general-purpose CPU using radix tries for the forwarding table (FIB). 
They don't have to (and can't) handle both high data rates and large 
routing tables at the same time. 

For a better understanding how the big iron works, check out 
https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see 
folks here talk about TCAM. This stands for Ternary Content 
Addressable Memory. It's a special circuit, different from DRAM and 
SRAM, used by most (but not all) big iron routers. The TCAM permits an 
O(1) route lookup instead of an O(log n) lookup. The architectural 
differences which balloon from there move the router cost from your 
$100 router into the hundreds of thousands of dollars. 

Your BGP advertisement doesn't just have to be carried on your $100 
router. It also has to be carried on the half-million-dollar routers. 
That makes it expensive. 

Though out of date, this paper should help you better understand the 
systemic cost of a BGP route advertisement: 
http://bill.herrin.us/network/bgpcost.html 

Regards, 
Bill Herrin 

-- 
William Herrin ................ herrin at dirtside.com bill at herrin.us 
Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/> 