AW: AW: AW: /27 the new /24

Mike Hammett nanog at ics-il.net
Sat Oct 3 13:10:36 UTC 2015


I don't think we are talking different things, though I think we are talking in circles and thus the thread probably needs to die. 


People keep thinking I want Level 3 to replace a loaded 6500 with a CCR and that's simply not what I'm saying at all. The point of rattling off the newer\smaller hardware was to say that if the site doesn't require 40G\100G, doesn't have the revenue to support an MX480, etc. you should put in a smaller\cheaper box. Cost is a non-issue at that point because the smaller gear that's all you need will have far less operational cost. Someone thought a particular POP was going to be a big hit... and wasn't. On the flip side, if there are 200 ports of customers chances are you need the big interfaces that aren't on the old boxes. You have the bigger revenue. Heck, the new big boxes probably still use less power than the old big boxes anyway. 



What I learned from this thread: Once you mention MT\UBNT routers, people assume you're using a MT\UBNT hammer everywhere. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


----- Original Message -----

From: "Jürgen Jaritsch" <jj at anexia.at> 
To: "Mike Hammett" <nanog at ics-il.net> 
Cc: "NANOG" <nanog at nanog.org> 
Sent: Saturday, October 3, 2015 6:06:59 AM 
Subject: AW: AW: AW: /27 the new /24 

Hi Mike, 

> but the boxes that have been there for 10 years have more than paid for themselves (unless they're a shitty business). 

No question about that! But why should they throw them away if they can still print $$$ with these boxes? They have to change nothing till the global routing table reaches at least 768k ... so let's say this will happen in 12-18 months. They have enough time to prepare, migrate, etc ... and while all the side stories are happening they are still able to print $$$ with the "old shit". 

> What I was saying is that my little business with meager means (and revenues) can afford a box to do it. 

This is definitely a question about sizing. Replacing a box with ~200 connected customers (only at this box!) is way more complex and this is nothing unrealistic. 

> If their business hasn't boomed, maybe it's time to replace that old 6500 with a 4500x or a QFX-5100 or an x670 or whatever. 

4500x => no MPLS features 

QFX-5100 => very nice box (I'm a big fan) but complicate (and expensive!) licensing. 

Extreme x670 => nice box too - we also use this. But it's simply too small and the BGP configuration on these boxes is horrible. It's also not possible to provide Ethernet over MPLS with LACP BPDU forwarding ... too less features. Nice for aggregation and POP interconnect. 

All three models are new and shiny but they can't replace a 6500/7600. Too less port density and too less features (people are still using SDH. You need SDH in an 6500/7600? Simply install the required line card ...). If you really plan to replace a 6509 or even a 6513 you have to go with something like Juniper MX480/960 (I'm in love ... :D) or Cisco Nexus 7k/9k. 

One thing that will more and more happen: physical separation. There will be boxes with 10G/40G/100G only and boxes with 100M/1G only. Why? It's easier for vendors to remove old compatibility requirements (like electrical interfaces). So what we did in the past 3 years (replacing old boxes with new boxes with 1G/10G interfaces) was useless - we'll get our "old shit" back in place and bring them up and running. Of course: the "old shit" will be reduced to do aggregation layer or to something like "multihop instance" to transport the customers access port to the "real big and powerful router". Solving this with Layer2 extensions (like VLANs) is not practicable because you'll ran into other problems (like STP instances, etc). Probably it makes sense to solve it with Layer2VPN (Ethernet over MPLS, etc) to transport the physical interface to a virtual interface. 

Lots of things to think about :(. 


> Your decreased power bill alone will pay it off. If it has boomed, then ten years of revenues should get you whatever the bigger Ciscos are or an MX or whatever the bigger Extremes are. 

Power is no argument. You get power starting at 0,10 Eur /kWh. Another 0,10 Eur / kWh for cooling and we talk about 0,20 Eur / kWh => Cisco 6513 (configured with 11 line cards + 2x SUP) with 2x 6kW PSU uses 3,8kW. 3,8kW * 24 hours * 30 days = 2.736 kWh per month. 2.736 * 0,20 Eur = 547,2 Eur per month for power consumption + cooling. If you have a good sales engineer you earn the revenue for this "side cost" with 1 customer :). Realistic calculation is: 10 customers are required to earn the money for the footprint. 


> Don't whine about my choices in gear I mentioned. I was just throwing things out there. Old big, new small if no money or old big new big if money. 

Think the other way around: companies are earning Mio (or even Bil??) with the old equipment and everything is up and running. Only sometimes there is a small hick up because (of course!) also the "old shit" gets stuck from time to time and crashes. They did everything the right way (especially Level3 ...) from the commercial POV. 

> BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle full table imports in a significantly reduced time. Oh, and I'm not sure that you couldn't do at least three nines with MT\UBNT. Well, no experience with the EdgeRouters yet. 

Never tried the earlier versions - my last tests happened in the end of 2014. I think we're talking a little bit about different sizes: you're talking about the CCRs and EdgeRouters (which are nice of course - no question about that!) and I'm talking about customer access devices (not CEP!) at carrier grade networks. Boxes I'm talking about have at least a few hundred ports. 

I think it's very important what UBNT and MT does: they bring fresh wind at the customer/semi-pro market and they show up that you (as a vendor) could get in touch with customers and optimize your equipment with customers feedback. 


best regards 


Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: JJaritsch at anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-----Ursprüngliche Nachricht----- 
Von: NANOG [mailto:nanog-bounces at nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Samstag, 03. Oktober 2015 02:52 
Cc: NANOG <nanog at nanog.org> 
Betreff: Re: AW: AW: /27 the new /24 

I don't expect carriers to be running UBNT\Mikrotik, but the boxes that have been there for 10 years have more than paid for themselves (unless they're a shitty business). It's time to rip and replace with whatever is appropriate for that site. No, I obviously don't think I'm going to change anyone's opinion on the matter (at least not anyone that matters in one of these networks). What I was saying is that my little business with meager means (and revenues) can afford a box to do it. They can too. 



I don't doubt their situation sucks... but either you fix it or you don't. Time and the rest of the Internet won't wait for them. 


If their business hasn't boomed, maybe it's time to replace that old 6500 with a 4500x or a QFX-5100 or an x670 or whatever. Your decreased power bill alone will pay it off. If it has boomed, then ten years of revenues should get you whatever the bigger Ciscos are or an MX or whatever the bigger Extremes are. 

Don't whine about my choices in gear I mentioned. I was just throwing things out there. Old big, new small if no money or old big new big if money. 


BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle full table imports in a significantly reduced time. Oh, and I'm not sure that you couldn't do at least three nines with MT\UBNT. Well, no experience with the EdgeRouters yet. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


----- Original Message ----- 

From: "Jürgen Jaritsch" <jj at anexia.at> 
To: "Mike Hammett" <nanog at ics-il.net> 
Cc: "NANOG" <nanog at nanog.org> 
Sent: Friday, October 2, 2015 6:11:47 PM 
Subject: AW: AW: /27 the new /24 

Hi Mike, 

sorry, this was probably sent to quick ... let me please explain my POV of your statement: 

I want to concentrate my detailed answer only to the backbone situation which is often handled by the 6500/7600 - I guess all of us know that the 6500/7600 has a ton of additional features ... 


6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone capacities built with platforms that only had n*100M interfaces another 3-5 years before. Their only invest in these 3-5 years was to add the Gig line cards, install some software updates and add new fibre optics (GBICs). Chassis, cabling, management interfaces etc could be remain in the cabinet - they only had to replace ONE line card (let's say for a few thousand bucks) and with this invest they were able to scale up their capacities. Of course: at some point they also had to replace the SUPs, PSUs, FANs, etc. But the invest in the surrounding stuff is nothing compared with completely new machines. 

So what all these companies did was buying a machine with an basic configuration and since 10(!) years they are able to expand this machines with (more or less) small and cheap upgrades. 

In backbone situations the 6500/7600 are definitely at the end of the resources the platform can provide. Most of the carriers (and of course also the bigger ISPs) had a real chance to evaluate a new model/vendor to ran future networks (with possibly also a very good scale-up path and scaling- and upgrade-options). Most of the before mentioned are already in an migration process (let's take a look at Seabone ... they are migration from Cisco to a mix of Juniper and Huawei). 

Summary: there are strict limitations within the Cisco 6500/7600 platform and these limitations forces the big players to move this boxes out (or move them into other parts of their network). The limitation with 1Mio routes is not a secret and the admins of these boxes decide what they want to use (e.g. 768k routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If the global routing table reaches the 768k mark (I guess this will happen in the next 12-18months) most of the boxes will crash again (as it happened in Aug 2014). 


Regarding the words "I have a small router which handles multiple full tables ...": push and pull a few full tables at the same time and you'll see what's happening: the CCRs are SLOW. And why? Because the software is not as good as it could be: the BGP daemon uses only one core of a 36(?) core CPU. Same problem in the past with the EoIP daemon (not sure if they fixed it on the CCRs - they fixed it on x86). 

Routerboards are nice and cool and to be honest: I'm a big fan of this stuff (also Ubiquiti). But with this boxes you're not able to ran a stable enterprise class carrier network with >99,5% uptime. And that’s thei MAIN reason why "the old shit" is still online :). 

Hopefully my words explained my hard "you know nothing" blabla ? 

Best regards 


Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: JJaritsch at anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-----Ursprüngliche Nachricht----- 
Von: NANOG [mailto:nanog-bounces at nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 21:33 
Cc: NANOG <nanog at nanog.org> 
Betreff: Re: AW: /27 the new /24 

Hrm. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


----- Original Message ----- 

From: "Jürgen Jaritsch" <jj at anexia.at> 
To: "Mike Hammett" <nanog at ics-il.net>, "NANOG" <nanog at nanog.org> 
Sent: Friday, October 2, 2015 2:25:10 PM 
Subject: AW: /27 the new /24 

> Stop using old shit. 

Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about carrier grade networks. 




Jürgen Jaritsch 
Head of Network & Infrastructure 

ANEXIA Internetdienstleistungs GmbH 

Telefon: +43-5-0556-300 
Telefax: +43-5-0556-500 

E-Mail: JJaritsch at anexia-it.com 
Web: http://www.anexia-it.com 

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt 
Geschäftsführer: Alexander Windbichler 
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 

-----Ursprüngliche Nachricht----- 
Von: NANOG [mailto:nanog-bounces at nanog.org] Im Auftrag von Mike Hammett 
Gesendet: Freitag, 02. Oktober 2015 20:38 
An: NANOG <nanog at nanog.org> 
Betreff: Re: /27 the new /24 

Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router. 

I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient. 


I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse. 

Stop using old shit. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


----- Original Message ----- 

From: "William Herrin" <bill at herrin.us> 
To: "Mike Hammett" <nanog at ics-il.net> 
Cc: "NANOG" <nanog at nanog.org> 
Sent: Friday, October 2, 2015 1:09:16 PM 
Subject: Re: /27 the new /24 

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog at ics-il.net> wrote: 
> How many routers out there have this limitation? A $100 router 
> I bought ten years ago could manage many full tables. If 
> someone's network can't match that today, should I really have 
> any pity for them? 

Hi Mike, 

The technology doesn't work the way you think it does. Or more 
precisely, it only works the way you think it does on small (cheap) 
end-user routers. Those routers do everything in software on a 
general-purpose CPU using radix tries for the forwarding table (FIB). 
They don't have to (and can't) handle both high data rates and large 
routing tables at the same time. 

For a better understanding how the big iron works, check out 
https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see 
folks here talk about TCAM. This stands for Ternary Content 
Addressable Memory. It's a special circuit, different from DRAM and 
SRAM, used by most (but not all) big iron routers. The TCAM permits an 
O(1) route lookup instead of an O(log n) lookup. The architectural 
differences which balloon from there move the router cost from your 
$100 router into the hundreds of thousands of dollars. 

Your BGP advertisement doesn't just have to be carried on your $100 
router. It also has to be carried on the half-million-dollar routers. 
That makes it expensive. 

Though out of date, this paper should help you better understand the 
systemic cost of a BGP route advertisement: 
http://bill.herrin.us/network/bgpcost.html 

Regards, 
Bill Herrin 




-- 
William Herrin ................ herrin at dirtside.com bill at herrin.us 
Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/> 







More information about the NANOG mailing list