Mikrotik in the DFZ (Was Re: AW: AW: /27 the new /24)

Sat Oct 3 13:29:44 UTC 2015

Sure MT has issues, but so does everyone. As someone that has used them for 10+ years, the past six months has seen a bit of a re-awakening over there. You can see this in the time to completion of many feature requests, bug fixes, new features, etc. I'm not sure they're going to do everything everyone is after, but they certainly have shown a huge increase in willingness to go the right direction. 

Of course it's easy for someone running big iron to scoff at the lack of feature X or feature Y. To that I say, what are the capabilities of your $200 router? Your $2k router? I haven't priced out new low-end gear from the big iron vendors, but I can't imagine at what price point you need to be at to have a multi-gig capable VPLS router. For Mikrotik you're in the $200 - $1k range, depending on what you mean by "multi-gig". One thing I miss as I start to use more non-Mikrotik hardware... Torch. I wish everything had Torch. Put Packet Sniffer in the list of things I'd like to see everywhere. I don't want port mirror as who's to say I have something to mirror to everywhere that can also capture? Put a few basic filters and drop the PCAP right on the damn box. Now obviously with something running BSD you could code up whatever you'd like or have an array of open-source packages to work with , but that wouldn't have the nice feature integration of a router OS. 

I have no problem running Mikrotik in the DFZ. Mine pull down full tables in 30 - 35 seconds, can handle somewhere in the 30 - 60 gb range when firewall rules are applied and so on. They'd cost under $1,500 new, but I got mine put together for a fraction of that. They're so cheap you can run two. Run two and now you have the advantage of being able to do maintenance without downtime. It's a little kludgey, but can get get the job done at a price point the others can't. Maybe with newer CCRs and ROS7 I could drop the need for the x86 boxes. We'll see. 

----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest Internet Exchange 
http://www.midwest-ix.com 

----- Original Message -----

From: "William Waites" <wwaites at tardis.ed.ac.uk> 
To: jj at anexia.at 
Cc: nanog at ics-il.net, nanog at nanog.org 
Sent: Saturday, October 3, 2015 3:23:49 AM 
Subject: Mikrotik in the DFZ (Was Re: AW: AW: /27 the new /24) 

On Fri, 2 Oct 2015 23:11:47 +0000, Jürgen Jaritsch <jj at anexia.at> said: 

> Regarding the words "I have a small router which handles 
> multiple full tables ...": push and pull a few full tables at 
> the same time and you'll see what's happening: the CCRs are 
> SLOW. And why? Because the software is not as good as it could 
> be: the BGP daemon uses only one core of a 36(?) core CPU. 

To expand on this, the problem is worse than being single-threaded. I 
had one of these in the lab and fed it 2x full tables. Sure it wasn't 
the fastest at accepting them but then I noticed that even in steady 
state one of the CPUs was pegged. What was happening -- and this was 
confirmed by Mikrotik -- was that it was recalculating the *entire* 
FIB for each update. The general background noise of announce / 
withdraw messages means it is doing this all the time. Any churn and 
it would have a very hard time. 

There are other serious bugs such as not doing recursive next hop 
lookup for IPv6 (it does for IPv4). This makes them unuseable as BGP 
routers even for partial tables with most non-trivial iBGP 
topologies. All of which may be fixed one day in version 7 of their 
operating system, which will inevitably have many bugs as any software 
project .0 release will, so we'll have to wait for 7.x for it to be 
reasonably safe to use. 

That said, we use a lot of Mikrotik kit for our rural 
networks. They're weird and quirky but you can't beat them on price, 
port density and power consumption. With 16 ports and 36 cores surely 
they should be capable of pushing several Gbps of traffic with a few 
full tables. 

I wish it were possible today to run different software on their 
larger boxes. If some like-minded small providers wanted to get 
together with us to fund a FreeBSD port to the CCR routers that would 
be great. Please contact me off-list if you are interested in this, 
I'll coordinate. 

As it is we don't let them anywhere near the DFZ, that's done with PCs 
running FreeBSD and BIRD which can easily do the job but is still an 
order of magnitude more expensive (and an order of magnitude less 
expensive than what you need if you want 10s of Gbps). 

-w 

-- 
William Waites <wwaites at tardis.ed.ac.uk> | School of Informatics 
http://tardis.ed.ac.uk/~wwaites/ | University of Edinburgh 
https://hubs.net.uk/ | HUBS AS60241 

The University of Edinburgh is a charitable body, registered in 
Scotland, with registration number SC005336. 