Route Reflector architecture and how to get small customer blocks in to BGP?

Sat Jan 27 18:39:54 UTC 2007

Hey all,

I know this will start an argument about relevance, but think of it  
as one network operator trying to help another operator do good and  
have a network where the potential for stupid routing leaks is  
minimized...

I've got a friend (really, it's not me!) who works at a regional  
ISP . The previous network designer didn't seem to have much of a  
clue, and has tons of customer nets in OSPF. So he's trying to  
migrate them properly to BGP. I know opinions will vary widely, but  
that's exactly what we're looking for. What pros/cons haven't we  
thought of?

First, they've got a BGP full mesh of all their routers. They're  
considering moving towards route reflectors. There's 2 core routers  
per-POP. And anywhere between 5 and 15 edge/aggregation routers in a  
POP. The current thought is to move to a route-reflector full mesh  
between all the dual-core routers in each POP. The other alternative  
is to deploy just 2 route reflectors for the entire network. Can  
anyone point me towards real-world info on the pros and cons of each  
approach? There seems to be little public info on why people do what  
they do, it's more info on how to do it.

Next I'm looking for differing advice on getting small customer- 
assigned blocks in to BGP. Like /29s that are given to fractional T1  
customers.

Assume that there are /29s assigned to customers. It would be a  
beautiful world if these /29s could be easily rolled up in to /24s,  
then have a static route /24 pointed towards the bitbucket, and  
advertise the /24 in to BGP. But alas the network is not that neat. / 
29s are scattered all over various customer aggregation routers.  
Sometimes it's just one /29 out of a a /24 block that's on another  
router. Also, there's /22s to /19s per-pop in nice little  
aggregatable subnets, so at least that's good.

- Should he use a permanent static route to the /29, then use a  
network statement to bring it in to BGP?
- Should he use a permanent static route, and redistribute *very*  
carefully, so that he can do tagging with communities?
- Should he use a static route which would be withdrawn if the link  
went down? This would mean traffic to a down customer would be  
dropped quicker, but flaps cause more BGP churn.

This would mean he's have hundreds to thousands of /29 routes  
floating around internally. But the /29s would be portable between  
edge aggregation routers.

- Should he have a /24 routed to the bitbucket, and aggregate /29s  
where he can, and have some exception /29s floating around randomly?  
The problem here is what if the customer needs to move to another  
router, because he wants a bigger pipe and his circuit can't be  
connected to the existing router?

- Should he take the hard line approach and force customers to  
renumber? If they upgrade their service, and have to move routers,  
they get assigned a new /29 subnet. In this world of NAT, it should  
be easy! ;) That way he can have nice contiguous blocks to announce.

- Would confederations help? Seems like overkill, but he could  
aggregate at the POP level instead of the router level.

How do the rest of you assign out customer blocks? On a per-router  
basis, on a per-pop basis? How do you keep the number of routes down  
to a manageable level? How do you make it easy for installation /  
provisioning engineers to bring up a customer, but never get near the  
BGP config (assume they'll login to put a /30 on the link interface)?

Any pointers to sites on the net that show how real world ISPs setup  
their route/policy maps? I'm not talking BGP intro stuff like "make  
sure you don't announce a default to your upstreams" but examples of  
what bgp communities are used for, what real world filtering is done  
beyond basics and bogon filtering, etc. I can't even find a good  
nanog presentation on this. They're all about the basics.

Let the floodgates begin! If you want to tell me this is off-topic,  
please do us all a favor and email me directly. Or don't read it and  
simply trash it. Nobody wants to hear why you're the on-topic police  
and what you say goes.

-pete