Outbound Route Optimization

Fri Jan 23 19:01:14 UTC 2004

I have been on a personal crusade for the last 8 months to address this
very issue!

We identified the exact same issues and questions as we grew from a
single backbone to 7 backbones, each of various sizes ranging from gig
connections to DS3s. In total I have almost 3GB of total available
capacity, but two small DS3 links make routing decisions very
interesting :-)

It was becoming a nightmare for my engineers to manage the BGP for all
of these backbones in such a way that dealt with both the business case
as well as the performance case. In the end, it was becoming a customer
service problem when we had spikes that saturated some of our smaller
links and left our larger links untouched. BGP simply did not care about
my capacity issues.

In our specific setting, we are an ISP that buys all of our connectivity,
and has spent a tremendous amount of time searching for total connectivity
as opposed to total capacity. While most of our bandwidth per mb costs
the same, our commit levels with our different carriers are different
and required constant vigilance to maintain the levels we needed to see
without overloading any particular link. We have no private peering at
all.

After some very unfortunate dealings with a bandwidth provider in the "performance
based routing" business, I decided to do it on my own.

Its important to note that in my world, my mandate was simple - get us
the best possible performance from our network as you can possibly get.
Worry about cost after performance. We house some large VoIP, Gaming and
E-Commerce farms and cost was the lowest concern on our plate - keeping
the customers happy was the primary concern.

I started out by going from 2 backbones to  buying backbone bandwidth
from a total of 7 carriers, spreading those out among Cisco 7507s and
Juniper M20s and basically relying on BGP and my engineering staff to
monitor and manage those resources.

In the end I discovered that it was a huge job to keep all of those
balls in the air while not upsetting some of our larger customers,

I spent months researching and talking to friends that drive some of the
largest networks in the world. In the end, it was very clear to see that
BGP was not up to the task of dealing with my network requirements. Best
path simply did not equate to best performance and BGP had no
provisions for determining saturation on my links. 

My engineers and I spent months talking to vendor after vendor about
their products, doing research and trying to find the closest thing to a
'silver bullet' that we could find.

An engineer friend of mine at Google turned me onto RouteScience and we
put them into the mix of vendors we were testing. Our needs were simple
- 100% performance based routing until we came within 15% max
utilization on any given backbone, then next best performance path. In
my world, cost based routing was the last thing we needed to deal with.

We enlisted the help of several of our larger data center customers in a
kind of blind trial of the various manufacturers as well as utilized
KeyNote locations around the world for testing. After four months of
testing and evaluation, we choose the RouteScience box.

In my mind, the question about utilizing route optimization boxes is moot.
Until we build into BGP (or some other method) the ability to sense
latency and capacity issues, optimize bandwidth allocation based on our
preferences, and maintain service level agreements by keeping our
traffic heading down the best performance path automatically, we have to
employ and dedicate an increasing number of engineers to these tasks.
Route Optimization equipment plays a critical part in keeping my
customers happy and myself and my other  expensive engineers focused on
other tasks more closely related to the bottom line. 

No smoke, no mirrors, no BS - these are real world numbers from our
network. For me the proof was in the performance. After four months of
baseline reporting, we were seeing an average performance increase
(measure in decrease in latency) of 40 to 50% between the routes my
pathcontrol box is selecting and standard BGP routes. My backbones
include carriers such as Level3, UUNet, Qwest, XO, Verio - decent
backbones with major connectivity.

In reality, I learned that BGP is simply not up to the task of handling
anything beyond its limited scope - best path routing. In today's world,
we need to look beyond best path as it simply has nothing to do with
best performance, at least not in 40 to 50% of my traffic routing
decisions. You can do that with bodies (if your a purest) or you can
utilize route optimization equipment. In either case, you have to do it.

I think for the time being, route optimization equipment, and the
companies that utilize them will have an edge over those doing things
the manual way. Regardless of which box I could have chosen, the end
result is that myself and my  backbone engineers have far more time on
their hands for other tasks and my customers are much happier than they
were before.

On Wed, 21 Jan 2004 12:27:16 -0800
"Jim Devane" <jim at powerpulse.cc> wrote:

> Hello,
> 
>  
> 
> I am trying to determine for myself the relevance of Intelligent Routing
> Devices like Sockeye, Route Science etc. I am not trying to determine who
> does it better, but rather if the concept of optimizing routes is addressing
> a significant problem in terms of improved traffic performance ( not in cost
> savings of disparate transit pipes )
> 
> I am interested in hearing other views ( both for and against ) these
> devices in the context of optimizing latency for a small multi-homed ISP. I
> want to make sure I understand their context correctly and have not missed
> any important points of view. 
> 
>             My questions are these:
> 
>  
> 
> "Is sub-optimal routing caused by BGP so pervasive it needs to be
> addressed?" 
> 
> "Are these devices able to effectively address the need?"
> 
>  
> 
> Thanks,
> 
> Jim
> 
>  
> 

******************************************
Richard J. Sears
Vice President         
American Digital Network                          
----------------------------------------------------
rsears at adnc.com
http://www.adnc.com
----------------------------------------------------
858.576.4272 - Phone
858.427.2401 - Fax
----------------------------------------------------

I fly because it releases my mind 
from the tyranny of petty things . . 

"Work like you don't need the money, love like you've
never been hurt and dance like you do when nobody's
watching."