Connectivity to Brazil

Wed Feb 2 14:22:59 UTC 2011

Algar looking Glass:

http://lg.ctbc.com.br/lg/

I found the response from the GC engineer.  It won't mean much cut&pasted verbatim and out of context, so I'll just summarize.

Basically, I was seeing vastly different response times to given hosts in the same subnet on CTBC's network.  My source ip was the same.  Traceroute revealed to me the packets taking different paths, after hitting a particular GLBX router.

The GC engineer said CTBC is a customer that hangs off of this particular router, and that traffic was hitting an interface and hair-pinning back out to the customer's segment.  He pointed to possible congestion on CTBCs switching fabric as the cause for the varied response times (conceivable) and different routes (um...ok maybe some kind of load-balancing at play).

I managed to call CTBC (Algar) and confirmed they were experiencing congestion issues and they had a scheduled circuit capacity upgrade with GLBX in a few weeks.

As a network admin, i find it sometimes sadly humorous how we can poke and prod at a problem, go through various carrier support channels and scratch our heads for hours/days when all it would normally take is a RO userid on a couple routers in the path to figure things out.  that's not too much to ask, right?  ;)

-matt

> CC: vinny at abellohome.net; nanog at nanog.org
> From: the76posse at gmail.com
> Subject: Re: Connectivity to Brazil
> Date: Wed, 2 Feb 2011 08:07:14 -0500
> To: gourmetcisco at hotmail.com
> 
> That thread detail would be very interesting to me.  Thanks for the heads up.   
> 
> Sent from my iPhone
> 
> On Feb 2, 2011, at 7:14 AM, Matt Disuko <gourmetcisco at hotmail.com> wrote:
> 
> > Very interesting.  I have had similar issues with connectivity to my Brazil office, and oddly enough it involved GBLX and CTBC (now called Algar Telecom).  I also vastly divergent paths to a couple hosts in the same subnet.  I ended up communicating with GBLX via email (who were actually really great in corresponding with  me)...the engineer pointed to some sort of link capacity issue...i'll dig up the thread...
> > 
> > > Date: Wed, 2 Feb 2011 01:21:09 -0500
> > > From: vinny at abellohome.net
> > > Subject: Re: Connectivity to Brazil
> > > To: the76posse at gmail.com
> > > CC: nanog at nanog.org
> > > 
> > > We saw similar issues with IKE through Global Crossing (as odd as that sounds) out of the NYC market at the same time. We routed around them and problem solved. Still scratching our heads on that one... In my experiences, GLBX has numerous odd issues to the point where it's become a bad joke anytime something breaks with connectivity... we blame them. It's kind of not funny though because it's almost always true. Taking them out of the equation usually fixes the problem. One of our customers who is frequently affected by GBLX problems jumps to the (often correct) conclusion that they are causing problems. :-/
> > > 
> > > -Vinny
> > > 
> > > On Feb 1, 2011, at 3:57 PM, Steve Danelli wrote:
> > > 
> > > > Thanks for the response. 
> > > > 
> > > > Ike had worked great up until Monday. The provider did a local test and our box saw the Ike packets so it appears to lie somewhere upstream. (GLBX may be a good guess)
> > > > 
> > > > Also - the paths are stable and we are sourcing from the same ip - very strange behaivor. Hope someone from GLBX or CTBC (or someone who had experienced an issue like this) can assist
> > > > 
> > > > 
> > > > Thanks to all for their feedback so far. 
> > > > 
> > > > SD
> > > > 
> > > > On Feb 1, 2011, at 3:19 PM, Valdis.Kletnieks at vt.edu wrote:
> > > > 
> > > >> On Tue, 01 Feb 2011 08:54:47 EST, Steve Danelli said:
> > > >> 
> > > >>> Some carrier, somewhere between us and the service provider is selectively
> > > >>> dropping the IKE packets originating from our VPN gateway and destined for
> > > >>> our Brazil gateway. Other traffic is able to pass, as are the IKE packets coming
> > > >>> back from Brazil to us. This is effectively preventing us from establishing
> > > >>> the IPSEC tunnel between our gateways.
> > > >> 
> > > >> Has IKE been known to work to that location before? Or is this something new?
> > > >> My first guess is some chucklehead banana-eater at the service provider either
> > > >> fat-fingered the firewall config, or semi-intentionally blocked it because it
> > > >> was "traffic on a protocol/port number they didn't understand so it must be
> > > >> evil".
> > > >> 
> > > >>> Also something else is awry, for two given hosts on the same subnet (x.y.z.52
> > > >>> and x.y.z.53), they take two wildly divergent paths:
> > > >> 
> > > >>> Anyone have any insight on to what may be occurring?
> > > >> 
> > > >> The paths appear to diverge at 67.16.142.238. I wonder if that's gear trying
> > > >> to do some load-balancing across 2 paths, and it's using the source IP as a
> > > >> major part of the selector function ("route to round-robin interface source-IP
> > > >> mod N" or similar?).
> > > >> 
> > > >> The other possibility is your two traceroutes happened to catch a routing flap in
> > > >> progress (obviously not the case if the two routes are remaining stable).
> > > >> 
> > > >> Sorry I can't be more helpful than that...
> > > > 
> > > 
> > > 