Anyone seen this kind of problem? SIP traffic not getting to destination but traceroute does

Wed Nov 9 19:13:49 UTC 2011

I saw the problems starting around 09:30 Eastern and continuing past 17:00.  Looking through ticket notes I had missed when writing my previous reply it seems that a fix was confirmed around 22:30 which involved a faulty piece of equipment being replaced.  I do not have specifics on what went wrong and when it was actually fixed though.
----------
Sean Harlow
sean at seanharlow.info

On Nov 9, 2011, at 2:04 PM, Preston Parcell wrote:

> What was the timeframe for your issues? Just curious since we saw some strangeness last night.
> 
> 
> Preston 
> 
> 
> 
> -----Original Message-----
> From: Sean Harlow [mailto:sean at seanharlow.info] 
> Sent: Wednesday, November 09, 2011 12:00 PM
> To: Jay Nakamura
> Cc: NANOG
> Subject: Re: Anyone seen this kind of problem? SIP traffic not getting to destination but traceroute does
> 
> I can't say I have a specific answer to your question, but yesterday I was seeing major packet loss on outbound audio from all my VoIP customers using Qwest and going in to servers on L3.  It's entirely possible that SIP was also being lost, just the audio was the more notable and pressing issue.  It seems to be resolved at this point, but we have not yet heard from Qwest what the actual problem was.
> 
> This was with sites in Northeast Ohio and the Chicago area connecting to servers in New York and LA for what it's worth.
> ----------
> Sean Harlow
> sean at seanharlow.info
> 
> On Nov 9, 2011, at 1:47 PM, Jay Nakamura wrote:
> 
>> We ran into a strange situation yesterday that I am still trying to 
>> figure out.  We have many VoIP customers but yesterday suddenly select 
>> few of them couldn't reach the SIP provider's network from our 
>> network.
>> 
>> I could traceroute to the SIP providers server from the affected 
>> clients' IP just fine.  I confirmed that the SIP traffic was leaving 
>> our network out the interface to the upstream provider and the SIP 
>> provider says they couldn't see the SIP traffic come into their border 
>> router.
>> 
>> SIP traffic coming from SIP provider to the affected customer came 
>> through fine.  It's just Us -> SIP server was a problem.
>> 
>> I thought there may be some strange BGP issue going on but we had 
>> other customers within the same /24 as the affected customers and they 
>> were connecting fine.
>> 
>> The traffic at the time traversed
>> 
>> Our network -> Qwest/century link -> Level 3 -> SIP provider
>> 
>> I changed the routing around so it would go through our other 
>> upstream, AT&T, and it started working.  With AT&T, the route was
>> 
>> Our network -> AT&T -> Level 3 -> SIP provider
>> 
>> So my questions is, is it possible there is some kind of filter at 
>> Qwest or Level 3 that is dropping traffic only for udp 5060 for select 
>> few IPs?  That's the only explanation I can come up with other than 
>> the whole Juniper BGP issue 2 days ago left something in between in a 
>> strange state?  I read the post about XO doing filtering on transit 
>> traffic, I haven't seen anyone say Level 3 or Qwest is doing the same.
>> 
> 
> 