routing issue for verizon dsl customers in western massachusetts

Thu Sep 15 20:13:02 UTC 2011

On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote:

> On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold <bgold at simons-rock.edu>  
> wrote:
>> Over the past week, we've discovered that there is an issue with  
>> the way
>> some Verizon DSL customers are being routed in Western  
>> Massachusetts that is
>> preventing them from reaching my employers public IPs. The problem  
>> is only
>> limited to Verizon DSL customers, everyone else can reach these IP  
>> addresses
>> just fine. After many hours on the phone with Verizon tech support, I
>> finally managed to get myself and one of my coworker's home dsl  
>> connections
>> switched from a "redback router" to a "juniper router" which  
>> resolved the
>> issue, but only for us.

[...]
>
> If you buy verizon services at your day job you can probably make
> noise through your sales droids better than here (sadly)... verizon
> likes to jump when customers have problems, if the customer is a large
> corporation or other 'important' customer.

That is just the problem! The college does not buy any Verizon network  
stuff directly, so we don't really have any access to their support.  
(We have a few cell phones, but not enough to be "important".)

Brian Gold (who first posted) happens to have their DSL to his house,  
and he was one of five who have reported the problem, so that gave him  
a slight in. But the only techs he could reach as an "end user" were  
not high enough up to fix this problem in a general way. After  
pressing them for literally hours, he was able to get transfered to  
their NOC, and get the problem resolved for his one address. But, they  
would not give him the NOC contact, and he had to repeat this multi- 
hour process to get it fixed for an other user. Verizon's DSL support  
suggested that we get our bandwith provider involved, and so they  
tried to pitch in, but they don't have any Verizon NOC contact either,  
especially since this issue is purely within a small corner of  
Verizon's DSL network, not on any of Verizon's links to our provider.

This issue hits only a few Verizon DSL users in NW Mass. It does not  
really seem like a routing problem, because the affected users can  
reach many of the servers in our AS, but not some addresses.  
Unfortunately, the "blocked" addresses include our web server and our  
mail server, so our staff who live out there noticed the issue pretty  
quickly. Traceroutes from Brian's house show that for our blocked  
hosts, the users don't get beyond Verizon's NAT.

The Verizon tech's "fix" of re-patching Brian's DSL line in to a  
different router feels to me like there is a config problem in the  
other router, but the tech we got is not authorized to alter the  
config. It would be nice if we could reach someone who could actually  
edit the broken config and make it right. Anyone from Verzion's NOC  
for Western Mass reading this? Or, does anyone else have useful  
contact info for them?

FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and  
a good trace from Brian's house, to different servers on our campus :

FAILS:
Tracing route to wilbur.simons-rock.edu [208.81.88.15]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2     1 ms     1 ms     1 ms  192.168.1.1
  3    53 ms   104 ms   116 ms  10.14.1.1
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.

WORKS:
Tracing route to dev.simons-rock.edu [208.81.88.25]
over a maximum of 30 hops:

1    <1 ms    <1 ms    <1 ms  192.168.10.1
2     1 ms     1 ms     1 ms  192.168.1.1
3    87 ms    54 ms    54 ms  10.14.1.1
4    99 ms   109 ms   103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon- 
gni.net [130.81.10.77]
5    16 ms    18 ms    16 ms  so-7-3-1-0.NY5030-BB-RTR2.verizon- 
gni.net [130.81.20.6]
6    19 ms    17 ms    17 ms  0.xe-3-1-0.BR3.NYC4.ALTER.NET  
[152.63.2.81]
7    18 ms    21 ms    18 ms  204.255.168.194
8   108 ms   188 ms   116 ms  pos5-0-2488M.cr1.BOS1.gblx.net  
[67.17.94.57]
9    24 ms    28 ms    23 ms  pos0-0-0-155M.ar1.BOS1.gblx.net  
[67.17.70.162]
10   121 ms   160 ms   127 ms  64.213.79.250
11    77 ms    77 ms    78 ms  208.81.88.25

Trace complete.

Anyways, thanks for any suggestions you can offer.

Steve Bohrer
Network Administrator
ITS, Bard College at Simon's Rock
413-528-7645