routing issue for verizon dsl customers in western massachusetts

Christopher Morrow morrowc.lists at gmail.com
Thu Sep 15 20:52:26 UTC 2011


On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer <skbohrer at simons-rock.edu> wrote:
> On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote:
>
>> On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold <bgold at simons-rock.edu> wrote:
>>>
>>> Over the past week, we've discovered that there is an issue with the way
>>> some Verizon DSL customers are being routed in Western Massachusetts that
>>> is
>>> preventing them from reaching my employers public IPs. The problem is
>>> only
>>> limited to Verizon DSL customers, everyone else can reach these IP
>>> addresses
>>> just fine. After many hours on the phone with Verizon tech support, I
>>> finally managed to get myself and one of my coworker's home dsl
>>> connections
>>> switched from a "redback router" to a "juniper router" which resolved the
>>> issue, but only for us.
>
> [...]
>>
>> If you buy verizon services at your day job you can probably make
>> noise through your sales droids better than here (sadly)... verizon
>> likes to jump when customers have problems, if the customer is a large
>> corporation or other 'important' customer.
>
>
> That is just the problem! The college does not buy any Verizon network stuff
> directly, so we don't really have any access to their support. (We have a
> few cell phones, but not enough to be "important".)
>
> Brian Gold (who first posted) happens to have their DSL to his house, and he
> was one of five who have reported the problem, so that gave him a slight in.
> But the only techs he could reach as an "end user" were not high enough up
> to fix this problem in a general way. After pressing them for literally
> hours, he was able to get transfered to their NOC, and get the problem
> resolved for his one address. But, they would not give him the NOC contact,
> and he had to repeat this multi-hour process to get it fixed for an other
> user. Verizon's DSL support suggested that we get our bandwith provider
> involved, and so they tried to pitch in, but they don't have any Verizon NOC
> contact either, especially since this issue is purely within a small corner
> of Verizon's DSL network, not on any of Verizon's links to our provider.
>
> This issue hits only a few Verizon DSL users in NW Mass. It does not really
> seem like a routing problem, because the affected users can reach many of
> the servers in our AS, but not some addresses. Unfortunately, the "blocked"
> addresses include our web server and our mail server, so our staff who live
> out there noticed the issue pretty quickly. Traceroutes from Brian's house
> show that for our blocked hosts, the users don't get beyond Verizon's NAT.

I wasn't aware verizon implemented CGN already... way to be a 'first
mover' in this field verizon!

>
> The Verizon tech's "fix" of re-patching Brian's DSL line in to a different
> router feels to me like there is a config problem in the other router, but
> the tech we got is not authorized to alter the config. It would be nice if
> we could reach someone who could actually edit the broken config and make it
> right. Anyone from Verzion's NOC for Western Mass reading this? Or, does
> anyone else have useful contact info for them?

you probably want someone in the NOC which is (I think) stil in
Reston, va... I don't think they have separate noc's per region. The
first-line tech folks you chat with on the phone really arent' able
(even to login really) to fix devices in the field :(

anyways, this looks crappy :( but yeah for CGN being all it's cracked up to be!

-chris

>
> FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and a good
> trace from Brian's house, to different servers on our campus :
>
> FAILS:
> Tracing route to wilbur.simons-rock.edu [208.81.88.15]
> over a maximum of 30 hops:
>
>  1    <1 ms    <1 ms    <1 ms  192.168.10.1
>  2     1 ms     1 ms     1 ms  192.168.1.1
>  3    53 ms   104 ms   116 ms  10.14.1.1
>  4     *        *        *     Request timed out.
>  5     *        *        *     Request timed out.
>  6     *        *        *     Request timed out.
>  7     *        *        *     Request timed out.
>
> WORKS:
> Tracing route to dev.simons-rock.edu [208.81.88.25]
> over a maximum of 30 hops:
>
> 1    <1 ms    <1 ms    <1 ms  192.168.10.1
> 2     1 ms     1 ms     1 ms  192.168.1.1
> 3    87 ms    54 ms    54 ms  10.14.1.1
> 4    99 ms   109 ms   103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon-gni.net
> [130.81.10.77]
> 5    16 ms    18 ms    16 ms  so-7-3-1-0.NY5030-BB-RTR2.verizon-gni.net
> [130.81.20.6]
> 6    19 ms    17 ms    17 ms  0.xe-3-1-0.BR3.NYC4.ALTER.NET [152.63.2.81]
> 7    18 ms    21 ms    18 ms  204.255.168.194
> 8   108 ms   188 ms   116 ms  pos5-0-2488M.cr1.BOS1.gblx.net [67.17.94.57]
> 9    24 ms    28 ms    23 ms  pos0-0-0-155M.ar1.BOS1.gblx.net [67.17.70.162]
> 10   121 ms   160 ms   127 ms  64.213.79.250
> 11    77 ms    77 ms    78 ms  208.81.88.25
>
> Trace complete.
>
> Anyways, thanks for any suggestions you can offer.
>
> Steve Bohrer
> Network Administrator
> ITS, Bard College at Simon's Rock
> 413-528-7645
>
>
>
>




More information about the NANOG mailing list