Fiber cut in SF area

Fri Apr 10 20:19:25 UTC 2009

On Apr 10, 2009, at 3:41 PM, Scott Doty wrote:
> George William Herbert wrote:
>> Scott Doty wrote:
>>
>>> (Personally, I can think of a "MAE-Clueless" episode that was  
>>> worse than this, but that was in the 90's...)
>>>
>> The gas main strike out front of the building in Santa Clara?
>>
>> Or something else?
>>
>>
>> -george william herbert
>> gherbert at retro.com

> No, it was when an AS took their full bgp feed & fed it into their  
> igp (which used RIP, iirc), which generated (de-aggregated) routes  
> into /24's, which they then announced back into bgp...

That was Vinny Bono of FLIX, the Fat man Little man Internet eXchange,  
as7007.  Happened in 1997, IIRC.  He used a Bay Networks router to  
redistribute BGP on one card into RIPv1 on another card, stripping the  
CIDR notations off each prefix, making them classful, and stripping  
the AS Path.  This means, for instance, 96.0.0.0 was a /8, not a /24.   
It also means   He then re-redistributed RIP into BGP on a third card,  
which then originated each route from as7007.

I have it on most excellent authority (the "Fat man" himself) that  
this was not possible on ciscos.  Wonder if it is now ... ?

Anyway, I did not know people were calling this the "MAE-Clueless"  
incident.  I've always called it the "7007 incident".  In fact, some  
people still have as7007 filtered.

> iirc, part of the chaos than ensued was due to a router bug, so that  
> the routes "stuck around" in global views, even after the AS killed  
> their announcements, and even after physically disconnecting from  
> their provider.

That was Sprint, as7007's transit provider.  Sprint only did AS Path  
filtering, and as every single prefix was ^7007$, they all passed the  
filter.

Vinny literally unplugged the router, no power, no fiber, no copper,  
but the prefixes were still bouncing around the 'Net for hours.   
Sprint kept the routes around for a long time as their routers would  
not honor withdrawals - or so the rumors said.  The rumors also  
claimed the IOS version was named "$FOO-sean".  Sean Doran was CTO of  
Sprint's Internet company at the time, and he supposedly specifically  
asked for the 'feature' of ignoring withdrawals to lower CPU on their  
AGS+s.  I have absolutely no way of confirming this as I haven't  
spoken to Sean in years & years, and wouldn't even know where to find  
him any more.

The most interesting rumor I heard is that Sprint had to shut down  
every single router simultaneously to clear the routes out of their  
network.  Personally I think that's probably a bit exaggerated, but  
who knows?

> We told our customers "the Internet is broken, please try again  
> later"...which was acceptable back then.  (But I doubt we would get  
> away with just that nowadays... ;-)   )

Really?  That's what some broadband providers say nearly daily.

-- 
TTFN,
patrick