NET99 sabotages IGC's routes NOW: The facts

David ``Joel Katz'' Schwartz stimpson at
Mon Mar 18 20:06:41 UTC 1996

On Mon, 18 Mar 1996, Peter Kline, Sr. Network Engineer wrote:

> At 10:21 PM 3/15/96 -0500, Joel Katz wrote:
> >
> >	IGC (Internet Gateway Connections, an ISP in Florida that was
> >multihomed between NET99 and MCI) just had an unexpected outage lasting in
> *The lie:*
> >excess of an hour. This outage was deliberately engineered by AGIS/NET99.
> *The fact:*
> IGC's routes were pulled from the Net99 net within 90 minutes of their being
> turned off *at IGC's request*.

	The outage had nothing to do with Net99 removing any routes, as I 
think I clearly stated.
> >First, a little history:
> >
> *The lie:*
> >	A few days ago, someone at Netrunner got upset with someone at 
> >NET99/AGIS and shut off _our_ service. For some reason (we strongly 
> >suspect a configuration error at NET99) our traffic failed to reroute 
> >over our MCI T1 for several hours. (They have a habit of broadcasting
> >static routes to route around outages, thus completely screwing their
> >multi-homed customers.)
> *The truth:*
> The referenced disconnection did occur, and was restored.  This kind of
> problem is a very real danger of the way Net99 colocated in certain cities.
> That's why AGIS only colocates in POP's and private spaces.  And in the
> event that a customer is dually homed and not running BGP, it's incumbent
> upon the customer to make sure that *both providers* are aware of the fact
> and properly setting local prefs.  IGC did not do that and bears full
> responsibility for any failure to route properly during an outage.
	We did not do *WHAT*? How were we BGP peering with NET99 in the 
first place if we didn't notify them that we were multi-homed? What are 
you claiming we didn't do? And what in the excerpt are you claiming is a lie?
> Aha: a true statement:
> >	We asked AGIS for a one month refund to compensate us for our 
> >losses due to the sabotage. Instead, they issued us a _one_day_ credit. 
> Ah, another true statement:
> >We got upset, both with the ability of our T to pass traffic, with 
> >NET99's continual routing problems, with NET99's backbone being 
> >disassembled from under us, and with NET99/AGIS's continual delays in 
> >migrating us to the AGIS network. We asked them to discontinue our service.
> True, but we certainly wouldn't have left them without a circuit.  We've
> tried mightily to install a new circuit to their premises, and ran into a
> b8zs facilities problem.  We could have installed it AMI as a temporary
> measure with no problem.

	If this is true, it was never communicated to me.

> *Another LIE*
> >	They did. They also deliberately and maliciously began 
> >broadcasting more specific routes for the individual class C networks in 
> >our CIDR block. (We confirmed this with MCI.)
> *The truth:*
> Over the months various static routes have been installed by various people
> on Net99 to route around some problems (a practice which predates our
> purchase).  You run into some curious situations like a static on a Chicago
> router pointing to Phoenix.  In this case, there were statics of differing
> prefix lengths on different routers.  The aggregate address statement got
> pulled, and we didn't catch the more specifics on another router.  As soon
> as we found out what happened, we pulled them.  In fact, we pulled them
> while we were on the phone with MCI so that they could confirm that they
> were gone.

	The only problem with this is that it is inconsistent with 
recorded statements made to me be people at AGIS when I called to discuss 
this issue with them. When I _specifically_ asked why AGIS was 
broadcasting routes for individual class C's in our CIDR block, I was 
told to ask Randy Epstein, the IGC employee who made the decision to 
terminate our service with NET99/AGIS.

> The statement of malicious intent is untrue and libellous and must be
> withdrawn immediately by Mr. Katz.

	I will clarify by stating that my judgment of malicious intent 
was based upon two facts:

	1) As I said above, AGIS specifically referred my to our own 
employees as the reason for the more specific routes orginating at AGIS, and

	2) Shortly after we were able to restore routing, AGIS pulled the 
reverse DNS from all of the class C's in our CIDR block, without warning. 
Was this another 'mistake' or 'oversight'?

> *Another LIE*
> >	I contacted MCI again and they contacted AGIS/NET99 with the
> >ultimate threat being for MCI to stop carrying AGIS' traffic if they would
> >not cease broadcasting the bogus routes. Shortly after being contacted by
> >MCI, AGIS/NET99 stopped announcing the bogus routes to MCI, but continued
> >advertising these routes to their other peers at the NAPs.
> Nope.  Like I said, MCI didn't threaten us at all, and worked with us to see
> that the routing was correct.  Net99 isn't announcing those routes anywhere.
> Both AGIS and Net99 announce all routes at all exchange points to all
> providers in compliance with our peering contracts.  If MCI had called back
> and said, "we still have a problem," we'd work with them until it was taken
> care of.

	Are you disputing that you continued to advertise routes for the 
Class C's in our CIDR block to your other peers at the NAPs? What in what 
I said are you claiming is a lie? I don't get it. Your statements seem 
entirely consistent with mine. 

> >	Fortunately, MCI was able to reconfigure our access lists at their
> >router to allow us to broadcast competing more specific routes for each
> >individual class C in the CIDR block. Thus (more or less) restoring IGC's
> >net connectivity.
> More specifics aren't competing.  They take precedence.  Sometimes it's
> better to be silent and thought dumb...

	More specific than our normal aggregate routes, competing with 
your new routes.

> Ah, more libel:
> >	NET99's deliberate, malicious actions are contrary to the very
> >spirit of the Internet. They resulted in DALnet (the third largest IRC
> >network) being split to ribbons, depriving 2,500 people of the ability to
> >converse with each other and cutting off DALnet's security and access
> >control system which is hosted here. 
> Another retraction required ASAP, Joel.

	Restore our reverse DNS for those class C's and give us a 
reasonable six months to renumber and I'll retract my statements. 

> AND NOW, here's what Joel left out:
> IGC called our NOC up, screaming, the night this occurred, and "put us on
> notice" that "if we didn't disconnect our circuit to them immediately [we
> all know that phone companies do nothing immediately] they were going to
> flood ping [another Miami customer] until we did."  We had no reason to
> disbelieve them so we did turn down the port, and had the routing unwound
> within a couple of hours, which isn't bad given the nonexistent notice and
> other things going on.  
> In other words, Joel comes to the court of public opinion with unclean
> hands.  Joel threatened an illegal act against another customer.  Joel's
> righteousness rings hollow.  Joel must recant immediately.

	Umm, I didn't threaten anyone or anybody. I'll spare the 
rigamarole of 'is libelous, must be retracted'.

> And this is weird.  The sig and the FROM don't match.  Is this all a joke?
> >
> >	David Schwartz
> >	Director of Network Services
> >	Internet Gateway Connections
> >	(954)-430-3030

	We only wish. If you think this is a joke, call MCI or myself.

More information about the NANOG mailing list