Milo S. Medin (NASA Ames Research Center) medin at
Fri Sep 2 04:19:59 UTC 1994

Dave, I think you still are not understanding my point.  NAP's can't
do everything for everyone.  You have to pick what's important and 
optimize for that.  You can't do everything well.

	 Here's a couple of other things we thought about.
	 FDDI NAP - good short term prospects, tried and true technology.  Capa
	 even more of a problem than SMDS, won't even support a 155 Mb/s vBNS
	 connection.  Switched FDDI over ATM helps the bandwidth problem a bit,
	 but I don't really understand how that will extend to a large-scale ne

2 things.  FDDI is available on almost everyone's router hardware.  It's
mature and stable.  You can deploy a concentrator based solution, then if
capacity gets tight, you install a DEC gigaswitch.  This is also a mature
product, and has good congested state behavior (degrades gracefully).  This
sort of switched FDDI solution should provide reasonable capacity at the

Next, the IMPORTANT thing to realize is that this transition has to go well.
You are talking about the entire operational core of the Internet being changed
out.  It's better to make things easier up front, get moved to a new
architecture while protecting existing service.  That means you have to 
make absolutely sure that reliability is the #1 criteria for an interconnect.

	 The LAN-emulation folks in the Forum have lots of work yet to do with 
	 LAN emulation networks.  The equipment for LAN emulation over ATM is i
	 the same stage of "immaturity" as the IP/ATM equipment.
Bletch.  Multimedia bridging isn't the answer.  The network management 
issues here are atrocious to deal with.

	 Multimedia NAP - My best reply to this is Peter Ford's statement, "NAP
	 Don't Route".  They're a layer two service.  Some layer 2 interworking
	 be possible in the future, and we're looking at FR<->ATM interworking 
	 right now.

Agreed.  You shouldn't route.  And I think multimedia bridging is a very
bad idea, at least for this use.  I shudder to think about debugging problems
because of different MTU settings in different media types.

	 Cheap, low speed connections to the NAP - We got squeezed by product
	 availability here.  We know that there is a bigger market in selling T
	1 speed
	 connections to the NAPs than T3 and above.  However, we had a requirem
	ent to
	 support multiple T3's initially (remember that the minimum purpose of 
	 NAPs is to support the existing NSFNET backbone for transition purpose
	s, the
	 NSPs who are providing access to RNP's and who have a requirement to c
	onnect to
	 all of the priority NAPs, and the vBNS).  FDDI and SMDS weren't going 
	to survive
	 for long under this type of load, but ATM doesn't yet have T1 capabili
	 (although it's coming).  So, we proposed ATM, starting at 45 and 155 M
	 initially, and extending to 1.5 Mb/s and 622 Mb/s as switching equipme
	 becomes available.  Believe me, we'll be offering T1 speed connections
	 the SF and Chicago NAPs just as soon as we can make it work.
Why offer T1 connections?  The primary goal of the NAP's in the first 
phase of the transition is to support the replacement of 1 core NSP with
a few NSP's with minimum disruption.  THIS is what should be optimized 
for.  Are you confident that all the management procedures and technologies 
we are deploying will support a gob of T1 NAP attached AS's?  This is highly
questionable in my mind, esp. with PVC's that have to be configured by hand.

	 ATM immaturity - ATM itself isn't immature, it's the router and ADSU t
	 that's immature.  We've been running ATM in our lab since (believe it 
	or not)
	 1986, and switch vendors have been shipping equipment for about 3 year
	s now.

I can't believe you said this!!  The routers and ADSU's are what your 
customers care about.  You are picking technology that's optimized for
your own world view, not that of what your customers can best deal with.  

Also, most of the ATM switch vendors think they can get away with not
learning from the lessons of the router people and build an ATM switch
like a voice switch, with only enough buffering to support CBR type
applications.  When you push these switches even into light congestion, 
they start dropping cells like mad because there isn't enough buffering
in them to allow the remote TCP's enough time to see the congestion building
up and flow control their sending rate.  This has to deal with some
reasonable delay since most of these high speed streams won't be coming from
geographically close to the NAP.  

And when you do drop cells, you won't drop all the cells out of a frame,
you'll deliver 99% of them, and all that data will be tossed by the endpoint
when it tries to reassemble the frame.  Alan Romanow and Sally Floyd did
a nice paper on this problem, and proposed a workaround, which no ATM
switch vendor has implemented.  So whatever problems you have from the 
previous problem, it's exacerbated by this issue.  It's really good you have
all that bandwidth to deliver these dead cells.  Oh, that's right, your 
customers are paying for this capacity right?  I'm sure your cost scheme
gives people credits for incompletely delivered frames, right?

	 We're beginning to see second generation products on the market right 
	 Since the Toronto IETF, we've been running tests on the SF NAP configu
	 and the problems we've found (and fixed) have all been with the ADSU's
	 routers.  We do have a working configuration for the 1490/AAL5 protoco
	 stack, and are running load tests on it now.
I'd really like to know the exact hardware configurations that were
tested and what loads you ran through them.  Most of our experiences show
bad things happen when you put things under stress.

	 Bi-lateral/multi-lateral NAP traffic agreements - There's no rules her
	e at
	 all.  The agreements are completely up to the NSPs who connect to the 
	 A CIX-like arrangement where all parties agree to exchange traffic w/o
	 settlements is just as possible as N**2 bi-lateral agreements with tra
	 sensitive settlements.  The NAP managers and NSF have nothing to say h
The issue isn't the policy, the issue is whether or not this is expected
to work well with manual configuration of all the VC's, and gobs of NAP
attached AS's that you've sold all those T1 connections to.  If there 
are 40 attachees, then do you expect that people are going to establish 
bilateral relationships with all 39 others?  And that the RS will never
send them a next hop address that they don't have a VC open to right?  Seems
to me this all gets more complicated with the larger number of attachees.  

	 As far as taking a long-range view on the Internet, I'll plead guilty.
	 already seeing congestion problems on the Internet today, and that's e
	 before we turn the video and audio applications completely loose.  My 
	 is that if we don't get enough capacity into the Internet, and really 
	 the growth rate is going to drop badly.  I've lived on badly congested
	 Internets (remember the 56 Kb/s NSFNET?), and it's not an experience I
	 wish to repeat.

And what happens if the system doen't work right on day 1?  I wonder how 
long you guys will be given a chance to fix it.  You guys are STILL
not optimizing for the right things.  That is the issue.  

Now, perhaps it might be useful to describe a little bit about some
of these issues are dealt with by the FIXes.  Maybe some of this experience
is useful in this discussion.

First off, the FIXes aren't designed to handle anyone attaching.  Too many
peers would be a problem for the peering approach used there.  However,
use of the RS could allieviate many of these problems, but you still 
have increased network management load from each party that attaches.

Next, you start off with a colocated facility for the interconnect.  You
pick a standard mature media for the interconnect, and all the attachees
routers have to use that interface.  When you have to evolve to
a new media, because of capacity, or some other issue, since the routers
are colocated, attachees simply add a new interface to their routers, and
then move to peering over the new media, while retaining the older
media as backup.

Historically, the FIXes started as ethernet based.  All the routers there were
plugged into a multiport transciever.  As traffic grew and AS's added
more trunk capacity to their FIX routers, the ethernet got congested, and 
so more capacity was required.  So an FDDI concentrator was attached, 
and the folks who had the biggest load issues ADDED FDDI interfaces to
their routers, and offloaded this traffic onto the ring.  Noone deactivated
their ethernet interfaces, which allowed the ethernet to be used for backup.
Some FIX attachees are more than adequately served by the ethernet, and as
long as some AS's are ethernet only connected, EVERYONE has to maintain
their ethernet interfaces.  Eventually, traffic will grow and the next
step will likely be a GIGASWITCH.  You could move everyone off the concentrator
into the switch, or if everyone has FDDI interfaces available, then maybe
you throw away the ethernet, and then have people add a new FDDI interface
for the GIGASWITCH, and use the concentrator FDDI for backup.  As this gets
congested, you then might move to ATM (if reliable and multiple interfaces
are available), and use the GIGASWITCH as backup.  Etc...

Key points:

Colocation buys you a lot.  It means that folks can support whatever
long haul media and speed they like to connect to their router at the 
FIX.  This means that some can connect at T1, or DS-3, or SMDS, or ATM,
or whatever.  It's THEIR choice, and this can change and evolve over time.
Since this trunk is part of the attachees network, they use their own
normal network management procedures that they would use to interconnect
to a customer site.  The interconnect isn't over a long haul link.  This greatly
reduces the network management requirements for the interconnect, now
that the long haul link is connected at both ends by a single AS's routers.

Next, since the routers are colocated, evolution can occur gradually.  
New attachment media can be supported with NO additional long haul trunking
required.  The ability to support multiple interfaces in the router provides
for gradual transitions and evolution, as long as a base interconnect 
technology is used by everyone.  Note that customers who need the extra capacity
have the incentive to do the work to add interfaces, and the older technology
used as a backup.  And as people move off the older technology, capacity
is reclaimed.  And when everyone has moved to a new technology, then you
can toss the old interconnect.  

This above point is very important.  You shouldn't attempt
to force people to use the end all technology solution on day one.  This
is what you are doing, and it's a broken approach, esp. given the 
importance of these attachements.  What you need to do is to build in
evolvability, because whatever answer people think is right today will
be wrong.  Things change too quickly for this.  And that evolvability needs
to occur with gradual transitions, transitions that can be accomodated by
multiple interfaces.  You are trying to optimize for everything and in
the process, will end up with a design that doesn't do anything well. Toss
out some of your goals, but NOT those of reliability, operational manageability
(do you guys still not have any SNMP access to the switch's port statistics
available to NAP attachees?), and use of mature technology which will be
absolutely required to make this transition work well. Then reevaluate your
design, and I think you and your customers will all be better off.  

Does this style of interconnect solve all problems?  No.  Can you reasonably
expect such an colocated facility to house hundreds of attachees?  No, but
then it's not clear to me that the routing technology that we have 
will scale to this sort of scale either.  So it's not clear to me why this
should be a hard requirement.  Are any of these advantages derived from
the fact that it's a Fed interconnect?  No, they apply to any interconnect
that uses a similiar architecture, with an RS or not.

Also, I should point out that the FIXes aren't in the profit making business.
That means that noone is out there trying to get people to attach directly
to FIXes when they could adequately be served by connecting to one of
the attached NSP's.  This naturally results in fewer connections rather 
than more.  I have nothing against profitmaking of course, (I'm a Republican
after all), but I think engineering and operational concerns need to
take priority over getting the maximum number of people to attach to a 
given interconnect point.  And in the end, if things don't work because 
you've oversold the system, people will find somewhere else to pass their
traffic.  This is too important an issue to not get done right.


PS Sorry for the long note.

More information about the NANOG mailing list