NAP Solutions

Fri Nov 14 18:18:33 UTC 1997

> Date: Fri, 14 Nov 1997 10:57:39 -0500 (EST)
> From: Brian Horvitz <horvitz at shore.net>
> To: nanog at merit.edu
> Subject: NAP Solutions
> 
> Has anybody though about a packet over sonet solution for an exchange?
> Seems like you could get a pretty effective answer out of a GSR with OC3
> and OC12 interfaces...

I think your question provides an interesting opportunity for comparing
SONET solutions with, for example, ATM solutions.

First, some thoughts on a local-area interconnect.

If you want a fully-meshed interconnect, you will need n-1 SONET
interfaces on each router, when "n" is the number of routers at the
interconnect, [assuming one router per ISP].

You might feel that this is an excessive number of router interfaces,
(assuming n > 2), so you might consider creating a ring instead.  
In this case, each router would require only two interfaces.  However,
most packets will need to be forwarded through several routers before
reaching their final destination [within the interconnect].  As a result,
a certain amount of the capacity of your router and your interfaces will
be used to forward packets between your competitors.

Obviously, there is a spectrum of configurations between a full mesh and a
ring.  However, determining the best trade-off between the cost of
additional interfaces, additional bandwidth used to forward packets
between routers not directly connected, and additional routing
capacity to forward packets within the interconnect is probably
worthy of a Master's thesis.  (Of course, you might have to write
another thesis every time you add another router to the interconnect.
Creating a general solution might be worthy of a Doctoral dissertation.)

[Note that this assumes that no router vendor integrates a SONET mux
into the router.  As far as I know, no router vendor has an integrated
SONET mux.]

Using an ATM switch (or most any other switch, for that matter) has
a number of advantages.  First, each router needs only one interface
to the interconnect.  This interface should probably be as large as
the router can support.  (This is the problem facing many of the 
existing interconnects, namely that the router can support more than
100 Mbps, but the interconnect media runs at only 100 Mbps.)
Second, adding additional routers is fairly straight-forward, (e.g.,
you don't have to add another SONET interface to every existing router,
in the rather unlikely event that you have a full-mesh topology).
Rather, you merely plug the new router into the switch.  Finally, adding
additional capacity is probably a lot easier.  Each router [ATM] interface
can be upgraded as that router requires additional capacity.  For example,
an interconnect could simultaneously support both OC-3c and OC-12c
connections.  In a similar fashion, the aggregate capacity of the
interconnect can be increased by swapping in a new ATM switch, 
(rather than, for example, swapping out the FDDI interfaces on the
routers for gigabit ethernet interfaces).

Now, there is a certain price for using ATM rather than SONET, namely
the overhead incurred by ATM.  However, as I have said before, the
best decision criteria is an analysis of cost/performance, not
merely looking at overheads.  In this case, you would need to compare
the cost of the unnecessary SONET interfaces with the cost of the
bandwidth used for ATM.

So, it seems to me that SONET makes sense for a private interconnect
between two parties, but that a switched technology, such as ATM or
perhaps gigabit ethernet, makes sense when the number of parties is
greater than two, (like three, for example).

Now, this same analysis can be applied in the wide area.

The next question is, if you have decide to use ATM, whether to 
create a local interconnect, (i.e., put the switch and all the routers
in one room) or to create a distributed interconnect, (e.g., leave
the routers distributed throughout a LATA or even throughout the country).
I think there are a number of good arguments both ways.  Both local
interconnects and intra-LATA interconnects have been tried.  I don't
know of anyone who has created a nationwide interconnect, (although
I think all of the original NAP proposals suggested nationwide NAPs).

On the other hand, the question of SONET versus ATM in a wide-area
environment is still being explored in, for example, the Internet2 efforts.
I don't understand how wide-area SONET solutions are supposed to scale,
but perhaps someone will figure it out.  Of course, another interesting
question is why, with services like ATM with distance-insensitive
pricing, one would build a regional wide-area interconnect (e.g., a
regional, wide-area gigapop), rather than a nationwide interconnect.
(Note that some claim that propagation delays are an issue in a
nationwide interconnect, but I believe that, e.g., routing through the
opposite coast to get to the adjacent city, would result from poor
design or configuration, rather from any inherent defect in the
concept of nationwide interconnects/gigapops).

Wide-area SONET certainly has its place, depending on the relative price
of SONET versus ATM services, the number of locations which need to be
interconnected and the amount of bandwidth required.  For example, in
a campus environment where private fiber is available, point-to-point
SONET solutions are easier to justify than when you need to lease SONET
links from a carrier.

Hopefully, projects like Internet2 and NGI will explore both SONET and
ATM wide-area interconnects so we can gain a better understanding
of their advantages and capabilities.

And finally, I don't know if I answered you question.  But, if you do
decide to build a SONET interconnect, please let us know how it works.

-tjs