concern over public peering points [WAS: Peering point speed publicly available?]

Mon Jul 5 08:18:25 UTC 2004

On Sun, 4 Jul 2004, Bill Woodcock wrote:

> Go back and think about the purpose of an exchange: it's an economic
> optimization over transit.  It's the value-add that lets someone who buys
> transit sell a service that's of greater value yet lesser cost than what
> they buy.  Now, what's an exchange that costs more money?  Less effective.
>
> So charging more money is just a way to make an exchange less effective,
> not more effective.
>
> Some people who don't bother to run the numbers think that reliability is
> a justification for charging money at an exchange.  I'd encourage them to
> run the numbers.  An exchange which costs twice as much money needs to be
> twice as reliable to be equally effective.  Ask yourself how many
> exchanges there are that drop more than half the bits on the floor,
> before you think of charging twice as much money.
>
> In other words, if it's not broke, don't fix it.

I'm going to quibble with Bill a bit on this.  I'm going to attempt to do
so carefully, both because I work for him, and because this is an area he
knows much better than I do. ;)

Peering is often said to help a network (and thus add value) in two ways,
by increasing performance and by reducing costs.  Performance is said to
increase because other networks are brought closer.  Costs are said to go
down because a network starts getting connectivity for free that it would
otherwise be paying for.  In well connected urban areas of the US, for all
but the biggest networks, it's no longer clear that either of these
arguments are valid.

The performance arguments are probably more controversial.  The arguments
are that shortening the path between two networks increases performance,
and that removing an extra network in the middle increases reliability.
The first argument holds relatively little water, since it's in many cases
only the AS Path (not really relevant for packet forwarding performance)
that gets shortened, rather than the number of routers or even the number
of fiber miles.  If traffic goes from network A, to network A's router at
an exchange point, to network C, that shouldn't be different
performance-wise from the traffic going from network A, to Network B's
router at the exchange point, to Network C.  Assuming none of the three
networks are underprovisioning, the ownership of the router in the middle
shouldn't make much difference.  The reliability argument is probably more
valid -- one less network means one less set of engineers to screw
something up, but the big transit networks tend to be pretty reliable
these days, and buying transit from two of them should be quite safe.

The pricing issues are simpler.  There's a cost to transit (which is, to
some degree, paying some other network to do your peering for you), and
there's a cost to peering.  Without a clear qualitative difference between
the two, peering needs to be cheaper to make much sense.  The costs of
transit involve not just what gets paid to the transit provider for the IP
transit, but also the circuit to the transit provider, the router
interface connecting to the transit provider, engineering time to maintain
the connection and deal with the transit provider if they have issues, and
so forth.  Costs of peering include not just the cost of the exchange
port, but also the circuit to get to the exchange switch, sometimes colo
in the exchange facility, engineering time to deal with the connection and
deal with the switch operator if there are issues, and time spent dealing
with each individual peer, both in convincing them to turn the session up,
and dealing with problems affecting the session.  Even if the port on the
exchange switch were free, there would be some scenarios in which peering
would not be cheaper than transit.

The situation changes considerably in less developed areas.  The transit
costs tend to be a lot higher (largely due to increased long-haul circuit
costs), and there's a significant performance cost to having your traffic
go hundreds or thousands of miles to get across town.

The argument against free exchanges is, I think, more of an argument in
favor of full-service facilities, and the savings they provide in terms of
operational engineering time.  If there's a problem at 3 am at PAIX,
Equinix, or NOTA (to pick three well-known North American commercial
exchange operators), it's easy to pick up the phone and get it resolved.
When dealing with volunteers, or with an organization that doesn't have
the budget for a 24/7 paid staff, there's at least a perception that it
may be hard to find somebody who will make fixing somebody else's problem
their top priority.  Again, it becomes a matter of plugging that cost into
the cost comparison, and figuring out whether it costs more to peer with
or without that level of service.

>     > I would bet that there is more than enough business available to
>     > cover the costs of intelligent spending.
>
> While I won't categorically dub that an oxymoron, I'd say that the
> possibility of their being "intelligent spending" at an exchange is, um,
> extremely rare.  Military intelligence.  Jumbo shrimp.  Microsoft Works.

To use the Seattle Exchange as an example again, it's not really fair to
say there's no spending there.  The spending is just hidden a bit better.
The Equinix/PAIX/NOTA model involves the exchange operator operating the
switch, as well as a big datacenter and cable plant for the customers.
Customers generally buy the exchange port, the colo space, and some
private crossconnects, and pay the exchange operator money for all of
that.  At the SIX, all the exchange organization runs is the stack of
switches.  The closet the exchange switches are in is provided by the
landlord, who presumably more than makes up for it in rent charged to the
exchange participants for their colo space.  Several colo providers in the
building provide remote hands service to exchange participants, which they
presumably charge for.  Then there's the donated switches, donated labor,
and various other donations, all of which have a cost to their donors.
The donors apparently feel that they get enough out of the exchange to
justify those expenditures, even if it means they're paying considerably
more for the exchange than some of the other participants are, so it all
works out, but it's important to note that that the exchange isn't free
for everybody.

>     > You could probably still give away FastE ports for free
>
> Any economist, and most practical thinkers generally, will tell you that
> creating artificial inequities isn't terribly wise.  Likewise, incenting
> bad behaviors like using too-small ports isn't terribly wise.  In an
> exchange, if you want to avoid congestion, and have to charge money for
> some reason, you probably want to charge the same amount per port,
> regardless of port speed.  But you're a lot better off just solving
> whatever problem is costing money in the first place.

Assuming there really is a reason to charge for the exchange ports (and
certainly not advocating charging just for the sake of charging), a
business model based on saving the customers money needs to keep in mind
that different customers may have different costs for alternatives.  If
small amounts of transit are cheaper than large amounts of transit, it may
be important to keep the small exchange ports cheaper than the small
transit connections.  If the amount of money required is such it's
sufficient to charge all participants an amount less than the smallest
participant's transit bill would be, charging a flat rate may work.  If
more money is required, it's probably better to get the extra from those
who are saving more money by peering at the exchange.

-Steve