Did we forget that ISP's are businesses?

Thu Oct 6 12:29:14 UTC 2005

As Randy pointed out, this conversation has been fairly clue free.
Working as a peering coordinator for a large ISP I can tell you
that most of the posts in this thread have been so wrong it makes
me laugh.

ISP's are businesses, and let me tell you that peering is no
exception.  People seem to like to think that "settlement free
interconnections" are "free", but nothing could be further from the
truth.  You have to buy routers and the cards that go in them,
provision transport services to the peering location, and then on
to the peer.  Provide enough backhaul in your network from where
your customers are located to where the peers are located.  You
have to pay lawyers to review contracts, engineers to configure and
troubleshoot sessions, and managers to negotiate the whole deal.

The budget for "settlement free interconnections" at a major ISP
can run into the millions of dollars.  Two major ISP's may have
8xOC-48 between them.  That's probably $200,000 in router costs
alone.  Yes, sometimes you can get a $200 cross connect, but sometimes
you also have to have the $6k/month circuit, for each one, creating
a $42,000/month cost.  That's a half million dollars a year.

When you look at the requirements, geographic diversity, volume,
ratio, number of routes what is really happening is the companies
are trying to make sure there is some balance of costs.  It doesn't
have to be a 50/50 split...everyone uses their own assets to reduce
their costs, but there has to be some equality.  Personally, I'm
not a fan of the technical requirements to make them equal, but
rather like to negotiate equality, but everyone has their own
approach.

Back to the issue at hand.  What I can tell from this situation is
that Level 3 thought they were paying a lot of costs for very little
return on investment.  The idea that Level 3 would take this action
because Cogent was selling cheaper seems a bit far fetched to me.
Level 3 knows this is going to hurt their customers as well.  Indeed,
I'll bet this went all the way to the Level 3 CEO for approval
first, because they knew their was going to be pain.  This isn't
some router cowboy going nuts in the middle of the night, this is
a business backed into a corner.

Why?  We'll never know the real story.  Maybe L3 is paying for third
party circuits because cogent doesn't touch their network and it's
costing them a boatload.  Maybe L3 wanted to move to GigE peering
for cheap high density ports, and Cogent wouldn't budge from using
OC-3's because their routers don't have great GigE density.  Maybe
traffic between the two has dropped to 20 megs, and so L3 doesn't
think maintaining ports is a justified cost.  Maybe the ratio is
20:1, and that was finally enough for L3 to feel they were carrying
too much of the transport cost.  Most likely it's a combination of
all of these issues.

Bottom line is some engineer had to dress up and go over to the
land of suits and explain to them that yes, Level 3 was going to
totally screw their own customers, but it was also going to save
$X, where $X is probably fairly large, and so they really had no
choice.

Least I seem like I'm on Level 3's side, it may well be that they
have high costs due to their own stupidity.  Perhaps they cut a
deal with Equinix for $5,000/month cross connects.  Perhaps they
pay list price for their routers.  Perhaps they are about to go
down the tubes.

As for those who want to re-architect the Internet to "fix" this
problem, please go away.  It's not a technical problem.  It's a
business problem.  Two companies, each responsible for their own
bottom line couldn't find an economically feesable way to interconnect.
Any attempt to "force" the interconnection (either via regulation,
transit through third parties, etc) will RAISE prices for all
involved.  The key here is that the peering was not economically
viable for some reason.

It will be interesting to see how this is resolved in the end.  As
time passes, there will be increased pressure on both companies to
fix this problem.  Single homed customers are going to think twice
about connecting to either one.

My own observations?  This appears to be happening to Cogent a lot
lately.  This makes me wonder if part of the reason they have been
able to offer lower costs is by finding ways to take advantage of
peers and transfer costs to them which is causing the peers to
notice and take action.  I also find Cogent's practice of offering
Level 3 customers free service unseemly.  They are partially
responsible for the outage, and are trying to use that fact to lure
customers.  That makes me wonder if they've written off Level 3
entirely already...after all if you're planning on working out a
deal with someone you don't rub salt in their wounds as a first
step.

-- 
       Leo Bicknell - bicknell at ufp.org - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
Read TMBG List - tmbg-list-request at tmbg.org, www.tmbg.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20051006/4583893e/attachment.sig>