Coop Peering Fabric??

Deepak Jain deepak at
Mon Aug 11 23:45:51 CDT 2008

Patrick W. Gilmore wrote:
> As a big ra-ra guy around peering, I thought this might be interesting, 
> but I do not think I agree with the numbers.

I think you read this thinking I meant something I didn't mean; perhaps 
I should have used a different set of prepositions.

[deleted the discussion about utilization and CIRs, that is up to 
everyone to engineer and negotiate, my point shouldn't be so fragile as 
to require quoting spot pricing in markets @ various commits].

>> Assuming 10G PNI model: Assuming even nominal cross-connect fees of 
>> $100-$300/month per fiber pair, plus router port costs for each 
>> private peer (assuming you aren't at >10% utilization on the port) at 
>> a commercial exchange, you are eating a pretty significant cost per 
>> megabit you are actually moving. (plug in your numbers here). 
>> Assumption: Above 1Gb/s utilization, this makes sense or you are 
>> counting on growth.
> Define "significant"?

If you are running << 1Gb/s per PNI it is "expensive".

> Taking 1 Gbps (10%), and assuming even 20K per 10G port over 2 years, 
> adding in $300/month for the x-conn, you are still looking at barely 
> over $1/Mbps.  If you have more than 10% utilization, that number goes 
> down.  Is that significant?  Compared to what?  Transit?

You have costs. Below 1Gb/s I'm stating they are "significant". The 
assumption above says it "automatically makes sense" above 1Gb/s or you 
are counting on growth with a starting point below 1Gb/s. The idea of 
presenting the PNI case was to avoid this sort of response. Obviously I 
didn't draw enough attention to the assumptions.

> I would say a 75% price reduction is pretty significant.  Plus you 
> haven't considered CapEx cost for the transit ports.

Depends on the business. 75% may not be enough if a network's opex costs 
(PP&E) are high enough that this doesn't help. Again, I was trying to 
avoid painting the picture for any particular network, but more of the 
industry's interested parties as a whole.

>> Below 10% you would normally go to a paid peering fabric where you are 
>> paying cross connect + a flat port charge + router port for 1->N peers 
>> and hoping that enough utilization occurs that you get >10% 
>> utilization (to recover capex, opex, etc) and then whatever additional 
>> utilization you need to cover the flat port charge or you are counting 
>> on growth.
> Here we agree.  The port fee even on european IXes is measured in 1000s 
> of $$ per month.  And don't get me started on US or Japanese ports....

I was thinking of US ports, but modeling based on LINX pricing.

>> A "coop", best-effort switch fabric colo'd at a few sites would allow 
>> participants to peer off traffic at a price of the order of a single 
>> cross-connect (~$500/month per 10G port is possible, maybe less), 
>> private-VLANs all-around, or to only-mutually approved peers (e.g. via 
>> an automated web interface, prior art) to avoid many of the /old/ 
>> issues. No requirement for multi-lateral peering. You could peer, sell 
>> transit, buy transit, multicast, etc.
>> The way I figure it, it removes approximately an order of magnitude 
>> from the operational cost of peering with more than a handful of your 
>> largest single talkers. Especially as 100G LAN Ethernet becomes 
>> production before 100G WAN connections become commonplace. Economic 
>> theory (assuming that worked on the Internet) suggests this would 
>> allow for the increase in number of peers by approximately an order of 
>> magnitude (maybe more).
> Sorry, I can't get there.
> First, the "largest single talkers" would not be on a shared fabric, 
> they'd be on dedicated ports, so this idea doesn't help.

I excluded "largest single talkers" by saying "more than a handful of 
your largest single talkers. Semantically, the assumption in PNI was 
that at 1Gb or above PNI makes sense [or you'd soon get there]." The 
question I asked was really related to the idea. If you have a few 
sensible PNIs, presumably you have enough traffic that you could 
conceivably have many potential peers at levels below the PNI case, but 
above the degenerate traffic case. (0, maybe 20mb/s, some number that 
isn't interesting enough to engineer for [or even consider] unless you 
have a nearly zero marginal cost to approach it)

> For the medium to small guys, I think it's a great idea.  Look at SIX, 
> TorIX, PaNAP, etc.  But shaving an _order of magnitude_ off?  No, I 
> don't see it.  CapEx alone is more than 10% of your cost.  (Well, unless 
> you get Japanese IX ports or the most expensive US IX ports.)
> Perhaps I'm lost or confused?  Can someone help me understand?

You and Woodcock make a good point (re: SIX @ zero cost). However, 
~20Gb/s aggregate is at most saving $80K/month between all participants 
for the additional traffic, which is pretty academic given the costs of 
operating networks of any sufficient scale [without looking at the 
constituency of the participants or the traffic].  If *each* network 
were saving $80K/month through the use of a few of these in multiple 
cities, that would be interesting to me.

I guess they would be more interesting deployed in Ashburn or some place 
similar because you could exclude the cost of "bringing" traffic to the 
exchange if the equipment (and bits) are already transported through 
that facility.

That said, I can't get with "Capex alone is more than 10% of your cost". 
  I see 4 port Cisco WS-6704s with 4 XENPAKs on Ebay for like $3K/port, 
but hey, YMMV. There is no real reason to use deep buffers as an 
interface to a low-cost, low latency fabric, especially when you (and 
the fabric) can just add ports cheaply.

So I guess this is now meandering. I can present it differently.

Take the most rudimentary part of the SIX model, put it in a few cities 
where more traffic is exchanged then Seattle, bake, then taste. Wouldn't 
this be far more preferable (with scale) than the "expensive" US IX 
ports -- especially for new [rather than existing] traffic -- and as 
Woodcock mentions, anyone can run it, just requires some scale to be 
valuable enough; and more importantly, since the effective price of coop 
IX traffic would be lower than current major IX traffic, wouldn't this 
encourage more exchange to all participants benefit?

Or (back to my original post) are these costs essentially insignificant 
to the modern business case given the current set of market dynamics?



More information about the NANOG mailing list