IXP

Sat Apr 18 00:08:04 UTC 2009

Arnold Nipper <arnold at nipper.de> writes:

> On 18.04.2009 00:04 Paul Vixie wrote
>
>> ... has anybody ever run out of 1Q tags in an IXP context?
>
> Why? You only need 1 ;-)

really?  1?  at PAIX we started with three, two unicast (wrongheadedness)
and one multicast, then added another unicast for V6.  then came the VNI's,
so i'm betting there are hundreds or thousands at most PAIX nodes today.
are others just using one big shared network for everything?

i should expand on something i said earlier on this thread.  the progression
i saw at PAIX and later saw from inside MFN was that most new peerings would
happen on a shared port and then as that port filled up some peerings would
move to PNI.  given that success in these terms looks like a PNI, i'm loathe
to build in any dependencies on the long term residency of a given peering on
a shared multiaccess subnet.

i should answer something said earlier: yes there's only 14 bits of tag and
yes 2**14 is 4096.  in the sparsest and most wasteful allocation scheme,
tags would be assigned 7:7 so there'd be a max of 64 peers.  it's more
likely that tags would be assigned by increment, but it's still nowhere
near enough for 300+ peers.  however, well before 300 peers, there'd be
enough staff and enough money to use something other than a switch in the
middle, so that the "tagspace" would be per-port rather than global to the
IXP.  Q in Q is not how i'd build this... cisco and juniper both have
hardware tunnelling capabilities that support this stuff...  it just means
as the IXP fabric grows it has to become router-based.

i've spent more than several late nights and long weekends dealing with the
problems of shared multiaccess IXP networks.  broadcast storms, poisoned ARP,
pointing default, unintended third party BGP, unintended spanning tree,
semitranslucent loops, unauthorized IXP LAN extension... all to watch the
largest flows move off to PNI as soon as somebody's port was getting full.

conventional wisdom says a shared fabric is fine.  conventional wisdom also
said that UNIX came only from bell labs, that computers and operating systems
were bought from the same vendor on a single PO, that protocols built for T1
customers who paid $1000 MRC would scale to DSL customers who paid $30 MRC,
that Well and Portal shell users should be allowed to use outbound SMTP, that
the internet would only be used cooperatively, and that business applications
were written in COBOL whereas scientific applications were written in FORTRAN,
and that the cool people all used BSD whereas Linux was just a toy.  so i
think conventional wisdom isn't perfectly ageless.
-- 
Paul Vixie