Deploying IPv6 in a datacenter (Was: Awful quiet?)

Thu Dec 22 01:59:15 UTC 2005

On Dec 21, 2005, at 4:18 PM, Daniel Roesen wrote:
>> 1) IPv6 on the internet overall seems a bit unreliable at the moment.
>> Entire /32's disappear and reappear, gone for days at a time.
>
> That's certainly true for people not doing it "in production". But  
> that
> ain't a problem as they aren't doing it... in production. :-)
>

We had a case where a somewhat decent sized provider that was  
actually using IPv6 accidentally stopped announcing their space  
without realizing it. After a couple of days of waiting for them to  
fix it, I emailed their NOC and got the impression that I was the  
first to notice they had killed IPv6.

>> The most common path over IPv6 from the US to Europe is US->JP->US- 
>> >EU.
>
> Sorry, but that's not true anymore on grand scale. That might still be
> valid for some exceptionally bad IPv6 providers who still "do it 6bone
> style". Fortunately, those don't play any too significant role  
> anymore in
> global IPv6 routing (which was hard work to achieve).
>

I admit, my experiences are with only a tiny number of users, so I  
may have just had bad luck. But, I had trouble finding any of our  
IPv6 guinea pigs that didn't take a perceptibly slower route to us  
over 6 than they do for 4. (50-100ms)

>> I realize this may be specific to our connection itself, but browsing
>> looking glasses seems to back up that it's not just us.
>
> That'd suprise me. Could you give examples?

Right now, I can't remember, this was a couple of months ago now...  
But next time I encounter one, I'll drop you an email.

>
>> 5) Our DNS software(djbdns) supports IPv6, kind of. WIth patches you
>> can enter AAAA records, but only by entering 32 digit hexadecimal
>> numbers with no colons or abbreviations. We were never able to get it
>> to respond to queries over IPv6, so of all our DNS is still IPv4.
>
> Then stop using incomplete and cumbersome software from authors with
> strong religious believes and a disconnection from any technological
> advances of the last $many years. :-)
>
> "Use the right tools for the job".
>

I don't doubt that there are better tools for IPv6 DNS, but we were  
already using djbdns for a couple of reasons and I didn't want to  
undergo a switch to something else JUST to add AAAA records when what  
we had was working well enough for us. I wasn't trying to document  
how to do IPv6 right, just what problems we hit when we tried to  
switch to IPv6 with no thought to IPv6 being done beforehand.

>> 10) Smaller than normal MTUs seem much more common on IPv6, and it is
>> exposing PMTUD breakage on a lot of people's networks.
>
> It is, but we have tracked down most of them... at least the ones we
> noticed. I don't experience PMTUD problems anymore since long... the
> last one is prolly over half a year ago. And I use IPv6 on all my
> servers, desktops and laptop. :-)
>

Our test network was running through a GRE tunnel inside an IPIP  
tunnel, so our MTU was abnormally tiny. I'm guessing that hit some  
people with PMTUD problems that didn't normally see them because they  
had a short MTU to start with.

>> 11) Almost without fail, the path an IPv6 user takes to reach us (and
>> vice-versa) is less optimal than the IPv4 route. Users are being
>> penalized for turning on IPv6, since they have no way to fall back to
>> IPv4 on a site-by-site basis when using a web browser.
>
> That is indeed a problem. How big the penalty is, depends heavily on
> your choice of upstream provider(s). The isle of sanity gets bigger  
> and
> bigger, and networks with bad IPv6 connectivity become more seldom
> (relatively).
>

Out of all of our transit providers, only one could sell us IPv6  
transit(not faulting those who don't yet). Out of 100+ peering  
connections, only 2 wanted to do IPv6 peering. So, I don't have many  
different angles to view things from.

That said though, the provider we are using for IPv6 seems to be  
doing it right, it just doesn't feel like IPv6 has the same "mesh"  
yet where who is connected to who doesn't really matter that much.

> Thank you for sharing your experience!
>
> BTW, what timeframe are we talking about? Things have changed  
> massively
> over the last 12-18 months.

We threw in the towel (pulled AAAA records) about 6 weeks ago, and  
started IPv6 experimentation about 16 weeks ago.

I'll be writing up a paper going into a lot more detail about what  
went right, what went wrong, and why the decision was made to revert  
back to IPv4 soon, if anyone is interested.

-- kevin