Deploying IPv6 in a datacenter (Was: Awful quiet?)

Kevin Day toasty at dragondata.com
Wed Dec 21 13:50:14 UTC 2005



On Dec 21, 2005, at 2:09 AM, Jim Popovitch wrote:
>
> With the thousands of datacenters that exist with IPv4 cores, what  
> will it take to get them to move all of their infrastructure and  
> customers to IPv6?  Can it even be done or will they just run IPv6  
> to the core and proxy the rest?
>
> -Jim P.
>


We've just gone through a pretty decent sized attempt to convert our  
infrastructure and applications to IPv4/IPv6 dual stack, and was  
asked by someone else to write up the successes and problems I had  
doing this. I'm no where near done writing everything up, or even  
finished with the migration attempt, but here's some bits of my  
notes, describing our real world experience to throw in. I'll add  
that I'm small potatoes to you guys out there, I'm sure those with  
much larger networks will face some additional challenges.

Our biggest client had a need for IPv6 for a couple of reasons.

The first was an application (which surprisingly was IPv6 ready) that  
required a unique IP address "per instance", they needed to be able  
to handle tens of thousands of "instances" to work around some  
brokenness in the application, and RFC1918 wouldn't cut it. Using  
made up IPv6 addresses with no IPv6 connectivity worked, but we  
wanted to do it "right".

The second was because of IRC and some Japanese students. This client  
has a pretty thriving chat community going, based around IRC. One  
niche of customers and users for this site that suddenly exploded in  
popularity was with some east asian(mostly Japanese) students, using  
these services from their dorm rooms or computer labs. The  
workstations themselves run IPv4, but the university's backbone was  
IPv6 only. The side effect of this was that all non HTTP IPv4  
connections were going through some kind of proxy server, that had a  
time limit on how long you could keep a session open. They were  
getting booted off constantly, and the IT staff at the university was  
asking them to find an IPv6 IRC server they could use instead. (It's  
possible I'm misrepresenting the situation here, I didn't have direct  
contact with the people involved, so the technical details here might  
be wrong).

The third was for VPN. A new product is going to use potentially  
hundreds of VPNs, again RFC1918 addresses won't work for this  
application, and we're trying to be a good neighbor and not blow  
several /24s worth of space on something that didn't really need it.  
GRE tunnels worked fine for this, and allowed us to do IPv6 inside  
the tunnel over IPv4 transit and preserve our tiny IPv4 allocations.

But yeah, those are three really weak needs for requiring IPv6, but  
I'm guessing it's going to be niches like this that start the need  
for IPv6 at all. So, we decided we'd try to make our network IPv6  
enabled, and see how hard it would be to run everything dual stack  
that would support it.

The first wakeup call that this wasn't going to be easy was that only  
one of our transit providers supported IPv6 in any way at all. After  
we got IPv6 turned up, the problems we discovered, roughly in order:

1) IPv6 on the internet overall seems a bit unreliable at the moment.  
Entire /32's disappear and reappear, gone for days at a time. The  
most common path over IPv6 from the US to Europe is US->JP->US->EU. I  
realize this may be specific to our connection itself, but browsing  
looking glasses seems to back up that it's not just us.

2) Lots of providers who are running IPv6 aren't taking it as  
seriously as IPv4. Manual prefix filters, NOC staff that doesn't even  
know they're running IPv6, etc.

3) Some key pieces of internet infrastructure are IPv6 oblivious.  
ARIN's route registry doesn't support the "route6" objects, for example.

4) Even though we went through our applications and software before  
starting this to check for IPv6 readiness, there's a huge difference  
between "Supports IPv6" and "actually works with IPv6".

5) Our DNS software(djbdns) supports IPv6, kind of. WIth patches you  
can enter AAAA records, but only by entering 32 digit hexadecimal  
numbers with no colons or abbreviations. We were never able to get it  
to respond to queries over IPv6, so of all our DNS is still IPv4.

6) Some software supports IPv6 and IPv4 at the same time by using  
IPv4 addresses on IPv6 sockets. (i.e. they bind to tcp6 on port 80,  
and an incoming ipv4 connection appears as ::ffff:192.168.0.1). Other  
applications want to bind to two different sockets. Others want you  
to run two copies of themselves, one for IPv6 an one for IPv4.  
However, on a BSD system, the setting net.inet6.ip6.v6only is a  
system-wide configuration option. If you turn it off (allow IPv4  
connections to come in on IPv6 sockets) for one application running  
on the server that requires it off, and you're running a difference  
service on the same server that wants to run IPv4 and IPv6  
separately, you have to make sure the IPv6 daemon doesn't start  
before the IPv4, or it will bind to both protocols and the IPv4  
daemon won't be able to bind to the port.

7) We found that even if applications support IPv6, they default to  
disabling it during compiling 95% of the time. We had to go back and  
recompile our HTTP servers, PHP, all sorts of libraries, etc. Since  
IPv6 defaults to off on a lot of packages, we had issues just getting  
the software to BUILD with ipv6 turned on unless we were using the  
exact same libraries as were current when it was last released.

8) Once we got everything on the network and server side ready for  
and usable on IPv6 we discovered that a lot of our client's  
applications just had no idea what to do with IPv6 connections. Many  
PHP applications broke because they expected $_SERVER['REMOTE_ADDR']  
to fit within 15 characters at most. Databases had to have their  
columns widened (if they were storing the address as text), or  
functionality had to be rewritten if they were storing IPs as 32 bit  
integers. Web server log analyzers claimed that the log was  
"corrupted" if it had an IPv6 address in it. Lots and lots of  
application logic just wasn't IPv6 aware at all, and either had  
serious cosmetic problems with displaying IPv6 addresses, or simply  
didn't work when an IPv6 address was encountered.

9) Once we started publishing AAAA records for a few sites, we  
started getting complaints from some users that they couldn't reach  
the sites. Some investigating showed that they had inadvertently  
enabled IPv6 on their desktop without having any IPv6 connectivity.  
In some cases it was traced to users reading about 6to4 and wanting  
to play with it but not correctly installing it, then not  
UNinstalling it. Others had turned on IPv6 on OS's that make it easy  
to do so (OS X for example) without realizing what it was. Some  
browsers/OSes seem better than others at figuring out that they don't  
have IPv6 working and falling back to IPv4.

10) Smaller than normal MTUs seem much more common on IPv6, and it is  
exposing PMTUD breakage on a lot of people's networks.

11) Almost without fail, the path an IPv6 user takes to reach us (and  
vice-versa) is less optimal than the IPv4 route. Users are being  
penalized for turning on IPv6, since they have no way to fall back to  
IPv4 on a site-by-site basis when using a web browser.


In the end, we've backed out almost all of our changes to make sites  
IPv6 visible for now. It broke things for far more IPv4 users than it  
helped IPv6 users. We've left the IRC services running on a dual  
stack system, since we were able to partition IPv6 off well enough  
not to hurt things. Everything else is back to IPv4 only.

I'm also personally a bit concerned with how IPv6 allocation and  
routing works with respect to small to medium sized networks like  
ours. I know this is still a hot topic and several proposals are  
being passed around to resolve some of these issues, but it seems  
like I *lose* functionality with IPv6 that I have with IPv4, mostly  
due to the "don't deaggregate your allocation" mantra, and how far  
the bar was raised to get PI space.  We do a lot of things things in  
IPv4 land with regard to routing and addressing that I don't believe  
we can do in IPv6, which worries me more. Shim6 and other proposals  
are creative, but don't replace a lot of the functionality I'd be  
losing. This is another story though, that is getting really off topic.


Getting your network running IPv6 doesn't seem to be the challenge  
anymore. None of our L2 devices cared at all. Our L3 devices took  
some configuration, but moved pretty easily. it's the server and  
application software that needs a lot more work. I don't think we're  
even close to the point where an end-user can go to their provider  
and say "IPv6 me!" and get it working for more hassle than it's worth  
to them.

-- Kevin







More information about the NANOG mailing list