shim6 @ NANOG (forwarded note from John Payne)

Kevin Day toasty at
Wed Mar 1 18:32:33 UTC 2006

On Mar 1, 2006, at 9:07 AM, Joe Abley wrote:

> On 1-Mar-2006, at 02:56, Kevin Day wrote:
>> If you include "Web hosting company" in your definition of ISP,  
>> that's not true.
> Right. I wasn't; I listed them separately.
> It's important to note that even if you are a hosting company who  
> *does* qualify for PI v6 space, you still need shim6-capable  
> servers, if you want to make them optimally available to multi- 
> homed, shim6-capable hosts. The difference PI makes is in the  
> distribution of addresses to servers (the servers only need a  
> single set).

Which isn't a point to be glossed over. PA v.s. PI is a big deal to  
people for many reasons. (Portability between providers being the  

>> I'm just one guy, one ASN, and one content/hosting network. But I  
>> can tell you that to switch to using shim6 instead of BGP speaking  
>> would be a complete overhaul of how we do things.
> You are not alone in fearing change.

It's not so much fearing change, as the bar of effort/difficulty in  
transitioning from 4 to 6 being really high.

If IPv6 is made as painless as possible to people to transition to it  
NOW, people will. If you can tell every ISP, NSP, hosting company and  
end site that they can continue doing what they're doing now in 4,  
but with vastly more address space, you'd have a lot of convertors.  
Every thing that you make people do differently (even if it is for a  
greater good, and even if it benefits them directly) is one more  
reason people will give NOT to do it until they have to.

Telling a hosting company "Here, you can get a /32 or /44 of your own  
which is virtually unlimited space for your needs, continue using BGP  
and TE as you do now, just deploy IPv6 on top of IPv4 and you're  
live" is an easy sell.

Telling a hosting company "You have to lose the independence of PI  
space. You need to completely start over with your traffic  
engineering and do it in a way totally orthogonal to how you have to  
continue doing it in IPv4 space(adding workload instead of replacing  
what you're doing in IPv4). You now need to trust your customers/end  
users to do the right thing with routing. Routing now involves  
routers, servers and DNS - instead of a handful of devices making  
routing decisions now ALL of your devices need to make routing  
decisions. Even if you do get PI space somehow, you can't deaggregate  
it even if you truly run multiple isolated networks.(Don't say 'Get  
additional allocations then!' because that still results in the same  
number of routes added to the global table, while wasting even more  
space)" is a really hard sell. The only carrot you can offer someone  
is "You can have lots more space", which I personally don't think is  
worth even half of those negatives.

If significant percentages of networks are going to heavily push back/ 
delay deployment of IPv6, IPv4 will be exhausted before a critical  
mass has switched to IPv6, making the whole "how do we protect the  
long term future of IPv6" rational for these policies a little less  

It reminds me of a story from engineer who worked for AT&T/Bell when  
touch tone dialing was first being tested in a few markets. AT&T  
wanted to offer touch tone dialing as a convenience for users, but  
they also had a desire to standardize the dialing procedure  
everywhere. In some markets you could make local calls by just  
dialing the last 4 or 5 digits of their number, others required the  
full 7. Some required a 1+XXX-XXXX to make calls outside their city,  
some only required the 1 if it was a toll call. AT&T really wanted to  
change their network where everyone in every city used the same  
procedure to make outbound calls. They decided that they'd make the  
new dialing plan mandatory with touch tone service. The carrot was  
"Look how much easier it is to dial compared to a rotary phone!", and  
they got the benefits of forcing a standardized system on everyone  
everywhere at the same time. The customer gets something that makes  
their life easier, and the operator of the network gets to make their  
job easier by standardization and reduced overhead of supporting  
hundreds of incompatible dialing plans in each exchange that people  
were used to.

They tested it in a few cities, with a few customers(business and  
residential). A large number of people perceived touch tone dialing  
negatively, so much so that they asked for their rotary phones back.  
It had nothing to do with the push-button interface, it was asking  
people to take a perceived negative along with a positive they  
weren't sure they needed in the first place. Asking people to make  
too many changes at once outweighed the convenience of faster  
dialing. Users found it too confusing to have the new system (touch  
tone dialing, a.k.a. IPv6) work so much differently than what they  
were used to (rotary dialing a.k.a. IPv4) that they couldn't see past  
the change into what the advantage was. In the end, they gave the  
customers touch tone dialing, then gradually deployed the new dialing  
plan, using permissive grace periods where both dialing plans worked  
for as long as possible.

I'm worried the same will happen with IPv4/IPv6. The temptation of  
virtually unlimited addressing is really nice. But I think the  
negatives of the allocation policy and the direction of how  
multihoming is going will scare off the willing participants, and  
we'll be stuck with only getting people to switch when they're forced  
to due to IPv4 exhaustion.

My advice: Get people to switch now.

If you have IPv4 PI space, you can get IPv6 PI space. If you have a / 
21 now, you get a /48. If you have a /20 now, you get a /47. If you  
have a /19 now you get a /46. Etc.

If you have anything bigger than a /48, you can rely on people  
accepting deaggregated prefixes of /48 or shorter.

Push shim6 for people who don't need fancy traffic engineering, as a  
tool for small/medium business. Heck, even residential if you want to  
go that far.

Get a critical mass of people using IPv6 as soon as possible. Once  
it's there, once people are comfortable with it, and once IPv6-acting- 
as-much-like-IPv4-as-possible has proven itself, let people WILLINGLY  
deploy shim6 if it truly would be advantageous to them. I'd have no  
problem with raising the initial/yearly fee per ASN if it made  
everyone comfortable that they were only being used where it was  
truly needed.

On top of all of that, I'm still not convinced that IPv6-acting-as- 
much-like-IPv4-as-possible isn't going to have significantly less  
routes than IPv4, even if everyone moved today. Aren't a large number  
of the routes being advertised due to people having to go back and  
ask for more/bigger allocations again and again? If everyone out  
there right now had to announce only 1 route per POP, and 1 route per  
multihomed customer with PI space, wouldn't your outbound routes  
shrink pretty significantly? That would be a huge step in the right  
direction, and would buy loads of time to allow for a much more  
gradual transition.

>> Putting routing decisions in the control of servers we don't  
>> operate scares me. I wouldn't rely on 90% of our customers to get  
>> this right unless it was completely idiot proof. Even if it was, I  
>> don't see how we can trust that users aren't messing with things  
>> to "game the system" somehow.
> This is the kind of feedback that the shim6 architects need. There  
> is talk at present of whether the protocol needs to be able to  
> accommodate a site-policy middlebox function to enforce site policy  
> in the event that host behaviour needs to be controlled. The scope  
> of that policy mediation function depends strongly on people like  
> you saying "at a high level, this is the kind of decision I am not  
> happy with the hosts making".

While I'm happy to give that feedback anywhere it's needed/welcome,  
I'm kind of surprised alarm bells didn't go off already about this. :)

>> We deal with long lived TCP sessions (hours/days). I don't see how  
>> routing updates can happen that won't result in a disconnect/ 
>> reconnect, which isn't acceptable.
> One of the primary objectives of shim6 is to provide session  
> survivability over re-homing events. Since routing protocols are  
> not used to manage re-homing, the speed at which a session can  
> recover from a topological event depends on the operation of the  
> shim6 protocol between client and server.

I'd... be really curious to see how that works, without having to add  
intelligence to the application layer and stateful firewall layer to  
handle this.

I don't mean to take an "I'll believe it when I see it" stance, but I  
think a lot of layers(that may exist on other hosts) would have to be  
changed to support doing that.

>> We have peering arrangements with about 120 ASNs. How do we mix  
>> BGP IPv6 peering and Shim6 for transit?
> You advertise all your PA netblocks to all your peers.

Ok, I was a bit too vague there...

How do we ensure that peering connections are always used instead of  
transit connections?

Currently for outbound, we can localpref peer-learned routes over  
everything else. How do all of our servers on our end know that  
routes learned from a BGP session on our own routers are desirable?

For inbound, we can either rely on our peers to do the same, prepend  
what we send out to transit to make peer routes look even better to  
our peers, or if we want to force the issue we can send peers more  
specific routes than we send to transit (operating on the principle  
of "most specific wins", no matter what else is done).

I'm not seeing how BGP routing information can enter into Shim6's  
decision making, in any scalable fashion, and again.. something that  
updates near instantly.

>> So far it looks like Shim6 is going to rely on DNS. The DNS  
>> caching issue is a real problem. We need changes to happen faster  
>> than DNS caching will allow.
> Well, not quite.
> If you change a transit provider, then you need to remove a set of  
> AAAA records from the servers you operate, and substitute a new  
> set. The time taken for this change to propagate in the DNS is non- 
> zero, assuming you use reasonable TTLs. This is your point above, I  
> think.

Reasonable TTLs on our end or not, lots and lots of people don't  
respect TTLs.

Seriously, try this sometime. Set the TTL for a very busy site to 5  
minutes. Wait 2 weeks, to make SURE everyone is seeing the new TTL.  
Change the IP in the A record. Watch how long it takes for traffic to  
stop coming in the old IP.  If you see 90% of your traffic moved in  
less than 4 hours, I'd be surprised. If you saw 99% of your traffic  
moved in less than a day, I'd be even more surprised.

I don't know if this is intentional misbehavior of DNS caches, broken  
software, broken OSes, or where the issue is. But, I can attest that  
having to make sudden changes from one provider's PA space to another  
with no grace period will result in support issues for at least a day  
of "Why can't I reach your site?".

> Renumbering for hosting providers can be a monstrous pain in the  
> neck, especially for hosting providers who rely on third parties  
> (or, horrors, their customers) to maintain the zone files within  
> which services are named.

Yep. We don't control the DNS for quite a number of the sites we're  
hosting. Making our customer get involved every time we add/remove a  
transit connection isn't going to be fun. Especially when "big"  
providers who can qualify for PI space of their own don't have to do  

> Some hosting providers of my acquaintance insist on customer zones  
> being redelegated to the hosting providers' nameservers, so that  
> any renumbering that needs to happen can be coordinated by the  
> hosting provider directly. Hosting providers who don't do this, and  
> who use PA addresses with shim6 to multi-home, are definitely going  
> to face some challenges.

That's possible for some hosting providers, not for others. It's not  
uncommon for a customer to use one provider for some services(web  
hosting, for example) and another provider for others(secure/e- 
commerce, for example). Trying to make two competing providers play  
together nicely when managing DNS for one domain is a recipe for  
disaster, even with sub-delegation.

>> Some of our applications are extremely sensitive to jitter/ 
>> latency. We've spent ages tweaking route-maps manually (and  
>> through automated continual tweaking) to make sure we avoid any  
>> congested links. [...]
> The site-policy middleware that I alluded to earlier seems like the  
> analogous place to specify this policy. Such a facility might  
> actually give you more control than you have now -- tweaking BGP  
> attributes to accomplish this kind of thing is often like a game of  
> whack-a-mole; if you were able to control the route taken by  
> traffic in both directions by influencing the locator selection for  
> each and every session, you'd have far greater, and more fine- 
> grained, control over your external traffic than BGP/swamp-abuse  
> gives you currently.

While I don't doubt that there are advantages and disadvantages to  
each way of doing things, I'd much prefer to be given the choice of  
selecting the one that works best for us, possibly a mix of both.

>> We'd still be relying on PA space. No matter how great dhcp6 is,  
>> there will be significant renumbering pain when providers are  
>> changed. Static ACLs, firewall rules, etc. If you're including  
>> customer machines in the renumbering, many simply won't do it.
> Agreed, renumbering is a pain. Dhcp6 sounds like a scary thing to  
> use with servers. Customers suck. Change in operational practices  
> will be required.
> Lest I sound too much like a foam-at-the-mouth shim6 advocate, I  
> think it would be perfectly fine if, in the final analysis, the  
> conclusion was that shim6 and PA/renumbering was not an option for  
> hosting providers. A reasoned technical argument which came to that  
> conclusion would provide a solid basis for the RIRs to modify their  
> allocation policies such that hosting providers could use PI space  
> instead. As perhaps the recent attempt to change the v6 PI policy  
> indicates, the chances of making changes without such a reasoned  
> argument are slim.

And that's kind of my overall point I've been not-so-successfully  
trying to make.

IPv4 is running out. We need people to switch to IPv6, sooner rather  
than later. Instead of trying to make the process as painless as  
possible, and with using tools that are available now, a large swath  
of what probably would be the FIRST people(content/hosting companies)  
who would want to move to IPv6 are being told to "hang tight until we  
figure out this multihoming thing, just don't expect it to work at  
all the same as how you do it now." The additional bite of "You can't  
do what you used to do, because if everyone did it the internet would  
break. Oh, and your bigger competitors can still do things like that,  
just not you." isn't helping matters.

Stopping the routing table from exploding in the future is obviously  
a goal everyone needs to have. Everyone needs to think about this NOW  
rather than when it's too late, I agree.

But, policies shouldn't be written depending on tools that don't  
exist yet, which is basically what's happening now. If IPv6 were  
permitted to be used in the same manner that IPv4 is now (PI space is  
accessible to just about everyone, everyone with an ASN can run BGP,  
you're trusted that if you deaggregate your space it's for a good  
reason, etc), people could actually begin moving things now. Taking  
an existing IPv4 network and overlaying an IPv6 network over the top  
of it is relatively easy, we went from planning to full deployment in  
a week.

Even if 100% of the IPv4 networks moved to IPv6 today, we'd still  
have a smaller table size in 6 than 4. Growth would be slower (ISPs  
and NSPs wouldn't continually be adding more networks as they grew,  
initial allocations should be enough for just about everyone).

Then when Shim6 is developed, you can rely on the current release of  
every major OS supporting it, router/middleware/etc vendors  
supporting it, and all the kinks are worked out, you can let the  
people who find Shim6 appropriate for their needs to use it. Make one  
of the requirements to get an ASN a justification for why non-ASN/non- 
public-routing doesn't work for your organization. Then let each  
network operator choose the right tools for the job.

Without totally upending my network, I need:

1) PI space
2) The ability to deaggregate that PI space where truly needed. (or  
the ability to request multiple PI blocks, but I don't see how that  
helps matters)
3) BGP announcements to the world

If IPv6 can't give me those, and the only thing it's offering is more  
space... That's just not worth the serious amount of labor and  
reengineering of our network. If that's the value proposition, we'd  
hold out on IPv4 as long as possible... And I'm *FOR* IPv6. :)

-- Kevin
(wondering how many people are muttering 'kook' right now)

More information about the NANOG mailing list