AWS Elastic IP architecture

Tue Jun 2 09:01:38 UTC 2015

> On Jun 1, 2015, at 4:30 PM, Christopher Morrow <morrowc.lists at gmail.com> wrote:
> 
> On Mon, Jun 1, 2015 at 3:06 AM, Owen DeLong <owen at delong.com <mailto:owen at delong.com>> wrote:
>> 
>>> On May 31, 2015, at 7:46 PM, Christopher Morrow <morrowc.lists at gmail.com> wrote:
>>> 
>>> On Sun, May 31, 2015 at 9:07 PM, Owen DeLong <owen at delong.com> wrote:
>>>> As I said before:
>>>> 
>>>> Host Virtual (vr.org <http://vr.org/>)
>>>> Softlayer (softlayer.com <http://softlayer.com/>)
>>>> Linode (Linode.com <http://linode.com/>)
>>>> 
>>>> All have full dual-stack support.
>>> 
>>> <snip>
>>> 
>>>>> At the risk of feeding the troll...
>>>>> 
>>>>> This isn't just an AWS problem.
>>> 
>>> So... ok. What does it mean, for a customer of a cloud service, to be
>>> ipv6 enabled?
>> 
>> It means that I should be able to develop my cloud application(s) with full native IPv6 support and expect to be able to serve IPv4-only, IPv6-only, and dual-stack customers using native IP stacks on my virtual machines without requiring external proxies, translators, etc. from the cloud service provider.
>> 
> 
> Ok, I suppose as a long term goal that seems fine. I don't get why
> 'ipv6 address on my vm' matters a whole bunch (*in a world where v4 is
> still available to you I mean), but as a long term goal, sure fine.

Short term, I don’t want to have to pay to develop my application twice.

I want to develop once for IPv6 and be done. Using IPV6_V6ONLY=FALSE
socket option, I should be able to support both protocols from an application
developed for IPv6 on machines which have both protocol capabilities.

If you deliver to my native IPv6 or native IPv4 address, then i get real source
addresses of the originator of the connection and I can develop my application
normally and not have to re-engineer it around a protocol change later.

If you don’t, then, if I care about the source of the connection request in my
application, I either have to write the application to look elsewhere for it if the
connection came in on IPv4, or, I have to get even more creative about it.
Worse, it means that I’m having to maintain IPv4-specific cruft in my application
some of which may be necessary to support IPv6 connections that arrive over
IPv4 because I can’t get native IPv4.

No, this is not a long-term goal, it’s a short-term need. Otherwise, my development
costs are increased and my development time is extended. Additionally, the
extra code creates additional opportunities for errors, security holes, etc.

In short, there’s nothing good about not being able to develop a native application
and there are many draw-backs.

> In the short term though, that was my question: "What should be
> prioritized, and then get that information to aws/gce/rackspace/etc
> product managers" because I suspect their engineers are not silly,
> they know that v6 should be part of the plan... but the PM folk are
> saying: "No one is asking for this v6 thing, but lots of people want
> that other shiney bauble...so build the shiney!!”

I have little doubt that this is true. However, but the time all of their customers
start screaming for IPv6, do you think they’re going to be ready/willing/able
to wait the 6-18 months it will take to deploy it after that point? I think they’re
going to be coming in crisis mode wondering how they can get IPv6 turned
on yesterday.

> 
>>> What really matters for a cloud service user? What information could
>>> be surfaced to the cloud providers in order to get the most important
>>> ipv6 'stuff' done 'now’?
>> 
>> Ideally, simple native routing of IPv6 to provisioned hosts should suffice in most cases.
>> 
>>> o Is it most important to be able to address ever VM you create with
>>> an ipv6 address?
>> 
>> Yes.
>> 
> 
> long term sure, short term though.. really?? isn't it more important
> to be able to terminate v6 services from 'public' users? (and v4 as
> well because not everyone has v6 at home/work/mobile)

Yes, but I need to be able to terminate the v6 services on the v6 socket
on my VM, not on some proxy that masks the source address.

If you have a 6->4 proxy/nat/whatever that somehow hands me a packet
with an IPv6 from address in the IPv4 packet header, that’s a pretty neat trick.
If you have operating system stacks that will support this and hand said source
address to the application through the standard accept() API, that’s an even
better trick.

If you don’t have that for every platform supported on AWS (Linux/BSD/Win/etc.),
then I think we need the above mentioned native connectivity, no?

>>> o Is it most important to be able to talk to backend services (perhaps
>>> at your prem) over ipv6?
>> 
>> It’s hard to imagine how you could provide the first one above without having this one come along for the ride.
>>> 
>>> o Is it most important that administrative interfaces to the VM
>>> systems (either REST/etc interfaces for managing vms or 'ssh'/etc) be
>>> ipv6 reachable?
>> 
>> This would be the one where I would most be able to tolerate a delay.
>> 
>>> o Is it most important to be able to terminate ipv6 connections (or
>>> datagrams) on a VM service for the public to use?
>> 
>> If you can’t get the first one, this might be adequate as a short-term fallback for some applications. However, it’s far from ideal and not all that useful.
>> 
> 
> I agree it's not ideal, and I was making a list of 'short term goals'
> that could be prioritized and get us all to the v6 utopia later.

The thing is that in most cases, the sum of implementing this and managing it over a 3-6 month period is more effort than simply routing native IPv6 and adding IPv6 to your provisioning systems. As such, it’s hard for me to justify putting this short term goal in place vs. just deploying native IPv6 connectivity.

>>> I don't see, especially if the vm networking is unique to each
>>> customer, that 'ipv6 address on vm' is hugely important as a
>>> first/important goal. I DO see that landing publicly available
>>> services on an ipv6 endpoint is super helpful.
>> 
>> Here’s the thing… In order to land IPv6 services without IPv6 support on the VM, you’re creating an environment where:
>> 
>>        1.      The services basically have to be supported by some form of proxy/nat/etc.
>>                So long as you are using a supported L4 protocol, that might be fine, but not everything fits in TCP/UDP/ICMP.
>>                Generally supporting GRE, IKE, and application-specific protocols becomes an issue.
>> 
> 
> I figured the simplest path from v6 to v4 was: "Rip the header and
> extension headers off, make a v4 packet, deliver to vm”

How do you do that without masking the source address of the IPv6 host at the other end of the connection?

If you don’t have a good answer to that question… This fails.

> aside from "yes, your request came from <ipv6 literal>" that should
> 'just work', you do have to maintain some state to deliver back,
> depending on the implementation (but even that is probably able to be
> hidden from the 'user' and provisioned/capacity planned independent of
> the 'user’).

Sure, this is all fine and dandy from the user perspective, but it sucks in very big ways from the application developer perspective.

Imagine running a service (let’s say not a web-based service for the moment) with 100s of thousands or even millions of users from all over the world. Now, imagine if every connection in your application logs came from 192.9.200.5. How, exactly, do you track where your users are coming
from? How do you tell who connected from what network? How do you track down the abuser after you find him in log entries which share the
same source address as everything else?

> 
>>        2.      The developer has to develop and maintain an IPv4-compatible codebase rather than be able to use
>>                the dual stack capabilities of a host with IPV6_V6ONLY=FALSE in the socket options. This delays the
>>                ability to produce native IPv6 applications.
>> 
>>    3.      Proxies and translators add complexity, increase fragility, and reduce performance.
>> 
> 
> I think this point is cogent, but ... also part of the pie that 'aws'
> pays for you the user... Or rather they run a 'service' which takes
> care of this, has slas and slos...
>   "All packets delivered with 99.99% having Xms extra latency!"
>   "Service has 99.999% uptime!"
>   "Throughput comparable (99.99% at peak, 99% of the time) to
> straight/native v4”

I have yet to see a proxy/nat implementation that delivers on those numbers.

Forgive my skepticism, but until this is demonstrated, I think I’d rather focus efforts on native deployments than miserable bandaids.

>>> Would AWS (or any other cloud provider that's not currently up on the
>>> v6 bandwagon) enabling a loadbalanced ipv6 vip for your public service
>>> (perhaps not just http/s services even?) be enough to relieve some of
>>> the pressure on other parties and move the ball forward meaningfully
>>> enough for the cloud providers and their customers?
>> 
>> For the reasons outlined above, I don’t really think so.
> 
> ok, I figured that for short-term while the providers figure out: "Oh
> yea, this v6 thing is pretty simple in our encap'd world... why didn't
> we just turn it on originally?"
> 
> getting v6 endpoints with little hassle for the 'user' (vm user) would
> help us all a bunch.

It might help some, but given the extent to which it complicates application development and breaks things unless you’re running the simplest case of web server, I don’t think it helps much.

Owen