Carrier Grade NAT

Tue Jul 29 22:42:55 UTC 2014

On Tue, Jul 29, 2014 at 06:19:31PM -0400, Lee Howard wrote:
> Thanks for sharing your experience; it's very unusual to get the
> perspective of an operator running CGN (on a broadband ISP; wireless has
> always had it).
> 
> On 7/29/14 5:28 PM, "Tony Wicks" <tony at wicks.co.nz> wrote:
> 
> >OK, as someone with experience running CGNAT to fixed broadband customers
> >in
> >general, here are a few answers to common questions. This is based on the
> >setup I use which is CGNAT is done on the BNG (Cisco ASR1K6).
> >
> >1. APNIC ran out of IPv4 a couple of years ago, so unless you want to pay
> >USD $10+ per IP then CGNAT is the only option.
> 
> Eh, a bit over US$7 now, but whatever. Higher in APNIC.
> 
> >2. IPv6 is nice (dual stack) but the internet without IPv4 is not a viable
> >thing, perhaps one day, but certainly not today (I really hate clueless
> >people who shout to the hills that IPv6 is the "solution" for today's
> >internet access)
> 
> It's viable, it's just not a substitute for IPv4 yet.
> Except for specific scenarios.  For instance, you mention gaming below; if
> two users are playing on Xbox ONE, they can use IPv6 and they're off the
> CGN.  Or if a bank has blacklisted an IPv4 address on the CGN, but the
> bank is dual-stack, some users can still get there.
> Of course, that snowballs.
> 
> >3. 99.99% of customers don't notice they are transiting CGNAT, it just
> >works.
> 
> Surprised it's that high.
> 
> >4. You need to log NAT translations for LI purposes. (IP
> >source/destination,
> >Port source/destination, time) Surprisingly this does not produce that
> >big a
> >database burden. However as Cisco's Netflow NAT logging is utterly useless
> >you need to use syslog and this ramps up the ASR CPU a bit.
> 
> Can you quantify?
> The log entry has to be at least:
> 32 bits	source address
> 16 bits source port
> 32 bits destination address
> 16 bits destination port
> 64 bits? timestamp
> ---
> 160 bits = 20 bytes per flow
> You have to log the end of the flow, too, right?  Another 20 bytes?
> 40 bytes per flow.  Not including syslog severity and message text.

You can get it down a bit smaller, if you're OK with having to find the
records again to update them at the end of the connection (either TCP FIN,
or UDP mapping timeout):

32 bits NAT endpoint ip
16 bits NAT endpoint port
32 bits dest ip
16 bits dest port
32 bits start timestamp
32 bits end timestamp
16 bits customer ID (you could store the customer's internal IP, but that's
bigger)

That's 22 bytes per flow (maybe 24 if you're planning on having more than
64ki customers in your CGNAT's lifetime).

You could drop the timestamps by another 16 bits each if you don't mind
reducing granularity (if you guarantee you won't reuse a given IP/port pair
for, say, 30 seconds, you can define the timestamp to be, say, 15 second
increments) and/or changing the epoch -- 15 second granularity + rolling
epoch every week => 16 bit timestamps do just fine.

> As I recall, a site like cnn.com opens 80 flows, so 3200 bytes of log data.
> If, as you say in #6, 10,000 customers = 200,000 active translations,
> that's 8,000,000 bytes of syslog. . . per second?  Not sure if "active"
> indicates how fast those sessions churn.
> 180 days of log retention would be. . . 124TB of data.  Per 10,000 users.

Of course, getting anything back *out* of that again in any sort of
reasonable timeframe would be... optimistic.  I suppose if you're storing it
all in hadoop you can map/reduce your way out of trouble, but that's going
to mean a lot of equipment sitting around doing nothing for 99.99% of the
time.  Perhaps mine litecoin between searches?

> >7. CGNAT protects your customers from all sorts of nasty's like small DDOS
> >attacks and attacks on their crappy CPE
> >8. DDOS on CGNAT pool IP's are a pain in the rear and happen often.
> 
> Between #7 and #8, do they balance out?

I'd doubt it.  A customer getting DDoS'd counts against their usage limit;
you can't bill traffic pointed at a CGNAT address against any particular
customer.  <grin>

- Matt

-- 
If only more employers realized that people join companies, but leave
bosses. A boss should be an insulator, not a conductor or an amplifier.
		-- Geoff Kinnel, in the Monastery