The Making of a Router

Sun Dec 29 09:10:19 UTC 2013

On Sun, 2013-12-29 at 03:31 +0100, Baldur Norddahl wrote:
> (...)
> The users each have a unique VLAN (Q-in-Q). The question is, what do I put
> on those VLANs, if I do not want to put a full IPv4 subnet on each?
> 
> My own answer to that is to have the users share a larger subnet, for
> example I could have a full class C sized subnet shared between 253
> users/VLANs.
> 
> To allow these users to communicate with each other, and so they can
> communicate with the default gateway IP, I will need proxy arp. And in a
> non-OpenFlow solution, also the associated security functions such as
> DHCP-snooping to prevent hijacking of IP addresses.
> 
> Which devices can solve this task?

Hi Baldur,

Assuming you manage 1.1.1.0/24 and 2001:db8:0::/48 and
have a Linux box on both ends you can get rid of
IPv4 and v6 interco subnets and arp proxy the following way:

1/ on the gateway
ip addr add 1.1.1.0/32 dev lo

for all client VLAN "NN" on eth0 :
ip -6 addr add fe80::1/64 dev eth0.NN
ip -6 route add 2001:db8:0:NN00::/56 via fe80::1:NN dev eth0.NN

2/ on user CPE number "NN" CPE WAN interface being eth0 :
ip addr add 1.1.1.NN/32 dev eth0
ip route add 1.1.1.0/32 dev eth0
ip route add default via 1.1.1.0
ip -6 addr add fe80::1:NN/64 dev eth0
ip -6 route add default via fe80::1 dev eth0
# ip -6 addr add  2001:db8:0:NN00::1/56 dev eth0 # optional

Note: NN in hex for IPv6

The trick in IPv4 is that linux by default will answer to ARP requests
for "1.1.1.0" on all interfaces even if the adress is on the loopback.
And in IPv6 use static link local on both ends. You can replace
"1.1.1.0" by any IPv4, but since ".0" are rarely assigned to end users
it doesn't waste anything and keep traceroute with public IPv4.

The nice thing of this setup is that it "virtualizes" the routing from
the client point of view: you can split/balance your clients on multiple
physical gateways and not change a line to the client configuration
while it's being moved, you just have to configure your IGP between
gateways to properly distribute internal routes.

We (AS197422 / tetaneutral.net) use this for virtual machines too (with
"tapNN" interfaces from KVM instead of "eth0.NN"): it allows us to move
virtual machines around physical machines without user reconfiguration,
not waste any IPv4 and avoid all issues with shared L2 (rogue RA/ARP
spoofing/whatever) since there's no shared L2 anymore between user VM.
It also allows us to not pre split our IPv4 space in a fixed scheme,
we manage only /32 so no waste at all.

Of course you still have work to do on PPS tuning.

Sincerely,

Laurent GUERBY
AS197422 http://tetaneutral.net peering http://as197422.net

PS: minimum settings on a Linux router
echo 1 > /proc/sys/net/ipv4/ip_forward
for i in /proc/sys/net/ipv6/conf/*; do for j in autoconf accept_ra; do echo 0 > $i/$j; done;done
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
echo 65536 > /proc/sys/net/ipv6/route/max_size
for i in /proc/sys/net/ipv4/conf/*/arp_announce; do echo 2 > $i;done

PPS: we also like to give /56 to our users in IPv6, it makes a nice /24
IPv4 <=> /48 IPv6 correspondance (256 users).