Starting a greenfield carrier backbone network that can scale to national and international service. What would you do?

Thu Apr 3 20:52:40 UTC 2014

Hello everyone,

It's been some time since I've been subscribed/replied/posted here (or 
on WISPA for that matter). I've been pretty busy running a non profit 
startup (protip: don't do that. It's really really terrible) :) I'm 
cofounder and CTO of the Free Networking Foundation. Our goal is to 
bring broadband (5 mbps symmetric to start) bandwidth to the 2/3 of 
Americans who currently can't get it (rural, urban core, undeserved, 
"$ILEC stops on otherside of street" etc).

Efforts so far primarily have consisted of WiFI last (square) mile 
delivery using Ubiquiti hardware and the qmp.cat firmware (also meraki 
access points that were donated, for some reason this seems to happen 
quite a bit). We've helped numerous networks get started, grow and (soon 
we hope) become self sustaining in Austin, Kansas City, Los Angeles, 
Detroit, New York and a few other places throughout the US. The networks 
are in various stages of maturity of course, but a number of them are 
fully operational and passing real traffic. Especially the one in Kansas 
City (it spans both states).

These are (point to point, routed) access/distribution networks which 
connect into colocation providers blended networks.

So that's the background and current state of affairs. Not really NANOG 
material.

The next step is to secure our v6 space and AS number. Now that's not 
horribly difficult or really worthy of NANOG (though I do greatly 
appreciate folks on the list who helped me through the theory/practice 
of that process sometime ago). It appears to be fairly straightforward 
if you are not an LIR. Simply go through the paperwork (LOA, submit to 
ARIN, get out the credit card, textbook BGP config and done). And if FNF 
was operating the networks (we don't, we just help with 
organizing/consulting/software guidance/hardware spend 
optimization/logistics etc) and if there was just one POP (and 
associated administrative body), then again it wouldn't be that 
interesting or worth cluttering up NANOG.

FNF goal is to serve as an LIR, SWIPing out /48 chunks to neighborhood 
level operators. They would then peer with whatever upstream ISPs are 
regionally close and announce out the space. This of course would be 
associated with a training program, registration in an IPAM tool etc.

Regarding the above?

What do the operators on this list wish they could of been trained in 
starting out? I mean obviously they should have good mastery and working 
experience of CCNA level material, along with exposure to higher level 
concepts of WAN networking. What are the tricks, the gotchas, the "man 
that would of saved my company a million bucks in transit costs". Yes I 
realize these sort of things are usually closely held. I also am 
striving to create an entirely new breed of operators running BGP 
enabled sites with ipv6. The more I can do to help ease those folks 
integration into the internet, the better. In short, the often debated 
issue on this list of v6 endpoint explosion is going to be very very 
very real.

What IPAM tools out there can scale to a multi hundred million node, 
distributed, "eventual consistency" national level? (I've been working 
closely with guifi.net, and we are attempting to relaunch that as a very 
slick Apple like experience with a libremap (couchdb based) system.

I'd love to hear from folks across the spectrum of experience and 
network size. From folks who have been dual homed for <~1 year at a 
single site, to tier1 operators who were there when it all started.

So what would you like to see done in a greenfield, open source, open 
governance carrier backbone network? What would a dream TIER1 (and I use 
that in the default free zone sense of the word) look like to you?

Also how the heck would one get this bootstrapped at a sustainable pace? 
Would one create numerous tier2 regional carriers, and they would feed 
into an over arching tier1? I'm thinking something like a 501c8 type 
structure ( 
http://www.irs.gov/Charities-&-Non-Profits/Other-Non-Profits/Fraternal-Societies[1] 
)

As far as I know, this is the first time that an intentional community 
type approach is taken and a tier1 is the end goal. Not evolving into 
one, buying ones way into it, but a manifest destiny type approach to 
building a backbone.

Please feel free to reach out to me directly (charles at thefnf.org[2] ) if 
you wish to have a one on one discussion. In particular I'm interested 
in legal expertise in these sort of areas 
(law/compliance/contracting/negotiations for right of way etc etc etc).

Thanks for reading. I look forward to the discussion!

PS: Yes, I'm young and idealistic. I'm also grounded/practical/focused. 
I'm currently working on making the access portion of the network as 
smooth and turnkey as possible. (That basically means packaging up 
zeroshell/observium/powerdns/libremap/trigger and other bits/bobs into a 
nice livecd/ova/openvz package). I also like to think about the next 
wave of issues while working on the current one. It will take another 
year or so before we need to really be building out the backbone (if 
nothing else, to link up the rapidly growing regional networks).

This is about physical, layer 1 infrastructure. This isn't yet another 
overlay network (CJDNS/GNu FreeNet etc). Yes it's messy, yes it's all 
about non technical end users, yes it's about taking a rather complex 
stack (auth/network awareness/routing platform) and making it accessible 
to power users/"IT professionals". It's also a whole lot of fun!

Please feel free to visit us at https://www.thefnf.org for more 
information.