[Nanog-observer] Re: Starting a greenfield(ish) small (10k subs?) multihomed (two ASN) , dual stacked, wireless ISP - i can haz advice?

Fri Jul 25 17:08:28 UTC 2014

Hey,

I have started this kind of organization with my friends about 11 years ago
(oh time flies) in Czech Republic in my small hometown. Nowadays it has
around 3000 users. Each user has to pay small membership fee about 8EUR.
Everyone shares 1GBit connectivity to the Internet.

We have started with 4 people on old PCs running Linux (mainly Slackware)
connected via 2.4GHz and backbone running on 2.4 as well with 64Kbit/s
connectivity to the Internet. We went afterwards from 5GHz and 10GHz
backbone, laser backbone and since last year or so ended up with fibres.

In last few years we are forcing users to move towards 5GHz as 2.4GHz is
very noisy in our area and therefore very hard to manage. In part of the
town with apartments we have connected the whole buildings with fibre.

We have seen our organization moving from fully volunteered workforce
towards volunteering organization with two full-timers to give support to
end users as it was unsustainable to support so many end users by
volunteers.

Since we have started as high school students and nowadays half of us has
own families or live/work out of the town it reminds me another thing.
Always share your knowledge with other volunteers and look for other young
and smart people as one day you won't be able to do it (for any reason:)
Our experience is that it is very hard to find other young people to
continue on our work.

Our 'business model' was always volunteering organization which supports
other non profit or non gov organizations. We also supported OSS projects
and help to build hockey pitch in our town et cetera et cetera:)

My last recommendation would be to just have fun. We have seen many points
in our history where we didn't have time or energy to do something but it
has all paid off - it helps us with our careers and most importantly to
other people.

I'm sorry if you were expecting any technical advice;)

Good luck and if you have any other (even technical) questions please let
know:)

Matyas

On Fri, Jul 25, 2014 at 4:08 PM, Blake Hudson <blake at ispn.net> wrote:

>
> charles at thefnf.org wrote the following on 7/23/2014 11:58 AM:
>
>
>> This is a greenfield network. We've got Ubiquiti gear for the backbone.
>> Running a mix of QMP routers with BMX6 as the IGP linked over AirOS l2
>> bridge "pseudowires". We'll be homed to two AS upstreams. Using pfSense as
>> the WAN edge routers.
>>
>> From all my reading of the list, it seems like key things to do in this
>> scenario:
>>
>> 1) Have full flow telemetry at all points to help with (D)DOS mitigation.
>> 2) Do CGN in pools (so perhaps ~500 to 1k users behind each IP)?
>> 3) Provision a /56 of v6 space to each end user. I was thinking of having
>> the CPE with CeroWRT and be multi SSID with a /64 per. I'm interested in
>> folks thoughts on this?
>> 4) Upsell a public v4 address if someone requires it
>> 5) Of course implement bcp38
>>
>> I'm mostly interested in technical feedback. Business model etc type
>> feedback is welcome as well, but not the primary purpose of this message. :)
>>
>
> Charles, it sounds like you've got a lot of the technical items on your
> radar.
>
> I highly recommend pfsense for a firewall (been using pfsense and m0n0wall
> for years), but do have some concerns about using it at scale for (several)
> thousands of users. Most of this relates to NAT/State tracking, some of it
> hardware related, some of it software. If possible, I would suggest you
> obtain a routable IP address per user and avoid the pitfalls of NAT (I know
> at some point this may become expensive). If you start with IPv6 from day 1
> you are in a lot better place to encourage customers to upgrade to IPv6
> capable gear. I would also suggest using stateless firewall rules and
> routing on your WAN devices. This should simplify the functions performed
> by these boxes to reduce the need to troubleshoot, apply updates, etc
> (resulting in better availability). I haven't used pfsense in an ISP WAN
> router capacity, and personally feel a router from Cisco, MikroTik, or
> Ubiquiti's EdgeOS devices, etc may be more appropriate in this role. If
> you've automatically discounted big name gear due to upfront costs, you
> might consider buying from a used equipment reseller (I can recommend a
> few, if needed).
>
> If you do need to use NAT, I feel like 500+ users sharing a single NAT IP
> will result in poor quality of service and more admin overhead. My gut
> feeling is that <50 may be more appropriate, depending on the quality of
> service you want to provide. This provides some headroom if one user makes
> many connections (p2p, virus infection, DoS attack) and also lessens the
> number of subs you need to look at in cases of abuse that are reported as
> an IP/port. Individual pfsense servers in a cluster may provide scalable
> CGN services. I'm not sure how you want to handle logging of all that data,
> but pfsense should allow you to define rules that allow stateless auditing
> (ip 1.2.3.4, ports 1000-2000 always NAT to sub A). The XML config file or
> possibly the shell is probably the easiest way to define such rulesets at
> scale.
>
> I didn't see it mentioned, where (and to whom) are you multihoming? Do you
> have a good working relationship with these folks (cell phone, email
> contacts that reach someone promptly)? Will you be considered a facilities
> based ISP (and subject to CALEA or other regulation)?
>
> --Blake
>
> --
>
>