Greenfield Access Network

Thu Jul 31 19:58:20 UTC 2014

Scott,

Thank you for your input.

What do you recommend for network segmentation? A VLAN per Chassis, a VLAN
per service, or a VLAN per customer/port? When you say qinq VLANs are you
referring to the CVLAN model?

I am really interested to know how the largest providers, like Comcast,
AT&T, and Verizon setup their residential networks. Little information is
available on the internet besides the access platforms they use.

On Thu, Jul 31, 2014 at 1:51 PM, Scott Helms <khelms at zcorum.com> wrote:

>
>
> On Thu, Jul 31, 2014 at 2:25 PM, Colton Conor <colton.conor at gmail.com>
> wrote:
>
>> I have read both the Juniper MX and Cisco ASR9K do support this advanced
>> BRAS functionality, what Juniper calls Subscriber Feature Management and
>> what Cisco calls BGN. These software functions run on the router itself,
>> however the are not free or included with the base chassis. To enable these
>> you must pay a hefty fee. So you are saying that these advanced feature
>> packs that the largest networking markers in the world sell are really not
>> needed anymore due to advancements on the access vendor side of the house?
>> From the reading I have done about these solutions, it is kind of like
>> PPPoE with a radius setup, but instead DHCP option 82 with a radius setup.
>> These routers are also capable of running a local DHCP server, but I am not
>> sure if that is recommended.
>>
>
> Yeah, that's it in a nutshell.  There are several options, like matching
> on Option 82 or redirecting to a web page, but at the end of the day I
> don't believe they're worth the time or expense.  Keep in mind that earlier
> in my career I was a huge proponent of BRAS architecture and I've put in
> everything from Nortel Shasta's to Lucent Terminators, to Redbacks, to
> Juniper ERXs and several more models I can't remember.  Once you get past
> the whole lack of authentication, which was never very secure, and
> understand that you can depend on Option 82 to tell you where a session
> came from physically the rest is just finding away to count and account for
> bits.
>
> Oh, and I never recommend running the DHCP daemon on a piece of networking
> gear for service providers.
>
>
>>
>> The DPoE DOCSIS provisioning of your GPON network is interesting, but is
>> that really relevant for a new provider if they don't have cable CMTS
>> systems already deployed. Sure, it makes sense for the cable compaines who
>> have already bought billing systems and are used to living in
>> a DOCSIS world. But if you were starting fresh from the group up are you
>> recommending we look at GPON providers like Calix because they support DPoE
>> so we can buy DOCIS billing systems? That is an interesting concept.
>>
>
> I'd strongly recommend finding a vendor that says they will support it on
> the shelves you're going to buy even if they don't today.  Even if you're
> not doing DOCSIS cable modems and don't ever plan to the provisioning
> paradigm (DHCP, TFTP, ToD) is much simpler than the proprietary north bound
> (usually SOAP) API that direct integration requires.  You can even build
> your own provisioning system with a little scripting and there are many
> more commercial options than there are for direct integration to the
> shelves.
>
>
>>
>>
>>
>>
>>
>> On Thu, Jul 31, 2014 at 12:59 PM, Scott Helms <khelms at zcorum.com> wrote:
>>
>>>
>>> On Thu, Jul 31, 2014 at 12:07 PM, Colton Conor <colton.conor at gmail.com>
>>> wrote:
>>>
>>>> Scott,
>>>>
>>>> Thanks for the long post.
>>>>
>>>> We will use a layer 2 10G aggregation switch then to aggregate the
>>>> chassis at the core location. Do you have any recommendations on 10G
>>>> switches?
>>>>
>>>
>>> Not really, just stick with one of the major brands and you _should_ be
>>> fine.
>>>
>>>
>>>>
>>>> Yes I realize the math is a little backwards as this is
>>>> all hypothetical at this point. We would provision each ONT as a shared
>>>> 1Gbps offering similar to Google Fiber. We know there will be a large
>>>> amount of oversubscription as no one really uses a full Gbps or anywhere
>>>> close to it. I just wanted to stress the point that carrier redundancy at
>>>> the 10G level would be a requirement for the core router, and it should of
>>>> course have 10G links going to the uplinks on the aggregation switch. I
>>>> think the Cisco ASR9k and the Juniper MX line will do well. Not sure if
>>>> there are any others that can handle this level of traffic on the BGP side?
>>>>
>>>
>>> That's reasonable IMO and yes, I think the Juniper MX can handle that as
>>> well as some other functions for you related to subscriber management if
>>> you want.  The MX line has a full BRAS set of capabilities built into it
>>> that it inherited from the older ERX line, but they're commonly deployed
>>> without using any of them of as well.
>>>
>>>
>>>>
>>>> So we have a 10G aggregation switch to aggregate the chassis uplink
>>>> connections, and a 10G router BGP capable router.
>>>>
>>>> I really liked your article on DHCP vs PPP for DSL networks. We
>>>> definitely agree the way to go is with a DHCP server. A couple of items
>>>> your article left as big questions:
>>>>
>>>
>>>
>>>> 1. The article mentioned DHCP doesn't do the other part of what PPPoE
>>>> or PPPoA does, which is generate RADIUS accounting records that give us the
>>>> bandwidth information. So that’s one of the main challenges in switching to
>>>> a DHCP based system. So, how do you handle bandwidth tracking in an all
>>>> DHCP environment then? If I want to track how many GB a customer used last
>>>> month, or the average Mbps used how do you do so?
>>>>
>>>
>>> There are a few ways to get at that problem.  You can use Netflow/IPFIX
>>> collection to gather the usage from your router, accepting that you're only
>>> going to get information on layer 3 traffic, which generally isn't a
>>> problem.  You will need to match the IPs up against your Option 82 parsing
>>> which will give you the circuit ID, IP address, and WAN MAC of the ONT.
>>>  You can also poll your shelves via SNMP, CLI, TL-1, and/or Netconf to
>>> collect the data and put it into a database in much the same way you can
>>> use RADIUS accounting data.
>>>
>>>
>>>>
>>>> 2. I liked your option 82 example, and that works well for DSL networks
>>>> where one port is tied to one customer. But how does option 82 work when
>>>> you have multiple customers hanging off a GPON port? What does GPON use a
>>>> subport identifier?
>>>>
>>>
>>> Yep, the different vendors implement it slightly differently, usually
>>> the ONT MAC/serial will be included or the ONT ID will be included.  Talk
>>> with your vendor, all the major OLT vendors are very familiar with Option
>>> 82 and in many cases they can tailor what their boxes send to make it
>>> easier for you.
>>>
>>>
>>>
>>>> 3. You mentioned, DHCP is again, not a authentication protocol. So what
>>>> handles authentication then if only DHCP is used, and there are no
>>>> usernames and passwords? I guess for DSL networks you can enable or disable
>>>> the port to allow or disallow access, and Option 82 for identification? I
>>>> assume you wouldn't want to shut off the GPON OLT port if one customer
>>>> wasn't paying their bill as it would affect the other customers on that
>>>> port. I assume access vendors allow you to shut down the sub port or ONT in
>>>> this situation for GPON? Still that seems messy having to login to a shelf
>>>> or EMS system or API to an EMS system especially if you have multiple
>>>> access vendors in a network. Is there a way to do authentication with DHCP?
>>>> What about open networks like wifi where anyone can connect, so you don't
>>>> have the ability to turn of the port or disable the end device?
>>>> 4. I don't think anyone is buying a BRAS anymore, but looks like Cisco,
>>>> Juniper, and ALU have what they call BGN, Broadband Subscriber Management,
>>>> and other similar software. How are these different from BRAS functionality?
>>>>
>>>
>>> First, if you can manage it turn on DOCSIS provisioning of your GPON
>>> network.  AFAIK only Calix has announced this functionality, but I expect
>>> the others to follow suit now that there is an official effort at CableLabs
>>> to allow that.
>>>
>>>
>>> http://www.lightreading.com/cable-video/docsis/calix-launches-docsis-provisioning-of-gpon/d/d-id/709859
>>>
>>> The notion of managing ports and profiles via (an ever changing) shelf
>>> API is one of the main reasons that telco billing systems cost so much
>>> compared to cable billing systems.  If you can't swing DPoG then you're
>>> kind of stuck, either you can implement the API your vendor supplies with
>>> your billing system, manage the profile assignment manually (yuck), or just
>>> provision everyone with the same speed (only works for data only
>>> deployments), or go down the older route of putting in a BRAS and making
>>> sure the ONTs you're deploying have a PPPoE client embedded in them.
>>>  Having to deploy an external router for each customer, which I've seen
>>> some operators do, makes your install costs higher and makes
>>> troubleshooting harder.
>>>
>>>
>>>> So it looks like there are open source and commercial solutions for
>>>> DHCP and DNS. Some providers like Infloblox seems to integrate all these
>>>> into one.
>>>>
>>>> So if we have a core router that speaks BGP, a 10G aggregation switch
>>>> to aggregate the the chassis, and a device like Infloblox or the other
>>>> commercial solutions you mentioned that do DHCP/DNS, is there anything else
>>>> that is needed besides the access gear already mentioned in the
>>>> assumptions?  Are these large and expensive commercial BGN/Broadband
>>>> Subscriber management products a thing of the past or still very relevant
>>>> in todays environment?
>>>>
>>>
>>> They're not very relevant, once the the OLT vendors realized they could
>>> snoop the DHCP session and enforce what the server provided the need for
>>> subscriber management pieces really dropped.  You've listed the bare
>>> essentials for a functional network, there are lots of things that are
>>> helpful or useful, but what you have is functional.  Having said all of
>>> that, there are some relatively unobtrusive ways to have some level of
>>> authentication, I just don't think they're very valuable.
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jul 31, 2014 at 8:54 AM, Scott Helms <khelms at zcorum.com> wrote:
>>>>
>>>>> "What is the ideal way to aggregate the 40 10G connections from the
>>>>> uplinks
>>>>> of the chassis? I would guess a 10G switch since 10G ports on a router
>>>>> would be much more expensive?"
>>>>>
>>>>> Definitely aggregate into a switch first unless you want to run a
>>>>> Layer 3 switch as your router, which I don't recommend.
>>>>>
>>>>>
>>>>> "Which router is recommended to handle 4 10G internet connections with
>>>>> full
>>>>> tables, and then at least 4 10G ports going back to the 10G aggregation
>>>>> switch?"
>>>>>
>>>>> Your math is a little backwards, its very unlikely that you're going
>>>>> to have 40 Gbps of Internet (or other interconnection) for the router to
>>>>> actually have to process.  What is the average provisioned speed for each
>>>>> of the 10k PON ports?  What over subscription rate are you planning for?
>>>>>  What, if anything, will you be carrying on net, ie bandwidth consumption
>>>>> that won't come from or go to the public Internet?  Your own video, voice,
>>>>> or other service are examples of things that are often on net.  In any case
>>>>> you're probably in the ASR family with Cisco and I can't remember
>>>>> the equivalent from Juniper.
>>>>>
>>>>>
>>>>> How do you handle IP address management? a /20 is only 4096 IP
>>>>> addresses,
>>>>> but the network would have potentially 10,000 customers. Assume that
>>>>> getting more space from ARIN is not an option. Is CGN an option?
>>>>>
>>>>> CGN is the option of last resort IMO, but you may have to consider it.
>>>>>  A better approach is to see if your backbone providers will agree to give
>>>>> some blocks that you can announce and use those blocks for dynamic
>>>>> customers only.  Your static IP customers should come from your direct ARIN
>>>>> allotment in case you need to choose a new backbone provider, which is
>>>>> extremely common over time.
>>>>>
>>>>>
>>>>> "Dynamic IP
>>>>> addresses? DHCP?"
>>>>>
>>>>> DHCP with enforcement from the shelves.  All the major OLT vendors
>>>>> support doing this so that a customer can only use the address assigned to
>>>>> him by DHCP and nothing else, except for those customers that you choose to
>>>>> hard code.  Make most of your "static" customers actually DHCP reservations
>>>>> and only hard code those that you must.
>>>>>
>>>>> "How do you separate users and traffic? VLANs, Service VLANs, Per
>>>>> Customer
>>>>> VLANs, Usernames? Passwords? PPPoE? MAC Separation?
>>>>> Is a BRAS or BGN functionally really needed or are these older
>>>>> concepts?"
>>>>>
>>>>> DHCP, with Option 82 logging for the circuit ID is the better path
>>>>> than a BRAS (PPPoE) these days.  Here's a paper we put together on that
>>>>> topic a while back:
>>>>>
>>>>>
>>>>> http://www.zcorum.com/wp-content/uploads/Why-Should-I-Move-from-PPPoA-or-PPPoE-to-DHCP.pdf
>>>>>
>>>>> Depending on your OLT vendor you can either use their built in port
>>>>> isolation or QinQ tagging, both are reliable and scalable, just ask your
>>>>> vendor which is the best option for your specific gear.
>>>>>
>>>>>
>>>>>
>>>>> "If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service?
>>>>> The
>>>>> core router, a linux box, or something else?"
>>>>>
>>>>> I wouldn't have those two services connected personally, though there
>>>>> are hooks for some of the CGN boxes to talk to DHCP servers.  I would hope
>>>>> you can get another 6k addresses and avoid the need for CGN altogether.
>>>>>  Having said that, have you tested your OLTs and ONTs for IPv6
>>>>> interoperability?  If they don't handle it well then you're going to have
>>>>> to think about alternatives like 6RD (
>>>>> http://en.wikipedia.org/wiki/IPv6_rapid_deployment)
>>>>>
>>>>> For DHCP at your scale you can run ISC DHCP (
>>>>> http://www.isc.org/downloads/dhcp/) which is the most common open
>>>>> source DHCP daemon if you someone who can take care of a Linux server,
>>>>> parse the Option 82 information for logging, and handle the configuration
>>>>> of the DHCP daemon itself.  Otherwise you might want to look at commercial
>>>>> products designed for the service provider market like Incongito's BCC and
>>>>> Cisco's BAC (CNR replacement)
>>>>>
>>>>> http://www.incognito.com/products/broadband-command-center/
>>>>>
>>>>> http://www.cisco.com/c/en/us/products/cloud-systems-management/broadband-access-center/index.html
>>>>>
>>>>>
>>>>> "What about DNS?
>>>>> Is a firewall needed in the core?
>>>>> What else is needed?"
>>>>>
>>>>> There are two kinds of DNS, caching (recursive) and authoritative.
>>>>>  The first is what your customers will use to resolve things on the
>>>>> Internet and the second is used to provide caching name servers on the
>>>>> Internet with information about domains you control (are authoritative
>>>>> for).  The first needs good performance, availability, and
>>>>> scalability since your customers will use your caching name servers
>>>>> constantly.  Most people can run BIND at your scale, again if you have
>>>>> someone with Linux experience, but there are other alternatives.  PowerDNS
>>>>> has both caching and authoritative modules and there are some commercial
>>>>> offerings out there both as cloud hosting and local deployments.  Your
>>>>> backbone provider will also often have caching name servers your customers
>>>>> can use, but the quality varies quite a bit.  You can also, especially at
>>>>> first, leverage some of the free offerings like Google's DNS.  I
>>>>> don't recommend firewalls for service provider networks, but you should
>>>>> make sure your gear can run (and is configured to do so) BCP 38.
>>>>>
>>>>>
>>>>> Scott Helms
>>>>> Vice President of Technology
>>>>> ZCorum
>>>>> (678) 507-5000
>>>>> --------------------------------
>>>>> http://twitter.com/kscotthelms
>>>>> --------------------------------
>>>>>
>>>>>
>>>>> On Thu, Jul 31, 2014 at 9:23 AM, Colton Conor <colton.conor at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> If a new operator or city is building a greenfield access network
>>>>>> from the
>>>>>> ground up, what software and hardware is needed in the core network to
>>>>>> provide and manage residential and business internet services similar
>>>>>> to
>>>>>> the likes of AT&T, Comcast, and Google Fiber? Television and Telephone
>>>>>> services are not to be considered only internet.
>>>>>>
>>>>>> Assume hypothetically the operator already has the following in place:
>>>>>> 10 GPON OLTs Chassis from an access vendor in 10 POPs around town
>>>>>> (each POP
>>>>>> has 1 Chassis). Each OLT Chassis has 4 10G Uplinks back to the core.
>>>>>> Dark fiber going from the POP locations back to the core location
>>>>>> Assume a 32:1 way split, and each OLT chassis has enough ports
>>>>>> populated to
>>>>>> serve the area.
>>>>>> 10,000 GPON ONTs. The ONTs can be put in routed gateway or bridged
>>>>>> mode.
>>>>>> Assume you are building a network designed to serve 10,000 subs
>>>>>> All the fiber splitters, ducts, fiber, etc connecting the OLTs to the
>>>>>> ONTs
>>>>>> is already in place
>>>>>> ASN from ARIN
>>>>>> /20 of IPv4 space and /32 of IPv6 space from ARIN
>>>>>> 4 burstable 10G internet connections from 4 tier 1 internet providers
>>>>>>
>>>>>> Questions are:
>>>>>>
>>>>>> What is the ideal way to aggregate the 40 10G connections from the
>>>>>> uplinks
>>>>>> of the chassis? I would guess a 10G switch since 10G ports on a router
>>>>>> would be much more expensive?
>>>>>> Which router is recommended to handle 4 10G internet connections with
>>>>>> full
>>>>>> tables, and then at least 4 10G ports going back to the 10G
>>>>>> aggregation
>>>>>> switch?
>>>>>> How do you handle IP address management? a /20 is only 4096 IP
>>>>>> addresses,
>>>>>> but the network would have potentially 10,000 customers. Assume that
>>>>>> getting more space from ARIN is not an option. Is CGN an option?
>>>>>> Dynamic IP
>>>>>> addresses? DHCP?
>>>>>> How do you separate users and traffic? VLANs, Service VLANs, Per
>>>>>> Customer
>>>>>> VLANs, Usernames? Passwords? PPPoE? MAC Separation?
>>>>>> Is a BRAS or BGN functionally really needed or are these older
>>>>>> concepts?
>>>>>> If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service?
>>>>>> The
>>>>>> core router, a linux box, or something else?
>>>>>> What about DNS?
>>>>>> Is a firewall needed in the core?
>>>>>> What else is needed?
>>>>>>
>>>>>> Is there a guide out there somewhere? I know many cities are looking
>>>>>> at
>>>>>> building their own network, and have similar questions. Access
>>>>>> vendors are
>>>>>> willing to sell gear all day long, but then they leave it up to the
>>>>>> operator/city to answer these harder questions.
>>>>>>
>>>>>> How would you build a access network from the ground up if you had the
>>>>>> resources and time to do so? Would you even use GPON? Even if GPON
>>>>>> was not
>>>>>> used and another access technology like AE, VDSL2, or wireless was
>>>>>> used I
>>>>>> think many of these questions would be the same.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>