Broadband Router Comparisons

Mike mike-nanog at tiedyenetworks.com
Sun Dec 27 05:49:43 UTC 2015


On 12/23/2015 06:49 PM, Lorell Hathcock wrote:
> All:
>
> Not all consumer grade customer premises equipment is created equally.  But end customers sure think it is.  I have retirement aged customers buying the crappiest routers and then blaming my cable network for all their connection woes.  The real problem is that there were plenty of problems on the cable network to deal with, so it was impossible to tell between a problem that a customer was having with their CPE versus a real problem in my network.
>
> Much of that has been cleared up on my side now, but customers were used to blaming us for everything so that they don't even consider that their equipment could be to blame.
>
> I want to be able to point out a third party list of all (most) broadband routers that rates them by performance.  Or that rates them by crappiness that I can send them to so they can look up their own router and determine if other users have had problems with that router and what can be done to fix it.
>
> So far my search has been in vain.
>
> Any thoughts?
>
>

As a service provider with largely residential/small business customers, 
I certainly have some thoughts on broadband routers. Sorry if this is 
overly long.

Firstly, they are all junk. Every last one of them. Period. Broadband 
routers are designed to be cheap and to appeal to people who don't know 
any better, and who respond well (eg: make purchasing decisions) based 
on the shape of the plastic, the color scheme employed, and number of 
mysterious blinking lights that convey 'something important is 
happening'. Further, the price point is $45 - $70 thereabouts, putting 
some definite constraints on the actual quality of the engineering and 
components that go into them. I feel that we, the service provider, 
endure a significantly high and undue burden of cost associated with 
providing ongoing support to customers as a result of the defects 
contained therein.

The laundry list of general operational issues for broadband routers, 
the ones that seem to be universal to every last one of them, goes 
something like this:

     * Device lock ups
     * Lost Settings
     * Abysmal device security
     * Inconsistent forwarding performance

     I will try to describe these:

Device lock up is by far the most damming problem there is. The lights 
are on, the cables are plugged in, but you aren't going anywhere 
therefore the Internet must be down. This condition typically can be 
resolved by powercycling the device, and whaever problem it was 
encountering is magically remedied and all is well again. The concept of 
the device developing 'a problem' that can only be resolved by power 
cycling it, is foreign and completely blows end users minds. And yet, it 
is very common, and leaves end users stranded since they don't have even 
the most basic of troubleshooting abilities. We have had people who wait 
days or even a week or two before calling in to ask for support, because 
they think the problem will fix itself or that we the provider are 
simply down (and, in their eyes, we're frequently down anyways and this 
is just routine...) and so it's out of their hands.

We've noted that there are waves of device lockups that occur nearly 
every time the weather turns, which I attribute to brownouts and other 
variations in the power grid which occur at these times and when coming 
into the office after a stormy weekend we know to expect our phones to 
be lit up all day with enormous numbers of people all screaming about 
being 'down the whole weekend!' and every last one of them being able 
restore themselves via powercycling. We try to counsel these customers 
and educate them that 'power cycling' is always a good "first responder" 
step to try, and secondly, that they always should employ a good quality 
standby UPS in order to avoid these types of issues in the future, but 
they never listen and blame us anyways. Broadband routers are not 
designed with quality robust power supplies, which certainly lowers the 
costs, but contributes substantially to this problem. This particular 
issue, I think, is one of the greatest deficiencies shared by all.

Other times, 'lockup' simply resolves to router software problems, such 
as  a kernel panic, a crashed or bugged system process such as 
pppoe/pppd or dhcp, an overfull nat state table, memory leaks, or other 
purely software related troubles. The recovery procedure is the same, 
eg: power cycle the device, but as before, it doesn't actually "fix" the 
underlaying problem (bugged software), it merely alleviates the current 
symptom...until next time later when it happens again. Many of these 
troubles are simply outstanding bugs in the versions of the opensource 
code that the SDK is built on, which never seems to get updated and 
instead just uses the same old buggy code. Some custom kits also have 
just crap buggy protocol implementations that also just never get fixed. 
And usually, (although this is improving), many of these cheap devices 
never have updated firmware available for them. 3 months after purchase 
the product is discontinued and it's on to the next newest thing so if 
you got bugs, tuff cookies. But even for those devices where firmware 
updates are made available, you would be hard pressed to find any end 
user which regularly reviews and applies same.

I should point out that an exception to the above are the dd-wrt and 
variant firmwares which will work on a subset of cpe devices. Generally 
dd-wrt is maintained much better and usually far superior to stock 
manufacturer firmware. A downside however is that it may not have that 
hot new wireless capability for your particular device or only support 
wireless in a generic way. It also doesn't support any adsl or vdsl 
modems that I know of, which precludes it from being able to be used in 
an integrated modem/router combo, forcing you still to have your cpe in 
bridge mode (and hope at least bridge mode can work well enough for 
you), and a second device at additional expense to be your router / 
wireless access point.

Lost settings is another very common symptom. One minute everything is 
great and fine, but then the next time you go to use the service... your 
wireless network name can't be found (or has been replaced by the 
ubiquitous ssid 'linksys'), and even if you can connect to your router, 
you still can't get on... only 20 minutes later when you are on the 
phone you are told that your device no longer appears to be configured 
for pppoe as it has a blank username / password credential now. And 
sometimes worse, the factory default ip range is different than what you 
use and so now the router is handing out foreign dhcp addresses but your 
printer with it's static IP is now on a different subnet and you can't 
print. This problem is even more devastating because it requires 
black-arts magic to correct; !!! Shudder !! YOU HAVE TO CONFIGURE IT AGAIN!

I have observed there seems to be a strong connection between 
brownouts/blackouts and lost settings (or, more accurately, reset to 
factory defaults). I suspect that the issue is flash memory corruption 
and the device firmware deciding it needs to format the flash (perhaps a 
reasonable assumption). We combat this at least on some of our dsl 
modem/routers by making the 'customer settings' the 'factory default' 
settings, which is stored in another bank of flash. But still it happens 
to other devices with frequent regularity. FURTHERMORE, some customers 
(a majority it seems), seems to be under the impression that 'if it 
don't work, reset it!', which means to use the paperclip in the special 
recessed hole. Usually these people are suffering from the first problem 
above, lockup, but they engage this factory default restore procedure 
not knowing they have just compounded their problem and ensured that it 
won't work now that it's no longer setup appropriately. We also try to 
ensure that every cpe that leaves our office has a red dot sticker over 
the hole to discourage this behavior. Sometimes it helps, sometimes the 
customer swears up and down they didn't touch it but when it's presented 
we see the tell-tell hole thru the sticker (or remnants of removed sticker).

The security of broadband routers is absolutely abysmal and there have 
been many documented cases now of customer home dsl modems having all 
sorts of issues, Secret remote root login exploits, default factory 
passwords. exposed internet facing management interfaces that in the web 
ui are 'turned off' but still reachable anyways, exploitable deamons 
such as dns, ntp and ssdp that are participants in DDoS attacks, and 
more. We have had direct experience with a particular malware that knew 
(when we didn't!) the default manufacturing passwords to our customer 
CPE and would change the dns settings of the device so that the resolver 
IP's handed out would be ones under the control of the bad guys, to 
support phishing attacks and other goals. Recovering from this was 
painful but a very good lesson - on my network, I now (per user), filter 
a list of inbound ports in order to secure by default these devices by 
denying Internet access to the CPEs themselves. I haven't had any 
complaints or requests for the filtering to be removed and I can clearly 
see it's a win. Still however, these kinds of games shouldn't be necessary.

Lastly, the forwarding performance of these devices is wildly 
inconsistent. Some devices slow down the more nat connections they are 
tracking (and keeping old closed connection info in their 
tables...blarf!), sometimes other bugs create situations in which 
pinging the upstream gateway thur the router takes thousands of ms (and 
that number immediately drops back to normal upon, you guessed it...a 
power cycle!), sometimes buffer bloat is a factor.

As I indicated earlier, there is dd-wrt (and other router firmware 
replacements) which are available and which will address some of these 
issues if you need to use consumer hardware. However, some other choices 
do include Mikrotik as well as aftermarket Cisco such as the 2600 
series. But there needs to be a lot more development in this area. The 
google onhub looks to have a great hardware design as far as it's radio 
array goes, but it lacks basic features found in low end linksys 
routers. Im sure it will catch up but for today it's really at 'gee 
heres what we can do' stage and not really a full featured broadband 
router device in the current sense of the term.

I apologize for length.

Mike-



More information about the NANOG mailing list