Broadband Router Comparisons
mike-nanog at tiedyenetworks.com
Sun Dec 27 05:49:43 UTC 2015
On 12/23/2015 06:49 PM, Lorell Hathcock wrote:
> Not all consumer grade customer premises equipment is created equally. But end customers sure think it is. I have retirement aged customers buying the crappiest routers and then blaming my cable network for all their connection woes. The real problem is that there were plenty of problems on the cable network to deal with, so it was impossible to tell between a problem that a customer was having with their CPE versus a real problem in my network.
> Much of that has been cleared up on my side now, but customers were used to blaming us for everything so that they don't even consider that their equipment could be to blame.
> I want to be able to point out a third party list of all (most) broadband routers that rates them by performance. Or that rates them by crappiness that I can send them to so they can look up their own router and determine if other users have had problems with that router and what can be done to fix it.
> So far my search has been in vain.
> Any thoughts?
As a service provider with largely residential/small business customers,
I certainly have some thoughts on broadband routers. Sorry if this is
Firstly, they are all junk. Every last one of them. Period. Broadband
routers are designed to be cheap and to appeal to people who don't know
any better, and who respond well (eg: make purchasing decisions) based
on the shape of the plastic, the color scheme employed, and number of
mysterious blinking lights that convey 'something important is
happening'. Further, the price point is $45 - $70 thereabouts, putting
some definite constraints on the actual quality of the engineering and
components that go into them. I feel that we, the service provider,
endure a significantly high and undue burden of cost associated with
providing ongoing support to customers as a result of the defects
The laundry list of general operational issues for broadband routers,
the ones that seem to be universal to every last one of them, goes
something like this:
* Device lock ups
* Lost Settings
* Abysmal device security
* Inconsistent forwarding performance
I will try to describe these:
Device lock up is by far the most damming problem there is. The lights
are on, the cables are plugged in, but you aren't going anywhere
therefore the Internet must be down. This condition typically can be
resolved by powercycling the device, and whaever problem it was
encountering is magically remedied and all is well again. The concept of
the device developing 'a problem' that can only be resolved by power
cycling it, is foreign and completely blows end users minds. And yet, it
is very common, and leaves end users stranded since they don't have even
the most basic of troubleshooting abilities. We have had people who wait
days or even a week or two before calling in to ask for support, because
they think the problem will fix itself or that we the provider are
simply down (and, in their eyes, we're frequently down anyways and this
is just routine...) and so it's out of their hands.
We've noted that there are waves of device lockups that occur nearly
every time the weather turns, which I attribute to brownouts and other
variations in the power grid which occur at these times and when coming
into the office after a stormy weekend we know to expect our phones to
be lit up all day with enormous numbers of people all screaming about
being 'down the whole weekend!' and every last one of them being able
restore themselves via powercycling. We try to counsel these customers
and educate them that 'power cycling' is always a good "first responder"
step to try, and secondly, that they always should employ a good quality
standby UPS in order to avoid these types of issues in the future, but
they never listen and blame us anyways. Broadband routers are not
designed with quality robust power supplies, which certainly lowers the
costs, but contributes substantially to this problem. This particular
issue, I think, is one of the greatest deficiencies shared by all.
Other times, 'lockup' simply resolves to router software problems, such
as a kernel panic, a crashed or bugged system process such as
pppoe/pppd or dhcp, an overfull nat state table, memory leaks, or other
purely software related troubles. The recovery procedure is the same,
eg: power cycle the device, but as before, it doesn't actually "fix" the
underlaying problem (bugged software), it merely alleviates the current
symptom...until next time later when it happens again. Many of these
troubles are simply outstanding bugs in the versions of the opensource
code that the SDK is built on, which never seems to get updated and
instead just uses the same old buggy code. Some custom kits also have
just crap buggy protocol implementations that also just never get fixed.
And usually, (although this is improving), many of these cheap devices
never have updated firmware available for them. 3 months after purchase
the product is discontinued and it's on to the next newest thing so if
you got bugs, tuff cookies. But even for those devices where firmware
updates are made available, you would be hard pressed to find any end
user which regularly reviews and applies same.
I should point out that an exception to the above are the dd-wrt and
variant firmwares which will work on a subset of cpe devices. Generally
dd-wrt is maintained much better and usually far superior to stock
manufacturer firmware. A downside however is that it may not have that
hot new wireless capability for your particular device or only support
wireless in a generic way. It also doesn't support any adsl or vdsl
modems that I know of, which precludes it from being able to be used in
an integrated modem/router combo, forcing you still to have your cpe in
bridge mode (and hope at least bridge mode can work well enough for
you), and a second device at additional expense to be your router /
wireless access point.
Lost settings is another very common symptom. One minute everything is
great and fine, but then the next time you go to use the service... your
wireless network name can't be found (or has been replaced by the
ubiquitous ssid 'linksys'), and even if you can connect to your router,
you still can't get on... only 20 minutes later when you are on the
phone you are told that your device no longer appears to be configured
for pppoe as it has a blank username / password credential now. And
sometimes worse, the factory default ip range is different than what you
use and so now the router is handing out foreign dhcp addresses but your
printer with it's static IP is now on a different subnet and you can't
print. This problem is even more devastating because it requires
black-arts magic to correct; !!! Shudder !! YOU HAVE TO CONFIGURE IT AGAIN!
I have observed there seems to be a strong connection between
brownouts/blackouts and lost settings (or, more accurately, reset to
factory defaults). I suspect that the issue is flash memory corruption
and the device firmware deciding it needs to format the flash (perhaps a
reasonable assumption). We combat this at least on some of our dsl
modem/routers by making the 'customer settings' the 'factory default'
settings, which is stored in another bank of flash. But still it happens
to other devices with frequent regularity. FURTHERMORE, some customers
(a majority it seems), seems to be under the impression that 'if it
don't work, reset it!', which means to use the paperclip in the special
recessed hole. Usually these people are suffering from the first problem
above, lockup, but they engage this factory default restore procedure
not knowing they have just compounded their problem and ensured that it
won't work now that it's no longer setup appropriately. We also try to
ensure that every cpe that leaves our office has a red dot sticker over
the hole to discourage this behavior. Sometimes it helps, sometimes the
customer swears up and down they didn't touch it but when it's presented
we see the tell-tell hole thru the sticker (or remnants of removed sticker).
The security of broadband routers is absolutely abysmal and there have
been many documented cases now of customer home dsl modems having all
sorts of issues, Secret remote root login exploits, default factory
passwords. exposed internet facing management interfaces that in the web
ui are 'turned off' but still reachable anyways, exploitable deamons
such as dns, ntp and ssdp that are participants in DDoS attacks, and
more. We have had direct experience with a particular malware that knew
(when we didn't!) the default manufacturing passwords to our customer
CPE and would change the dns settings of the device so that the resolver
IP's handed out would be ones under the control of the bad guys, to
support phishing attacks and other goals. Recovering from this was
painful but a very good lesson - on my network, I now (per user), filter
a list of inbound ports in order to secure by default these devices by
denying Internet access to the CPEs themselves. I haven't had any
complaints or requests for the filtering to be removed and I can clearly
see it's a win. Still however, these kinds of games shouldn't be necessary.
Lastly, the forwarding performance of these devices is wildly
inconsistent. Some devices slow down the more nat connections they are
tracking (and keeping old closed connection info in their
tables...blarf!), sometimes other bugs create situations in which
pinging the upstream gateway thur the router takes thousands of ms (and
that number immediately drops back to normal upon, you guessed it...a
power cycle!), sometimes buffer bloat is a factor.
As I indicated earlier, there is dd-wrt (and other router firmware
replacements) which are available and which will address some of these
issues if you need to use consumer hardware. However, some other choices
do include Mikrotik as well as aftermarket Cisco such as the 2600
series. But there needs to be a lot more development in this area. The
google onhub looks to have a great hardware design as far as it's radio
array goes, but it lacks basic features found in low end linksys
routers. Im sure it will catch up but for today it's really at 'gee
heres what we can do' stage and not really a full featured broadband
router device in the current sense of the term.
I apologize for length.
More information about the NANOG