PoE, Comcast Modems, and Service Outages
jgreco at ns.sol.net
Tue Mar 29 20:24:08 UTC 2022
On Tue, Mar 29, 2022 at 11:21:28AM -0700, Aaron de Bruyn via NANOG wrote:
> I just got off the phone with a Comcast tech, and wanted to double-check my
> Somehow in the last 6 months I've managed to reach the exact same rep twice
> when dealing with an outage or a degraded service event.
> I asked him to remotely reboot the modem because there was high packet loss.
> Both times I've talked with him, he noted the high packet loss, started to
> reboot the modem, and then asked me point-blank if we had any PoE switches
> on our network.
> When I said "yes", he said I needed to disable PoE because it messes with
> the Comcast modems and he can see "buildups" in his graphs that show power
> is "leaking" to the Comcast modem every 24 hours.
> For reference, our setup is:
> Internal Network ?????? PoE Switch ?????? My Router (FreeBSD Box) ?????? Comcast Modem
> I told him the Comcast modem isn't plugged into the PoE Switch, it's
> plugged into My Router (FreeBSD box) and My Router does not negotiate PoE+
> and the switch shows PoE isn't being send to My Router's LAN port. While
> the switch is capable of outputting old-school 24v PoE, it must be
> specifically turned on for a port, and it's not enabled or used anywhere on
> the networks I manage.
> When provided with that information, the Comcast tech still insisted that
> the switch was sending PoE to My Router and it was "leaking through" to the
> Comcast modem and that's why every 4-6 weeks the Comcast modem needs to be
> reset. The tech insisted that switches that *are* PoE-capable *always* send
> PoE even if the device doesn't request it or negotiate it. Attempts to
> explain the difference between the old 24-volt PoE and PoE+/++ were met
> with arguing that he's been in the industry for decades and I don't know
> what I'm talking about...and that all my problems would go away if I just
> disabled PoE everywhere on the switch.
> Again, I double-checked the port and said "It's not sending PoE to my
> router, but even if I were, I highly doubt PoE would leak through a PCI
> card to the opposite side of the chassis to the on-board NIC and out to
> your modem".
> He insisted it happened "all the time" and he had previously fried
> equipment by plugging it into a PoE switch. He insisted that he's also
> handled quite a few calls relating to this magic PoE problem over the years
> and Comcast has internal tools that show graphs of how much PoE power
> "builds up" inside their modems and he "can see a buildup in my router that
> resets every 24 hours".
> I didn't have the heart to tell him that I manage about 40 networks that
> have Comcast connections...and they *all* have identical FreeBSD boxes
> acting as their router, and they are *all* using the exact same PoE
> switches at every location with all ports set to PoE+...and we only have
> degraded service or outages after ~30 days at ~3 locations.
> Slightly off-topic, but if I call Comcast about outages or degraded service
> and any *other* tech but this guy answers, they all say "you need to unplug
> your Comcast modem and plug it back in once every 3-4 weeks" and they act
> like it's normal to reboot the modems every few weeks. In fact, last week I
> wanted Comcast to check on a modem setting at one location and they said
> the modem had been up for over 127 days and it should be rebooted. I said
> "it's up and working fine, why would I reboot it?".
> Anyways, am I insane for thinking the tech was flat-out wrong? I
> mean...occasionally some really bizarre stuff happens in IT...but this
> seems extremely far-fetched and contrary to everything I know about the PoE
That's ridiculous, as you already know.
Their crappy equipment needing rebooting every few weeks, not ridiculous.
Their purchasing gear from incompetent vendors who cannot be standards
compliant for PoE PD negotiation, tragically plausible.
Ethernet electronic differential signalling not being handled properly
with respect to grounding or other issues, not unheard-of.
Ghosts of PoE floating around a network through other devices, causing
weird problems on the far side of properly installed and standards compliant
gear, ah, super-unlikely, I'll go so far as to say nah, but in this audience
I am positive someone has a counterexample SOMEWHERE, cuz, you know, the
world's a strange place and there's broken stuff out there.
He's got graphs showing it every 24 hours? Liar, liar, pants on fire,
lazy SOB is looking for an excuse to clear you off the line. Where the
heck does this "24 hour" cycle even come from? What SNMP OID is there
for "ghostly PoE build-up"? What crontab is there that would clear out
such buildups in the router's daily run? What capacitor would store up
juice for precisely 24 hours? What's the mechanism here? CURIOUS MINDS
WANT TO KNOW!
Been doing PoE everywhere for years and this is the stupidest thing I've
heard this year so far in the networking category.
Next time, play dumb. People who can, do. People who can't, tech support.
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"The strain of anti-intellectualism has been a constant thread winding its way
through our political and cultural life, nurtured by the false notion that
democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov
More information about the NANOG