Rogers Outage Canada

Mon Jul 11 14:09:27 UTC 2022

What in depth analysis have you seen? Seems to me, this was a failure in a
known maintenance activity, and they simply disconnected the devices under
maintenance from the network.

Shane

On Mon, Jul 11, 2022 at 5:41 AM Jon Sands <fohdeesha at gmail.com> wrote:

> Given the outage was so bad it was disrupting select E911 services
> nationwide for something like 24+ hours, it's great to see such an in depth
> analysis and plan of action to prevent such things in the future. bravo
> rogers
>
> On 7/10/2022 2:55 PM, L F wrote:
>
> fyi - see BOLD.
>
> A Message from Rogers President and CEO
>
>
> Dear Valued Customer,
>
> As you know, we experienced a service outage across the Rogers, Fido,
> Chatr and Cityfone wireless networks on Friday.
>
> I am reaching out to share that our services have been restored, and our
> networks and systems are close to fully operational. Our technical teams
> are continuing to monitor for any remaining intermittent issues. I also
> want to outline an action plan we are putting in place to address what
> happened.
>
> I want to share what we know about what happened on Friday. We now believe *we’ve
> narrowed the cause to a network system failure following a maintenance
> update in our core network, which caused some of our routers to
> malfunction. We disconnected the specific equipment and redirected traffic,*
> which allowed our network and services to come back online over time as we
> managed traffic volumes returning to normal levels.
>
>
>
> On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas at gmail.com> wrote:
>
>>
>> Lest we ALL MOVE ON….
>>
>>
>> Yes he said
>>
>> “RETARD” =
>>
>> The delay in processing the current status of a Situation…
>>
>>
>> Lets move on to bigger n scarier Post mortems:
>>
>>
>> Was This or Was this NOT
>>
>> A CYBERATTACK ???
>>
>> Done.
>>
>> Lets deal with REAL threats not PC references.
>>
>> Time to evaluate n re eval Our NTS!!
>>
>> Veni vidi vici 2022
>>
>>
>>
>>
>> On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke at gmail.com> wrote:
>>
>>> Can we have a discussion with the list admins about a list member
>>> appending a threat of violence to their outbound emails?  Whether serious
>>> or not.
>>>
>>> Does this person need directions to some local mental healthcare
>>> resources?
>>>
>>>
>>> On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf at dessus.com> wrote:
>>>
>>>>
>>>> >I can't either, but the reality right now seems to be that 911 calls
>>>> are
>>>> >failing for anyone on a Rogers cellphone.
>>>>
>>>> This is par for the course.  These people chose to deal with Rogers
>>>> despite knowing the consequences.  It is like if you bought a Rogers
>>>> Snowblower and it did not work.  That would mean that people who bought the
>>>> Rogers Snowblower will not be using it to get rid of the snow that is
>>>> preventing them from leaving their house.
>>>>
>>>> Mutatis mutandis when Rogers is down things that are Rogers dependent
>>>> will not work.
>>>>
>>>> Some people are so retarded it is astonishing!
>>>>
>>>> --
>>>> (CAUTION) You are advised that if you attack my person or property, you
>>>> will be put down in accordance with the provisions of section 34 & 35 of
>>>> the Criminal Code respectively.  If you are brandishing (or in possession)
>>>> of a weapon then lethal force will be applied to your person in accordance
>>>> with the law.  This means that your misadventures may end in your death.
>>>> Consider yourself cautioned and govern your actions appropriately.
>>>>
>>>> >-----Original Message-----
>>>> >From: NANOG <nanog-bounces+kmedcalf=dessus.com at nanog.org> On Behalf Of
>>>> >Eric Kuhnke
>>>> >Sent: Friday, 8 July, 2022 13:34
>>>> >To: jim deleskie <deleskie at gmail.com>
>>>> >Cc: NANOG list <nanog at nanog.org>
>>>> >Subject: Re: Rogers Outage Canada
>>>> >
>>>> >
>>>> >I have seen anecdotal reports that the mobile network is in a half
>>>> broken
>>>> >state that phones remain registered to, so a 911 call will attempt and
>>>> >then fail.
>>>> >
>>>> >
>>>> >This is unlike what would happen if you had a US/Canada cellphone with
>>>> >battery power but no SIM card in it that would search for any available
>>>> >network in RF range for a 911 call if needed.
>>>> >
>>>> >
>>>> >On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie at gmail.com
>>>> ><mailto:deleskie at gmail.com> > wrote:
>>>> >
>>>> >
>>>> >       i cant see BGP taking out SS7.
>>>> >
>>>> >       -jim
>>>> >
>>>> >       On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004
>>>> ><greenjosh6499 at gmail.com <mailto:greenjosh6499 at gmail.com> > wrote:
>>>> >
>>>> >
>>>> >               According to Cloudflare Radar
>>>> ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> ,
>>>> Rogers
>>>> >BGP announcements spiked massively to levels 536,777% higher than
>>>> normal
>>>> >(343,601 vs 64 normally) just minutes before the outage. I would not be
>>>> >surprised if this happened to be the culprit.
>>>> >
>>>> >               Regards,
>>>> >               Josh Green
>>>> >
>>>> >               On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG
>>>> ><nanog at nanog.org <mailto:nanog at nanog.org> > wrote:
>>>> >
>>>> >
>>>> >                       In the early hours of the morning around 2-3am
>>>> my modem
>>>> >got hit with a configuration update that caused a DHCP release that
>>>> >wasn't renewed for about two hours, after rollback the connection was
>>>> >fine for 3 hours before this network wide outage.
>>>> >
>>>> >
>>>> >                       Maybe a failed night time update was attempted
>>>> again
>>>> >during office hours, I've heard daytime guys are still WFH and night
>>>> >shift is in building.
>>>> >
>>>> >
>>>> >                       I expect we'll never get a real explanation.
>>>> Rogers is
>>>> >notorious for withholding any type of helpful or technical information.
>>>> >
>>>> >
>>>> >                       Sent from my inoperable Rogers Mobile via
>>>> emergency eSIM.
>>>> >
>>>> >
>>>> >                       Regards,
>>>> >
>>>> >                       Andrew Paolucci
>>>> >                       -------- Original Message --------
>>>> >                       On Jul. 8, 2022, 1:48 p.m., Jay Hennigan <
>>>> jay at west.net
>>>> ><mailto:jay at west.net> > wrote:
>>>> >
>>>> >
>>>> >                               On 7/8/22 07:44, Robert DeVita wrote: >
>>>> Does anyone
>>>> >have information on a widespread Rogers outage in Canada. I > have
>>>> >customers with multiple sites down. There's discussion on the Outages
>>>> >mailing list. Seems widespread, affecting all services, mobile, voice,
>>>> >Internet. No cause or ETR posted yet. -- Jay Hennigan - jay at west.net
>>>> ><mailto:jay at west.net>  Network Engineering - CCIE #7880 503 897-8550 -
>>>> >WB6RDV
>>>> >
>>>> >
>>>> >
>>>> >               --
>>>> >
>>>> >               Josh Green.
>>>>
>>>>
>>>>
>>>>
>
>
> --
> Jon Sands
> MFI Labshttps://fohdeesha.com/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20220711/cbc7870b/attachment.html>