<div dir="ltr"><div dir="ltr">This is the most they can and will say. For liabilities reasons, specifics are likely not in the cards. As most services ride over common service networks, its quite possible that a network substrate failure can have a number of upstream service impacts. The point here is that the CEO is directly addressing the customer base, which is needed here.</div><div dir="ltr"><div><br></div><div>regards,</div><div><br></div><div>Victor K</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 11, 2022 at 10:11 AM Shane Ronan <<a href="mailto:shane@ronan-online.com">shane@ronan-online.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">What in depth analysis have you seen? Seems to me, this was a failure in a known maintenance activity, and they simply disconnected the devices under maintenance from the network. <div><br></div><div>Shane</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 11, 2022 at 5:41 AM Jon Sands <<a href="mailto:fohdeesha@gmail.com" target="_blank">fohdeesha@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Given the outage was so bad it was
disrupting select E911 services nationwide for something like 24+
hours, it's great to see such an in depth analysis and plan of
action to prevent such things in the future. bravo rogers<br>
</div>
<div><br>
</div>
<div>On 7/10/2022 2:55 PM, L F wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">fyi - see BOLD.</div>
<div dir="ltr">
<table role="presentation" style="margin:0px auto;padding:0px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:inherit;vertical-align:baseline;border-spacing:0px;width:460px" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">
<tr style="margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">
<td style="font:inherit;padding:0px;border:0px;vertical-align:baseline" align="left">
<p>A
Message from Rogers President and CEO</p>
</td>
</tr>
<tr style="margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">
<td style="font-family:inherit;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:1px;line-height:1px;vertical-align:baseline;height:30px" height="30"><br>
</td>
</tr>
<tr style="margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">
<td style="font-family:inherit;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:1px;line-height:1px;vertical-align:baseline;height:12px" height="12"><br>
</td>
</tr>
<tr style="margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">
<td style="font-style:inherit;font-variant:inherit;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:inherit;padding:0px;border:0px;vertical-align:baseline" align="left">
<p><span style="font-weight:inherit">Dear Valued Customer,</span><br>
<br>
<span style="font-weight:inherit">As you know, we
experienced a service outage across the Rogers,
Fido, Chatr and Cityfone wireless networks on
Friday.</span><br>
<br>
<span style="font-weight:inherit">I am reaching out
to share that our services have been restored, and
our networks and systems are close to fully
operational. Our technical teams are continuing to
monitor for any remaining intermittent issues. I
also want to outline an action plan we are putting
in place to address what happened.</span><br>
<br>
<span style="font-weight:inherit">I want to share
what we know about what happened on Friday. We now
believe </span><b>we’ve narrowed the cause to a
network system failure following a maintenance
update in our core network, which caused some of
our routers to malfunction. We disconnected the
specific equipment and redirected traffic,</b><span style="font-weight:inherit"> which allowed our
network and services to come back online over time
as we managed traffic volumes returning to normal
levels.</span><br>
</p>
</td>
</tr>
</tbody>
</table>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, Jul 9, 2022 at 9:09
PM L F <<a href="mailto:liz.fazekas@gmail.com" target="_blank">liz.fazekas@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><br>
</div>
<div dir="auto">Lest we ALL MOVE ON….</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Yes he said</div>
<div dir="auto"><br>
</div>
<div dir="auto">“RETARD” = </div>
<div dir="auto"><br>
</div>
<div dir="auto">The delay in processing the current status
of a Situation…</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Lets move on to bigger n scarier Post
mortems:</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Was This or Was this NOT</div>
<div dir="auto"><br>
</div>
<div dir="auto">A CYBERATTACK ??? </div>
<div dir="auto"><br>
</div>
<div dir="auto">Done.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Lets deal with REAL threats not PC
references.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Time to evaluate n re eval Our NTS!!</div>
<div dir="auto"><br>
</div>
<div dir="auto">Veni vidi vici 2022</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote" dir="auto">
<div dir="ltr" class="gmail_attr">On Sat, Jul 9, 2022 at
5:01 PM Eric Kuhnke <<a href="mailto:eric.kuhnke@gmail.com" target="_blank">eric.kuhnke@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div>Can we have a discussion with the list admins
about a list member appending a threat of
violence to their outbound emails? Whether
serious or not. <br>
</div>
<div><br>
</div>
<div>Does this person need directions to some
local mental healthcare resources?</div>
<div><br>
</div>
</div>
</div>
<div dir="ltr"><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, 9 Jul
2022 at 08:48, Keith Medcalf <<a href="mailto:kmedcalf@dessus.com" target="_blank">kmedcalf@dessus.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
>I can't either, but the reality right now
seems to be that 911 calls are<br>
>failing for anyone on a Rogers cellphone.<br>
<br>
This is par for the course. These people chose
to deal with Rogers despite knowing the
consequences. It is like if you bought a Rogers
Snowblower and it did not work. That would mean
that people who bought the Rogers Snowblower
will not be using it to get rid of the snow that
is preventing them from leaving their house.<br>
<br>
Mutatis mutandis when Rogers is down things that
are Rogers dependent will not work.<br>
<br>
Some people are so retarded it is astonishing!<br>
<br>
-- <br>
(CAUTION) You are advised that if you attack my
person or property, you will be put down in
accordance with the provisions of section 34
& 35 of the Criminal Code respectively. If
you are brandishing (or in possession) of a
weapon then lethal force will be applied to your
person in accordance with the law. This means
that your misadventures may end in your death.
Consider yourself cautioned and govern your
actions appropriately.<br>
<br>
>-----Original Message-----<br>
>From: NANOG <nanog-bounces+kmedcalf=<a href="mailto:dessus.com@nanog.org" target="_blank">dessus.com@nanog.org</a>>
On Behalf Of<br>
>Eric Kuhnke<br>
>Sent: Friday, 8 July, 2022 13:34<br>
>To: jim deleskie <<a href="mailto:deleskie@gmail.com" target="_blank">deleskie@gmail.com</a>><br>
>Cc: NANOG list <<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>><br>
>Subject: Re: Rogers Outage Canada<br>
><br>
><br>
>I have seen anecdotal reports that the
mobile network is in a half broken<br>
>state that phones remain registered to, so a
911 call will attempt and<br>
>then fail.<br>
><br>
><br>
>This is unlike what would happen if you had
a US/Canada cellphone with<br>
>battery power but no SIM card in it that
would search for any available<br>
>network in RF range for a 911 call if
needed.<br>
><br>
><br>
>On Fri, 8 Jul 2022 at 12:31, jim deleskie
<<a href="mailto:deleskie@gmail.com" target="_blank">deleskie@gmail.com</a><br>
><mailto:<a href="mailto:deleskie@gmail.com" target="_blank">deleskie@gmail.com</a>>
> wrote:<br>
><br>
><br>
> i cant see BGP taking out SS7.<br>
><br>
> -jim<br>
><br>
> On Fri, Jul 8, 2022 at 2:45 PM
Snowmobile2004<br>
><<a href="mailto:greenjosh6499@gmail.com" target="_blank">greenjosh6499@gmail.com</a>
<mailto:<a href="mailto:greenjosh6499@gmail.com" target="_blank">greenjosh6499@gmail.com</a>>
> wrote:<br>
><br>
><br>
> According to Cloudflare Radar<br>
><<a href="https://radar.cloudflare.com/asn/812?date_filter=last_24_hours" rel="noreferrer" target="_blank">https://radar.cloudflare.com/asn/812?date_filter=last_24_hours</a>>
, Rogers<br>
>BGP announcements spiked massively to levels
536,777% higher than normal<br>
>(343,601 vs 64 normally) just minutes before
the outage. I would not be<br>
>surprised if this happened to be the
culprit.<br>
><br>
> Regards,<br>
> Josh Green<br>
><br>
> On Fri, Jul 8, 2022 at 2:19
PM Andrew Paolucci via NANOG<br>
><<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>
<mailto:<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>>
> wrote:<br>
><br>
><br>
> In the early hours of
the morning around 2-3am my modem<br>
>got hit with a configuration update that
caused a DHCP release that<br>
>wasn't renewed for about two hours, after
rollback the connection was<br>
>fine for 3 hours before this network wide
outage.<br>
><br>
><br>
> Maybe a failed night
time update was attempted again<br>
>during office hours, I've heard daytime guys
are still WFH and night<br>
>shift is in building.<br>
><br>
><br>
> I expect we'll never
get a real explanation. Rogers is<br>
>notorious for withholding any type of
helpful or technical information.<br>
><br>
><br>
> Sent from my
inoperable Rogers Mobile via emergency eSIM.<br>
><br>
><br>
> Regards,<br>
><br>
> Andrew Paolucci<br>
> -------- Original
Message --------<br>
> On Jul. 8, 2022, 1:48
p.m., Jay Hennigan < <a href="mailto:jay@west.net" target="_blank">jay@west.net</a><br>
><mailto:<a href="mailto:jay@west.net" target="_blank">jay@west.net</a>>
> wrote:<br>
><br>
><br>
> On 7/8/22
07:44, Robert DeVita wrote: > Does anyone<br>
>have information on a widespread Rogers
outage in Canada. I > have<br>
>customers with multiple sites down. There's
discussion on the Outages<br>
>mailing list. Seems widespread, affecting
all services, mobile, voice,<br>
>Internet. No cause or ETR posted yet. -- Jay
Hennigan - <a href="mailto:jay@west.net" target="_blank">jay@west.net</a><br>
><mailto:<a href="mailto:jay@west.net" target="_blank">jay@west.net</a>>
Network Engineering - CCIE #7880 503 897-8550 -<br>
>WB6RDV<br>
><br>
><br>
><br>
> --<br>
><br>
> Josh Green.<br>
<br>
<br>
<br>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
</div>
</blockquote>
<p><br>
</p>
<pre cols="72">--
Jon Sands
MFI Labs
<a href="https://fohdeesha.com/" target="_blank">https://fohdeesha.com/</a></pre>
</div>
</blockquote></div>
</blockquote></div></div>