CenturyLink RCA?

Tue Jan 1 12:05:29 UTC 2019

On 12/31/18 3:31 PM, Keith Medcalf wrote:
>> It could have been worse:
>>    https://www.cio.com.au/article/65115/all_systems_down/
> 
> "Make network changes only between 2am and 5am on weekends."
> 
> Wow.  Just wow.  I suppose the IT types are considerably different than Process Operations.  Our rule is to only make changes scheduled at 09:00 (or no later than will permit a complete backout and restore by 15:00) Local Time on "Full Staff" day that is not immediately preceded or followed by a reduced staff day, holiday, or weekend-day.
> 
We had fairly extensive discussion on this list decades ago.

Deploy non-emergency changes early Tuesday morning local time,
a few hours before regular working hours.

Agreed about "not immediately preceded or followed by a reduced
staff day, holiday, or weekend-day."

Because you won't know it's really working until the actual
users have no reported problems.  Tuesdays give a couple of
working days to ensure that there were no hidden ill affects.

Weekends are terrible for that reason.  As are Mondays
and Fridays, because actual users aren't around in overseas
locations.  Even those of us who have operated regional ISPs
still can effect the world.  And those of us with multiple
datacenters world-wide have to ensure that changes in one
place don't affect the others....

As I remember, at the time this NANOG wisdom propagated to
legal blogs.  Because so much of what network operators do
has legal implications.