Mitigating human error in the SP

Larry Sheldon LarrySheldon at
Thu Feb 4 23:22:23 UTC 2010

On 2/4/2010 5:13 PM, Larry Sheldon wrote:
> On 2/4/2010 3:30 PM, Scott Weeks wrote:
>> A recent organizational change at my company has put someone in charge
>> who is determined to make things perfect. We are a service provider,
>> isn't a common occurrence, and the engineer in question has a pristine
>> track record.
>> This outage, of a high profile customer, triggered upper management to
>> react by calling a meeting just days after. Put bluntly, we've been
>> told "Human errors are unacceptable, and they will be completely
>> eliminated. One is too many."
>> ----------------------------------------------------------------
>>> From experience...
>> At one point this will become overwhelming. You'll wake up every
>> morning dreading going to
>> work instead of looking forward to it. Chain shot will be put in the
>> 'blame cannon' and
>> blasted regularly and at everyone. Update your resume and get
>> everything in place just in
>> case it gets to the point you can't take it anymore sooner than you
>> expect. ;-)
> This is a golden opportunity.
> Prepare a pLan for building the lab necessary to pre-test EVERYTHING.

Plan.  Prepare a plan.

> Cost it out.
> Present the cost and the plan in a public forum or widely distributed
> memorandum (including as a minimum everybody that was at the meeting and
> everybody in the chain(s) of command between you and the edict giver.

"Government big enough to supply everything you need is big enough to 
take everything you have."

Remember:  The Ark was built by amateurs, the Titanic by professionals.

Requiescas in pace o email
Ex turpi causa non oritur actio
Eppure si rinfresca

ICBM Targeting Information:

More information about the NANOG mailing list