Mitigating human error in the SP

Larry Sheldon LarrySheldon at
Thu Feb 4 23:13:30 UTC 2010

On 2/4/2010 3:30 PM, Scott Weeks wrote:
> A recent organizational change at my company has put someone in charge
> who is determined to make things perfect.  We are a service provider,
> isn't a common occurrence, and the engineer in question has a pristine
> track record.
> This outage, of a high profile customer, triggered upper management to
> react by calling a meeting just days after.  Put bluntly, we've been
> told "Human errors are unacceptable, and they will be completely
> eliminated.  One is too many."
> ----------------------------------------------------------------
>> From experience...
> At one point this will become overwhelming.  You'll wake up every morning dreading going to
> work instead of looking forward to it.  Chain shot will be put in the 'blame cannon' and
> blasted regularly and at everyone.  Update your resume and get everything in place just in
> case it gets to the point you can't take it anymore sooner than you expect.  ;-)

This is a golden opportunity.

Prepare a pan for building the lab necessary to pre-test EVERYTHING.

Cost it out.

Present the cost and the plan in a public forum or widely distributed 
memorandum (including as a minimum everybody that was at the meeting and 
everybody in the chain(s) of command between you and the edict giver.

"Government big enough to supply everything you need is big enough to 
take everything you have."

Remember:  The Ark was built by amateurs, the Titanic by professionals.

Requiescas in pace o email
Ex turpi causa non oritur actio
Eppure si rinfresca

ICBM Targeting Information:

More information about the NANOG mailing list