Mitigating human error in the SP

Joe Provo nanog-post at
Tue Feb 2 14:26:49 UTC 2010

On Mon, Feb 01, 2010 at 09:21:52PM -0500, Chadwick Sorrell wrote:
> Hello NANOG,
> Long time listener, first time caller.
> What measures have you taken to mitigate human mistakes?
> Have they been successful?
> Any other comments on the subject would be appreciated, we would like
> to come to our next meeting armed and dangerous.

Define your processes well, have management sign off so no blame game 
and people realize they are all on the same side Use peer review.  
Don't start automating until you have a working system, and then get 
the humans out of the repetitive bits.  Don't build monolithic systems.
Test your automation well.  Be sure have the symmetric *de-provisioning* 
to any provisioning else you will be relying on humans to clean out 
the cruft instead of addressing the problem. 

Extend accountability throughout the organization - replace commission-
minded sales folks with relationship-minded account management. 

Always have OoB.  Require vendors to be *useful* under OoB conditions,
at least to your more advanced employees.

Expect errors in the system and in execution; develop ways to check
for them and be prepared to modify methods, procedures and tools 
without multiple years and inter-departmental bureaucracy.  Change and
errors happen, so capitalize on to those events to improve you service
and systems rather than emphasizing punishment.


             RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

More information about the NANOG mailing list