Real network failure causes Was: What do you want your ISP to block today?

Thu Sep 4 13:59:09 UTC 2003

At 22:30 03/09/2003, Rob Thomas wrote:
[snip]

>effects.  We all know better.  Bugs aren't restricted only to
>products from Redmond, typos happen, and the performance hit can
>be quite painful.

In my experience more network downtime is caused by configuration errors 
that all other causes together.

The best diagnostic tool I've ever had is a script I cobbled together over 
two hours one night. Once an hour, it simply collected all the router 
configs across the network, did a 'diff' between the current and last 
config, and if there were changes, emailed them to me, along with a TACACS+ 
log summary that showed who had logged into which router when.

Experience with this quickly taught me to check these summary change logs 
whenever a problem was escalated to me. Most times the problem was related 
to a config change, not an external cause. Further experience taught me to 
look out for one particular engineers name in the logs but that's another 
story.