Operations task management software?

Wed Jul 27 23:27:23 UTC 2016

Full automation is planned but does not eliminate the need for the software.  Zero human auditing of fully automated processes and data collection are not acceptable to various certifying entities, the relevant auditors, the inevitably involved lawyers, and won’t pick up on bad data, like a bad thermometer or snmp counter that says a CRAC is 65 degrees when it’s really 90.  So I’m still going to need a management solution to the issue whether it’s to tell someone to do the work or to tell someone to check the automated work.

David

On 7/27/16, 7:19 PM, "Lee" <ler762 at gmail.com> wrote:

    On 7/27/16, David Hubbard <dhubbard at dino.hostasaurus.com> wrote:
    > Hi all, curious if anyone has recommendations on software that helps manage
    > routine duties assigned to operations staff?

    Have computers do the routine scut work - not people.

    > For example, let’s say we have a P&P that says someone from the netops group
    > must check that Rancid is successfully backing up all router configs
    > bi-weekly.

    You've got the source code for rancid, so change rancid-run to do something like
      LOGFILE=$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S`; export LOGFILE
    change the
      ) >$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S` 2>&1
    to
      ) >$LOGFILE 2>&1

    and then in control_rancid do something like
      grep "clogin error:" $LOGFILE | sort | uniq -c >$TMP.fail
      if [ -s $TMP.fail ]; then
         # got some output, mail the report
         ...

    Do the same type thing for checking on
    > backup failures, backup internet circuit status, out of band interfaces, etc.

    Automate the checks, put the scripts in crontab & mail out an
    "OhNoes!" or "all clear" msg at the end.   At which point you're left
    with the problem of making sure the managers are looking at the emails
    & making sure whatever problems are found actually get fixed :)

    Regards,
    Lee