Software Bugs

Sun Feb 20 19:43:36 UTC 2011

On Sun, 20 Feb 2011 18:05:44 +0200, Kasper Adel said:

(Disclaimer - I've never filed a bug report with Cisco or Juniper,
but I've spent 3 decades filing bugs with almost everybody else in
the computer industry, it seems...  Questions like the ones you asked
are almost always pointless unless the asker and answerer are sharing
a set of base assumptions.  In other words, "which one is best/worst?"
is a meaningless question unless you either tell us what *your* criteria
are in detail, or are willing to listen to advice that uses other
criteria (without stating how they're different from yours).

> 1) Which vendor has more bugs than others, what are the top 3

More actual bugs, more known and acknowledged bugs, or more serious bugs that
actually affect day to day operations in a major manner?

The total number of actual bugs for each vendor is probably unknownable, other
than "there's at least one more in there".  The vendor probably can produce a
number representing how many bug reports they've accepted as valid. The
vendor's number is guaranteed to be different than the customer's number - how
divergent, *and why*, probably tells you a lot about the vendor and the
customer base. The vendor may be difficult about accepting a bug report, or the
customer base may be clueless about what the product is supposed to be doing
and calling in a lot of non-bugs - almost every trouble ticket closed with RTFM
status is one of these non-bugs. If there's a lot of non-bugs, it usually
indicates a documentation/training issue, not an actual software quality
issue.

And of course, bug severity *has* to be considered.  "Router falls over if
somebody in Zimbabwe sends it a christmas-tree packet" is different than "the
CLI insists on a ;; where a ; should suffice".  You may be willing to tolerate
or work around dozens or even hundreds of the latter (in fact, there's probably
hundreds of such bugs in your current vendor that you don't know about simply
because they don't trigger in your environment), but it only takes 2 or 3 of
the former to render the box undeployable.

> 2) Who is doing a better job fixing them

Again, see the above discussion of severity.  If a vendor is good about fixing
the real show-stoppers in a matter of hours or days, but has a huge backlog of
fixes for minor things, is that better or worse than a vendor that fixes half
of both serious and minor things?

In addition, the question of how fixes get deployed matters too.  If a vendor
is consistently good about finding a root cause, fixing it, and then saying
"we'll ship the fix in the next dot-rev release", is that good or bad?
Remember that if they ship a new, updated, more-fixed image every week, that
means you get to re-qualify a new image every week....

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20110220/02c12aad/attachment.sig>