Is Cisco equpiment de facto for you?

Mon Jan 10 17:26:06 UTC 2011

 >I try to follow the Tolly Group who compares products, and they 
 >continually show that Cisco equipment
 >is a poor performer in almost any equipment compared to others, I find 
 >that so hard to believe.....

Just a rough comment here.  Tolly's business model is a sponsored test 
one, and Cisco is the dominant vendor in the marketplace.  So when 
anyone has a new product they want to promote, they can hire Tolly's lab 
to show how a particular feature set is implemented and can often show 
how one model exceeds Cisco's performance on some or all benchmarks.

While Tolly never cheats, remember that the guys who "win" the test are 
the ones that are helping to design the test methodology, and so they 
are focusing on features that they know that they can beat Cisco on. 
Thus, the Tolly reports are true, but may not be telling the whole story 
or may be focusing on a particular feature more than anything else.  No 
one is lying to you, but you may not be given the whole picture.

There's nothing wrong with having Tolly show that your product is in the 
same league as Cisco's, but I would not necessarily read those reports 
to say that Cisco is always the "loser" in any new product comparison. 
The real value in Tolly's reports is to give credibility to the new and 
up-coming products by having a mostly-independent source show that the 
new products are comparable with the market leaders' in some ways.

Tolly is also very very important at doing spec verification: showing 
that a product actually DOES what the specification sheet says it does. 
  Remember that those numbers are often made up by product managers 
based on poorly-done in-house tests, so having a third-party, even a 
paid one, say "yes, it does do that in our test lab" is WAY more 
information than you get from most vendors.

Tolly provides a valuable service to our community but you need to look 
at a combination of benchmarks from Tolly as well as other very-unbiased 
sources (IDG and CMP magazines are pretty thorough) AND YOUR OWN TESTING 
to know what is best for you.

On the other hand, compared to the analyst firms like Gartner who don't 
even have test labs and generate their reports based on phone calls, 
Google searches, and unverified specification sheets, Tolly is a source 
of infinite knowledge about products.

Finally, also remember that performance is only one part of system 
design and deployment.  Management and feature completeness are, for the 
operator community, often more important than whether the box can do 80 
Gbps or 85 Gbps.  Product selection is best done by YOU setting YOUR 
requirements and doing YOUR OWN testing, not by looking at point tests 
to see if one product version/model/device beats another in a particular 
small set of performance tests.

Obligatory disclaimer: we don't compete directly with Tolly, but we do 
get paid by the magazines to do testing and there is some overlap, so we 
do compete indirectly.

jms

-- 
Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
Senior Partner, Opus One       Phone: +1 520 324 0494
jms at Opus1.COM                http://www.opus1.com/jms