Networking Pearl Harbor in the Making

Tue Nov 8 19:49:36 UTC 2005

On Mon, Nov 07, 2005 at 05:03:32PM +0000, Christopher L. Morrow wrote:
> How do the operators/engineers explain to 'management', or whomever asks,
> the 'training issues' that always crop up when more than one vendor are
> proposed? Has anyone had good luck with this arguement? (my answer is sort
> of along the lines of: "Its just a router, no matter the vendor and they
> all have command-line help" but that's not always recieved well :) )
> 
> Just curious as I'm sure there are folks stuck in an all vendor X shop who
> look over the electronic fence and see vendor Y with 'so much better' or
> 'so much faster' or 'so much more blinkly lighty'... and try to have their
> management agree to purchasing new devices :)

It really depends on your management.  Good management will either
understand your arguments and support them or be non-technical enough that
they'll let you make the technical decision because that's what they're
paying you a lot of money to do.

Back in 1999, I was the director of IT Engineering for Macromedia.  We were
putting up a new colo location at Exodus (the story of our relationship
with them is worthy of another rant), and had purchased something on the
order of $500K of Cisco equipment to provision it -- two Cisco 7513s and
two Cisco 6009s.  Forgive me, but I can't reliably tell you why we chose
those specific models (six years ago!) -- but we needed to be able to
sustaing gigabit bandwidth and our Cisco VAR, in combination with the Cisco
district sales droids, said "get this!"

The provisioning took a while, during which time the equipment gathered a
bit of dust.  Getting close to the installation date, our technical staff
(including me) took Exodus' technical staff to lunch during which they said
"Cisco ?! Ha! They're a piece of crap.  You should look at Foundry."  We
talked about it in more details, and left lunch persuaded we should look at
Foundry.

The next 2-3 weeks were interesting.  Amusingly, our salesrep at a Cisco
VAR had left earlier to go join Foundry, so we already knew someone there.
We had him come in and discuss their product line, got some references from
him and talked to them (they were glowing), and priced out the competing
platform.  We then told Cisco that we were goig to go with Foundry.
Amusingly, A) They did not deny their equipment could actually not sustain
gigabit speeds; B) An Exoduds SVP called my CIO to let him know that the
engineer I spoke with had 'over-communicated', that Exodus and Cisco had a
strong partnership, and that in no way would they actually recommend going
with someone else.  Yeah, OK.  

We spent about 25%-40%, I believe, of the Cisco price on the two BigIron
8Ks we got.  We had to deal with some minor training issues, obviously, and
there were some things we were counting on being able to do that they
weren't able to do at the time -- I still remember the meeting where the
Foundry sales rep came in and said "I talked to my CEO about this feature
you need; we can't do it right now, but it's scheduled to be put in in
about 8 weeks.  Meanwhile, we've gotten an OK to loan you this other
equipment that will let you do what you need."  In the end, though, the
transition was remarkably smooth on a technical basis.

Up until this time we were a Cisco-only shop for any purchased equipment
(I'm making this subtle distinction because we still had to manage a bunch
of shared-media 10-baseT hubs in some of our older locations.  Don't get me
started).  Making the decision to go with Foundry required me to go to my
boss and have a grand total of one informal meeting with him that largely
consisted of "hey, I'm thinking of going with Foundry for the new colo.  If
they work, I might start considering them for Corp use also.  They're
cheaper, faster, and promise better service, and so far I see them
delivering on this."  

Of course, I also promised that we'd be able to return the Cisco 7513s (we
repurposed the 6009s) -- and that's when Cisco started pissing us off with
an insane "no, wait, you've got to return it to the VAR ... But the VAR
needs our approval!" game.  It was about 6 weeks of conference calls with
Cisco and our VAR and some meetings; apparently, the big issue was that we
had unpacked the routers and, much like Best Buy, Cisco required us to
return the equipment with all the original packaging.  I suggested that
maybe, since we were looking at about $150K of equipment, we could actually
pay for replacement packaging, but no deal.  Note that at the time we were
OK with just a credit -- we knew we were going to spend more money on Cisco
(at the time) anyway.  

Long story short, one day my Foundry salesrep and his SE were visiting and
noticed I was pissed off.  I told them about this issue I was having with
Cisco, and on the spot he offered to do a trade-in -- he'd give me my full
spent cost on the Ciscos for the 7513s as credit toward any Foundry
equipment.  The day after, I rented a van and took the two 7513s to
Foundry's facilities, and dropped them off at shipping and receiving, where
I got some curious looks.  That evening, when we called the Cisco district
manager and told him "don't worry about it -- we gave them to Foundry for a
credit," both my boss and I enjoyed the resulting shock and dismay.  

So sometimes, moving away from being a one-vendor shop can be relatively
painless.  Other than Cisco trying desperately to hold on to their
exclusivity in this case, we didn't really have too many problems.  The key
was mutual trust within my organization, and the ability of each layer in
it -- my network engineers, me, my boss, my CIO -- to trust the other
layers and let them do their job*.  

-roy

* No IT story is complete without an unhappy ending.  A management shakeup
resulted in the replacement of the CIO, who ended up replacing management
with his own people.  My replacement was a Cisco guy and they ended up
ripping out perfectly functioning Foundry equipment to put Cisco back in
there.  Of course by then it wasn't my problem anymore, but I got to hear
the grumblings from my guys over beers.