Rack rails on network equipment

Mon Sep 27 21:32:57 UTC 2021

Folks,

I kind of started to doubt my perception (we don't officially calculate
it) of our failure rates until Mel provided this:
"That’s about the right failure rate for a population of 1000 switches.
Enterprise switches typically have an MTBF of 700,000 hours or so, and 1000
switches operating 8760 hours (24x7) a year would be 8,760,000 hours.
Divided by 12 failures (one a month), yields an MTBF of 730,000 hours." At
least I'm not crazy and our failure rate is not abnormal.

I really don't buy the lack of failure in 15 years of operation or w/ever
is the crazy long period of time that is longer than a standard
depreciation period in an average enterprise. I operated small colo cages
with a handful of Cisco Nexus switches - something would fail once a year
at least. I operated small enterprise data centers with 5-10 rows of racks
- something most definitely fails at least once a year. Fun fact: there was
a batch of switches with the Intel Atom clocking bug. Remember that one a
couple of years ago? The whole industry was swapping out switches like mad
in a span of a year or two... While I admit that's an abnormal event, the
quick rails definitely made our efforts a lot less painful.

It's also interesting that there were several folks dismissing the need for
toolless rails because switching to those will not yield much savings in
time compared to recabling the switch. Somehow it is completely ignored
that recabling has to happen regardless of the rail kit kind, i.e. it's not
a data point in and of itself. And since we brought up the time it takes to
recable a switch at replacement point, how is tacking on more time to deal
with the rail kit a good thing? You have a switch hard down and you are
running around looking for a screwdriver and a bag screws. Do we truly take
that as a satisfactory way to operate? Screws run out, the previous tech
misplaced the screw driver, the screw was too tight and you stripped it
while undoing it, etc, etc...

Finally, another interesting point was brought up about having to rack the
switches in the back of the rack vs the front. In an average rack we have
about 20-25 servers, each consuming at least 3 ports (two data ports for
redundancy and one for idrac/ilo) and sometimes even more than that.
Racking the switch with ports facing the cold aisle seems to then result in
having to route 60 to 70 patches from the back of the rack to the front.
All of a sudden the cables need to be longer, heavier, harder to manage.
Why would I want to face my switch ports into the cold aisle when all my
connections are in the hot aisle? What am I missing?

I went back to a document my DC engineering team produced when we asked
them to eval Mellanox switches from their point of view and they report
that it takes 1 person 1 minute to install a Dell switch from cutting open
the box to applying power. It took them 2 people and 15 min (hence my 30
min statement) to install a Mellanox switch on traditional rails (it was a
full width switch, not the half-RU one). Furthermore, they had to install
the rails in reverse and load the switch from the front of the rack,
because with 0-U PDUs in place the racking "ears" prevent the switch from
going in or out of the rack from the back.

The theme of this whole thread kind of makes me sad, because summarizing it
in my head comes off as "yeah the current rail kit sucks, but not enough
for us to even ask for improvements in that area." It is really odd to hear
that most folks are not even asking for improvements to an admittedly
crappy solution. I'm not suggesting making the toolless rail kit a hard
requirement. I'm asking why we, as an industry, don't even ask for that
improvement from our vendors. If we never ask, we'll never get.

--Andrey

On Mon, Sep 27, 2021 at 10:57 AM Mel Beckman <mel at beckman.org> wrote:

> That’s about the right failure rate for a population of 1000 switches.
> Enterprise switches typically have an MTBF of 700,000 hours or so, and 1000
> switches operating 8760 hours (24x7) a year would be 8,760,000 hours.
> Divided by 12 failures (one a month), yields an MTBF of 730,000 hours.
>
>  -mel
>
> > On Sep 27, 2021, at 10:32 AM, Doug McIntyre <merlyn at geeks.org> wrote:
> >
> > On Sat, Sep 25, 2021 at 12:48:38PM -0700, Andrey Khomyakov wrote:
> >> We operate over 1000 switches in our data centers, and hardware failures
> >> that require a switch swap are common enough where the speed of swap
> starts
> >> to matter to some extent. We probably swap a switch or two a month.
> > ...
> >
> > This level of failure surprises me. While I can't say I have 1000
> > switches, I do have hundreds of switches, and I can think of a failure
> > of only one or two in at least 15 years of operation. They tend to be
> > pretty reliable, and have to be swapped out for EOL more than anything.
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210927/f4aca0c0/attachment.html>