keeping your cabinet clean (was Re: Looking for help @ 60 Hudson)

Tue Nov 14 09:12:01 UTC 2017

Care to share some pics?

David

On 14 November 2017 at 02:04, Ken Chase <math at sizone.org> wrote:

> Some tricks I've learned managing multicustomer/shared cabinets over the
> last
> 20+ years...sorry it's long, but I think there's some good info on keeping
> things clean and maintaining sanity. Please send your protips.
>
> Most of this is lower-end 1-4U sized mixes of gear specific and specific to
> cabinets that have 2-6U+ flux per quarter with some rushed installs. Huge
> one-time 12U blade installs of $1M appliances usually lend to gorgeous
> cable
> management schemes (and proper budgets) being included. No such lux here!
>
> TL;DR: thin premade ethernet of exact lengths and multiple random colours
>   (never black!), use min gauge required power cable thickness of exact
>   length, face A/B PDU's backwards on one side of rack cable management on
>   other side, never get less than 30" wide x 36" deep cabinet (if not,
> wider
>   better than deeper), premeasure vert mount rail positions to be
> compatible
>   with rail length/ front of server clearance, prewire front of rack
>   power/ether if needed (leave string too), practice tooless rail removal
>   while you can still get in above/below, rack similar-depth gear together,
>   switches face backwards (with front-to-back airflow switch config option
> of
>   course) on rail-shelves not ears (that bend over time anyway) so they
> can be
>   extracted out front and easily replaced in emergency.
>
> Details:
>
> Installing in 30" wide x 36" long cabinets makes all the diff over 24" x
> 30".
> A/B 0U PDUs on one side, cable wrangling ladders on other.  More room =
> more
> flexbility. (If have no side panels and no neighbours, 24x30" is ok). 36"
> deep
> allows facing the PDUs backwards not sideways - cableheads extend
> backwards,
> not into the rail-tail path/airflow/etc. Worth getting the
> 90degree-bent-head
> cables too if you need the spare inches. (I ofset my PDUs vertically by
> 1/2 a
> plug-spacing distance so cable from left one fits between cableheads of
> right
> one.)
>
> Avoid racks that don't use cagenuts. Prethreaded holes get abused and
> stripped. Try to get the right size of cagenut, there's a few standards out
> there. Some will fit - poorly. (Either they fall out under weight or you
> end
> up trying to force them in with a thin screwdriver - I've seen people stab
> themselves in the hand. Ask for a cagenut tool (J-hooked shaped piece of
> metal that
> looks like a bent desktop-case PCI slot cover.)
>
> Having many power cables of varying lengths is key (but why doesnt anyone
> make
> 15" and 21" power cables?). Not having ziptied loops of 12ga wire hanging
> around made things much nicer (and better airflow).  More $ but worth
> sanity.
> Esp. with varied coloured heads. Great for tracing (see ethernet below).
> Wire
> the gauge required - I find 10ga (6' long..) wire delivered with 100VA-max
> server
> configs often. Too thick to manage properly and usually unnecessary. But
> check
> your warranties and theoretical max power envelopes.
>
> Yes, full rack solns w/extendible arms exist but generally require vendor
> compatibility. Expensive too. Great for one time well-funded installs. Not
> practical for varied species installed over longer periods.
>
> Prewire any front-of-rack-powered gear when you first get the rack. I have
> 5
> pairs (A/B) going to the front permanently ziptied and labelled - 3x2 in
> use
> for my back-facing switches, 1 for a small piece of gear (low watt
> microtik),
> others spare.  Also prewire some proper length (multiple colours of) ether.
> Fishing ether through the side can be impossible in a full cabinet in a
> dense
> row (we're in APC pods). I leave string in there too (probably will use it
> for a twinax pull to the microtik soon, and pull more string with).
>
> Curse vendors for not picking a standard side (left vs right) for power
> ingress!
> (ibm and dell vs supermicro, sun and hp, IIRC?)
>
> Beware Dell's long fins/tails on their rails - won't fit in a 30" cabinet
> if
> your vert mount rails are too far back - or it blocks the power cord head
> on
> the pdu if it faces sideways/etc. And beware max/min rail extension - Dell
> seems 'longest', with many min. rail lengths of 25.5". I think I saw min.
> 26.5" once.
>
> Also had a cx jam a long Dell rail's tail into his fully assigned cabinet
> - in
> between a powercable head and the pdu body it was plugged into.  BZZT!
> Took 20
> min to get a monkey to reset at central panel. Thank proper cabinet
> grounding
> cables, right?)
>
> If you have an entirely empty cabinet to start with, grab a few different
> rails and ensure your cab's vertical mount rails are all within spacing
> spec.
> and give door-closing clearance to server noses. (See reference tables.)
> Moving them later can be impossible (though with sunfire rails that slid to
> varying lengths, it worked out luckily!)
>
> Must admit Dell's tooless rail installs are awesome now. Better than
> supermicro's and sun's (Sunfires). Learn how to derack them before you
> install, and practice a few times while you can still get your
> fingers/tools
> in from above/below. Make notes on how it works. Trying to guess how to
> derack
> a single U rail sandwiched in with no other access can be nearly
> impossible,
> especially an old EOL one with no identifying marks to google.
>
> Try to rack similar depth gear together. Nothing like having 1U 27" long
> Intels sandwiching 1U of 3/4 depth supermicro between them - cant get your
> hands in to do anything. 2U minimum together, but 3+ is better. Stash a
> headlamp
> in the rack too to see into these wells.
>
> Having more switchports than necessary (3 redundant 48 port switches with
> avg
> 2xc/box, avg 1.5U per box, 42U cabinet) allows 4' ethernet cables to be
> put in at
> exactly 3.75' extension into a switchport - minimal extra slack, just
> enough
> to manage the cable and keep it unstressed.  Folding spare cable into
> management ladders makes manual tracing hard, and is bad form. Crimping
> your
> own rj45 ends for exact lengths is risky. Done it many times (and once saw
> a tech do
> it before a bgp session timed out on a damaged live cable! :), but results
> are
> poorer than factory crimps and dont resist stress well. ('Sides, do you
> have 8
> different colour spools? See below.)
>
> Switch racking - I'd rather them on shelves not ears - then you can pull
> out
> your backwards-facing switch ass-first from front of the cabinet and slide
> the
> replacement in without moving cables too far -- trying to get the ears past
> the cabling can be heinous (and most ears cause major cantilevering due to
> deep heavy switches - after a few years I find things are bent from weight,
> impinging on the U below). There are thin little 3" wide rail-edge
> 'shelves'
> you can get, but your switch may not give the spare vertical mm required
> (about 3-4mm) - works best with 2u+ sized gear where there's spare mm play.
>
> Proper cable management trays/guides between the switches is great too
> (though
> eats U). Your density/financials will determine if that's viable. Its worth
> the U/sanity usually.
>
> Out of U for cable mgmt but have spare depth? Bungee cords won't work as
> anchor points to ziptie too - they sag in the heat over time. But rubber
> bungees ('tarp straps') work great.
>
> Though not for full 100m ether lengths, narrow gauge ethernet is key.  You
> can
> fit like 8-12 of them into a run of what 4-5 took previously. I've had zero
> issues with them with rack-lengths of cable. Worth the $.
>
> I wish there was thin bicolour ethernet - it's sexy to have all your ether
> coloured the same (or same per category - red mgmt, yellow public, white
> internal) -- but then you need to know where the failed port 34 cable is...
> tracing identically coloured cables can suck, your eyes start playing
> tricks
> looking at 30 cables in a twisting bundle that you ziptied too tight
> trying to
> be pretty - and under tension, yanking on them may not identify the right
> cable properly -- even with labels (you stop trusting them anyway after the
> first couple errors or when job security is thin).
>
> I tend to use as many random colours as possible -- and note it in the
> switch
> configs. If the switch config doesnt match, slow down and check things
> twice
> from scratch. (Keep many colours and lengths handy for new installs! Blue
> is so
> ugly - peach ftw, amirite?).
>
> Note that no pattern of colour use will work - wire all mgmt ports as one
> colour,
> or all ports of one server one colour - either way, you assume the colour
> means
> the cable is wired a certain way, and that leads to errors. With no
> pattern,
> no dangerous assumptions are made - must check with the switch config.
> Maxing out
> #s of different colours leads to easier identification/fewer chances of
> neighbouring
> cables of same colour.
>
> If the management layer of your company doenst have to slave over racks,
> this
> is likely not an option. They like the pretty and impractical stuff best
> of course.
>
> Labelling only works so well - in 100-120F exhaust paths most glue just
> melts
> off. One client I had no input on the install for has 100s of little
> silvery
> labels littering the floor/lower U of their cabinet. So pretty! And goey
> cables. And thank god they're all white! Makes tracing so easy! :P
>
> (Of course no one ever wires black ethernet, right? That's a capital
> offence!)
>
> /kc
>
>
> On Mon, Nov 13, 2017 at 05:05:35PM -0500, Chuck Anderson said:
>   >On Mon, Nov 13, 2017 at 01:30:25PM -0800, Seth Mattinen wrote:
>   >> On 11/13/17 12:49, Mike Hammett wrote:
>   >> >Keep the humans out of the rack and you should be fine.
>   >> >
>   >> >Where should I send the invoice?:-P
>   >>
>   >>
>   >> It's easy to keep a rack nice if you take the time. I've spent hours
>   >> removing and replacing cables in neatly dressed bundles because
>   >> equipment changes required a different length/type cable, but
>   >> sometimes that's what you gotta do to keep things neat and tidy.
>   >
>   >Exactly.  Most people do not want to spend the time to do it properly.
>
> --
> Ken Chase - math at sizone.org Guelph Canada
>

-- 
--
My opinion is mine.