Building a new NOC: Summary

Tue Jul 17 16:33:23 UTC 2001

Hi folks!

Several people asked for my summary of information I learned in the NANOG thread
"building a new NOC" - I decided to assemble the summary and post it here,
since enough people seemed to want it.

I learned about several contractors who do NOC buildouts.  To avoid seeming like
a plugger, I won't give their names here.  If you want the names of the contractor
list i've assembled, please e-mail me privately.

There were three major vendors of furniture that seemed to provide NOC-level
stuff.  APC-Wrightline, SMC, and Evans were the people that seemed to make
items specifically built for use in an IT center or a NOC.  I don't know where
they compare on price; we're going through the process with each to determine
who is the best bet.  Evans by far seems to make the nicest furniture, though.
I wasn't all that impressed with APC.

Obviously a lot of the considerations for building a NOC come down to where
you are building it at.  Several people consider the same criteria for NOCs
as they do for datacenters:  Things like earthquake protection, good power,
redundant everything, HVAC, etc.  Several people also mentioned that you want
to be careful that the building you're building your NOC in isn't required
to use union labor, since that ups the cost significantly.

For our NOC, we considered the following criteria:

* Breakrooms
* Lighting
* Sleeping areas
* Televisions and projection screens
* Consoles with "roaming" capability - leave any console
  and move to another
* Access control in and out of the facility
* Laptops: For those people who must leave the NOC and go to a customer
  immediately
* Whiteboards and conference rooms, inc. conference phones
* Good chairs, keyboards, and required breaks for NOC personnel

I roughly use the criteria of 50-100 machines/clusters per on-duty engineer
(we're a systems shop).  There's at least one senior engineer on staff for
each shift.  This probably varies widely among NANOG NOC managers.

I considered several projector manufacturers.  I haven't gotten all the way
down to the lower level of detail on the projector scene, but we are told
by an AV guy we're retaining that we should buy something between 700 and 1400
ANSI  lumens.  The big, 3000-3400 lumen units will run about $15K, and will
project about 100+ ft in bright sunlight - obviously not necessarily for our
purposes.  The less lumened units, when placed closer to your projection
screens, will provide more sharpness and clarity.  (Any comments from AV
people?)  DLP projectors are the way to go.  We're ordering three projectors.
I'm still looking for either software or hardware that will allow me to
multiplex PC signals onto one projector, and looking for what NOCs use
to control the machines the projector is on.  If anyone has info on this
stuff, please let me know.

In my pricing sheet for CFO consumption, I included the common items as well
as the expensive technology stuff.  We decided to include things like
blankets and pillows.  Rather than buy cots (there are lots of lumpy NOC
people, like myself) we chose to build bunks out of sheets of plywood, and
just lay the standard bed mats you find at camp stores over them.  NOCs
usually have pretty good AC, so we included lots of blankets.  Cold and
tired engineers have a hard time working - we included a fridge, a good
microwave, and nice coffee machines.  I'm getting into minutiae[sic] here.

Our layout was for seven operators and three senior engineers.  The
engineers are officed behind the NOC people, with a good view of the
screens through windows in their area.  There's a dedicated staircase
just to this room, to avoid traffic.  The room is very low-light - just
enough to see where you're going.  We haven't decided on lighting
fixtures yet, even for the overheads, but I think we've all come to
the conclusion that we're just going to install some soft track
lighting and let the engineers decide what kind of task lights they
want over their consoles.

The consoles are arrayed in such a way that even though they curve
well so that everyone can see the screen, there is some privacy in the
two-monitor setup we have.  I chose Sun's line of Sun Ray workstations.
These are really cool machines, folks.  It's basically a dumb X terminal
with a smart card reader - you take your smart card out, move to
another machine, and your desktop is right there with no loss of
anything.  You can show another engineer something just by running
your card over to his workstation.   This was critical - in a shifted
environment, you want everyone to have their own space.  NOC people
are picky.  The workstations provided a good mix of that stuff.
Fairly cheap, too.

Everyone gets their own locker, including the senior engineers, to
store personal items.  We have a rule that nothing personal comes
into the NOC except for books and cell phones - this keeps our
engineers working.  There is some music service available from an
MP3 library, but there isn't any audio CD stuff on the
workstations.

The Sun Ray stuff really only seems good for terminal serving, and
it requires its own segment.  We run all our applications via
tunneled SSH to separate PC or SPARC servers running Linux or
Solaris.   This helps us so that nobody can screw up the terminal
server by installing software or changing parameters - everyone
pretty much gets redirected to an appropriate machine upon login
where they do their sandbox stuff.

One thing I did like about the Sun Ray setup is that it does voice
over IP.  When you plug in your smartcard, your phone and voicemail
also gets redirected to that location.  It remains to be seen whether
we'll use this solution, but I thought it was appropriate for NANOG
consumption.

I didn't really skimp on ergonomics.  Since there is a high turnover
in NOC personnel, we included every creature comfort we reasonably
could.  This includes Playstation 2 and Dreamcast, DVD, and large-screen
TV with DSS for the guys.  They're required to take 20 minute breaks
every two hours, not including any Nazi bathroom permission or anything
like that.  This keeps them loose and relaxed.  That rule isn't
violated, even if everything's falling down around our ears.

Personally, I think the biggest investment we make in the NOC is in
people and creature comforts.  The second biggest investment is in
standard operating procedures - don't make people think if they don't
have to.  Make sure you invest on training your folks right when they
come in.  This is probably old hat to lots of you, and it's old hat
to me, but it doesn't hurt to mention it.

The hardest thing in a NOC is to make sure that things are kept quiet
so that everyone can work, and make sure that the business types
don't bring people in for tours without approval and/or scheduling.
You don't want people walking through your NOC when there's a problem,
whether it's obvious or not.

I'm going to be building a web page to contain all of this information,
as well as a mailing list, over the next couple of weeks.  I'll post
when available.

Thanks very much to Brennan Murphy @NAI, Derrick Daugherty, 
Matt McGuirl, the folks over at NOCbuilder, and herb at tomobiki.urusei.net
for excellent commentary and help.  Thanks also to everyone else who
contributed privately - there were so many responses.

Tim