Building a NOC

Sun Mar 22 09:31:40 UTC 1998

At 22:01 -0600 3/21/98, Sean Donelan wrote:
>Both good ideas.  But be sure to learn and understand when, where and why
>the operational experiences given in these sources apply to building a NOC.

No argument, and no argument that the practices have to be current.  But
there are things to be learned from the past, as well.  In about 1981 or
so, I built the NOC for what was then the US Agriculture Department
Washington Computer Center.

As you point out, one must consider the specific requirements.  In this
case, it was with horrible fascination I realized that my client was the
Federal agency with official responsibility for bull, and presumably their
output, in the US government.

>
>One difficulty in designing a NOC for an ISP is they operate across
>several different problem domains, telecommunications, data processing,
>lan, wan, etc.  Be careful if you hire a consultant with expertise in
>one particular area.  NEBS and the like have several good ideas, but
>they also have a built-in set of assumptions.  If you, or your consultant,
>don't understand when the assumptions apply and don't apply, when you
>apply the solutions to a different problem domain, you can actually
>make things worse.  For example, the National Electrical Code doesn't
>apply to communications equipment located inside a communication utility
>building.  But the building inspector will get very grouchy if an ISP's
>electrical wiring doesn't meet NEC standards.

Yes.  And your consultant must know local practices as well.  If you are
building a facility in Fairfax County, Virginia (a major technology suburb
of Washington DC), avoid, in your plans or anything the building inspector
may see, the term "computer room."  "Communications room," "network room,"
etc., all prevent the problem: if they see "computer room," they will
demand a mainframe-style central emergency power off control, which greatly
increases electrical wiring cost.

I haven't looked at this recently, but another Fairfax County practice was
if you used halon, regardless if the room was also sprinklered, you had to
do a live test with halon before the inspector would approve it.  The
refill of the halon had to be built into your budget...an extra $16,000 as
I remember from about 1986.  The specific isn't that important, but if you
are going to have to deal with irrational requirements, you need to know
that.
>
>Also, there have been changes in 'standard' practices in the last 5 years.
>If your sources for designing data processing centers were written prior
>to 1990, I would look for a second source.  The military has been updating
>its standards to reflect revised commercial practices, but the civilian
>sellers of the MILSTDs don't always show when an old practice has been
>superceded.

Wasn't thinking of MILSTDs as much as some of the DISA documents, and often
relating to base practices.  The DoD manual on grounding, bonding, and
shielding is excellent.

There has been some very good work on filtering alarms in radio and
multiplex algorithms, for reducing the noise level of alarms triggered in
response to a single component failure.  I'd have to dig through my
library, but IIRC this was done by GTE (Sylvania?) for European Command.

>
>My basic problem with the original question is I don't really understand
>what kind of NOC the person is trying to build.  Are we talking about
>how build a NOC staff, or a NOC building to monitor a network of 70,000
>nodes?

Another big question, frequently ignored, is staff training and retention.
Someone who can quickly diagnose complex problems tends not to want to
answer phones.  Some organizations come up with reasonably creative ways to
give a growth path to NOC people, arrange rotations off the help desk,
etc., to avoid losing some very valuable knowledge.  Other organization
simply assume their senior operations support people will leave every year
or so.

>Are we talking about building a NOC theatre so management/customers
>can view CNN on the projection TV wall when they visit, or a lights-out
>server/equipment room designed for limited human occupation?  ...

Speaking of irrational requirements, your point about NOC theatre is well
taken.  I've dealt more recently with some organizations that outsourced
the NOC function for state governments, and was told flatly "we are going
to be visited by politicians who want to see where their money went.  While
the capacity planning projections are the really useful things, they simply
won't believe the contractor isn't cheating them unless the place looks
like something out of Star Trek."

Of course, the bridge of the NCC-1701 series Enterprise(s) here show
incredibly bad human factors design, but that is often ignored.  Perhaps
more realistic is the display over the medical bed.  While one of the bar
graphs has text too small to read on the TV screen, the Star Trek technical
manual shows how it is labeled "Medical Insurance Remaining."

Surely there is a NOC equivalent of that bar graph.