Fiber cut in SF area

Tue Apr 14 00:42:01 UTC 2009

Though I think networked environmental monitoring has its merits, it's
clear the technology is unproven in monitoring fiber tunnels, and my
inexperience in running and managing such tunnels makes this thread
bordering on off-topic.

I'm happy to continue conversations via email, but this will be my last
on-list reply regarding the topic I started.

On Mon, 13 Apr 2009, Crist Clark wrote:

> But would alarms prevent any, or what proportion, of these incidents?

  It's hard to say without researching.  Sometimes such research shows
  amazing results that shock people in the industry.  Hospitals were shocked
  to see surgical mistakes reduced by 80+% after implementing a checklist
  that both doctors and nurses had to go through prior to starting the
  procedure, and having the patient also go over and approve what was to be
  done.  The stories you hear of people who are getting amputated writing
  "this leg" and "X X X NOT THIS LEG" before surgery is a result of these
  studies and checklists.  RFID-tagged surgical components and gauze pads
  are another tech tool being used after such research.

  You'd think a checklist wouldn't really help, but in reality it made
  industry changing and life-saving differences.

  While active alarms and monitoring of fiber tunnels would do the same, but
  without research, nobody can say for sure how effective or ineffective
  such a system would be.

> From what we know of this specific one, would an alarm have stopped the
> perpetrator(s)? It would have bought the NOC five, ten minutes tops
> before they got the alarm on the circuit. And in practice would a manhole
> alarm translate to a call to Homeland Security to have the SEALs descend
> the site pronto, a police unit to roll by when it has the time, or is it
> going to be an AT&T truck rolling by between calls? I'm guessing number
> two or three, probably three. So what would it get them in this case. If
> it doesn't deter these guys, who does it deter?

  It's not there as a deterrent.  It's there to allow a NOC to know that
  something is going on in a tunnel where potentially critical
  infrastructure resides.  Maybe it doesn't prevent the malicious cut, but
  combined with video surveilence, it could identify the cutters.  Audio
  recording devices could record voices.

  I assume large networks have large 24/7 crews.  Get a truck to roll (once
  you sufficiently trust the system) or get a contractor who resides nearby
  to check out the area.  When the alarm goes off, you go check it.  If you
  welded the manholes shut, and there are no scheduled maintenance windows
  for that area, you can be pretty damn sure something untoward is going on,
  or it'll be a company truck roll that didn't follow procedure.

> And what are the costs of false alarms? What will the ratio of "real"
> alarms to false ones be? Maybe lower-stakes vandals take to popping the
> edge of manhole covers as a little prank.

  Weld 'em shut.  Use one of those special screws that you can only unscrew
  with the right equipment (worked wonders for the tire industry with the
  "lock nut").  It won't stop anyone determined, but 13 year olds with M80s
  will move on.  If you get a certain location that continues to get false
  alarms due to vandals, put in a highpowered webcam to monitor the
  location.  Use ZoneMinder to monitor and record motion.  Make sure the
  camera does nighttime well.  Then when you have an alarm, check the video.

> Or that one that triggers whenever a truck tire hits it right.

  I would envision that though every device would report the same data with
  the same sensitivity, false alarms could be mitigated through filters for
  a given location.  Tunnels near train tracks would be filtered differently
  than tunnels in the middle of a field under high power lines.

> Or the whole line of them that go off whenever the temperature drops
> below freezing.

  The device would go through a lot of environmental testing, so that its
  upper and lower operating limits could be known.  Hardened where
  necessary.

> Or, what I am absolutely sure will happen, miscommunication between
> repair crews and the NOC about which ones are being moved or field crews
> opening them without warning the NOC (or even intra-NOC communication).
> Will they be a boy who cried wolf?

  Maybe.  Maybe the whole idea is way too far fetched.  Maybe my impression
  of the state of affairs when it comes to fiber tunnels is really not that
  big of a deal, and that outages due to physical access (humans, backhoes,
  floods) don't make up a significant portion of outages, and this is not a
  problem that fiber companies want to solve.

  Clearly there are a lot of problems that this sort of monitoring could
  face.  Given sufficient time to mature, I think cheap, repeatable
  monitoring devices networked together can be a valuable asset, rather than
  yet another annoying alarm NOC folk and maintenance crews grow to hate and
  simply not be effective.

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                 http://www.angryox.com/
---------------------------------------------------------------------------