Fiber cut in SF area

Mon Apr 13 16:27:46 UTC 2009

It all comes down to money... It will cost them lots of it to get power and some type of readers installed to monitor manhole access... There has always been a lack of security on the telco side, this incident just brings it to light... In my town many of the verizon fios boxes are not locked and the wiring frame boxes for pots line neither.. Its all of a matter of how much cash they wanna throw at it...
Sent on the Now Network™ from my SprintÂ® BlackBerry

-----Original Message-----
From: "Dylan Ebner" <dylan.ebner at crlmed.com>

Date: Mon, 13 Apr 2009 09:57:30 
To: <nanog at nanog.org>
Subject: RE: Fiber cut in SF area

One thing that is missing here is before we can define "security" we
need to define the "threat" and the "obstruction" the security creates.
With an ATM machine, the threat is someone comes and steals the machine
for the cash. The majority of the assailants in an ATM case are not
interested in the access passwords, so that is not viewed as a threat by
the bank. Then bank then says, "If we set really complicated passwords,
our repair guys (or contractors) will not be able to fix them." So
setting hard passwords is an obstruction. This happens every day, in
every IT department in the world. 

So lets define the "Threat" to the fiber network? We know it isn't
monetary as their isn't much value in selling cut sections of fiber. So
that leaves out your typical ATM theif. That leaves us with directed
attack, revenge or pure vandalism.

In a directed attack or revenge scenario, which is what this case looks
like, how are manhole locks going to help? If it is was the fiber union,
wouldn't they already have the keys anyway? If this was some kind of
terrorism scenario wouldn't they also have the resources to get the
keys, either by getting employed by the phone company or the fiber union
or any one of the other thousand companies that would need those keys?

Manhole locks are just going to stop vandalism, and I think the threat
to obstruction calculation just doesn't add up for that small level of
isolated cases.

Here in Qwest territory, manhole locks would be disasterours for repair
times. We have had times when our MOE network has an outage and Qwest
cannot fix the problem because their repair guys don't have the keys to
their own buildings. Seriously. Their own buildings.

Ultimately, what really needs to be addresses is the redundancy problem.
And this needs to be addresses by everyone who was affected, not just
ATT and Verizon, etc. 

A few years ago we had a site go down when a sprint DS-3 was cut. This
was a major wake-up call for us because we had 2 t-1's for the site and
they were suppose to have path divergence. And they did, up to the qwest
CO where they handed off the circuit to sprint. In the end, we built in
workflow redundancies so if any site goes down, we can still operate at
near 100% capacity. 

My point is, it is getting harder and harder to gurantee path divergence
and sometimes the redundancies need to be built into the workflow
instead of IT. 

But that does't mean we cannot try. I remember during Katrima a
datacenter in downtown New Orleans managed to stay online for the
duration of disaster. These guys were on the ball and it paid off for
them. 

In the end, as much as I like to blame the phone companies when we have
problems, I also have to take some level of responsibility. And with
each of these types of incidents we learn. For everyone affected, you
now know even though you have two carriers, you do not have path
divergence. And for everyone who colos at an affected Datacenter and
get's your service from that center, you know they don't have
divergence. So we need to ask ourselves, "where do we go from here?"

It will be easier to get more divergence than secure all the manholes in
the country. 

Dylan Ebner, Network Engineer
Consulting Radiologists, Ltd.
1221 Nicollet Mall, Minneapolis, MN 55403
ph. 612.573.2236     fax. 612.573.2250
dylan.ebner at crlmed.com
www.consultingradiologists.com

-----Original Message-----
From: Joe Greco [mailto:jgreco at ns.sol.net] 
Sent: Sunday, April 12, 2009 7:12 AM
To: Mike Lewinski
Cc: nanog at nanog.org
Subject: Re: Fiber cut in SF area

> 
> Joe Greco wrote:
> 
> > My point was more the inverse, which is that a determined, equipped,

> > and knowledgeable attacker is a very difficult thing to defend
against.
> 
> "The Untold Story of the World's Biggest Diamond Heist" published 
> recently in Wired was a good read on that subject:
> 
> http://www.wired.com/politics/law/magazine/17-04/ff_diamonds

Thanks, *excellent* example.

> > Which brings me to a new point:  if we accept that "security by 
> > obscurity is not security," then, what (practical thing) IS
security?
> 
> Obscurity as a principle works just fine provided the given token is 
> obscure enough.

Of course, but I said "if we accept that".  It was a challenge for the
previous poster.  ;-)

> Ideally there are layers of "security by obscurity" so compromise of 
> any one token isn't enough by itself: my strong ssh password (1 layer 
> of obscurity) is protected by the ssh server key (2nd
> layer) that is only accessible via vpn which has it's own encryption 
> key (3rd layer). The loss of my password alone doesn't get anyone
anything.
> The compromise of either the VPN or server ssh key (without already 
> having direct access to those systems) doesn't get them my password
either.
> 
> I think the problem is that the notion of "security by obscurity isn't

> security" was originally meant to convey to software vendors "don't 
> rely on closed source to hide your bugs" and has since been mistakenly

> applied beyond that narrow context. In most of our applications, some 
> form of obscurity is all we really have.

That's really it, and bringing us back to the fiber discussion, we are
forced, generally, to rely on obscurity.  In general, talk to a hundred
people on the street, few of them are likely to be able to tell you how
fiber gets from one city to another, or that a single fiber may be
carrying immense amounts of traffic.  Most people expect that it just
all works somehow.  The fact that it's buried means that it is
sufficiently inaccessible to most people.  It will still be vulnerable
to certain risks, including backhoes, anything else that disrupts the
ground (freight derailments, earthquakes, etc), but those are all more
or less natural hazards that you protect against with redundancy.  The
guy who has technical specifics about your fiber network, and who picks
your vulnerable points and hits you with a hacksaw, that's just always
going to be much more complex to defend against.

... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI -
http://www.sol.net "We call it the 'one bite at the apple' rule. Give me
one chance [and] then I won't contact you again." - Direct Marketing
Ass'n position on e-mail spam(CNN) With 24 million small businesses in
the US alone, that's way too many apples.